Development of new filesystem for f7/f12e

More
02 May 2015 15:58 - 02 May 2015 16:02 #31964 by PhracturedBlue
Development of new filesystem for f7/f12e was created by PhracturedBlue
the Devo F12E and F7 both have tiny 64kB external flash storage. This flash also has a 4kB sector size. That makes it completely unsuitable for the current FAT filesystem since only ~12 files could be stored on the flash device.

I went looking for a better solution, but was unable to find a filesystem that met all of my criteria. Specifically, I need:
* very low per-file/directory overhead
* low RAM usage (can't keep a whole sector in RAM)
* wear-leveling (can't continually use the same sector for temporary storage during erase)
* small code size

the best solution I came up with was the MicroMonitor TFS: www.umonfw.com/
However, it is not very well documented, and it fails my3rd point above.

In the end I spent the last couple days and developed my own filesystem: DevoFS

It is optimized for very small flash devices, and will not work well on the > 4Mbit flash found in other Walkera devices. So don't expect it to be used in the Devo8 or Devo10. It also comes with the disadvantage that you can't access it with USB (theroetically a FUSE driver could be written, but that wouldn't help on Windows, and I have no inclination to do so)

Currently I am developing it in the devof7 branch. It isn't quite ready, but is passing my basic tests, so I expect to put it on my F7 soon.

The filesystem is log based, which helps with wear leveling. It also does not have a dedicates 'temp' sector, so no single sector will be over-abused. Each file/directory has an overhead f just 15 bytes (11 of which is the file name) so it is well optimized for space, although it does require one sector to be kept free, so the filesystem size is actuallyon 60kB.

Using this, It should be possible to have maybe 4 or 5 protocol modules and maybe 15-20 models installed on the 64kB stock flash (depending on modle complexity). For the F12E, the protoocols will fit on the MCU flash, so we can use the remaining space for bitmaps or more models.

While installing a larger flash (and using FAT) will still be supported, this will enable using Deviation of the F7 and F12E without needing a hardware hack.

Here is the readme:
The DevoFS filesystem is designed to provide a filesystem to very small FLASH
memories with large sector sizes (ex 64kB flash with 4k sector-size). It is
optimized for flash layout efficiency over all else.

NOTE:  It is not recommended to use this filesystem on devices with
                  larger than 512kB of FLASH space
The only supported file attributes are file/directory bit and file-size. No
other data is stored. **** The maximum file size is 65535 **** (this could be
changed by adding an extra byte to the file-system header)

The file-system is a log-based filesystem with built in wear-leveling.
Sequential read and write speeds will be very fast, however, open and read-dir
functions will be quite slow.

DevoFS will slow down linearly with the number of files present, as well as
with the number of files written since the last compact() event

The filesystem will automatically run compact() when necessary, butit is
recommended to run it more frequently to optimize performance

The filesystem has an overhead of 15 bytes for each file and directory, and does
not require file alignment to sector boundaries. Additionally, one sector must
be left vacant to allow for running the compact operation (unless the equivalent
amount of RAM is available)

A very simple algorithm is used:
* find current filesystem start
- will be sector aligned
- not restricted to the 1st sector. Will move backwards by one sector
every time compact() is run
* read file header (15 bytes)
* use current file-header to identify start of next file-header
* repeat until requested file/directory is found

Support for creating new files or directories is not currently implemented, but
could easily be added in the future. Also, file-size cannot currently be
changed.

For efficiency, it is required that a directory be created before any files that
reside within that directory. The code does not check this condition, and it
leaves the burden to the developer

Reads and writes will automatically be partitioned to sector boundaries so the
sector-id bytes are not inadvertently read

DevoFS layout:

The constant values below assume the FLASH resets to 0xff. negate all values if reset is 0x00
sector boundary
id/counter: 0x00 - start of filesystem
          : 0xff - empty sector
file
type/status:    file/dir/currently-writing/deleted
                                ff: no more data in the sector
                                80: directory
                                0c: writing file (no size info)
                                08: file
                                00: deleted
parent-dir:             0 - 255
name:                   8.3 (11 bytes)
size:                   2 bytes
data:
directory
type:                   80
parent_dir:             0-255
name:                   8.3
id:                             1-255
empty byte:       00

compacting
cursec = find sector 0
nextsec = cursec+1
emptysec = find sector 255
while (1) {
        copy all non-deleted data from cursec to emptysec
        while (! full) {
                fill emptysec with non deleted data from nextsec
                if (no more files) {
                        clear cursec
                        return
                }
                nextsec++
        }
        offset = nextsec index of uncopied data
        nextfile = bytes in nextsec to next file boundary
        clear cursec (now sector 255)
        cursec = nextsec
        nextsec = nextsec + 1 (may not be old cursec due to compressing multiple setcors)
}
Last edit: 02 May 2015 16:02 by PhracturedBlue.

Please Log in or Create an account to join the conversation.

More
03 May 2015 20:57 - 03 May 2015 20:58 #32017 by IMback!
Replied by IMback! on topic Development of new filesystem for f7/f12e
very intersting, i might have some non deviation uses for that too. how do you plan alow users to wirte models to devo FX? i also dont get why walkera is going backwords in specs with their txes
Last edit: 03 May 2015 20:58 by IMback!.

Please Log in or Create an account to join the conversation.

More
03 May 2015 21:44 #32019 by PhracturedBlue
Replied by PhracturedBlue on topic Development of new filesystem for f7/f12e
I think Walkera is entirley focused on the drone market now. They know exatcly how little they need to invest and still provide the needed capabilities.

I don't think I undersrand your question regarding DevoFS. Do you mean how does a user copy models from a PC to the transmitter? I plan to start with a utility that will convert a set of directories into a dfu file that can be uploaded. The next step would be to enhance the deviation-uploader app to allow you to drag-and-drop files on and off it.

Please Log in or Create an account to join the conversation.

More
09 May 2015 00:14 #32272 by PhracturedBlue
Replied by PhracturedBlue on topic Development of new filesystem for f7/f12e
I have written a bunch of tests to ensure the devofs file system is working properly. I found several bugs, and now have confidence that all of the basic functions work as intended. I haven't yet tested recovery (what happens if the tx loses power during data compaction?) but other than that everything seems to be working well.

As I mentioned before, the bug issue is how to get a filesystem onto the devof7 in the 1st place. i haven't yet solved that issue.

Please Log in or Create an account to join the conversation.

More
13 May 2015 16:42 #32503 by Djhg2000
Replied by Djhg2000 on topic Development of new filesystem for f7/f12e
If I understand correctly we still have USB flash disk emulation, right? If so then I'd suggest using a bootstrap image and writing it with something like dd or Win32 Disk Imager , basically the same tools and method used to write bootable Linux installers to USB drives.

This way Windows doesn't have to understand the file system at all, we can read and write any arbitrary data to/from the flash through the image file.

Please Log in or Create an account to join the conversation.

More
11 Dec 2021 15:14 - 11 Dec 2021 15:23 #77624 by aerocarv
Replied by aerocarv on topic Development of new filesystem for f7/f12e
The F12 transmitter is a beauty inside and out. It is unique as far as I know that it operates on ONE LiPo at 3.7V. It is a 3000mah capacity. Measuring inside, it operates internally at 3.3V. None of the add-on Deviationx modules operate on that low voltage.
The 2.4G RF module is a tiny thing that sits on a 10 pin connector--again at 3.3V.
There is a plug-in space to the left of the battery compartment under a rubber flap. It has two separate connections. A two pin and a three pin. Wonder what for?
I was hoping to simply change out the 2.4g module with an Orange DIY 2.4G. Not possible at all.
So,
I can only hope your work to bring Deviationx to the F12 will succeed.
By the way, the F7 transmitter is powered at 7.4V with two LiPos in series. Inside it may be quite different from the F12.
Ed
Last edit: 11 Dec 2021 15:23 by aerocarv.

Please Log in or Create an account to join the conversation.

Time to create page: 0.039 seconds
Powered by Kunena Forum