INI Handler gone!
- mwm
-
Topic Author
- Offline
Less
More
29 Jul 2015 21:56 - 29 Jul 2015 22:13 #36329
by mwm
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.
My remotely piloted vehicle ("drone") is a yacht.
INI Handler gone! was created by mwm
PB has apparently decided to rip out the INI handler in order to make room on the 7E. This does create some interesting problems.
After writing the model file switch tweaking stuff, I'm all for this. INI files suck. I had to write an INI file handler for this, because none of the available libraries dealt with the things that deviationTx did in those files. Similarly, I'm thinking about writing an INI file GUI editor, because the ones I can find can't handle what deviation does. Well, except for emacs config mode, but that's not really a GUI editor.
My question is whether or not we can replace the INI files with something for which a standard exists? Preferably one with schema, so that we can get intelligent editors!
After looking at the INI file code, I figured any such parser would have to be larger than what's there. But since we're going to be doing this on the desktop, that shouldn't be an issue. If I'm wrong about that, maybe there's a smaller library for parsing some standard format which would let us avoid going to needing a desktop tool for dealing with model files.
After writing the model file switch tweaking stuff, I'm all for this. INI files suck. I had to write an INI file handler for this, because none of the available libraries dealt with the things that deviationTx did in those files. Similarly, I'm thinking about writing an INI file GUI editor, because the ones I can find can't handle what deviation does. Well, except for emacs config mode, but that's not really a GUI editor.
My question is whether or not we can replace the INI files with something for which a standard exists? Preferably one with schema, so that we can get intelligent editors!
After looking at the INI file code, I figured any such parser would have to be larger than what's there. But since we're going to be doing this on the desktop, that shouldn't be an issue. If I'm wrong about that, maybe there's a smaller library for parsing some standard format which would let us avoid going to needing a desktop tool for dealing with model files.
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.
My remotely piloted vehicle ("drone") is a yacht.
Last edit: 29 Jul 2015 22:13 by mwm. Reason: Hit submit by accident.
Please Log in or Create an account to join the conversation.
- greenfly
-
- Offline
29 Jul 2015 23:51 #36335
by greenfly
Replied by greenfly on topic INI Handler gone!
That's an interesting question.
Of course some sort of XML format comes to mind, But that may be too much overhead for the little 7E. Then there is JSON, but that may be difficult to hand edit. (All the nested data). Any other ideas?
I think we need to realize that model files are not exactly true INI files. That is one reason they may be difficult to work with. They are similar to an object dump that just happens to look like an INI file. There is a schema for the file in a sense... in the code that read and writes the files. Not that there is anything wrong with this approach.
Of course some sort of XML format comes to mind, But that may be too much overhead for the little 7E. Then there is JSON, but that may be difficult to hand edit. (All the nested data). Any other ideas?
I think we need to realize that model files are not exactly true INI files. That is one reason they may be difficult to work with. They are similar to an object dump that just happens to look like an INI file. There is a schema for the file in a sense... in the code that read and writes the files. Not that there is anything wrong with this approach.
Please Log in or Create an account to join the conversation.
- PhracturedBlue
-
- Offline
Less
More
- Posts: 4402
30 Jul 2015 00:19 - 30 Jul 2015 00:20 #36337
by PhracturedBlue
Replied by PhracturedBlue on topic INI Handler gone!
1st note that I am not planning to remove support for ini files. For all Tx other than the 7e/f7, I plan to keep the ini files.
yaml would have been my choice, but parsers for it are too complicated, and took up too much room on the tx. The ini parser we use is actually very efficient, but converting it to a schema is complicated, and requires a lot of code.
The new code is a binary dump of the struct. This is both very small in file size (good for the f7) and very small in ROM usage (just a simle RLE compressor)
The downside is that the format changes every time the ABI changes, so each time you install a new firmware, you need to read each binary file, convert it to ini, then reconvert it to the new ABI.
To make that possible, I'm using emscripten, which compiles the deviation ini readers/writers into javascript. I can then load and run that javascript in Java in the deviation-uploader.
It is a really complicated process, and requires that we keep track of the ABI currently on the Tx, so we have a means to decode it. It is made more complicated because we need to know the layout on the STM32 which has different alignment rules than the PC the encoder/decoder run on.
I'm really unhappy with the solution to be honest, but I've been trying to come up with something for over a month now, and this is the 1st solution I found that I can get all the pieces to work. But because it is very complicated, it will probably take quite a while before I have it all working properly (One plus side is that I can now compile the emulator to javascript which means you can run the emulator in a browser (well, almost, the filesystem code is still broken)
Anyhow, I don't think we'll be switching away from ini anytime soon, as the solution I'm using for the 7e is not one I'd want to choose for a more capable Tx.
yaml would have been my choice, but parsers for it are too complicated, and took up too much room on the tx. The ini parser we use is actually very efficient, but converting it to a schema is complicated, and requires a lot of code.
The new code is a binary dump of the struct. This is both very small in file size (good for the f7) and very small in ROM usage (just a simle RLE compressor)
The downside is that the format changes every time the ABI changes, so each time you install a new firmware, you need to read each binary file, convert it to ini, then reconvert it to the new ABI.
To make that possible, I'm using emscripten, which compiles the deviation ini readers/writers into javascript. I can then load and run that javascript in Java in the deviation-uploader.
It is a really complicated process, and requires that we keep track of the ABI currently on the Tx, so we have a means to decode it. It is made more complicated because we need to know the layout on the STM32 which has different alignment rules than the PC the encoder/decoder run on.
I'm really unhappy with the solution to be honest, but I've been trying to come up with something for over a month now, and this is the 1st solution I found that I can get all the pieces to work. But because it is very complicated, it will probably take quite a while before I have it all working properly (One plus side is that I can now compile the emulator to javascript which means you can run the emulator in a browser (well, almost, the filesystem code is still broken)
Anyhow, I don't think we'll be switching away from ini anytime soon, as the solution I'm using for the 7e is not one I'd want to choose for a more capable Tx.
Last edit: 30 Jul 2015 00:20 by PhracturedBlue.
Please Log in or Create an account to join the conversation.
- greenfly
-
- Offline
30 Jul 2015 03:29 #36342
by greenfly
Replied by greenfly on topic INI Handler gone!
PB, thanks for the update on what you've been struggling with. I can see the point of moving to a simple compressed binary dump of the struct... that would be much smaller than any INI parsing code could be. And I see your dilemma with changing the structure from version to version. That is a tough one.
For me, with my lowly 7E, I'm going to miss the hackable nature of the INI file model definition. While it is completely secondary to it's primary purpose (of persisting the model data structure) messing around with the model files is fun.
For me, with my lowly 7E, I'm going to miss the hackable nature of the INI file model definition. While it is completely secondary to it's primary purpose (of persisting the model data structure) messing around with the model files is fun.
- I like to tweak the GUI section to take advantage of settings that are not editable on the 7E version.
- I like to view and compare how others have configured their models. Exchanging model files will not be as transparent.
- Some things are just faster and easier to change with a text file
Anyway...
1. Could we dump the structure but still have it be "readable" in some way? Maybe a hybrid with some portion of the file that would still be be readable? Like some kind of "standard" header that would contain Model Name, structure version, etc...
2. If not, could someone be able to leverage your emscripten work to build PC-side model editors? I'm not fluent in any cross-platform languages, but can build anything you need on the Windows side.
I'm new around here and I have not really worked with the code (yet!), so my ideas may be way off base... but maybe they might spark another idea.
Please Log in or Create an account to join the conversation.
- Richard96816
-
- Offline
Less
More
- Posts: 208
30 Jul 2015 05:03 #36343
by Richard96816
Replied by Richard96816 on topic INI Handler gone!
Perhaps it's time to do the ini file editing on Android.
A small bluetooth dongle for the radio and UI parts of Deviation could be moved to your phone or tablet. A nice pocketable GUI with plenty of resources.
A small bluetooth dongle for the radio and UI parts of Deviation could be moved to your phone or tablet. A nice pocketable GUI with plenty of resources.
Please Log in or Create an account to join the conversation.
- PhracturedBlue
-
- Offline
Less
More
- Posts: 4402
30 Jul 2015 11:34 #36352
by PhracturedBlue
Replied by PhracturedBlue on topic INI Handler gone!
Sorry if I wasn't clear. You will continue to work with ini files on the 7e. you will upload and download ini files to the tx. The uploader app will convert from the ini to the binary format and back for you in the background. So you won't lose any functionality, but you will be forced to use the DeviationUploader Java App to upload to the Tx (instead of Walkera's Dfuse tool).
The Uploader shows you the file system so you can copy files to and from the tx like you do today using USB. The interface is a little different, and it is a lot slower (due to how the bootloader works), but again, it should have parity functionality to what you have now.
As for using an Android, it is already planned, but will require the UniversalTx with bluetooth dongle. I'm not sure that functiality will ever make it to the 7e either. If your Android phone can act as USB master, it is probably possible to port the Deviation Uploader to it, but that is beyond my scope.
The Uploader shows you the file system so you can copy files to and from the tx like you do today using USB. The interface is a little different, and it is a lot slower (due to how the bootloader works), but again, it should have parity functionality to what you have now.
As for using an Android, it is already planned, but will require the UniversalTx with bluetooth dongle. I'm not sure that functiality will ever make it to the 7e either. If your Android phone can act as USB master, it is probably possible to port the Deviation Uploader to it, but that is beyond my scope.
Please Log in or Create an account to join the conversation.
- mwm
-
- Offline
30 Jul 2015 12:04 #36353
by mwm
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.
My remotely piloted vehicle ("drone") is a yacht.
Replied by mwm on topic INI Handler gone!
I wasn't sure if you were planning on leaving INI files on the non-8E Tx's or not. Glad to have that cleared up.
And yeah, the current INI parser is pretty slim - completely context free. When I realized that, I gave up on replacing it with something that might be smaller. JSON, YAML, etc. keep the context around. They would more directly correspond to to the structures in the program, but that requires keeping around a lot of state during the process.
As for deviation not using "TRUE" INI files - exactly what does that mean? What sucks about INI files is that there is no standard. No definition about what characters are valid in names, when and where comments can appear, or how often sections and options can appear, etc. The latter is what killed all the extant Haskell libraries - they all turned the thing into a map of maps. There's no provision in them (or INI in general) for lists of things, so deviation using repeated entries to handle lists made them unusable.
I don't think deviation does anything that some other program doesn't do. It may be unique in the set of oddball things it does - but I suspect that's true of a lot of programs that use INI files in oddball ways.
And yeah, the current INI parser is pretty slim - completely context free. When I realized that, I gave up on replacing it with something that might be smaller. JSON, YAML, etc. keep the context around. They would more directly correspond to to the structures in the program, but that requires keeping around a lot of state during the process.
As for deviation not using "TRUE" INI files - exactly what does that mean? What sucks about INI files is that there is no standard. No definition about what characters are valid in names, when and where comments can appear, or how often sections and options can appear, etc. The latter is what killed all the extant Haskell libraries - they all turned the thing into a map of maps. There's no provision in them (or INI in general) for lists of things, so deviation using repeated entries to handle lists made them unusable.
I don't think deviation does anything that some other program doesn't do. It may be unique in the set of oddball things it does - but I suspect that's true of a lot of programs that use INI files in oddball ways.
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.
My remotely piloted vehicle ("drone") is a yacht.
Please Log in or Create an account to join the conversation.
- PhracturedBlue
-
- Offline
Less
More
- Posts: 4402
30 Jul 2015 12:53 #36357
by PhracturedBlue
Replied by PhracturedBlue on topic INI Handler gone!
We do 3 things that are non-standard:
1) we have multiple copies of the same section which are each processed separately (mixer definitions). A standard ini file will only use each section header once. I could have gotten around that by numbering the mixers in the ini, but there really wasn't much reason to and the numbers wouldn't mean anything since we recompact the mixers after load.
2) we depend on the order of the sections (channel, then the associated mixer) in order to effectively add extra hierarchy to the ini file.
3) we depend on the order that items appear in the section (I don't recall where but I'm pretty sure we still do). items within a section should be independent ideally.
All of the above are covered as exceptions in the wiki .
basically an ini would ideally represent a 2 level unordered hash structure:But again, it doesn't really matter; as you said, there is no true standard for how ini files are parsed, and everything we're doing has been done before.
1) we have multiple copies of the same section which are each processed separately (mixer definitions). A standard ini file will only use each section header once. I could have gotten around that by numbering the mixers in the ini, but there really wasn't much reason to and the numbers wouldn't mean anything since we recompact the mixers after load.
2) we depend on the order of the sections (channel, then the associated mixer) in order to effectively add extra hierarchy to the ini file.
3) we depend on the order that items appear in the section (I don't recall where but I'm pretty sure we still do). items within a section should be independent ideally.
All of the above are covered as exceptions in the wiki .
basically an ini would ideally represent a 2 level unordered hash structure:
section =>
key =>value
key =>value
key =>value
section =>
key =>value
key =>valuePlease Log in or Create an account to join the conversation.
- greenfly
-
- Offline
30 Jul 2015 14:53 #36360
by greenfly
Thanks PB for clarifying what I meant about the INI files. I used to be a big user of the old Windows API and GetPrivateProfileString...
So the model files will reside on the TX in binary files and we will have to use the Java Uploader to move INI-based model files to and from the TX. OK, I see now.
Is there anything that I can help with? Testing? Documentation? Should I learn Java?
Replied by greenfly on topic INI Handler gone!
PhracturedBlue wrote: We do 3 things that are non-standard:...
Thanks PB for clarifying what I meant about the INI files. I used to be a big user of the old Windows API and GetPrivateProfileString...
So the model files will reside on the TX in binary files and we will have to use the Java Uploader to move INI-based model files to and from the TX. OK, I see now.
Is there anything that I can help with? Testing? Documentation? Should I learn Java?
Please Log in or Create an account to join the conversation.
- PhracturedBlue
-
- Offline
Less
More
- Posts: 4402
30 Jul 2015 16:55 #36362
by PhracturedBlue
Replied by PhracturedBlue on topic INI Handler gone!
I need to make a bit more progress before I need any help. I have a proof of concept of all the pieces, but have not integrated them into anything usable as yet.
Please Log in or Create an account to join the conversation.
- mwm
-
- Offline
30 Jul 2015 17:10 #36364
by mwm
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.
My remotely piloted vehicle ("drone") is a yacht.
Replied by mwm on topic INI Handler gone!
We also accept # as a comment character, and allow comments in places other than the beginning of the line. And we put spaces in option names. The first two are also covered as common exceptions in the wiki. Not sure about unquoted spaces in option names, but none of the libraries I looked at allowed that.
The thing is that the only "standard" is the behavior of some library. Since it's an MS library, it only runs on windows, so "some library" will vary with your platform and language. Since it's largely used for configuration information for a program and not data exchange, there's no real need for a standard. Except we now want to swap files with desktop programs. Maybe we need a C library for use with an FFI.
The thing is that the only "standard" is the behavior of some library. Since it's an MS library, it only runs on windows, so "some library" will vary with your platform and language. Since it's largely used for configuration information for a program and not data exchange, there's no real need for a standard. Except we now want to swap files with desktop programs. Maybe we need a C library for use with an FFI.
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.
My remotely piloted vehicle ("drone") is a yacht.
Please Log in or Create an account to join the conversation.
- Richard96816
-
- Offline
Less
More
- Posts: 208
30 Jul 2015 18:41 #36368
by Richard96816
Replied by Richard96816 on topic INI Handler gone!
All for want of a $2 CPU change.
Please Log in or Create an account to join the conversation.
- PhracturedBlue
-
- Offline
Less
More
- Posts: 4402
30 Jul 2015 20:07 #36370
by PhracturedBlue
Replied by PhracturedBlue on topic INI Handler gone!
mwm, I'm not sure what you are looking for.
The code to convert the ini to/from a C-structure can easily be extracted and run on the PC. You can either do it in C or now you can do it via javascript as well (just look at the sdl2 branch for code to do this).
So getting the ini into a machine parseable structure is easy. I'm not sure what you want to do with it from there, but the ini file shouldn't be a limitation
The code to convert the ini to/from a C-structure can easily be extracted and run on the PC. You can either do it in C or now you can do it via javascript as well (just look at the sdl2 branch for code to do this).
So getting the ini into a machine parseable structure is easy. I'm not sure what you want to do with it from there, but the ini file shouldn't be a limitation
Please Log in or Create an account to join the conversation.
- mwm
-
- Offline
30 Jul 2015 21:35 #36377
by mwm
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.
My remotely piloted vehicle ("drone") is a yacht.
Replied by mwm on topic INI Handler gone!
Ideally, I'd like a format that's standardized enough that libraries are available in most languages for dealing with it. Getting one with schemas so OTS GUI editors could be used for editing the files on the PC would be a major win, but most people these days don't seem to care about valid data files, so that seems unlikely.. Failing that, an ansi C library that most languages can talk to via their FFI - though that would still require building tools to actually work with the things on the desktop.
Given that you can count the number of people writing code to work with them on the fingers of one hand even if you've lost fingers in a helicopter accident, it's clearly not gong to be a priority.
Butt if we're going to have to break what's on the Tx even if just for the 7e, that would be the time to do it.
Given that you can count the number of people writing code to work with them on the fingers of one hand even if you've lost fingers in a helicopter accident, it's clearly not gong to be a priority.
Butt if we're going to have to break what's on the Tx even if just for the 7e, that would be the time to do it.
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.
My remotely piloted vehicle ("drone") is a yacht.
Please Log in or Create an account to join the conversation.
- mwm
-
- Offline
31 Jul 2015 16:42 #36399
by mwm
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.
My remotely piloted vehicle ("drone") is a yacht.
Replied by mwm on topic INI Handler gone!
You said YAML was preferred but to complex to fit on a Tx. Was that any of them, or just the 4E?
And what about JSON? That should be simpler. And poking around, I can find online editors, including some that use schemas, for it. Like exavolt.github.io/onde/ and www.jsoneditoronline.org/ . I suspect that means that there are desktop editors as well, and probably similar for YAML as well.
And what about JSON? That should be simpler. And poking around, I can find online editors, including some that use schemas, for it. Like exavolt.github.io/onde/ and www.jsoneditoronline.org/ . I suspect that means that there are desktop editors as well, and probably similar for YAML as well.
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.
My remotely piloted vehicle ("drone") is a yacht.
Please Log in or Create an account to join the conversation.
Time to create page: 0.056 seconds
-
Home
-
Forum
-
Development
-
Development
- INI Handler gone!