Internationalization

I have started work on internationalization support.
My idea is as follows:
wrap each string with a '_tr()' call. Initially this is just a macro that does nothing.
Allow the definition of a strings.<lng> file that contains translations. The strings file will be UTF-8 encoded, and will basically consist of pairs of lines:this file will be parsed, and the english string will be hashed. The hash result will be used as a key to the translated string.

then the _tr() function will hash the input string, see if there is a match in the translation table and return the translated string if it exists or the english string if not.

I need to actually try it to see if the performance is acceptable though.

After that there will be issues with not enough memory being allocated for UTF-8 strings, issues with the fonts not containing internationalized characters, and issues with sprintf and friends not understanding UTF-8.

I expect this to be a long process to get right

Why not the same .ini format as usual?

because the strings can have special characters that would break the ini parser. This also potentially allows me to store file offset pointers if it turns out that we can't store all the strings in RAM.

Yep, they can contain at least %s, %d and \n type formatters, I guess...

I just committed a major code reorg to make handing internationalization easier (and to reduce some duplication for mulitple Tx, though a lot more is still needed on this front).

Here is a list of all the strings I've marked so far that need translation.
I generated this list via:Which probably won't work on windows. The output isn't in the necessary syntax, but gives a good idea of what is needed.

Introduced a compile error in the emulator:

FDR wrote: Introduced a compile error in the emulator:

+ Compiling 'target/emu_devo8/fltk.cpp'
target/emu_devo8/fltk.cpp: In function 'void LCD_Init()':
target/emu_devo8/fltk.cpp:235:63: error: 'asprintf' was not declared in this sco
pe
make: *** [objs/emu_devo8-w32/fltk.o] Error 1

That one is windows specific. I knew there was a reason I took it out before.
No time to look at it tonight though.

PhracturedBlue wrote:

FDR wrote: Introduced a compile error in the emulator:

+ Compiling 'target/emu_devo8/fltk.cpp'
target/emu_devo8/fltk.cpp: In function 'void LCD_Init()':
target/emu_devo8/fltk.cpp:235:63: error: 'asprintf' was not declared in this sco
pe
make: *** [objs/emu_devo8-w32/fltk.o] Error 1

That one is windows specific. I knew there was a reason I took it out before.
No time to look at it tonight though.

This is fixed now, but I made even more sweeping changes in the name of reducing duplication (mostly just code moves, no functionality changes). Hopefully I didn't break anything else.

PhracturedBlue wrote: This is fixed now, but I made even more sweeping changes in the name of reducing duplication (mostly just code moves, no functionality changes). Hopefully I didn't break anything else.

Yep, it compiles now...

Does that mean, that the tx fw and the emulator code have more common?
Than they should behave more alike, I guess, not that they were too much different...

FDR wrote: Does that mean, that the tx fw and the emulator code have more common?
Than they should behave more alike, I guess, not that they were too much different...

No, they were already about as close as I know how to make them. But it will allow me to merge the Devo10 code (someday) with minimal code duplication.

I've committed a bunch of Unicode related work.
With an expanded font (not yet included) I was able to actually display a utf-8 string (mostly).
Since I know mostly nothing about European character sets, I am unsure which characters need to be supported.
So far I have ascii characters 0x20 - 0x7f and 0xa1 - 0xff but I don't think that is sufficient. I don't have enough ROM to include complete fonts, so I need to figure out the best ranges of characters to meet our needs.

Still, the capability is coming along. I have no idea what to do about the keyboard yet. We'll likely start without an internationalized keyboard.

Yep, my language is outside that range, we use ISO 8859-2, but I don't want to use my tx in hungarian anyway. I've learned these terms in english, don't even know how they are in hungarian!

But I think that supporting the eastern languages would be important, because english is not always common knowledge there (and here ), not to mention far east...

I don't know how to load the font via USB without massive performance impact. The one possible option I can think of would be a 'font replace' capability that loaded a font from SPI and overwrite the ones in the STM32 ROM. We'd need to align the fonts on a page boundary, and it would make it hard to specify fonts in the ini file. I think I'm going to avoid that option for now.

So my thought is to provide a latin font that is mostly good enough. Other languages will require a firmware upgrade in order to add in the requisite font.

As it turned out, I tried Hungarian 1st (because I was using google translate on a string and none of the western europoean languages had any special characters for my string). It was not an ideal choice as it turns out, because my font did not have sufficient coverage.

The code supports a segmented font, so one font can support multiple non-contiguous ranges. However, if the character doesn't exist in the specified font, it just skips the character (we should probably do something better than that I guess)

Anyhow, I found this:
en.wikipedia.org/wiki/ISO/IEC_8859-1
so I think the initial font set will include 8859-1 with the listed special characters, and should give reasonable coverage to start with.

On the devo-8 LCD there is an option to write conditionally in flipped mode (I use this for bmps which are stored bottom-line-1st. You can also write right-to-left. But I'm not sure I want to enable that for fonts (since I think I thne need to actually store the characters backwards too) Easier might be to have the translation file written reversed (or to have a flag in the file that reverses it as we read it.

I'm also making a lot of assumptions here. Like that anyone will care enough to translate into those languages.

Well, I've just committed the first full internationalization code.
There is a new font '12narrow' that has all of 8859-1 and a smattering of other characters which should handle western-european languages pretty well. Building a font that could be internationalized and also fit within the 16-pixel height was quite challenging.

I've switched to using Ubuntu's font as the base, as it is a bit thinner than Arial, and was shorter in the tallest characters without being shorter in the smaller ones. The current font is composed of a composite of 12, 11, 10, an 9 point fonts which is a little weird, but seems to give a reasonable overall result.

I used FNV1 in 16bit mode as the hash function for the font. It seem sto work well so far.
The file language/lang.hu is just an testcase to see if it worked. It contains a google-translate version of the binding dialog. Obviously we won't keep it that way.

I have not tested on the Tx, just in the emulator so far.

You do need to reload the filesystem (at least config.ini and the language dir) in the Tx to use the latest release.

There is a lot of such warnings, is the ok:

I wanted to edit the language file, but it is quite hard to edit it because (if I'm right) carriage returns indicate the new lines in it and line feeds separate the entities. But in a text editor they are all the same, and I don't see the difference.
Wouldn't be simpler to keep the \n to accomplish the new lines and keep each string entity on one line pair?

FDR wrote: I wanted to edit the language file, but it is quite hard to edit it because (if I'm right) carriage returns indicate the new lines in it and line feeds separate the entities. But in a text editor they are all the same, and I don't see the difference.
Wouldn't be simpler to keep the \n to accomplish the new lines and keep each string entity on one line pair?

Simpler yes but moe complicated to implement. In linux there is a difference between CR and LF so it is easy to write the files there. I'll figure something out.

The warnings are expected for now. I'll be cleaning them up today.

I've made some changes to the binding message in both languages, and pushed the changes.
It seems, that it can't calculate right the string lengths with the new font.
My message is the same row count, but it draws it to a much larger box...