Compressed TrueType fonts for Micro-Controllers with Lcd display

Compressed TrueType fonts for Micro-Controllers with Lcd display
Posted by Krakenwak on 01 Feb, 2019 15:01
Hi People,

A few years ago I started playing with a STM32F411RE & 7 inch 800x480
Lcd display using an SSD1963 controller just to pass the time.

When I required larger fonts such as 72pt numeric characters I found
The memory requirements for the font to be prohibitively large. After
Looking around for a font compression app I was unable to find one,
but I did come across ‘The Dot Factory’ source by Pavius in C# on Github.

https://github.com/pavius/the-dot-factory

‘The Dot Factory’ did everything I wanted except give me a compressed
font output. After looking at the source I found a suitable place to break in
and add my own output code. (This is used via a dropdown menu)

I’ve only used MSVC6 & winapi and not worked with C# before so please
keep this in mind when critically viewing the source.

The new output (If selected) will now generate Run-Length-Encoded font
data and spit-out some size statistics to help decide if it’s worthwhile.

Snippet below:

( Ignore ‘Continuous Bitmap’ I lost interest before completing it)

//     *** Font Encoding Statistics ***
//
//    This font has been encoded with 'The Dot Factory'
//    using Run-Length-Encoding (RLE) from a 72-Pt
//    TrueType font called "Monofonto"

//   '+' (Ascii 43) has the smallest width at 43 Pixels Wide
//   '+' (Ascii 43) has the greatest width at 43 Pixels Wide
//   '+' (Ascii 43) has the smallest height at 84 Pixels high
//   '+' (Ascii 43) has the greatest height at 84 Pixels high

//    The total Horizontal RLE bytes created for this font = 2183
//    The total Vertical RLE bytes created for this font = 1653
//    The total Vert+Horiz RLE bytes created for this font = 1619
//    The total Continuous Bitmap bytes created for this font = 6328
//    The total Standard Bitmap bytes created for this font = 7056

As you can see from generating 14 72pt chars ‘0123456789.,-+’ using a
Fixed font for a digital meter display, RLE only needs 1619 bytes.

(There are also some house-keeping arrays which need to be added to that figure for RLE)

Normally I would have just uploaded the new code to Github and let
nature take it’s course, and it would be found sooner or latter.
But As I’m still on XP with an old version of Firefox, I found
I could no longer upload files to Github and so ended up zipping
It up and putting it on SourceForge.

https://sourceforge.net/projects/dot-factory-rle/files/latest/download

As I’m not sure whether SourceForge is really used anymore I thought
I’d let people know that the app exists. Hopefully someone will find it useful
and maybe improve it and even load it up to Github.

#1 Reply
Posted by mariush on 01 Feb, 2019 16:22
You should provide a binary as well, I don't have C# installed at the moment.

With such big fonts there are other tricks you can use, techniques similar to sprites etc

* arrange the image of a character in 4x4 pixel blocks, gather statistics over all the characters, which 4x4 sprites are most common ...
* use different amount of bits depending on how used sprite is ex 4 bits for the most used 7 sprites (0b0xxx) (use 0000 as sprite not in the "pallete") , 8 bits for the most common 7..63 sprites (0b1xxx0xxx) , 12 bits for 63..512 .... build palette of sprites as you go along ex [4 bits run length] [4-12 bits palette offset] [if data not preset in palette 16 bit follow with sprite info or if value is offset is 0 aka not in palette]
* could add two bits per block which tell you if the sprite is flipped vertical or horizontal (to improve run length encoding) etc

maybe investigate if you can "optimize" a font to have a limited number of sprites and overlap/xor a couple of sprites to get a third sprite..

Now that I think about it, palmdoc compression (the one used in mobi files, a kind of LZ77 compression) would probably work well for "big fonts"

Quote
Read a byte from the compressed stream. If the byte is

0x00: "1 literal" copy that byte unmodified to the decompressed stream.
0x09 to 0x7f: "1 literal" copy that byte unmodified to the decompressed stream.
0x01 to 0x08: "literals": the byte is interpreted as a count from 1 to 8, and that many literals are copied unmodified from the compressed stream to the decompressed stream.

0x80 to 0xbf: "length, distance" pair: the 2 leftmost bits of this byte ('10') are discarded, and the following 6 bits are combined with the 8 bits of the next byte to make a 14 bit "distance, length" item. Those 14 bits are broken into 11 bits of distance backwards from the current location in the uncompressed text, and 3 bits of length to copy from that point (copying n+3 bytes, 3 to 10 bytes).

0xc0 to 0xff: "byte pair": this byte is decoded into 2 characters: a space character, and a letter formed from this byte XORed with 0x80.

I'm reminded of this amazing presentation, where this talented guy makes clouds appear or buildings or have animated waterfalls just by playing with the color palettes and shifting colors in the images:

#2 Reply
Posted by T3sl4co1l on 01 Feb, 2019 16:41
This may be of interest:
https://github.com/T3sl4co1l/st7735_gfx

If you have the power to decode and use ttf (32F4, hell yeah you do!), and assuming you find a good library for it, absolutely, do that. It'll look great!

If you don't need as much flexibility, and overhead, it may be easier to draw the individual glyphs (in whatever font sizes are needed), in which case this may be useful. It still may not look great, since antialiasing (with the help of a frame buffer or canvas system) isn't so easy to incorporate with this.

I'm not actually using this method for fonts, on the project I've been playing with over the last yearish; that's just because I'm using a small LCD and the actual fonts are small (~10pt), better encoded differently (for which I also have a (less fancy) tool, that's not really in a publishable format, unfortunately). I am using this for large graphics (backgrounds, buttons..). (Example image: takes up about... oh what was it, 9kB data+code? https://www.seventransistorlabs.com/Images/ActiveLoad_ImageTest.png https://www.seventransistorlabs.com/Images/ActiveLoadLCD6.jpg Note that, if you need more and bigger images, and better compression, you'll be much better served with a proper LZ-based (or the like) library.) That's the thing that brings this to mind, the large glyphs with transparent (but not alpha-blended) areas should compress pretty nicely in this format.

Tim

#3 Reply
Posted by tooki on 01 Feb, 2019 21:07
Quote from: mariush on 01 Feb, 2019 16:22
I'm reminded of this amazing presentation, where this talented guy makes clouds appear or buildings or have animated waterfalls just by playing with the color palettes and shifting colors in the images:

Ummmm… palette shifting (aka color cycling) is a technique that was widely used in the era of 8-bit color gaming to create repeating animations that affected a lot of pixels while using nearly zero CPU. It was most common for things like backgrounds (waves in water and lava, etc), but a notable exception was SimCity 2000, which used palette shifting to create nearly all of its background motion.

Basically, giving the graphics card an updated color palette takes practically no CPU at all, and the graphics card takes care of actually updating the pixels. This way, you can animate an unlimited number of pixels with the same near-zero CPU hit.

#4 Reply
Posted by Krakenwak on 01 Feb, 2019 21:44
Hi,

Just to clarify.

There is an exe file included, it needs DotNet installed to run.
Also there is a Readme which contains more details on using the app, and
some code snippets showing how I contain the fonts in .h files and render
pixels in a putc & puts format.

Actual truetype fonts:- I did spend a few days trying to implement real ttf but
got no where due to memory limitations.

#5 Reply
Posted by bson on 03 Feb, 2019 17:23
One technique I've been using is to define:

Code: [Select]
#define FONTMAP(FONTLIST, CHARS) (CHARS)
Then I run a tool to extract all of these (sed will do it, but my font tools are in python so I use regex matching) during the build (with C++ preprocessing so only get that which is used). Being completely syntactically unaware, it finds every appearance of e.g. FONTMAP(big72, "0123456789") so I can put these either in code or in a comment:

Code: [Select]
// FONTMAP((big72pt, small5x7), "0123456789") // FONTMAP((small5x7), "abcddefghjijklmopqrstuvAEGJM") // Only some uppercase chars used // FONTMAP((graphic_elements), "\001\004\010")
or

Code: [Select]
const uchar _digits[10] = FONTMAP((big72pt, small5x7), "0123456789");
or both.

or

Code: [Select]
#ifdef DEBUG _panel.drawMessage(NOTICE_X, NOTICE_Y, Fonts::med10pt, FONTMAP((med10pt), "ADC timeout, reading #") "%u", adc_counter); #endif
This lets me automatically determine which characters are actually used in each font. Which in turn lets me have generic complete fonts but only embed the glyphs actually used. This means for large glyphs where I only use maybe digits and a few symbols I don't need the whole thing since even RLE'd it can take up a few tens of k. Maybe not a big deal on a part with 1MB flash, but in general I don't like to include any code or data that is unused if it can be prevented. The usage information for each font is then used while the font is turned into C data (arrays of bytes) that are compiled and relocated to a flash section.

This is not unlike how gettext() is canonically used to collect strings for translation.

(Actually, it's a bit more complicated than a #define, but it works out the same.)