Why is Intel Hex (and similar formats) in text and not binary?

[petert]

Hello,

I read about the history of Intel Hex starting on paper. But soon after it was possible to use it with floppies as well.
Executable files like PE or ELF are also in sections, and are in binary, so you could have gaps like in Intel Hex.

Is there an inherent reason why it remained a text only format, such as serial communication being only 7-bit, and therefore preferably use ASCII?

But it seems most(/all?) programmers communicate with microcontrollers in 8 bits, so why this limitation, on the PC to programmer side?

If anybody knows more about the rationale, I'd be curious.

[MIS42N]

I cannot say for sure, but my opinion is it stays in hex because there's no need to change it. I would suggest that every computer filing system ever invented can handle the characters in a hex file. So it can be passed from computer to computer as a text file and not get corrupted. And since it is a 2 character to one byte format, it is possible to make some sense of it (especially the address part). It is not so easy to make sense of data packed with base64 (packing 3 bytes into 4 characters).

[ataradov]

They have been invented by old timey people, who are used to used to computers only being able to handle text. Also, they may not have indented for it to get a wide use, so they used whatever worked at a time for a specific purpose.

There is also no particular reason to have all those CRCs. They should not be a part of the file format. If your medium or communication channel are so bad, you need to address the checking on that level.

[amyk]

https://en.wikipedia.org/wiki/Intel_HEX#History

tl;dr: punched cards and paper tape. The equipment used to create mask ROMs likely predates microcomputers, so the format was retained as a convention.

 I remember one Japanese mask ROM manufacturer requested ROM images be faxed(!) to them, as printouts in hex format...

[S. Petrukhin]

It seems to me that the main reason was that there were no compilers at the time of the birth of this format, or they were not available. People did the compilation manually and recorded in the form of text binary information - machine code. Writing each byte in decimal would be inconvenient and difficult to read. HEX file display format is still actively used, it is in some cases much more convenient than binary and decimal. However, for a long time all compilers have the BIN format.

[Kjelt]

There is also no particular reason to have all those CRCs. They should not be a part of the file format. If your medium or communication channel are so bad, you need to address the checking on that level.

Which is exactly what they did. In those time one bitflip would render the entire rom useless, a very expensive mistake. So the data should be intrinsincly verifiable if no mistake was made in the transfer, eg fax of an A4 paper with the data. Remember in those days faxes used silver thermal paper that faded over time, not really long term reliable medium and the fax quality was also terrible.
Even today if someone sends me software or binary files I am happy if they accompany it with a par2 file to verify the correctness of the files.

[retiredfeline]

OP, you also have to look at the OS support for files in those days. It was a pain in some OSes to write binary files, there were all sorts of details you had to attend to (I'm thinking of an OS like VMS where you had to tell the OS what the length of the records were, and lots of other details I forget, then you had to use special calls from the programming language to read and write), whereas even a BASIC or FORTRAN program could generate a text file. Once established, there was no incentive to move from text to binary for the sorts of uses IHX files (and other formats like S19) were put to. For the typical sizes of the files (a few KB), the overhead was not a problem. Of course OSes used binary format for executables, but they were OS specific as formats like ELF, etc. came later.

[S. Petrukhin]

OP, you also have to look at the OS support for files in those days. It was a pain in some OSes to write binary files, there were all sorts of details you had to attend to (I'm thinking of an OS like VMS where you had to tell the OS what the length of the records were, and lots of other details I forget, then you had to use special calls from the programming language to read and write), whereas even a BASIC or FORTRAN program could generate a text file. Once established, there was no incentive to move from text to binary for the sorts of uses IHX files (and other formats like S19) were put to. For the typical sizes of the files (a few KB), the overhead was not a problem. Of course OSes used binary format for executables, but they were OS specific as formats like ELF, etc. came later.

Even DOS didn't have a binary editor for a long time. After a while, Norton Commander appeared and ... already forgot the name of the utility for binary files DiskEdit, maybe.. And Basic can write both text and binary files equally well. 

[retiredfeline]

The other thing you write about, OP, is PCs and attached programmers. But remember PCs didn't arrive until 8-bit micros. Prior to that development for mask ROMs and such were done using mainframes and minicomputers. I remember the manual of an early micro dev kit (might have been Mostech) mentioning development software in FORTRAN. I seem to recall I mulled writing off to ask if I could get the software. Now I realise that even if they paid me any attention, it would have been sent out on magtape to be read on a mainframe. Transfer of the IHX files to the foundry would have been done using paper tape, punch cards, and later, floppies.

[ChristofferB]

It also makes hex editors extremely easy to implement in software.

I think the main point is that it was very logical for hand-assembly and computers with few utilities, the only disadvantage being larger file size, but that usually isnt really an issue either, so there was no need to improve upon it.

Nothing good comes from insisting on improving something that just works.

[Siwastaja]

Intel HEX format can signify memory areas to remain unchanged (no operation) from those to be written, because the address space doesn't need to be contiguous. This is impossible in raw binary. Sometimes this
is important. ELF contains section information, but for just programming certain memory areas, this is almost too much information and the programming needs to be directed.

A new "programming binary" format would be required, with headers and support for non-contiguous (sparse) data; coming up with new file formats is easy, but getting people to actually use them in masses typically requires there is significant benefit to using them. Intel HEX works fine, and embedded MCU binaries are often quite small even if the HEX format blows the size up by some 3x.

I use raw binary always when possible and try to arrange contiguous memory for the firmware (then the only parameter needed additional to the .bin file is the start address); then separate process (such as a custom piece of software the end-user or calibrator can also use) to update other parts of flash.

[AndersJ]

Intel HEX format can signify memory areas to remain unchanged (no operation) from those to be written, because the address space doesn't need to be contiguous.

Good point!

[0xdeadbeef]

Is there an inherent reason why it remained a text only format, such as serial communication being only 7-bit, and therefore preferably use ASCII?
But it seems most(/all?) programmers communicate with microcontrollers in 8 bits, so why this limitation, on the PC to programmer side?

The only real benefit of binary formats is that files are usually smaller. But this comes at a price: simply peeking into an Elf/Dwarf file is much more difficult than looking inside a text dump of it. Besides, using binaries makes it necessary to define byte order, byte size of types (enums) etc. In a nutshell, it makes things less portable, less readable, less robust.
So, if size, reading/writing speed or obfuscation is not an issue, there is no real reason to use binary formats.

Side note: At some point, programmers started to just serialize their objects to write them to a binary stream from which they could be restored later. Obviously, this was a bad idea in the first place and caused lots of issues. Small changes in their objects made the streams incompatible etc. So people started to export their stuff to XML which is (surprise!) a text format. Just one with a massive overhead that is barely used sensibly.
Most programs that use XML could just use some simple INI file format which would be more readable and smaller. But, yeah, it's not sexy.

[Siwastaja]

Quite frankly, moving to over-engineered, poorly applied, bloated text-based interchange formats such as XML has created much more problems than it has solved.

Even simple text based formats have problems time-to-time because their designers didn't take all possible footguns into account; classical example is, some text protocols require specific sequences of binary codes 10 and 13, and many text-based systems produce these control characters in different ways, and some layers convert them silently. Getting an HP power supply to communicate through UART properly comes into mind as an example where simple text-based protocol totally failed due to wrongly designed handling of CR/LF. No, it's not difficult to get right, but one thing to consider, and one thing that commonly fails; much more commonly than endianness on a binary system.

Simple binary formats have their pitfalls, but only a small number of them, they are widely known (we are reminded of them in every discussion, after all), easily enumerated, documented, and taken care of. For example, a binary protocol implemented in packed C struct has exactly one (1) footgun mechanism, endianness, to be taken care of. The added benefit of not having to write, debug, verify, learn to use, utilize parsers, and to maintain matching protocol definition in multiple places, pays back the concern of endianness a million times. Increased energy efficiency and performance is added benefit. Sometimes this matters, too.

But it's easy to sell complex overdesigned systems with hunderds of pitfalls because no one is going to enumerate those problems, and sell them by the argument of listing all known possible issues of binary formats (BYTE ORDER! PADDING!)

I have spent days fixing broken XML handling in a complex Java project. Never again.

So no, efficiency is not the only reason to use a binary interchange format. Another is simplicity and orders of magnitude smaller number of potential problems. Yet the argument against them is they are not completely free of possibilities for mistakes.

[PKTKS]

Quite frankly, moving to over-engineered, poorly applied, bloated text-based interchange formats such as XML has created much more problems than it has solved.
(..)

Absolute  true indeed.

Like several things that came with "modern" x86 interfaces
REGISTRY XML and  JSON sucks  more than solves a single bit.

Need to say that DOS was NOT (definitively NOT)  the first small
computer or Operating System of nothing..

DOS was bought to embed a BASIC interpreter in the 80s
and it was a shameless clone of CP/M with some ideas
(like directory extensions)  owned by CPM86.

Hex format is a regular CP/M format where most of the tasks
are actually made in ASM (not C like DOS) and DOS  also cloned
the  100H .COM  necessity  made by CP/M non positional loader.

Like the .COM  format (replaced by .EXE later)  the BIN format
replaced HEX  and  until today EEPROMs read/writers require 
both handlers..

I doubt it will just vanish..

Paul
 

[S. Petrukhin]

Intel HEX format can signify memory areas to remain unchanged (no operation) from those to be written, because the address space doesn't need to be contiguous. This is impossible in raw binary.

Yes, yes, yes, learn the right remark!

[S. Petrukhin]

Quite frankly, moving to over-engineered, poorly applied, bloated text-based interchange formats such as XML has created much more problems than it has solved.
(..)

Absolute  true indeed.

Like several things that came with "modern" x86 interfaces
REGISTRY XML and  JSON sucks  more than solves a single bit.

Need to say that DOS was NOT (definitively NOT)  the first small
computer or Operating System of nothing..

DOS was bought to embed a BASIC interpreter in the 80s
and it was a shameless clone of CP/M with some ideas
(like directory extensions)  owned by CPM86.

Hex format is a regular CP/M format where most of the tasks
are actually made in ASM (not C like DOS) and DOS  also cloned
the  100H .COM  necessity  made by CP/M non positional loader.

Like the .COM  format (replaced by .EXE later)  the BIN format
replaced HEX  and  until today EEPROMs read/writers require 
both handlers..

I doubt it will just vanish..

Paul

On the DOS side, I think you got a little overreacted. Yes, there are many similarities, but inside DOS is very different from the organization itself. And initially, in the days of DOS 1.0-2.01, they wrote more in Assembly language, at least in Russia. Or Basic.

These were simple and good times. At that time, merchants did not come running with their technologies, did not stir up the water that kills the brain with their loud names and presentations of useless things... Even many viruses were funny jokes.  "Hellow, World!" require about 20 bytes, not 2Mb.  Many things were inconvenient and difficult, but there was no noise and a huge pile of junk around. There are words of a Russian poet from the past about the modern information world: "A single word for a thousand tons of word ore." 

And XML-type formats aren't so bad. Before it was distributed, programmers had to agree on the transfer of data and formalize the agreement each time in each case. XML allowed everyone to agree once and for all. This is the same format for data exchange. But it became stupid to use it to store your own data, which no one else needs. This is just from the laziness of programmers. 

[Siwastaja]

And XML-type formats aren't so bad. Before it was distributed, programmers had to agree on the transfer of data and formalize the agreement each time in each case. XML allowed everyone to agree once and for all. This is the same format for data exchange.

Sorry but this makes absolutely zero sense, yet a common argument in management powerpoints. And management continues to believe in it because they don't need to do all the work.

XML defines the structure and delimitation of data. It defines the syntax to organize the data in a tree, but you need to define all the elements. So you still need application-specific agreement under what name, and within which tags (at which part of the tree) you store the asshole temperature or whatever, and even what datatypes are used, what the units are, what everything means. XML Schemas can formalize this process, but it still needs to be done. I don't see any way except self-programming AI to get rid of having to specify data formats.

The only thing XML does is to add more complexity to the process, necessitating longer design sessions and specification meetings. There are tools to help, but then you are learning to use them.

Now it's disputable whether there are some (complex) use cases when the XML and the associated libraries and tools make life easier in the end, and I fully accept someone feels it's the right way, but in no way it helps you get rid of doing the specification work, no way. In simple cases, it definitely makes your life a lot harder, compared to, for example, a C struct between two parties, which works as a specification and implementation at the same time without requirement to use a parser at all; but this does have its limitations. This has worked in networking for decades, and works wonders in embedded.

[S. Petrukhin]

And XML-type formats aren't so bad. Before it was distributed, programmers had to agree on the transfer of data and formalize the agreement each time in each case. XML allowed everyone to agree once and for all. This is the same format for data exchange.

Sorry but this makes absolutely zero sense, yet a common argument in management powerpoints. And management continues to believe in it because they don't need to do all the work.

XML defines the structure and delimitation of data. It defines the syntax to organize the data in a tree, but you need to define all the elements. So you still need application-specific agreement under what name, and within which tags (at which part of the tree) you store the asshole temperature or whatever, and even what datatypes are used, what the units are, what everything means. XML Schemas can formalize this process, but it still needs to be done. I don't see any way except self-programming AI to get rid of having to specify data formats.

The only thing XML does is to add more complexity to the process, necessitating longer design sessions and specification meetings. There are tools to help, but then you are learning to use them.

Now it's disputable whether there are some (complex) use cases when the XML and the associated libraries and tools make life easier in the end, and I fully accept someone feels it's the right way, but in no way it helps you get rid of doing the specification work, no way. In simple cases, it definitely makes your life a lot harder, compared to, for example, a C struct between two parties, which works as a specification and implementation at the same time without requirement to use a parser at all; but this does have its limitations. This has worked in networking for decades, and works wonders in embedded.

One moment... After describing the data in XML, you will not only list the values, but also specify their type and purpose. This will allow you to personally negotiate with no one. Anyone else can understand the content on their own and compare the data. Of course, you can describe it in a text file in human language and it will probably be even clearer. But let's look at another case: records with options. You can have a million different fields in the data structure, but only pass the relevant values in XML. And, of course, we will not lose sight of the fact that this format was driven by influential forces. Personally, I don't need it, but there may be people who give up, use it, and ask me to transmit data in this format because they have processing tools. 

[rstofer]

First reason:  We were using Teletypes and paper tape.  What was on the tape had to be printable on the paper.  There was a time before CRTs became available.  In grad school ('75-'76) we had Teletypes all over campus.  Noisy beasts!

Second reason:  Human readability.  I can see a pattern like C3 00 00 and immediately know that it is a jump instruction to address 0x0000 for an 8080 instruction set.  Same as 21 00 00 as LXI H,0x0000.  With a lot of effort, I could disassemble the program from the tape.  Some versions of Basic were distributed on paper tape and disassembly was a popular pasttime.

The biggest machines had 64k RAM and the Teletype was 10 cps or about 5 bytes/sec.  It took a very long tape a very long time to load the machine.  Dedicated tape readers were a lot faster.

That's why we went to audio cassette tape.  Shortly thereafter we had 8" floppies and just as soon as those worked (thanks to the Western Digital 1771 Floppy Controller chip) we had CP/M and the rest is history.

[Benta]

Back when Intel hex and Motorola S-records were introduced (70s), there was absolutely NO established binary standard type of file defined.
IBM had EBCDIC, the rest of the world ASCII character sets.

Text was the only way to do it.

Input devices were punched tape, teletypes, keyboards...; output was text screen, teletypes or line printers. (Although teletypes had their own 5-bit encoding, but that's diverging).

So Intel hex or S-records evolved to be the standard for binary exchange as well as the standard for all EPROM/Flash programmers that I've seen.

So why change it? It's simple and easy to understand and all machines can interpret it. And there's zero benefit in a change.

It's a bit like a utility saying "Hey, we don't like 120 VAC, let's change it to 147 V". Pointless.

[coppice]

For decades there were so many channels, both 7 bit and 8 bit, which were not 8 bit clean, that a text file made a lot of sense. When software tools were much cruder, there was also real merit in a format that was easy to manually edit. Sure, the lines have a checksum, but those are easy to calculate and patch up. Until MIME email came along, I couldn't even quickly exchange binary files with colleagues far away, unless we agreed to encode them in some way before inserting them in the email.

Starting now, there are far more options that would make sense, than made sense in the early 70s. You had to be there, man!

[rstofer]

It's a bit like a utility saying "Hey, we don't like 120 VAC, let's change it to 147 V". Pointless.

But the US did go through a progression:  110V, 115V, 117V, 120V (standardized in 1967)
I suspect we will stay with 120V.

[rstofer]

Starting now, there are far more options that would make sense, than made sense in the early 70s. You had to be there, man!

I was and it's been a heck of a ride over the last 50 years.

[PKTKS]

On the DOS side, I think you got a little overreacted. Yes, there are many similarities, but inside DOS is very different from the organization itself. And initially, in the days of DOS 1.0-2.01, they wrote more in Assembly language, at least in Russia. Or Basic.
(..)

Probably because they even bother to translate whatever was possible
from Z80 ASM directly to 8086 (which mostly inherited all Z80 good features
lacking on 8085)

"EXPEDITE"  lazy..  coffy..  whatever... they had 50% work done

CPM86 was very late at that time ...
lucky bastards saw the such called window..

Paul