EEVblog® Electronics Community Forum
Electronics => Projects, Designs, and Technical Stuff => Topic started by: petert on December 12, 2020, 03:22:00 am
-
Hello,
I read about the history of Intel Hex starting on paper. But soon after it was possible to use it with floppies as well.
Executable files like PE or ELF are also in sections, and are in binary, so you could have gaps like in Intel Hex.
Is there an inherent reason why it remained a text only format, such as serial communication being only 7-bit, and therefore preferably use ASCII?
But it seems most(/all?) programmers communicate with microcontrollers in 8 bits, so why this limitation, on the PC to programmer side?
If anybody knows more about the rationale, I'd be curious.
-
I cannot say for sure, but my opinion is it stays in hex because there's no need to change it. I would suggest that every computer filing system ever invented can handle the characters in a hex file. So it can be passed from computer to computer as a text file and not get corrupted. And since it is a 2 character to one byte format, it is possible to make some sense of it (especially the address part). It is not so easy to make sense of data packed with base64 (packing 3 bytes into 4 characters).
-
They have been invented by old timey people, who are used to used to computers only being able to handle text. Also, they may not have indented for it to get a wide use, so they used whatever worked at a time for a specific purpose.
There is also no particular reason to have all those CRCs. They should not be a part of the file format. If your medium or communication channel are so bad, you need to address the checking on that level.
-
https://en.wikipedia.org/wiki/Intel_HEX#History
tl;dr: punched cards and paper tape. The equipment used to create mask ROMs likely predates microcomputers, so the format was retained as a convention.
I remember one Japanese mask ROM manufacturer requested ROM images be faxed(!) to them, as printouts in hex format...
-
It seems to me that the main reason was that there were no compilers at the time of the birth of this format, or they were not available. People did the compilation manually and recorded in the form of text binary information - machine code. Writing each byte in decimal would be inconvenient and difficult to read. HEX file display format is still actively used, it is in some cases much more convenient than binary and decimal. However, for a long time all compilers have the BIN format.
-
There is also no particular reason to have all those CRCs. They should not be a part of the file format. If your medium or communication channel are so bad, you need to address the checking on that level.
Which is exactly what they did. In those time one bitflip would render the entire rom useless, a very expensive mistake. So the data should be intrinsincly verifiable if no mistake was made in the transfer, eg fax of an A4 paper with the data. Remember in those days faxes used silver thermal paper that faded over time, not really long term reliable medium and the fax quality was also terrible.
Even today if someone sends me software or binary files I am happy if they accompany it with a par2 file to verify the correctness of the files.
-
OP, you also have to look at the OS support for files in those days. It was a pain in some OSes to write binary files, there were all sorts of details you had to attend to (I'm thinking of an OS like VMS where you had to tell the OS what the length of the records were, and lots of other details I forget, then you had to use special calls from the programming language to read and write), whereas even a BASIC or FORTRAN program could generate a text file. Once established, there was no incentive to move from text to binary for the sorts of uses IHX files (and other formats like S19) were put to. For the typical sizes of the files (a few KB), the overhead was not a problem. Of course OSes used binary format for executables, but they were OS specific as formats like ELF, etc. came later.
-
OP, you also have to look at the OS support for files in those days. It was a pain in some OSes to write binary files, there were all sorts of details you had to attend to (I'm thinking of an OS like VMS where you had to tell the OS what the length of the records were, and lots of other details I forget, then you had to use special calls from the programming language to read and write), whereas even a BASIC or FORTRAN program could generate a text file. Once established, there was no incentive to move from text to binary for the sorts of uses IHX files (and other formats like S19) were put to. For the typical sizes of the files (a few KB), the overhead was not a problem. Of course OSes used binary format for executables, but they were OS specific as formats like ELF, etc. came later.
Even DOS didn't have a binary editor for a long time. After a while, Norton Commander appeared and ... already forgot the name of the utility for binary files DiskEdit, maybe.. And Basic can write both text and binary files equally well. :)
-
The other thing you write about, OP, is PCs and attached programmers. But remember PCs didn't arrive until 8-bit micros. Prior to that development for mask ROMs and such were done using mainframes and minicomputers. I remember the manual of an early micro dev kit (might have been Mostech) mentioning development software in FORTRAN. I seem to recall I mulled writing off to ask if I could get the software. Now I realise that even if they paid me any attention, it would have been sent out on magtape to be read on a mainframe. Transfer of the IHX files to the foundry would have been done using paper tape, punch cards, and later, floppies.
-
It also makes hex editors extremely easy to implement in software.
I think the main point is that it was very logical for hand-assembly and computers with few utilities, the only disadvantage being larger file size, but that usually isnt really an issue either, so there was no need to improve upon it.
Nothing good comes from insisting on improving something that just works.
-
Intel HEX format can signify memory areas to remain unchanged (no operation) from those to be written, because the address space doesn't need to be contiguous. This is impossible in raw binary. Sometimes this
is important. ELF contains section information, but for just programming certain memory areas, this is almost too much information and the programming needs to be directed.
A new "programming binary" format would be required, with headers and support for non-contiguous (sparse) data; coming up with new file formats is easy, but getting people to actually use them in masses typically requires there is significant benefit to using them. Intel HEX works fine, and embedded MCU binaries are often quite small even if the HEX format blows the size up by some 3x.
I use raw binary always when possible and try to arrange contiguous memory for the firmware (then the only parameter needed additional to the .bin file is the start address); then separate process (such as a custom piece of software the end-user or calibrator can also use) to update other parts of flash.
-
Intel HEX format can signify memory areas to remain unchanged (no operation) from those to be written, because the address space doesn't need to be contiguous.
Good point!
-
Is there an inherent reason why it remained a text only format, such as serial communication being only 7-bit, and therefore preferably use ASCII?
But it seems most(/all?) programmers communicate with microcontrollers in 8 bits, so why this limitation, on the PC to programmer side?
The only real benefit of binary formats is that files are usually smaller. But this comes at a price: simply peeking into an Elf/Dwarf file is much more difficult than looking inside a text dump of it. Besides, using binaries makes it necessary to define byte order, byte size of types (enums) etc. In a nutshell, it makes things less portable, less readable, less robust.
So, if size, reading/writing speed or obfuscation is not an issue, there is no real reason to use binary formats.
Side note: At some point, programmers started to just serialize their objects to write them to a binary stream from which they could be restored later. Obviously, this was a bad idea in the first place and caused lots of issues. Small changes in their objects made the streams incompatible etc. So people started to export their stuff to XML which is (surprise!) a text format. Just one with a massive overhead that is barely used sensibly.
Most programs that use XML could just use some simple INI file format which would be more readable and smaller. But, yeah, it's not sexy.
-
Quite frankly, moving to over-engineered, poorly applied, bloated text-based interchange formats such as XML has created much more problems than it has solved.
Even simple text based formats have problems time-to-time because their designers didn't take all possible footguns into account; classical example is, some text protocols require specific sequences of binary codes 10 and 13, and many text-based systems produce these control characters in different ways, and some layers convert them silently. Getting an HP power supply to communicate through UART properly comes into mind as an example where simple text-based protocol totally failed due to wrongly designed handling of CR/LF. No, it's not difficult to get right, but one thing to consider, and one thing that commonly fails; much more commonly than endianness on a binary system.
Simple binary formats have their pitfalls, but only a small number of them, they are widely known (we are reminded of them in every discussion, after all), easily enumerated, documented, and taken care of. For example, a binary protocol implemented in packed C struct has exactly one (1) footgun mechanism, endianness, to be taken care of. The added benefit of not having to write, debug, verify, learn to use, utilize parsers, and to maintain matching protocol definition in multiple places, pays back the concern of endianness a million times. Increased energy efficiency and performance is added benefit. Sometimes this matters, too.
But it's easy to sell complex overdesigned systems with hunderds of pitfalls because no one is going to enumerate those problems, and sell them by the argument of listing all known possible issues of binary formats (BYTE ORDER! PADDING!)
I have spent days fixing broken XML handling in a complex Java project. Never again.
So no, efficiency is not the only reason to use a binary interchange format. Another is simplicity and orders of magnitude smaller number of potential problems. Yet the argument against them is they are not completely free of possibilities for mistakes.
-
Quite frankly, moving to over-engineered, poorly applied, bloated text-based interchange formats such as XML has created much more problems than it has solved.
(..)
Absolute true indeed.
Like several things that came with "modern" x86 interfaces
REGISTRY XML and JSON sucks more than solves a single bit.
Need to say that DOS was NOT (definitively NOT) the first small
computer or Operating System of nothing..
DOS was bought to embed a BASIC interpreter in the 80s
and it was a shameless clone of CP/M with some ideas
(like directory extensions) owned by CPM86.
Hex format is a regular CP/M format where most of the tasks
are actually made in ASM (not C like DOS) and DOS also cloned
the 100H .COM necessity made by CP/M non positional loader.
Like the .COM format (replaced by .EXE later) the BIN format
replaced HEX and until today EEPROMs read/writers require
both handlers..
I doubt it will just vanish..
Paul
-
Intel HEX format can signify memory areas to remain unchanged (no operation) from those to be written, because the address space doesn't need to be contiguous. This is impossible in raw binary.
Yes, yes, yes, learn the right remark!
-
Quite frankly, moving to over-engineered, poorly applied, bloated text-based interchange formats such as XML has created much more problems than it has solved.
(..)
Absolute true indeed.
Like several things that came with "modern" x86 interfaces
REGISTRY XML and JSON sucks more than solves a single bit.
Need to say that DOS was NOT (definitively NOT) the first small
computer or Operating System of nothing..
DOS was bought to embed a BASIC interpreter in the 80s
and it was a shameless clone of CP/M with some ideas
(like directory extensions) owned by CPM86.
Hex format is a regular CP/M format where most of the tasks
are actually made in ASM (not C like DOS) and DOS also cloned
the 100H .COM necessity made by CP/M non positional loader.
Like the .COM format (replaced by .EXE later) the BIN format
replaced HEX and until today EEPROMs read/writers require
both handlers..
I doubt it will just vanish..
Paul
On the DOS side, I think you got a little overreacted. :) Yes, there are many similarities, but inside DOS is very different from the organization itself. And initially, in the days of DOS 1.0-2.01, they wrote more in Assembly language, at least in Russia. Or Basic. :)
These were simple and good times. At that time, merchants did not come running with their technologies, did not stir up the water that kills the brain with their loud names and presentations of useless things... Even many viruses were funny jokes. "Hellow, World!" require about 20 bytes, not 2Mb. :) Many things were inconvenient and difficult, but there was no noise and a huge pile of junk around. There are words of a Russian poet from the past about the modern information world: "A single word for a thousand tons of word ore." :)
And XML-type formats aren't so bad. Before it was distributed, programmers had to agree on the transfer of data and formalize the agreement each time in each case. XML allowed everyone to agree once and for all. This is the same format for data exchange. But it became stupid to use it to store your own data, which no one else needs. This is just from the laziness of programmers. :)
-
And XML-type formats aren't so bad. Before it was distributed, programmers had to agree on the transfer of data and formalize the agreement each time in each case. XML allowed everyone to agree once and for all. This is the same format for data exchange.
Sorry but this makes absolutely zero sense, yet a common argument in management powerpoints. And management continues to believe in it because they don't need to do all the work.
XML defines the structure and delimitation of data. It defines the syntax to organize the data in a tree, but you need to define all the elements. So you still need application-specific agreement under what name, and within which tags (at which part of the tree) you store the asshole temperature or whatever, and even what datatypes are used, what the units are, what everything means. XML Schemas can formalize this process, but it still needs to be done. I don't see any way except self-programming AI to get rid of having to specify data formats.
The only thing XML does is to add more complexity to the process, necessitating longer design sessions and specification meetings. There are tools to help, but then you are learning to use them.
Now it's disputable whether there are some (complex) use cases when the XML and the associated libraries and tools make life easier in the end, and I fully accept someone feels it's the right way, but in no way it helps you get rid of doing the specification work, no way. In simple cases, it definitely makes your life a lot harder, compared to, for example, a C struct between two parties, which works as a specification and implementation at the same time without requirement to use a parser at all; but this does have its limitations. This has worked in networking for decades, and works wonders in embedded.
-
And XML-type formats aren't so bad. Before it was distributed, programmers had to agree on the transfer of data and formalize the agreement each time in each case. XML allowed everyone to agree once and for all. This is the same format for data exchange.
Sorry but this makes absolutely zero sense, yet a common argument in management powerpoints. And management continues to believe in it because they don't need to do all the work.
XML defines the structure and delimitation of data. It defines the syntax to organize the data in a tree, but you need to define all the elements. So you still need application-specific agreement under what name, and within which tags (at which part of the tree) you store the asshole temperature or whatever, and even what datatypes are used, what the units are, what everything means. XML Schemas can formalize this process, but it still needs to be done. I don't see any way except self-programming AI to get rid of having to specify data formats.
The only thing XML does is to add more complexity to the process, necessitating longer design sessions and specification meetings. There are tools to help, but then you are learning to use them.
Now it's disputable whether there are some (complex) use cases when the XML and the associated libraries and tools make life easier in the end, and I fully accept someone feels it's the right way, but in no way it helps you get rid of doing the specification work, no way. In simple cases, it definitely makes your life a lot harder, compared to, for example, a C struct between two parties, which works as a specification and implementation at the same time without requirement to use a parser at all; but this does have its limitations. This has worked in networking for decades, and works wonders in embedded.
One moment... After describing the data in XML, you will not only list the values, but also specify their type and purpose. This will allow you to personally negotiate with no one. Anyone else can understand the content on their own and compare the data. Of course, you can describe it in a text file in human language and it will probably be even clearer. But let's look at another case: records with options. You can have a million different fields in the data structure, but only pass the relevant values in XML. And, of course, we will not lose sight of the fact that this format was driven by influential forces. Personally, I don't need it, but there may be people who give up, use it, and ask me to transmit data in this format because they have processing tools. :-//
-
First reason: We were using Teletypes and paper tape. What was on the tape had to be printable on the paper. There was a time before CRTs became available. In grad school ('75-'76) we had Teletypes all over campus. Noisy beasts!
Second reason: Human readability. I can see a pattern like C3 00 00 and immediately know that it is a jump instruction to address 0x0000 for an 8080 instruction set. Same as 21 00 00 as LXI H,0x0000. With a lot of effort, I could disassemble the program from the tape. Some versions of Basic were distributed on paper tape and disassembly was a popular pasttime.
The biggest machines had 64k RAM and the Teletype was 10 cps or about 5 bytes/sec. It took a very long tape a very long time to load the machine. Dedicated tape readers were a lot faster.
That's why we went to audio cassette tape. Shortly thereafter we had 8" floppies and just as soon as those worked (thanks to the Western Digital 1771 Floppy Controller chip) we had CP/M and the rest is history.
-
Back when Intel hex and Motorola S-records were introduced (70s), there was absolutely NO established binary standard type of file defined.
IBM had EBCDIC, the rest of the world ASCII character sets.
Text was the only way to do it.
Input devices were punched tape, teletypes, keyboards...; output was text screen, teletypes or line printers. (Although teletypes had their own 5-bit encoding, but that's diverging).
So Intel hex or S-records evolved to be the standard for binary exchange as well as the standard for all EPROM/Flash programmers that I've seen.
So why change it? It's simple and easy to understand and all machines can interpret it. And there's zero benefit in a change.
It's a bit like a utility saying "Hey, we don't like 120 VAC, let's change it to 147 V". Pointless.
-
For decades there were so many channels, both 7 bit and 8 bit, which were not 8 bit clean, that a text file made a lot of sense. When software tools were much cruder, there was also real merit in a format that was easy to manually edit. Sure, the lines have a checksum, but those are easy to calculate and patch up. Until MIME email came along, I couldn't even quickly exchange binary files with colleagues far away, unless we agreed to encode them in some way before inserting them in the email.
Starting now, there are far more options that would make sense, than made sense in the early 70s. You had to be there, man!
-
It's a bit like a utility saying "Hey, we don't like 120 VAC, let's change it to 147 V". Pointless.
But the US did go through a progression: 110V, 115V, 117V, 120V (standardized in 1967)
I suspect we will stay with 120V.
-
Starting now, there are far more options that would make sense, than made sense in the early 70s. You had to be there, man!
I was and it's been a heck of a ride over the last 50 years.
-
On the DOS side, I think you got a little overreacted. :) Yes, there are many similarities, but inside DOS is very different from the organization itself. And initially, in the days of DOS 1.0-2.01, they wrote more in Assembly language, at least in Russia. Or Basic. :)
(..)
Probably because they even bother to translate whatever was possible
from Z80 ASM directly to 8086 (which mostly inherited all Z80 good features
lacking on 8085)
"EXPEDITE" lazy.. coffy.. whatever... they had 50% work done
CPM86 was very late at that time ...
lucky bastards saw the such called window..
Paul
-
For example, a binary protocol implemented in packed C struct has exactly one (1) footgun mechanism, endianness, to be taken care of.
Oh actually there are a few more. You need to make sure that e.g. you never take a pointer to a member of a packed struct, because then you are going to violate alignment constraints on non-x86 processors (sometimes, even on x86). Also, if memory serves right, packed bitfields aren't portable. Of course, only fixed size types may be use (no "int").
And obviously, all of this depends on non-standard C compiler extensions, so I don't quite see how that compares to actually well-defined (binary or text) formats.
-
Thanks for all the replies and the discussion!
I wondered if Intel Hex (and similar formats), were meant to be used as a kind of stream-based format, a bit like AT commands for modems, which can be issued one by one (one per line).
Was/is there any device/software than can do something similar with individual Intel Hex records/stream of records, or do they all expect a complete file?
-
Thanks for all the replies and the discussion!
I wondered if Intel Hex (and similar formats), were meant to be used as a kind of stream-based format, a bit like AT commands for modems, which can be issued one by one (one per line).
Was/is there any device/software than can do something similar with individual Intel Hex records/stream of records, or do they all expect a complete file?
This is just a human-friendly display of a binary file, it's not a set of commands.
Of course, you can transfer this file in its entirety, or you can transfer it in parts.
There are programs that allow you to edit the contents of memory one byte at a random address.
But with Flash and the MCU (which has built-in Flash), we must not forget that Flash memory cannot be written one byte at a time. More precisely, you can "write" any Flash cell separately, but only by erasing the 1s that appear after erasing the entire page at once.
-
I wondered if Intel Hex (and similar formats), were meant to be used as a kind of stream-based format, a bit like AT commands for modems, which can be issued one by one (one per line).
It's a stretch to view the format a set of commands when really the lines just specify addresses and contents, and no return channel is required. It's just a text encoding of binary data as others have written.
-
Storage formats before disk like paper tape and punched cards were not always 8 bits and even if they were, serial communications links were often 7 bits anyway. The HEX format was dense enough, includes error detection, and is easy to work with.
-
Somehow, this is a thread where only us old guys that were there can contribute to.
Museum stuff... that always worked.
Makes me think of Voyager 1 and 2... 15 Billion km away now in interstellar space, transmitting at 50 bits/s with the power of a car brake light. Still works. No streaming.
-
Years before XML was introduced, the RIFF format was a relatively decent, and relatively easy-to-deal-with structured binary format. AFAIR, it got used quite a lot for all kinds of files in the 90's (way outside of MS and IBM where it originated) and in the early 2000's. Then XML appeared and progressively took over.
Of course a pure text format is easier to read and edit if you ever need to do this manually, and from a programming standpoint, you don't need to deal with endianness and related issues (well, at least you can avoid that). Drawback is that it takes a lot more space. But this can be mitigated by just compressing the files - XML files tend to compress nicely.
Then JSON became trendy and many switched from XML to JSON. Sure JSON is more frugal and maybe a bit more structured? But it's not ideal either.
I designed some proprietary structured binary format a few years ago, but even though I find it useful, it's probably far from being simple or universal enough to possibly become useful for all.
As to the Hex format, it's simple and pervasive enough that it's still widely used. There's also the Motorola SREC format ( https://en.wikipedia.org/wiki/SREC_(file_format) ) that many tools still support.
The benefit of a format that has become a de-facto standard, even if it's not perfect, is that it makes it easier for everyone - tool designers, users... and it can take a lot of time for a new format to become as widely accepted.
-
Years before XML was introduced, the RIFF format was a relatively decent, and relatively easy-to-deal-with structured binary format. AFAIR, it got used quite a lot for all kinds of files in the 90's (way outside of MS and IBM where it originated) and in the early 2000's. Then XML appeared and progressively took over.
There are still some important RIFF based file formats. Wave files for audio use a RIFF structure. TIFF files for images use a RIFF structure. XML is a useful format for text, but it sucks for binary content like audio, video and images.
-
Not only was paper tape the dominant means of interchange in the 1960s when the LSI revolution began, but it is still to this day the most durable information storage media. As long as the material is of high quality (mylar), it will remain readable for thousands of years, through nuclear wars, infrastructure collapse, global warming, etc.
-
Only very low data density.
-
First reason: We were using Teletypes and paper tape.
And, you know, actually re-typing things from the published article into a different device.
Intel Hex has enough readability and error checking that you can do that. (checksum every 16 bytes or so, so you don't even get very far before you get notified of an error. Theoretically, anyway.)
Was/is there any device/software than can do something similar with individual Intel Hex records/stream of records
Why yes. I wrote one relatively recently, in fact. I wanted to program some <obsolete microcontrollers that use HV parallel programming>, so I wrote an Arduino sketch to do it It started with simple write/read commands, and then I added a feature where if you type in a line of intel Hex, it'd program that data at that address. I can pretty much copy/paste from "edit foo.hex" to the Serial Monitor, one line at a time to program a chip, bypassing the need for anything that understands files, or serial ports, or ... anything.
(the current version of the code to be programed is about 100 bytes, so I haven't bothered to check whether I can paste whole .hex files.) I suspect that this sort of thing was common back when "ROM Monitors" were common (8051 boards, MIT Handyboard, etc.)
(Note that often, a chip will have a page-based programming algorithm, where you have to program 64 bytes at a time, or whatever. That gets more awkward...)
Serial.print("Cmd: ");
ttycli.reset();
ttycli.getLineWait();
clicmd = ttycli.tryihex(&addr, results);
if (clicmd > 0) { // We have an intel hex file line?
for (int i=0; i < clicmd; i++) {
writeFlashByte(addr, results[i]);
addr++;
}
return; /* next command */
}
// else try an "interactive" command.
-
I find it quite funny that if you know where to look you can still see parts of windows 3 UI in windows 10, and even many throwbacks fro DOS.It is all part of backwards compatibility and part of why PCs using Windows are far from as efficient as they could be. It is a bloated code base full of known bugs kept for compatibility or to comply with old standards.
I have for the last 10 years or so supported modern mainframes, and before that much older ones. Very little has changed, all fault codes are in hex, much of the operation interface is in hex. All for good reasons explained a few times in this thread. We could go as far back as punch codes working in nibbles rather than bytes.so with only four binary digits available we could only count to 16 which makes hex seem more obvious as to why it was the chosen method.
-
It's text because it's intel hex.
If you use binary it's not intel hex.
The format exists in the way it is because somebody some 50 years ago thought it was a good idea.
And apparently it was a good idea because it still exists and is being used, which is rare for such old file formats.
-
apparently it was a good idea because it still exists and is being used, which is rare for such old file formats.
Indeed. One of its major disadvantages, that the .hex file is more than twice the size of the binarydata , has become totally irrelevant because "bulk storage" has advanced at a higher rate than nearly any other aspect of computing. (and simultaneously, "sparse" data in .hex files has made it less likely that a .hex file for a given image will take less space than a pure binary file...)