How do you preserve the IDE/toolchain over many years?

[fourtytwo42]

Something I have not seen mentioned and has caught me on more than one occasion is programming hardware and it's compatibility with different pc's although I must confess this tended to be more common in the days of parallel printer ports it can and still does happen with USB ports particularly those weak on the 5V voltage.  So if your not going to keep the actual PC used for the project it may well be a good idea to go through the motions of programming with the new PC before binning the old one.

[coppice]

Something I have not seen mentioned and has caught me on more than one occasion is programming hardware and it's compatibility with different pc's although I must confess this tended to be more common in the days of parallel printer ports it can and still does happen with USB ports particularly those weak on the 5V voltage.  So if your not going to keep the actual PC used for the project it may well be a good idea to go through the motions of programming with the new PC before binning the old one.

The problem was mostly parallel ports in the last 5 years or so of them being a normal feature of PCs, and mostly in notebooks. A lot of development tools were made very quirky by the reduction in pin drive strength during that period. It was the final driver for many vendors to get their USB tools out the door, although the lack of parallel ports on notebooks would soon have created another impetus.

I wonder if there is a lesson here which will be repeated? The use of USB for most current tools means you no longer have the issue of using an interface for a non-intended purpose, and having to live with any quirks the evolution of the port hardware creates for you. Many high consumption USB devices, like hard disks, have required power from multiple USB sockets, which might leads to quirks as the USB ports evolve. However, the high consumption development tools I have seen use a power brick to get their power. So, they do not use USB in any kind of non-standard manner. As long as there are USB ports, the hardware of the tools should be OK. The software drivers, on the other hand.......

[tszaboo]

The most scary thing for me is embedded linux development. Our current procedure for setting up a new dev enviroment requires to pull data from a dozen online repositories. And the entire thing is a big blob of a black box, nobody really understanding what the actual hell going on inside.

If there is a single failure, like a server goes down, or files get deleted, or anything like that... well good luck finding out how to do it. Or what is actually missing. How on earth can anyone recommend embedded linux for anything is beyond me.

And then, you consider connecting your linux to the internet. Well good luck. Constant updates to every stupid thing, kernels coming faster than fully testing the functionality.

So we have a VM, and if that doesnt work, hope that nothing changed on the internet.

"I like to find out why someone else's code doesnt compile properly." -said nobody ever.

[coppice]

The most scary thing for me is embedded linux development. Our current procedure for setting up a new dev enviroment requires to pull data from a dozen online repositories. And the entire thing is a big blob of a black box, nobody really understanding what the actual hell going on inside.

If there is a single failure, like a server goes down, or files get deleted, or anything like that... well good luck finding out how to do it. Or what is actually missing. How on earth can anyone recommend embedded linux for anything is beyond me.

And then, you consider connecting your linux to the internet. Well good luck. Constant updates to every stupid thing, kernels coming faster than fully testing the functionality.

So we have a VM, and if that doesnt work, hope that nothing changed on the internet.

"I like to find out why someone else's code doesnt compile properly." -said nobody ever.

How is that worse than using any other platform?

[legacy]

Keeping a set of mothballed laptops isn't an awful idea, assuming that they really include everything that you need

for me it works great!
There is also tons of hw equipment which needs the LPT port, the PCMCIA port, etc
all tasks where a virtual machine sucks great, and I can't loose the working days to fix them.

The problem gets "different" in larger companies.   I'm sure it was someone's responsibility to make sure that entire build environments for all active branches continued to work, as the company transitioned from 680x0/HPUX build systems to Sparc/SunOS build systems to Sparc/Solaris to x86/Solaris to x86 Linux server farms, but I'm glad it wasn't mine!

That's (a part of) my duty. A freak Hell 

[legacy]

All of the toolchains are installed to /opt and binaries symlinked to /usr/local/bin/arm-none-eabi__VERSION_HERE__-gcc

We need to support gcc-v2.95, do you really believe it's a piece of cake?
gcc-v2.95 is broken on modern rootfs due to glibc incompatibility!

Thus, what you MUST do is having tons of minirootfs, to chroot into! Each one with its libraries, and tools and dependencies.

Old tools (even Makefile) has changed during these years, even the motorola assembly GAS for m68k: completely incompatible with legacy projects!

Do you want? Can you spend weeks of time to import legacy projects into modern toolchains? It depends IF the customers agree and IF they pay for it.

Usually no!

Oh, and beyond the rootfs we also have kernel incompatibilities! Thus what you MUST do is putting everything into a VM.


Your solution only works for a short window of time, say no more than five-six years 

[legacy]

Another thing we do is: cloning the whole hard drive partition into a chunk of external memory, usually stored into a server's array of disks. We usually choose 32GB for the root (linux or windows), thus it comfortably suits the size of 40GB offered by DLT-tapes (as backup).

We also have a special partition in the laptop for the bootstrapper. It's a linux partition, embedded profiled (forget the graphical GUI), which contains tools like "ssh", "scp", "dd", and tons of scripts usable to restore the OS-partition through the net.

"Draft-projects" are always stored in the second partition (VFAT), this they are never affected by any restore/clone procedure, while "in-trunk-projects" are stored into a server, and can be cloned back on demand.

These tricks have saved us from the need of buying a laptop for each project since we can group similar projects to the same laptop, where people can clone the required software into the harddrive.

[Howardlong]

The biggest risks nowadays comprise not just software, but parts too. My risk list is:

o Company instability, including mergers/acquisitions, bankruptcy or illiquidity.

o EOL'd parts

o Subscription- or time-limited software with either phone home or date/time lookup - AVOID if at all possible!

o Deprecated features, and quicksand software (frequent development-driven agile half-finished releases with shoddy QA [there is a definite correlation!])

I've been directly affected by all of these. At my worst point, I found myself having to completely redesign a product after a fabless supplier went bust, frustratingly with 1/4 million devices idly sitting at their fab assembler, but inaccessible due to the liquidator not releasing them. A year later I had a call asking me if I wanted any, but I'd already redesigned the product. Liquidators are stupid sometimes, I'd originally offered them £70k for the lot, they turned me down, and wouldn't release any at all: a year later they were worth bugger all, as all their customers, like me, had moved on.

EOL'd parts happen for me on pretty much a monthly basis, it's part of life. Often they're just jelly bean passives but for some parts you have to do a substantial re-characterisation. There's not a lot of practical difference between EOL or an out of stock item, you still have to re-test. For some parts there is an MOQ, which ties up capital for an extended period depending on your run rate, and that in itself has risk. While it's a reasonable practice to select more generic parts, it's not always practical, it's the usual set of balancing engineering requirements and priorities, and compromising appropriately.

Maintaining software builds are frustrating, particularly in mothball scenarios, and my solution is a frozen VMWare VM, but even that is not fool proof if you have phone home or subscription time limited dependencies. Even fiddling dates/times nowadays has its risks as it breaks things like Kerberos and certificates.

Deprecated software or quicksand software can be mitigated against with VMs, but there is a remaining problem in that you need to maintain currency on older technology, and its quirks. Personally I prefer that, to having to continually maintain dozens and dozens of mothballed environments.

The benefits of a VM over physical hardware for build environments is that the HALs are consistent, you can archive them more easily, and you can move them around. you can also fiddle dates and times, but this is becoming less easy to do. The downside is that if you have some older stuff that has dependencies on physical COM or LPT ports, and they're targeted directly, you're often out of luck.

[NorthGuy]

Arguably, in this context a VM is the single part, so that ought to be optimum w.r.t. "Less parts means more reliability".

"Too big to fail" some would say.

[Tomorokoshi]

I have a project where the code base goes back to 2003. Latest build was this summer. CCS compiler with old Microchip IDE. Easily copied to another computer if needed.

Another project is 10 years old, but it uses an IDE and a compiler that have a network license. This is something I like to avoid, but:

((micromanagement - management) + engineering) = microengineering

drove that decision, making eventual updates tenuously dependent on the availability of the network, the skills of the maintenance of the systems, etc.

Make full backup copies of source code in the clear. Include all development tools, including the source code and build tools of any conversion programs that are used. Also backup the programs that run the programmer. Don't rely on access to the repository. Those also can get lost, upgraded out of existence, or broken as systems move on.

For an embedded systems include a directory with datasheets of all the parts.

Finally, make a long, detailed document that covers every step of the build process, from installing the development tools to doing the build and programming the target. Include basic guidance of the source structure and key areas and functions. Perhaps write it in Word, use a lot of screenshots, and save to PDF.

[nctnico]

The most scary thing for me is embedded linux development. Our current procedure for setting up a new dev enviroment requires to pull data from a dozen online repositories. And the entire thing is a big blob of a black box, nobody really understanding what the actual hell going on inside.

Perhaps someone should think about that and -like me- keep all the downloaded source files together with the environment. Actually embedded Linux environments are pretty much self containing eco systems. Everything -including the compiler- is build from scratch. The biggest problem is that script languages like Python, Yacc, Bison, etc have incompatibilities between versions so it is best to have a virtual machine to also keep the OS environment stable.

Knowing how the build environment works is a matter of spending time on it. It isn't black magic.

[technix]

The most scary thing for me is embedded linux development. Our current procedure for setting up a new dev enviroment requires to pull data from a dozen online repositories. And the entire thing is a big blob of a black box, nobody really understanding what the actual hell going on inside.

If there is a single failure, like a server goes down, or files get deleted, or anything like that... well good luck finding out how to do it. Or what is actually missing. How on earth can anyone recommend embedded linux for anything is beyond me.

And then, you consider connecting your linux to the internet. Well good luck. Constant updates to every stupid thing, kernels coming faster than fully testing the functionality.

So we have a VM, and if that doesnt work, hope that nothing changed on the internet.

"I like to find out why someone else's code doesnt compile properly." -said nobody ever.

For embedded Linux just maintain only the parts of code you actually touched, and leave everything else to some community. If, for example, Debian works on your board and is not creating too much of a fuss, just go with Debian with your board’s specific kernel. All major package managers, be it Fedora’s yum or Debian’s apt, can be set up to pull updates automatically. Whenever there is an upstream change you just start by validating the binary package, which should be a quick pass in the CI against existing binaries.

Actually The two best existing distributions to base your own embedded systems in would be Debian and CentOS. Both distributions are known for their stability: Debian requires packages to be bug free for a few years before entering stable, and CentOS pull all source code from the commercial Red Hat Enterprise Linux. Those are both very much preferred in data centers for mission critical systems, and their respective maintainers try their best not to break things when upgrading packages.

[mac.6]

The most scary thing for me is embedded linux development. Our current procedure for setting up a new dev enviroment requires to pull data from a dozen online repositories. And the entire thing is a big blob of a black box, nobody really understanding what the actual hell going on inside.

If there is a single failure, like a server goes down, or files get deleted, or anything like that... well good luck finding out how to do it. Or what is actually missing. How on earth can anyone recommend embedded linux for anything is beyond me.

And then, you consider connecting your linux to the internet. Well good luck. Constant updates to every stupid thing, kernels coming faster than fully testing the functionality.

So we have a VM, and if that doesnt work, hope that nothing changed on the internet.

"I like to find out why someone else's code doesnt compile properly." -said nobody ever.

For embedded Linux just maintain only the parts of code you actually touched, and leave everything else to some community. If, for example, Debian works on your board and is not creating too much of a fuss, just go with Debian with your board’s specific kernel. All major package managers, be it Fedora’s yum or Debian’s apt, can be set up to pull updates automatically. Whenever there is an upstream change you just start by validating the binary package, which should be a quick pass in the CI against existing binaries.

Actually The two best existing distributions to base your own embedded systems in would be Debian and CentOS. Both distributions are known for their stability: Debian requires packages to be bug free for a few years before entering stable, and CentOS pull all source code from the commercial Red Hat Enterprise Linux. Those are both very much preferred in data centers for mission critical systems, and their respective maintainers try their best not to break things when upgrading packages.

And then you still have heartbleed-class issues.

There is no free lunch, supporting software for xx years cost yy in time and money. You can mitigate it (VM/redundant hw...) but not eliminate it.
And be sure to sanity test your environment often (jenkins & al) to be sure to catch failure as soon as they happen, not when urgent work is needed.

[Mr. Scram]

Sticking toolchains on laptops is a terrible idea. Hard drive bearings become stuck. SSDs are probably a bit more resilient, but aren't guaranteerd to work after long periods of time either.

You need to be as independent of hardware as you can be and especially specific hardware.

[technix]

And then you still have heartbleed-class issues.

That is why for Internet-connected devices you need a mechanism to push updates to the devices. And firewall things as much as possible.

There is no free lunch, supporting software for xx years cost yy in time and money. You can mitigate it (VM/redundant hw...) but not eliminate it.
And be sure to sanity test your environment often (jenkins & al) to be sure to catch failure as soon as they happen, not when urgent work is needed.

Using the FOSS community can be considered a way to outsource that burden. It wouldn't be surprising for me to see someone building a piece of custom hardware using other brand's silicon, yet still runs Raspbian as the user land environment. For that development team they effectively outsourced the maintenance of their base environment to the Raspberry Pi Foundation.

[coppice]

Sticking toolchains on laptops is a terrible idea. Hard drive bearings become stuck. SSDs are probably a bit more resilient, but aren't guaranteerd to work after long periods of time either.

You need to be as independent of hardware as you can be and especially specific hardware.

SSDs need to be powered up regularly to refresh themselves. A hard disk drive, properly stored, doesn't usually get stuck bearing. The nasty sticking issue with hard disks is the head sticking to the disks, and tearing themselves apart on restart.

[Mr. Scram]

SSDs need to be powered up regularly to refresh themselves. A hard disk drive, properly stored, doesn't usually get stuck bearing. The nasty sticking issue with hard disks is the head sticking to the disks, and tearing themselves apart on restart.

That SSD story mainly concerns older drives near the end of their lives. The author has stated how he regrets the actual context of the story generally is lost. Losing data through that failure mode at least is very unlikely if the disk is relatively fresh.

Stuck bearings is a real problem for drives stored for longer periods of time. It won't happen regularly after 2 years, but the whole point of the laptop idea is to have a surefire tool chain even after a decade.

https://www.pcworld.com/article/2925173/debunked-your-ssd-wont-lose-data-if-left-unplugged-after-all.html

[legacy]

A hard disk drive, properly stored, doesn't usually get stuck bearing. The nasty sticking issue with hard disks is the head sticking to the disks, and tearing themselves apart on restart.

I have harddrives bought in 1995. Still working!
Anyway, cloning the harddrive into a remote server (and into a backup media, i.e. tape and MO) is mandatory for us.

[Mr. Scram]

I have harddrives bought in 1995. Still working!
Anyway, cloning the harddrive into a remote server (and into a backup media, i.e. tape and MO) is mandatory for us.

How many hard drives bought in 1995 are currently still working?

[technix]

AFAIK the only really permanent data storage medium is actually mechanically pressed optical discs. That is, CD-ROM, DVD-ROM and BD-ROM inside their jewel cases, not the recordable or rewritable ones.

[Mr. Scram]

AFAIK the only really permanent data storage medium is actually mechanically pressed optical discs. That is, CD-ROM, DVD-ROM and BD-ROM inside their jewel cases, not the recordable or rewritable ones.

Pressed discs rot too.

[technix]

AFAIK the only really permanent data storage medium is actually mechanically pressed optical discs. That is, CD-ROM, DVD-ROM and BD-ROM inside their jewel cases, not the recordable or rewritable ones.

Pressed discs rot too.

Depending on material. There are archival optical discs made out of strengthened glass that does not rot. Also the metal or glass mold (master) used to press the disks does not rot easily too, which can be preserved to press more discs upon request.

[legacy]

How many hard drives bought in 1995 are currently still working?

From our Olivetti Philos 486DX2 DOS laptops? They are all work
I can take a pic. They are 80MB, 120M, 340MB, made by Seagate.

[legacy]

One of the most used DOS toolchains is Called "EVO" and is targeted for DSP56002 by Motorola.
It comes with a special PCMCIA type1 card, and it also requires the LPT port and the COM port.
But, I am willing to migrate the whole to a Pentium3 laptop, IBM T23.

[coppice]

AFAIK the only really permanent data storage medium is actually mechanically pressed optical discs. That is, CD-ROM, DVD-ROM and BD-ROM inside their jewel cases, not the recordable or rewritable ones.

CD-ROMs are hopeless for long term storage. The lacquer over the metalisation is porous. If you live in a place with high humidity, like HK or Singapore, pressed CDs fail at quite a high rate. Pressed DVDs and BDs are supposed to overcome this, as they use a more substantial covering for the metalisation. I don't know how big an improvement they offer, but my experience suggests they are genuinely better.