Year 2038 bug - how big an issue will it be?

[peter-h]

There is one other thread here on this, from 5 years ago.

There is just 16 years to go. In the consumer sphere this is ok but in the industrial sphere 16 years is nothing; one expects products to run for longer. I have thousands of products made in say 1995 still installed and running.

I am surprised because Y2K was a huge hassle. It wasn't always easy to fix because most RTC designs, including those on micros even today, have just 2 digits for the year, so one had to change the code interfacing to the RTC to interpret the epoch differently. It was harder if the sources were lost or could no longer be recompiled, etc. And most software engineers move on after a few years...

A lot of products were scrapped simply because the mfg could not prove Y2K compliance. An amazing waste. Especially as many of them had no RTC or any date-related code at all

It seems that the 2-digit year RTC will not be an issue this time - that's good until 2099. The issue seems to be in the C libraries, which have been only recently fixed, but that will help only with new products. Specifically it seems to be the functions which use milliseconds since 1970 or some such. I've never used these but they seem to be popular for e.g. calculating the number of days between two dates; the intermediate values use a signed int32.

[Siwastaja]

It's called "unix time" and it's not milliseconds after 1970, but seconds. It's used literally everywhere so it's really surprising you heard about it the first time.

In any case, any remotely sensible system and codebase have used 64-bit integer to hold Unix time for a long time; the problem is utterly trivial and known for a long time.

But as you can't fix stupidity, certainly some systems will contain such bug. The question is, is this relevant as its own "separate" issue? It's very likely those same contraptions are full of other, even more critical bugs.

[SiliconWizard]

Still, it'll never cease to amaze me how so many developers make choices based on the "it ought to be enough" paradigm - often collecting the minimum number of other opinions, if any.

There are several approaches to picking the "size" of a given object representing some quantity, when the max value is unknown, or could change over time:
- Pick the smallest size you can get away with (to save resources), claim the "ought to be enough" method, and move on (by the time it won't be "enough", you'll be long gone... );
- Pick a clearly overkill size, so that the limit ever being a problem becomes very unlikely;
- Use a variable-length scheme (this one is of course not as trivial to implement, but virtually "unbreakable".)

[ejeffrey]

Time_t has been 64 bit in the C ABI most of not every 64 bit platform ever made, and later backported to many 32 bit platforms with no native 64 bit type.

Probably some 32 bit microcontrollers still use 32 bit time_t since I think you can compile newlib and similar either way and you have a lot more inexperienced people configuring and compiling libc for microcontrollers vs general purpose OSes.  Still most systems have it available for years and I would guess that major RTOS platforms will use a 64 bit time_t at least.

Having time_t be 64 bit means all the c library functions correctly handle dates past 2038.  That doesn't keep programmers from storing timestamps in int, int32 or even long datatypes that are only 32 bit.  C doesn't effectively prevent this so undoubtedly some things will break.  But software engineering has changed a lot since Y2K.  Software is much more likely to be using higher level time/date libraries which handle things properly at least as far as range.  I would still expect some issues but nothing extraordinary.

[TimFox]

There was a short-lived TV animated program about Dilbert, that was often more angry and grim than the daily newspaper comic strip.
Back when Y2K was an issue, their treatment of the problem was to show Wally in 1975, when he was still an enthusiastic young engineer just starting at the company.  The older employees showed him that they used only two digits to encode the year.  He asked, "won't that be a problem in 25 years?", and they laughed at him.

[gf]

I already see that 2037 may be a good business year for the IT industry, with lots of extra hours - similar to 1999
People tend to ignore the problem as long as possible and panic in the last second - shortly befor it is too late.

[SiliconWizard]

You can literally call this "programmed obsolescence".

[peter-h]

The Y2K issue (I was very much involved then) was much more to do with the inability of equipment makers to prove Y2K compliance to their usually much bigger customers (whose "compliance managers" aggressively demand compliance certification "or else") than the equipment having an actual issue.

In the vast majority of cases the equipment didn't even have any kind of date processing. No more than your washing machine has.

So this can and will repeat itself just the same.

And, after say 10 years, how many companies will be able to find the sources, let alone find somebody who can read them (coders generally hate reading somebody else's code ) let alone be able to find somebody who can fix it.

In most cases the project won't even compile after say 10 years after it was finished. Or 5 years. Most people know they should archive sources but few know they also need to archive the tools. Maybe create a VM when a project is "finished" and then you can archive the VM - it is only about 10GB. But nobody does this. I had a taste of this with FPGA projects (I used to do FPGA / ASIC design work, using Xilinx FPGAs, and their newer tools could not load older sources. The tools could not be archived either because they were dongled In the end, even though I had the dongles cracked, I got out of it; it became a completely ridiculous business, if you were running your own company and had to think long term.

Project archival is a big issue in many relatively critical areas. For example when Bendix King was taken over and (as usual) trashed by Honeywell, anybody who could write Hello World left to Garmin, and products (e.g. autopilots) finished in say 1999 were completely dead in 2003. The sources "exist" somewhere (I am told by an ex insider) but nobody could touch them.

[ejeffrey]

The Y2K issue (I was very much involved then) was much more to do with the inability of equipment makers to prove Y2K compliance to their usually much bigger customers (whose "compliance managers" aggressively demand compliance certification "or else") than the equipment having an actual issue.

That might be how your were involved but there was a lot more to it than that.  And yes some of it was "just" certification but in most cases things that didn't actively process dates could be easily shown to have no issue -- either it has no clock or you can set the clock forward to show it works or you can show a simple work around   If your customers were making it harder than that for things that weren't doing scheduling, financial processing, or other work where date / time is directly relevant it was probably just a bit of "me tooism" where they saw a lot of other people working on something and thought they had to participate.  That certainly happened but there were a lot of real problems that got fixed and a lot of real certification to validate that systems were ok.

[peter-h]

I think this will be BIG.

For embedded systems, it will be bigger than Y2K whose actual effects were outside the embedded world.

And another with GPS, though I would hope currently working units will have coded around it.
https://en.wikipedia.org/wiki/GPS_week_number_rollover

[JohanH]

In any case, any remotely sensible system and codebase have used 64-bit integer to hold Unix time for a long time; the problem is utterly trivial and known for a long time.

Even though it sounds trivial it is a lot of work. It took until kernel version 5.6 (in 2020) until Linux supported 64-bit time_t on 32-bit systems. Even glibc isn't compliant by default (if you’re on a 32-bit system, you have to explicitly build your app with -D_TIME_BITS=64).

Lots of old industry systems will have issues (SCADA and the like). There will always be some part of some old system that isn't upgraded to work with this.

But it's not the end of the world. Lots of people will work on this and fix things. Critical systems will be tested, replaced and upgraded. Those that won't, probably don't matter that much and can be fixed afterwards if needed.

Now, why did Germany announce they will phase out all coal power stations before 2038... 

[ozcar]

It's called "unix time" and it's not milliseconds after 1970, but seconds. It's used literally everywhere so it's really surprising you heard about it the first time.
. . .

Way back in about 1993 (!), somebody at the place I was working was tasked with contacting all our suppliers in regard to potential y2k issues (yeah might have been a bit ahead of the pack). One of the suppliers responded that they were not aware of any y2k issues, but warned of the looming Unix time issue. I think they did not get the date and time that this would occur quite right, but hey, it is the thought that counts.

[peter-h]

The reason I had not got involved in this is because I never used "seconds since 1970" in anything. I use the unix tm structure (in various code like around the 32F417 RTC) but that doesn't contain such a value.

I think the secs since 1970 crops up when using unix library funcs to do date/time calculations and conversions, this value is an intermediate variable.

[Siwastaja]

But it's not the end of the world. Lots of people will work on this and fix things. Critical systems will be tested, replaced and upgraded

That's the point. What laymen do not realize when discussing about Y2K & friends, software is full of bugs. Time wraparound related bugs are no special case at all.

This happens at much shorter time periods. Boeing 787 must be rebooted every 51 days because some counter wraps around in such short time.

Failing to understand the numerical range required and use too small of integer type is one of the most classic types of bugs, and not limited to time.

And yes, solution is trivial: just use larger size. It only gets difficult when interchange formats or APIs use wrongly chosen number formats because the real solution is to update the whole API to a new version and then get the developers of all software make updates to use that new version.

But it's just software, it's full of bugs anyway, everything can be fixed or worked around. Try to do anything in modern days and our daily life is hindered by massive amounts of software bugs to the point we are so used to it we don't even see it. Rebooting products all the time is the norm. Not being able to call the emergency number without signing in to Microsoft Teams really happened. These are HUGE issues. In 2038, we may see a tiny increase in number of bugs affecting our daily life, but it likely gets lost in noise.

[peter-h]

Boeing 787 must be rebooted every 51 days

Another report says 284 days
https://www.euroga.org/forums/website/3942-a-boeing-787-must-be-rebooted-every-284-days

As you say, these overflow bugs are everywhere.

[Siwastaja]

I remember hearing even shorter number IIRC. 51 days was the first result found by Googling. Who knows.

[ozcar]

. . .

Failing to understand the numerical range required and use too small of integer type is one of the most classic types of bugs, and not limited to time.

. . .

So many boundary crossings to test and so little time.

Years ago the keepers of one particular program were grumpy with us because in their eyes the program had failed because we had changed something in the system it was running on. It turned out that if the memory address of a particular structure was greater than 0x5f5e0ff the program choked.

Just the sort of thing you would expect! Well, you might expect it if you knew that somewhere along the way they took a pointer to the structure,  converted it into an 8-character decimal string, and then back to a pointer.

[Ed.Kloonk]

Boeing 787 must be rebooted every 51 days

Another report says 284 days
https://www.euroga.org/forums/website/3942-a-boeing-787-must-be-rebooted-every-284-days

As you say, these overflow bugs are everywhere.

You'd just reboot it just before take off, wouldn't ya?

[TimFox]

Boeing 787 must be rebooted every 51 days

Another report says 284 days
https://www.euroga.org/forums/website/3942-a-boeing-787-must-be-rebooted-every-284-days

As you say, these overflow bugs are everywhere.

You'd just reboot it just before take off, wouldn't ya?

Quite a few years ago, the VP of Engineering and I took a work day trip on a nice clear day from Chicago to White Plains, NY, where the aircraft was a "commuter" jet.  We took the first flight on that route early in the morning out of Chicago, returning late afternoon from White Plains.  We got to White Plains on time, drove to the site, and returned, where we found the outgoing flight was substantially delayed.  Opening up WiFi at the bar, we checked the progress of that aircraft going back and forth, finding that each take-off was increasingly delayed.  The VP looked at the data and found that the plane had just started service on the airline, and deduced that the on-board computer had to be rebooted before every takeoff.

[peter-h]

A reboot before each flight is a standard procedure in some types, especially the smaller commuter jets. It is done to assure a clean config for the avionics.

The chances of any passenger jet remaining powered-up for a month or more depends on the operation; it is unlikely.

There have been worse things e.g. the navigation crashing when crossing the 180/-180 degree longitude line (not the 0 degree one; that one gets crossed a lot). I've done a fair bit of software in this area and some of the conventions are quite bizzare e.g. heading, or longitude, run from 0 to +179.9999 degrees for the 0-180 half, and 0 to -179.9999 for the 180-359.0000 half. The actual value is represented in ARINC429 using a signed int, typically 20 or 21 bits, IIRC, and you can imagine how careful you have to be handling the transitions from +179.something to -179.something which actually represent a miniscule movement in the heading (or longitude). Same when converting between say a GPS track around North, which NMEA represents as 359.something to 0.something, and as a float (probably a double, to be safe), whereas ARINC429 represents as 0, plus or minus very little. In some cases I had to use signed 64 bit ints so I had extra bits to play with to detect small changes and make them fit properly onto the signed int in ARINC429. So many ways to get bitten. Especially as, due to historical reasons, people writing software in aviation are not the sharpest knives in the drawer.

[TimFox]

I assume that the apparent re-boots I mentioned were not expected, since they caused a noticeable increasing delay in the back-and-forth operation of the individual aircraft.

[brucehoult]

There was a short-lived TV animated program about Dilbert, that was often more angry and grim than the daily newspaper comic strip.
Back when Y2K was an issue, their treatment of the problem was to show Wally in 1975, when he was still an enthusiastic young engineer just starting at the company.  The older employees showed him that they used only two digits to encode the year.  He asked, "won't that be a problem in 25 years?", and they laughed at him.

In December 1982 at the end of my 2nd year at university I got a summer holiday job at the local city council, computerising their debentures. At various times they had raised funds for works such as improvements to water treatment or building a library by getting 25 or 30 year loans from individuals and institutions such as pension funds. Once or twice a year the accountant would go through all the certificates and make a note of upcoming payments. This was pretty time-consuming.

At the time, some loans from the 1950s were still active, and the newest ones would be paid off around 2010. I had to handle dates spanning 2000. The bureau who owned the PR1ME computer on which I would write the loan system (in COBOL) insisted that "for storage reasons" I could only use two digits for the year.

I programmed in a modifiable parameter for what year was considered the base year. Initially I made this 1950, so dates from 1950 to 2049 could be represented. I made a menu option to adjust the parameter which, before doing so, checked that there were no still-active loans which were made in the years between the old base year and the new base year.

If this system is still in use now, and if new 30 year loans are being raised for capital works, they would have had to adjust the base year parameter for the first time in 2019. If only 25 year loans then they're good for a couple of years yet :-)

However I believe this whole system of raising funds for capital investment from the market (including individuals) was banned sometime around 1990 and a government body set up for councils to borrow from.

[peter-h]

Just think... today you can get £1000/day for COBOL66 coding

Even more than doing Ruby on Rails

I think everybody doing embedded in the later 20th century, using the RTC chips with a 2 digit year (which they still have now!), had to do something like this. You just can't span more than 99 years with it, and preferably cannot go back in time.

[brucehoult]

Just think... today you can get £1000/day for COBOL66 coding

If I knew where I could do that I'd happily do it. If I can use a modern fast computer, text editor etc to do the dev work with.

Well, I wouldn't go to the UK. Remote?

I think everybody doing embedded in the later 20th century, using the RTC chips with a 2 digit year (which they still have now!), had to do something like this. You just can't span more than 99 years with it, and preferably cannot go back in time.

I'm sure it was not at all a unique solution. I was chuffed because it was still 18 years until y2k and the term hadn't been invented ... and I was 19 myself, without experienced programmer supervision.

The funniest thing about that summer job was at one point it was the time for people to pay their property taxes ("rates") for the ... quarter (?) ... and a lot of people were still paying in cash. Normally the office girl took the daily cash take to the bank, but for about a week there was several million dollars of cash a day and I was asked (apparently as a strapping lad who might not get mugged?) to carry it to the bank in a suitcase.

[Ed.Kloonk]

Just think... today you can get £1000/day for COBOL66 coding

If I knew where I could do that I'd happily do it. If I can use a modern fast computer, text editor etc to do the dev work with.

Well, I wouldn't go to the UK. Remote?

I think everybody doing embedded in the later 20th century, using the RTC chips with a 2 digit year (which they still have now!), had to do something like this. You just can't span more than 99 years with it, and preferably cannot go back in time.

I'm sure it was not at all a unique solution. I was chuffed because it was still 18 years until y2k and the term hadn't been invented ... and I was 19 myself, without experienced programmer supervision.

The funniest thing about that summer job was at one point it was the time for people to pay their property taxes ("rates") for the ... quarter (?) ... and a lot of people were still paying in cash. Normally the office girl took the daily cash take to the bank, but for about a week there was several million dollars of cash a day and I was asked (apparently as a strapping lad who might not get mugged?) to carry it to the bank in a suitcase.