Accelerated software life testing?

[ocset]

Hello,

If  some software is supposed to do some action 100 days (or so) after being  started (ie the microcontroller being powered up) , then how can you test that this action will really happen?  (without  having to wait  for 100 days of software testing  to elapse)? 

..We have an offline lamp which has dimming software inside it. It is a lamp for people with large gardens. It dims itself as dawn approaches  so as to save  energy, ..just before it turns off at dawn, it has already dimmed well down (because its getting lighter as dawn approaches). It uses  its software  in order to gauge the  likely duration of darkness in any given night. 
No prizes for guessing how this is done…it simply times the time between the light sensor output changing rapidly either up (for approaching dawn), or down (for approaching dusk). It then averages this time over a few nights to gauge what the darkness time duration will be for the next-to-come night. This is the well known “night time length  forecaster”, very very well  known throughout the world of lighting. 
 (obviously you cannot just use a  cheap light sensor to gauge the exact darkness level, because light sensors are ridiculously wide in their tolerance).

The lamp   also gradually  uses more led current as time goes on, in order to counteract the fact that leds naturally get dimmer as time goes on. 
We are worried that the software engineer (who lives 1000 miles away from us) has correctly coded these things    …I mean, how are we supposed to know that the lamp will have adjusted its led current to the  correct level in 1 years time?    ...i mean, how do you test for that before  you ship the product?    How do you know that the software that executes in 100 days time doesnt go wrong and just turn the lamp off for good? 
Our software engineer says that its all very simple…and that we can do “accelerated life testing”. This is where he simply  reduces the timer interval (in the code) by a factor of 100, so that we can test what would happen  in 100 days time in just one day.
Is this right?...i mean, this kind of accelerated  time  testing is genuinely worthy? 
 

[chris_leyson]

In principle I don't see why you can't just reduce the timer interval to get a x100 speed up. Of course you would have to mimic a light sensor running at x100 speed.

[nctnico]

In principle I don't see why you can't just reduce the timer interval to get a x100 speed up. Of course you would have to mimic a light sensor running at x100 speed.

I would increase the frequency of the timer. That way you also test for number overflows in the software.

[IanB]

I would increase the frequency of the timer. That way you also test for number overflows in the software.

Yes, this would be the kind of thing.

A similar approach is to replace the actual real-time clock in the device with a simulated clock for testing. Then you can make the software clock run faster than real time to simulate long time periods quickly.

[tszaboo]

You place it in an oven, heat it up to 125 degrees, and then apply Arrhenius Equation to it.
Oh, wait a minute....

No, I'm working on a micro, where it is supposed to take measurements every 15 minutes, then send it to the cloud. It runs on a software timer, it has an RTC and NTP, I prime the alarms in timestamps. So I reduced the wait to 30 sec for the test.

But for this, you also need to test timer accuracy.

[Jeroen3]

You can probably simulate much of it.
- You can probably simulate the sun by replacing the light sensor with an equivalent circuit and a signal generator.
- You can probably read the dimming signal of your light with some filter and a scope/dmm.
- You can probably increase the software speed, find the clock tree and see if you can increase speed only by changing dividers.
This does not require recompilation, just changing 1 literal. Code structure won't change.

The signal generator and meter you can script and log.

The thing is.. if software is put up correctly, you can completely synthesize this test on your pc and have it run a million years.
It's called software testing. Ask your software guy about it. 

[hans]

Testing is not something you bolt on to your development. Your software developer that is off site needs to be an integral part in this, picture this model:


The V-model, although by some industries criticized ("go Agile!"), but for hardware purposes it works quite well.

I suppose you're think you are at the integration test or system test stage.
But can/did you do unit testing? Yes you can. If you have the code that is, or the software dev is willing to do this.

You could:

- Test all the code in your timer callback. Do the 'accelerated life' testing on a machine. Call the callback worth 10 or 100 days of time passed, then assert that indeed 10 or 100 has been added to the aging variables.
- Test the formulas for calculating increased light intensities due to aging. This should be a rather simple, "give this input [age], expect this output [light PWM modifier]" test.
- You could also combine these 2 units and write an integration test for this, that calls the timer callback worth 100 days, and then asserts that your light output is indeed increased. But testing the same parameter twice on 2 different places sucks, because more maintenance if you change 1 single business rule.
- You can also test a "virtualized sun" and on a mocked timebase, such that you can test if the night scheduling works correctly.

Personally I wouldn't really trust the interval /100 test on real hardware. This could work, but it really depends on how the software is written. It could exhibit bugs only present in this /100 test interval. You will never really know, until you spend hours debugging faults you have seen in "special firmware builds". What you're basically doing here is making the real-time application 100x faster, which may change some small timing detail in your software that could make or break things.

Timing is absolutely the worst thing to test. But you can unit test all software 'modules' that describe the functionality and behaviour, given a set of test vectors.

[rhb]

Turing addressed the subject long ago in the "halting problem".  It is the canonical example of an undecidable problem.

The best you can do is design the most difficult tests you can think of and run them.  That may or may not be sufficient.  But it beats the hell out of just being optimistic.

[andyturk]

I work on one project where we use traditional unit testing (with Googletest) to deal with some of this issue. This device collects data at regular intervals and records it in an in-memory database. We wrote the database code so that timestamps are explicitly passed with each sample to be recorded and made sure that part of the code didn't have references to any embedded code.

The result is that the database part is entirely unit testable. We have tests to simulate a year's worth of continuous operation, ensure that different sampling configurations all work, etc. In many ways, that part of the codebase is more complicated than the embedded stuff with timers and SPI transactions, and it's very comforting to know that after you've made a significant change, all the unit tests still pass.

However, driver code that touches machine registers and peripheral devices isn't unit tested. We might go that direction someday, but it seems like a lot of work.

[capt bullshot]

Our software engineer says that its all very simple…and that we can do “accelerated life testing”. This is where he simply  reduces the timer interval (in the code) by a factor of 100, so that we can test what would happen  in 100 days time in just one day.
Is this right?...i mean, this kind of accelerated  time  testing is genuinely worthy?

That's the way one usually would test such kind of stuff. Excactly the same way I already did (for other SW stuff, not lighting). You'd still have to trust your SW guy to have done everything right, there can be more subtle faults that won't show up with this kind of accelerating. So one would at least run one test with a non-accelerated unit until some point where one can validate the correct timing of your increased current ... Anyway, If I was you, I wouldn't increase the current at all, as this will also accelerate the failure of the LEDs. For a real professional unit, I'd use some kind of light sensor for feedback, as it is done e.g. for laser diodes.

[hans]

However, driver code that touches machine registers and peripheral devices isn't unit tested. We might go that direction someday, but it seems like a lot of work.

Yes this is a lot of work, and hard to get right.

- Some peripheral operations are so trivial, that it adds only unnecessary weight to the project. Many peripherals have almost no control-flow property or state to test for.
Above all each test added potentially has to be maintained at some point as well.

- Asynchronous peripherals will be hard to test because of timing details, interrupts etc. Some temporal properties are not measurable in an unit test (like latency or throughput) unless you can run them in a simulator that allows for probing these properties. Additionally modern micro's tend to contain deeper pipelines, caches, arbitrated memory buses, making it harder to give upper bounds for these temporal properties anyway.
At best you could test the drivers behaviour by calling the interrupt "manually", and see what happens.

- Emulating complex peripherals like DMA, USB, Ethernet, specific timer modes, trends towards writing simulator-like peripherals for the targeted platform... with all the translation errors accompanied with that.

Personally I do the same approach as you just described.

For a project I created a persistent data store that uses variable-size blocks in order to maximize utilization of the attached memory device. It uses a circular index to address those blocks. All of the "bookkeeping" logic in adding, reading, writing and removing pages is tested. In extension of these standard tests, it's 'easy' to add erroneous states (e.g. a power loss during writing) and test the recovery behaviour as well.

This kind of behaviour is exactly the kind of thing you want to avoid doing manually. It's cumbersome, time consuming, and very tempting to skip.