JohanEkdahl over at AVRFreaks.net suggested a time capsule idea. Something with an embedded message that gets lost if the device looses power or something interrupts the main loop.
Why would you want to make it lose the data if power was lost?
Also, it should be something that is continuously interesting, for as long as it lasts. Not just a one-pop wonder.
How about this:
Solar powered weatherproof object. Power is stored in a supercap so it can operate through brief intervals of darkness. It has a microphone, and a speaker (both impervious to weather), a VOX circuit with software analysis to detect human voices. Over the course of a day it records in CMOS ram just the first 30 seconds of any voice sequences that contain at least 30 seconds of continuous speech. At the end of each day it compresses and transfers ONE of those sequences (randomly chosen) to a large flash memory. Typical flash memories should be able to hold centuries of such once-a-day fragments. Organized as a ring buffer, the data doesn't start to get overwritten until full - in hundreds of years? Also the system should once a year or so, do a 'rewrite' of the entire memory to refresh the cell charges. Minimize the wear on the flash cells, but don't let the data 'evaporate' over time.
During daylight hours, every day, it will also now and then randomly select fragments from its sounds archive, and play them back. Maybe two or three times a day. Or maybe it would do it as payment for someone talking to it.
I'd imagine this as some kind of very solid stainless steel pillar, set into bedrock in scenic, not-so easy to reach places, that not so many people would visit. The idea would be that people could speak to it, knowing their voices *might* be preserved and sometimes played back, for centuries. If only one person visits in a single day, their words are guaranteed to be preserved.