Right, you can run a JFET oscillator from thermocouples (<< 1.0V). Depletion MOS may work as well. You can't really get any power from it though. Mainly because no one makes such a transistor wide enough to handle appreciable current.
If we're talking batteries, it's less than pointless as the amount of energy remaining in the already-dead cell is minuscule. Ditto with supercaps; from 2.5V down to 0.7V, you've consumed fully 92% of the total energy.
TEGs (including thermocouples, or Peltier modules in a thermal gradient) are somewhat useful*, and may generate low voltages in the process, for which a converter is required.
You're better off using such a circuit for a bias supply, then using that to power a controller that does the heavy lifting with a conventional approach (e.g., controller IC + switching MOSFET + output diode).
*Mostly if you happen to have access to a few pounds of Pu238 oxide pellets, as NASA does. RTGs are very robust and reliable power sources, if terrifically inefficient (a few percent or so). Of course, here on Earth you're better off, say, hooking up a Stirling or steam engine, or where fuel is involved, any of the usual types of combustion engine.
Tim