WTF, software updates breaking all my SW projs in 6 to 12 moths top!

[RoGeorge]

As the title says.  What is good and running today, won't run in a couple of months from now, because of software updates.

Examples:
- Installed some GPT transformers (AI, 15GB+ of HDD, with CUDA, PyTorch, inference models, etc.).  Made for it a venv to preserve the Python things, did some customisation, parked the project in good state 3 months ago.  Last week wanted to continue.  Doesn't run any longer.  nVidia updates, CUDA updates, Eleuther model updates.  TL;DR not working, spent hours reinstalling, still not working.

- Did some Jupyter Notebook in April this year, to SCPI control a power supply with PyVISA.  It's October and it doesn't run any more, had to disable some authentication token crap in Jupyter, and it's still not working like it was in April.

- Made a virtual machine for VirtualBox about half a year ago.  VirtualBox changed, had to Google why the errors for what was left in good working and nicely shut down, had to add the new guest addition in the VB machines, upgrade and so on.

This is not the first time when such things happen to me.  In fact, in the former years it becomes the rule that a project won't run any more when open after a few months.
 

I just want my project to run when I start them, next day or in a year from now.  How can I have that?

I don't want any software "improvements" and I don't care about security.  It all runs offline on the main PC (Ubuntu), nothing has to go outside of the LAN, ever.

How can I make a "time capsule" for the tools I write?
Even virtual environments aren't immune to updates.  For my use, containers are cumbersome to configure, and hard to make them run months later.  And it's ridiculous to create a many GB new VM for each 20 lines of Python I may write as a tool.  It's for a hobby lab at home, should be fun and easy.

Anybody else going back to use own scripts from months ago?  Does that happen to not run any more?  How to deal with that?  Do I need yet another PC as a "production" machine for the lab, isolated and never to be updated?

[Ed.Kloonk]

Virtual box can store incremental snapshots.

[jonpaul]

RoGeorge, see old Scientific American article on software complexity, 1960s(?)

The issues you have were bound to happen.

I am very glad that I am a hardware engineer.

Circuits and devices I designed and built back to 1960s, still work perfectly.

bon courage

Jon

[RoGeorge]

OK, but how do I avoid to continuously adjust my own software to whatever updates? Even a virtual box VM required additional installs.

I want to start the PC and run again what used to be working last month, or last year.  As simple as that.

Am I the only one returning to work/projs from more than a couple of months ago?
How is everybody else running their own SW tools and scripts?

[Bud]

How is everybody else running their own SW tools and scripts?

By not using Python.

[Karel]

Yep, python is a moving target...

[SiliconWizard]

 

[RoGeorge]

You mean to use C instead?

OK, though sometimes this might not be possible, or possible but very time consuming.  Often is only a few lines improvised, then tuned a little, then saved and that's it.  Rewriting all that in C would be too pedantic for home use.

From the answers it looks like I'm the only one going bacj to 6+ months old projects and having problems with backward compatibility after updates. 

[janoc]

Yep, python is a moving target...

Python is fine.

The problem is that most people (especially the ones using stuff like Jupyter - i.e. mostly research folks) don't have a clue (or care!) how to build software for distribution, how to manage dependencies properly, etc. But that's not fault of Python. Python has plenty of good tools designed to handle this for you, one only has to use them!

One can make the same or worse horrific mess in any language. Don't bring up the argument that "I wrote some C or assembler software 20 years ago and still runs" - sure and since it has never been updated it is likely full of security holes and bugs thanks to the linked in dependencies. Or (given the nature of this forum) it is some embedded stuff irrelevant to the debate. And btw, the same feat is possible perfectly fine with Python too - as long as you don't distribute only "naked scripts", letting the user figure out (tearing their hair out in the process) which versions of which dependency they have to install to make it work ...

Especially things like the Jupyter(Lab) notebook with its many plugins,  PyTorch, Tensorflow (the AI frameworks) that are using Python as glue to bring together hundreds or thousands of dependencies written in various languages (including C/C++, Fortran, Javascript ...) in a very fragile manner are a nightmare. Done typically with no proper dependency version control, so one package somewhere deep in the dependency tree gets updated because of something unrelated - and now the entire stack is broken. To make it worse, most of that code is written in a way of "throwaway" notebooks and scripts because the goal is to get a paper published as quickly as possible by a guy or girl who programs only because they have to, not because they are programmers. And not to produce any sort of "production quality" code.

And if something underlying like CUDA or Nvidia drivers (which are used by the above for acceleration of the computation) changes because of a forced automatic update, then even containers like Docker or a complete VM (e.g. VirtualBox) won't save you.

If you want to make sure something keeps running you need to "freeze" that version of the system in a VM - and never touch it anymore. Including never letting e.g. Windows to update anything. Linux is much easier in this regard because you have complete control over the update process. In Windows this is difficult unless the system is airgapped from the Internet.

EDIT: I have slept on this and to be constructive, here are a few suggestions how to make your Python life easier:

If  you don't need CUDA for AI, then being rigorous with dependency management (e.g. using tools like poetry, pipenv or at least a requirements.txt file!) together with virtual environments to isolate applications from each other (both pipenv & poetry manage that for you) will make sure your code remains installable and working. And if you distribute your thing, distribute it including those build configurations!. Even better - make a proper Python package and have the thing distributed via PyPi so that it can be installed using pip.

"Naked" scripts are only acceptable if they don't have any dependencies beyond what is in the standard library - and even then be clear about which Python version is required as a minimum if you use any sort of "exotic" features - such as asyncio (needs 3.6+), match keyword (3.10+), etc. And please, don't write new software requiring Python 2!

For end users who don't want to install Python and every dependency, build an executable - e.g. pyinstaller will do it for you. Far better than slapping a python script on Github and then people complain that they can't get it to work because it relies on this or that in version X but version X+2 has changed API and now your script doesn't work for them but since it is the latest, that's what gets installed.

Also, never distribute code that is meant to be used for actual "production" (i.e. not only research, quick tests) as a Jupyter notebook. The notebooks don't have the abovementioned version management, neither for Python dependencies by default and neither for the various Jupyter plugins that may be required e.g. to display some plots and what not. Use Jupyter to generate images for documentation but the actually useful code should be in properly managed Python modules instead that don't depend on 20 Jupyter javascript plugins to work. Even plots can be generated without Jupyter - e.g. matplotlib or bokeh work from "straight" Python too.

And for "users" of those python scripts - never install anything into the system-wide Python installation. I.e. don't just run "pip install foo"! Always set up a virtual environment first, to keep your Python applications isolated from each other, so that installing e.g. numpy version 1 doesn't break another tool that needs numpy version 2.

If everyone followed this it would have made everyone's life a lot easier and there would be far less bashing Python for things that are not its fault.

[alm]

- Installed some GPT transformers (AI, 15GB+ of HDD, with CUDA, PyTorch, inference models, etc.).  Made for it a venv to preserve the Python things, did some customisation, parked the project in good state 3 months ago.  Last week wanted to continue.  Doesn't run any longer.  nVidia updates, CUDA updates, Eleuther model updates.  TL;DR not working, spent hours reinstalling, still not working.

For something this elaborate Anaconda might help. For me it works better to go with the flow and update and fix things, rather than fighting things and trying to stick to old versions.

- Did some Jupyter Notebook in April this year, to SCPI control a power supply with PyVISA.  It's October and it doesn't run any more, had to disable some authentication token crap in Jupyter, and it's still not working like it was in April.

I have Python scripts and notebooks for instrument control that go back years. Some of it written over a decade ago. Still runs fine. The last minor change I remember was when I moved from Python 2.x to 3.x. Had to run a script to make minor changes like parentheses for the print statement and change from float to integer division. I use Poetry to install all dependencies in a virtual environment. If I'm really fussed, then I might use PyEnv to freeze the version of Python so I'm not dependent on the system Python version, but generally I don't bother. Getting Jupyter to use virtual environments is a bit of a hassle, but works once you set it up. If anything goes wrong with the dependencies, I blow away the virtual environment and have poetry install all dependencies from scratch. But I try to stay reasonably current on libraries, because generally many small updates are less painful than a single huge upgrade.

Circuits and devices I designed and built back to 1960s, still work perfectly.

How about if you tried to build another copy? Or if you had to build a thousand and had to find a contract manufacturer to build them? What size of feeder do you use for tube sockets? Hardware is not exactly immune to dependency (component availability) issues.

[james_s]

I hate this, software gets updated far too often and usually the updates cause more problems than they fix. I've gone back to the policy I had with things like BIOS updates back in the early 2000's, I only update if there is a specific problem that I'm trying to fix with the update. Almost every other time I have let something update I came to regret it. Far too often the changes just make something worse.

[janoc]

For something this elaborate Anaconda might help. For me it works better to go with the flow and update and fix things, rather than fighting things and trying to stick to old versions.

The problem with Anaconda is that it has its own packaging system separate from pip/Pypi and if some package isn't available in the Anaconda "channels", then you need to rely on various uncurated 3rdparty repositories that often have packages not compatible with the official ones. Or you use pip to install things from Pypi and then everything breaks because, of course, pip and conda aren't aware of each other.

Also most Python tutorials and recipes are designed around standard Python tooling, Anaconda does its own thing and then you can shoot yourself in your foot in a pretty big way if you don't know what you are doing.

Finally, I have found the dependency resolution by the conda tool in Windows to be extremely, painfully slow and it often fails. And then the fun starts ...

Anaconda is good if you need e.g. a workable installation of PyTorch or Jupyter and don't want to touch it. They do give you that out of box experience because this is the target market they focus on - researchers who want to do numerical calculations using numpy, pandas, scipy, PyTorch, etc. so they have a good well-rounded setup with all the common packages available. But if you need stuff that is not available in Anaconda, it will be a major pain.

I have used Anaconda in Windows for some AI training stuff before but ended up switching to standard Python install. It was a bit more work but in the end it was far easier to manage because of the standard, well documented tooling.

[alm]

I agree, that's why I specifically suggested it to get started quickly with PyTorch etc. I wouldn't recommend it otherwise.

[RoGeorge]

OK.  Now I need a new setup for this speech recognition tool from OpenAI, MIT licensed:
https://github.com/openai/whisper

It depends on
- Python
- PyTorch
- HuggingFace Transformers
- AI inference models
- ffmpeg, ffmpeg-python
- sound
- might need nVidia/CUDA, too, not sure about last one yet

Once the setup is running, I want to shut down the "whisper" thing, and never start it again until next year.  Meanwhile, I need to use the PC as a day to day home desktop.  One I start "whisper" application again, tomorrow or in a year from now, it should run like in day one, unattended, all offline and without Internet.

How would you setup this?

[Karel]

How would you setup this?

I wouldn't.

[alm]

OK.  Now I need a new setup for this speech recognition tool from OpenAI, MIT licensed:
https://github.com/openai/whisper

It depends on
- Python
- PyTorch
- HuggingFace Transformers
- AI inference models
- ffmpeg, ffmpeg-python
- sound
- might need nVidia/CUDA, too, not sure about last one yet

Once the setup is running, I want to shut down the "whisper" thing, and never start it again until next year.  Meanwhile, I need to use the PC as a day to day home desktop.  One I start "whisper" application again, tomorrow or in a year from now, it should run like in day one, unattended, all offline and without Internet.

How would you setup this?

Like I said, I prefer a more dynamic approach where I'll update dependencies and make whatever small changes might be necessary to track those updates. But if you truly want to freeze time, then these would be my suggestions depending on how deep the dependencies go:

• For projects with pure Python libraries that depend on pretty basic system libraries, which includes PyTorch and possibly ffmpeg, but not Cuda, I'd use virtual environments with pyenv to also freeze the Python version
• For projects with libraries that have more specific native dependencies, like a particular ffmpeg version (I haven't tracked that project closely enough to know how they are about updates), I would use Docker. Docker is not made for desktop use, so it will not do video, and probably sound only if you can do it over the network. But there is apparently Cuda support, although this obviously still depends on the driver on the host.
• For projects that need to show a UI but do not need Cuda, I'd use VirtualBox
• For projects that need full hardware access, like video, Cuda, etc, I'd make a bootable USB stick. There are ways to provide GPU pass-thru to virtual machines, but I don't think that's worth it for this scenario since it still depends on the host configuration.

[artag]

If you want consistency and stable interfaces, avoid two things like the plague - Windows and Python.

[Mechatrommer]

avoid two things like the plague - Windows and...

more shit smell post? did you read OP is running Ubuntu? if you experienced like i do for the past few decades about Windows and its paradigm its try to impose on software developers, you'll change your mind. non-backward compatibilitiness is only for rookies programmers who want/think to make the world better in short term thought process. another thing is if the developer dont allow you to do full offline installer download nor letting to install older version, like Python does.

[SiliconWizard]

If you want consistency and stable interfaces, avoid two things like the plague - Windows and Python.

I'd admit that for Python, but Windows? As much as I don't like where Windows has been headed for a number of years now, it has been one of the most stable platforms out there. Heck, if you stick to the Windows API, applications written 20 years ago can still be compiled for the latest Windows and run. They are even likely to run without requiring a recompilation.

[Zoli]

If you want consistency and stable interfaces, avoid two things like the plague - Windows and Python.

I'd admit that for Python, but Windows? As much as I don't like where Windows has been headed for a number of years now, it has been one of the most stable platforms out there. Heck, if you stick to the Windows API, applications written 20 years ago can still be compiled for the latest Windows and run. They are even likely to run without requiring a recompilation.

Released in 1999, still using daily.

[KE5FX]

If you want consistency and stable interfaces, avoid two things like the plague - Windows and Python.

Really?  I can run code on Windows 11 in 2022 that I wrote in 1995. 

That leaves Python.

[artag]

Sure, if you code for windows 3.11
But if you use any of the annual fashion APIs they produce, you'll be using it for 2 years before they drop it and try something else.
OLE anyone ?

[Mechatrommer]

then dont use half baked API invention, try to rely on more pro/established platforms (IDE) i cant tell which one i havent used the latest IDEs, i program old style. but back then M$ will tell much earlier about deprecated API's, not sure today when kids signing in...

[SiliconWizard]

What's that silly comment about Win 3.11? Windows has had a solid API for a very long time, and it still does. And it's still fine. It's well documented, along with which versions of Windows support a given function/type/... You can still perfectly write applications purely in with the Windows API these days. It's not rocket science. No need to go back to Win 3.11. Which was 16-bit anyway. But they have handled the Windows API very well over the years. It practically never broke anything.

Of course. If you want stable stuff, use stable stuff. Duh. If you use the latest trendy shit, you'll reap what you sow.

[janoc]

OK.  Now I need a new setup for this speech recognition tool from OpenAI, MIT licensed:
https://github.com/openai/whisper

It depends on
- Python
- PyTorch
- HuggingFace Transformers
- AI inference models
- ffmpeg, ffmpeg-python
- sound
- might need nVidia/CUDA, too, not sure about last one yet

Once the setup is running, I want to shut down the "whisper" thing, and never start it again until next year.  Meanwhile, I need to use the PC as a day to day home desktop.  One I start "whisper" application again, tomorrow or in a year from now, it should run like in day one, unattended, all offline and without Internet.

How would you setup this?

Like I said, I prefer a more dynamic approach where I'll update dependencies and make whatever small changes might be necessary to track those updates. But if you truly want to freeze time, then these would be my suggestions depending on how deep the dependencies go:

• For projects with pure Python libraries that depend on pretty basic system libraries, which includes PyTorch and possibly ffmpeg, but not Cuda, I'd use virtual environments with pyenv to also freeze the Python version
• For projects with libraries that have more specific native dependencies, like a particular ffmpeg version (I haven't tracked that project closely enough to know how they are about updates), I would use Docker. Docker is not made for desktop use, so it will not do video, and probably sound only if you can do it over the network. But there is apparently Cuda support, although this obviously still depends on the driver on the host.
• For projects that need to show a UI but do not need Cuda, I'd use VirtualBox
• For projects that need full hardware access, like video, Cuda, etc, I'd make a bootable USB stick. There are ways to provide GPU pass-thru to virtual machines, but I don't think that's worth it for this scenario since it still depends on the host configuration.

Yep, pretty much this.

If it doesn't need native code dependencies like ffmpeg then a Python virtual environment with frozen versions of the dependencies would be enough. However, as this needs external, non-python dependencies, then stuffing it into a VM or a container (Docker ...) is the safest option because if those external libraries change the Python packages that depend on them could get broken.

Tools like VirtualBox and I believe even Docker are able to give access to the raw hardware like the GPU, so it is possible to have CUDA and the matching driver installed inside of the VM, even though the host system is running a different version. As long as the GPU is compatible with the old driver it should work. However, this is unfortunately a rather non-trivial to set up and make play nicely with the host system. Graphic cards and their proprietary (the open source drivers usually work - but are useless for these use cases ...) drivers are notorious for not playing nicely with virtualization of any sort.

And finally, OP, if you need speech recognition - unless you are specifically doing research in this area, did you consider other tools? There are multiple mature and even open source speech recognizers that work well and have both acoustic and language models for various languages and applications, typically using hidden Markov models instead of neural networks. Using a month old university research project is bound to give you trouble, for exactly the reasons I wrote above - it is code meant to publish papers/get a PhD, not anything that is supposed to be "production quality".

E.g.:
- CMU Sphinx: https://cmusphinx.github.io/ (multiple versions, including a lite one that works on small computers like RPi - PocketSphinx)
- Vosk: https://alphacephei.com/vosk/install 
- Kaldi: https://github.com/kaldi-asr/kaldi

They may not be the bleeding edge state of the art like the many recent deep learning based ones but may be far less hassle to set up and use because there are no crazy fast moving dependencies like PyTorch, CUDA or Tensorflow involved.