EEVblog Electronics Community Forum
Products => Computers => Programming => Topic started by: RoGeorge on October 12, 2022, 01:08:31 pm
-
As the title says. What is good and running today, won't run in a couple of months from now, because of software updates.
Examples:
- Installed some GPT transformers (AI, 15GB+ of HDD, with CUDA, PyTorch, inference models, etc.). Made for it a venv to preserve the Python things, did some customisation, parked the project in good state 3 months ago. Last week wanted to continue. Doesn't run any longer. nVidia updates, CUDA updates, Eleuther model updates. TL;DR not working, spent hours reinstalling, still not working.
- Did some Jupyter Notebook in April this year, to SCPI control a power supply with PyVISA. It's October and it doesn't run any more, had to disable some authentication token crap in Jupyter, and it's still not working like it was in April.
- Made a virtual machine for VirtualBox about half a year ago. VirtualBox changed, had to Google why the errors for what was left in good working and nicely shut down, had to add the new guest addition in the VB machines, upgrade and so on.
This is not the first time when such things happen to me. In fact, in the former years it becomes the rule that a project won't run any more when open after a few months.
:horse: :horse: :horse:
I just want my project to run when I start them, next day or in a year from now. How can I have that?
I don't want any software "improvements" and I don't care about security. It all runs offline on the main PC (Ubuntu), nothing has to go outside of the LAN, ever.
How can I make a "time capsule" for the tools I write?
Even virtual environments aren't immune to updates. For my use, containers are cumbersome to configure, and hard to make them run months later. And it's ridiculous to create a many GB new VM for each 20 lines of Python I may write as a tool. It's for a hobby lab at home, should be fun and easy.
Anybody else going back to use own scripts from months ago? Does that happen to not run any more? How to deal with that? Do I need yet another PC as a "production" machine for the lab, isolated and never to be updated?
-
Virtual box can store incremental snapshots.
-
RoGeorge, see old Scientific American article on software complexity, 1960s(?)
The issues you have were bound to happen.
I am very glad that I am a hardware engineer.
Circuits and devices I designed and built back to 1960s, still work perfectly.
bon courage
Jon
-
OK, but how do I avoid to continuously adjust my own software to whatever updates? Even a virtual box VM required additional installs.
I want to start the PC and run again what used to be working last month, or last year. As simple as that.
Am I the only one returning to work/projs from more than a couple of months ago?
How is everybody else running their own SW tools and scripts?
-
How is everybody else running their own SW tools and scripts?
By not using Python.
-
Yep, python is a moving target...
-
:popcorn:
-
You mean to use C instead?
OK, though sometimes this might not be possible, or possible but very time consuming. Often is only a few lines improvised, then tuned a little, then saved and that's it. Rewriting all that in C would be too pedantic for home use.
From the answers it looks like I'm the only one going bacj to 6+ months old projects and having problems with backward compatibility after updates. :-//
-
Yep, python is a moving target...
Python is fine.
The problem is that most people (especially the ones using stuff like Jupyter - i.e. mostly research folks) don't have a clue (or care!) how to build software for distribution, how to manage dependencies properly, etc. But that's not fault of Python. Python has plenty of good tools designed to handle this for you, one only has to use them!
One can make the same or worse horrific mess in any language. Don't bring up the argument that "I wrote some C or assembler software 20 years ago and still runs" - sure and since it has never been updated it is likely full of security holes and bugs thanks to the linked in dependencies. Or (given the nature of this forum) it is some embedded stuff irrelevant to the debate. And btw, the same feat is possible perfectly fine with Python too - as long as you don't distribute only "naked scripts", letting the user figure out (tearing their hair out in the process) which versions of which dependency they have to install to make it work ...
Especially things like the Jupyter(Lab) notebook with its many plugins, PyTorch, Tensorflow (the AI frameworks) that are using Python as glue to bring together hundreds or thousands of dependencies written in various languages (including C/C++, Fortran, Javascript ...) in a very fragile manner are a nightmare. Done typically with no proper dependency version control, so one package somewhere deep in the dependency tree gets updated because of something unrelated - and now the entire stack is broken. To make it worse, most of that code is written in a way of "throwaway" notebooks and scripts because the goal is to get a paper published as quickly as possible by a guy or girl who programs only because they have to, not because they are programmers. And not to produce any sort of "production quality" code.
And if something underlying like CUDA or Nvidia drivers (which are used by the above for acceleration of the computation) changes because of a forced automatic update, then even containers like Docker or a complete VM (e.g. VirtualBox) won't save you.
If you want to make sure something keeps running you need to "freeze" that version of the system in a VM - and never touch it anymore. Including never letting e.g. Windows to update anything. Linux is much easier in this regard because you have complete control over the update process. In Windows this is difficult unless the system is airgapped from the Internet.
EDIT: I have slept on this and to be constructive, here are a few suggestions how to make your Python life easier:
If you don't need CUDA for AI, then being rigorous with dependency management (e.g. using tools like poetry, pipenv or at least a requirements.txt file!) together with virtual environments to isolate applications from each other (both pipenv & poetry manage that for you) will make sure your code remains installable and working. And if you distribute your thing, distribute it including those build configurations!. Even better - make a proper Python package and have the thing distributed via PyPi so that it can be installed using pip.
"Naked" scripts are only acceptable if they don't have any dependencies beyond what is in the standard library - and even then be clear about which Python version is required as a minimum if you use any sort of "exotic" features - such as asyncio (needs 3.6+), match keyword (3.10+), etc. And please, don't write new software requiring Python 2!
For end users who don't want to install Python and every dependency, build an executable - e.g. pyinstaller will do it for you. Far better than slapping a python script on Github and then people complain that they can't get it to work because it relies on this or that in version X but version X+2 has changed API and now your script doesn't work for them but since it is the latest, that's what gets installed.
Also, never distribute code that is meant to be used for actual "production" (i.e. not only research, quick tests) as a Jupyter notebook. The notebooks don't have the abovementioned version management, neither for Python dependencies by default and neither for the various Jupyter plugins that may be required e.g. to display some plots and what not. Use Jupyter to generate images for documentation but the actually useful code should be in properly managed Python modules instead that don't depend on 20 Jupyter javascript plugins to work. Even plots can be generated without Jupyter - e.g. matplotlib or bokeh work from "straight" Python too.
And for "users" of those python scripts - never install anything into the system-wide Python installation. I.e. don't just run "pip install foo"! Always set up a virtual environment first, to keep your Python applications isolated from each other, so that installing e.g. numpy version 1 doesn't break another tool that needs numpy version 2.
If everyone followed this it would have made everyone's life a lot easier and there would be far less bashing Python for things that are not its fault.
-
- Installed some GPT transformers (AI, 15GB+ of HDD, with CUDA, PyTorch, inference models, etc.). Made for it a venv to preserve the Python things, did some customisation, parked the project in good state 3 months ago. Last week wanted to continue. Doesn't run any longer. nVidia updates, CUDA updates, Eleuther model updates. TL;DR not working, spent hours reinstalling, still not working.
For something this elaborate Anaconda might help. For me it works better to go with the flow and update and fix things, rather than fighting things and trying to stick to old versions.
- Did some Jupyter Notebook in April this year, to SCPI control a power supply with PyVISA. It's October and it doesn't run any more, had to disable some authentication token crap in Jupyter, and it's still not working like it was in April.
I have Python scripts and notebooks for instrument control that go back years. Some of it written over a decade ago. Still runs fine. The last minor change I remember was when I moved from Python 2.x to 3.x. Had to run a script to make minor changes like parentheses for the print statement and change from float to integer division. I use Poetry (https://python-poetry.org/) to install all dependencies in a virtual environment. If I'm really fussed, then I might use PyEnv (https://github.com/pyenv/pyenv) to freeze the version of Python so I'm not dependent on the system Python version, but generally I don't bother. Getting Jupyter to use virtual environments is a bit of a hassle (https://www.geeksforgeeks.org/using-jupyter-notebook-in-virtual-environment/), but works once you set it up. If anything goes wrong with the dependencies, I blow away the virtual environment and have poetry install all dependencies from scratch. But I try to stay reasonably current on libraries, because generally many small updates are less painful than a single huge upgrade.
Circuits and devices I designed and built back to 1960s, still work perfectly.
How about if you tried to build another copy? Or if you had to build a thousand and had to find a contract manufacturer to build them? What size of feeder do you use for tube sockets? Hardware is not exactly immune to dependency (component availability) issues.
-
I hate this, software gets updated far too often and usually the updates cause more problems than they fix. I've gone back to the policy I had with things like BIOS updates back in the early 2000's, I only update if there is a specific problem that I'm trying to fix with the update. Almost every other time I have let something update I came to regret it. Far too often the changes just make something worse.
-
For something this elaborate Anaconda might help. For me it works better to go with the flow and update and fix things, rather than fighting things and trying to stick to old versions.
The problem with Anaconda is that it has its own packaging system separate from pip/Pypi and if some package isn't available in the Anaconda "channels", then you need to rely on various uncurated 3rdparty repositories that often have packages not compatible with the official ones. Or you use pip to install things from Pypi and then everything breaks because, of course, pip and conda aren't aware of each other.
Also most Python tutorials and recipes are designed around standard Python tooling, Anaconda does its own thing and then you can shoot yourself in your foot in a pretty big way if you don't know what you are doing.
Finally, I have found the dependency resolution by the conda tool in Windows to be extremely, painfully slow and it often fails. And then the fun starts ...
Anaconda is good if you need e.g. a workable installation of PyTorch or Jupyter and don't want to touch it. They do give you that out of box experience because this is the target market they focus on - researchers who want to do numerical calculations using numpy, pandas, scipy, PyTorch, etc. so they have a good well-rounded setup with all the common packages available. But if you need stuff that is not available in Anaconda, it will be a major pain.
I have used Anaconda in Windows for some AI training stuff before but ended up switching to standard Python install. It was a bit more work but in the end it was far easier to manage because of the standard, well documented tooling.
-
I agree, that's why I specifically suggested it to get started quickly with PyTorch etc. I wouldn't recommend it otherwise.
-
OK. Now I need a new setup for this speech recognition tool from OpenAI, MIT licensed:
https://github.com/openai/whisper
It depends on
- Python
- PyTorch
- HuggingFace Transformers
- AI inference models
- ffmpeg, ffmpeg-python
- sound
- might need nVidia/CUDA, too, not sure about last one yet
Once the setup is running, I want to shut down the "whisper" thing, and never start it again until next year. Meanwhile, I need to use the PC as a day to day home desktop. One I start "whisper" application again, tomorrow or in a year from now, it should run like in day one, unattended, all offline and without Internet.
How would you setup this?
-
How would you setup this?
I wouldn't.
-
OK. Now I need a new setup for this speech recognition tool from OpenAI, MIT licensed:
https://github.com/openai/whisper (https://github.com/openai/whisper)
It depends on
- Python
- PyTorch
- HuggingFace Transformers
- AI inference models
- ffmpeg, ffmpeg-python
- sound
- might need nVidia/CUDA, too, not sure about last one yet
Once the setup is running, I want to shut down the "whisper" thing, and never start it again until next year. Meanwhile, I need to use the PC as a day to day home desktop. One I start "whisper" application again, tomorrow or in a year from now, it should run like in day one, unattended, all offline and without Internet.
How would you setup this?
Like I said, I prefer a more dynamic approach where I'll update dependencies and make whatever small changes might be necessary to track those updates. But if you truly want to freeze time, then these would be my suggestions depending on how deep the dependencies go:
- For projects with pure Python libraries that depend on pretty basic system libraries, which includes PyTorch and possibly ffmpeg, but not Cuda, I'd use virtual environments with pyenv to also freeze the Python version
- For projects with libraries that have more specific native dependencies, like a particular ffmpeg version (I haven't tracked that project closely enough to know how they are about updates), I would use Docker. Docker is not made for desktop use, so it will not do video, and probably sound only if you can do it over the network (https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Network/). But there is apparently Cuda support (https://www.celantur.com/blog/run-cuda-in-docker-on-linux/), although this obviously still depends on the driver on the host.
- For projects that need to show a UI but do not need Cuda, I'd use VirtualBox
- For projects that need full hardware access, like video, Cuda, etc, I'd make a bootable USB stick. There are ways to provide GPU pass-thru to virtual machines, but I don't think that's worth it for this scenario since it still depends on the host configuration.
-
If you want consistency and stable interfaces, avoid two things like the plague - Windows and Python.
-
avoid two things like the plague - Windows and...
more shit smell post? did you read OP is running Ubuntu? if you experienced like i do for the past few decades about Windows and its paradigm its try to impose on software developers, you'll change your mind. non-backward compatibilitiness is only for rookies programmers who want/think to make the world better in short term thought process. another thing is if the developer dont allow you to do full offline installer download nor letting to install older version, like Python does.
-
If you want consistency and stable interfaces, avoid two things like the plague - Windows and Python.
I'd admit that for Python, but Windows? As much as I don't like where Windows has been headed for a number of years now, it has been one of the most stable platforms out there. Heck, if you stick to the Windows API, applications written 20 years ago can still be compiled for the latest Windows and run. They are even likely to run without requiring a recompilation.
-
If you want consistency and stable interfaces, avoid two things like the plague - Windows and Python.
I'd admit that for Python, but Windows? As much as I don't like where Windows has been headed for a number of years now, it has been one of the most stable platforms out there. Heck, if you stick to the Windows API, applications written 20 years ago can still be compiled for the latest Windows and run. They are even likely to run without requiring a recompilation.
Released in 1999, still using daily.
-
If you want consistency and stable interfaces, avoid two things like the plague - Windows and Python.
Really? I can run code on Windows 11 in 2022 that I wrote in 1995.
That leaves Python.
-
Sure, if you code for windows 3.11
But if you use any of the annual fashion APIs they produce, you'll be using it for 2 years before they drop it and try something else.
OLE anyone ?
-
then dont use half baked API invention, try to rely on more pro/established platforms (IDE) i cant tell which one i havent used the latest IDEs, i program old style. but back then M$ will tell much earlier about deprecated API's, not sure today when kids signing in...
-
What's that silly comment about Win 3.11? Windows has had a solid API for a very long time, and it still does. And it's still fine. It's well documented, along with which versions of Windows support a given function/type/... You can still perfectly write applications purely in with the Windows API these days. It's not rocket science. No need to go back to Win 3.11. Which was 16-bit anyway. But they have handled the Windows API very well over the years. It practically never broke anything.
Of course. If you want stable stuff, use stable stuff. Duh. If you use the latest trendy shit, you'll reap what you sow.
-
OK. Now I need a new setup for this speech recognition tool from OpenAI, MIT licensed:
https://github.com/openai/whisper (https://github.com/openai/whisper)
It depends on
- Python
- PyTorch
- HuggingFace Transformers
- AI inference models
- ffmpeg, ffmpeg-python
- sound
- might need nVidia/CUDA, too, not sure about last one yet
Once the setup is running, I want to shut down the "whisper" thing, and never start it again until next year. Meanwhile, I need to use the PC as a day to day home desktop. One I start "whisper" application again, tomorrow or in a year from now, it should run like in day one, unattended, all offline and without Internet.
How would you setup this?
Like I said, I prefer a more dynamic approach where I'll update dependencies and make whatever small changes might be necessary to track those updates. But if you truly want to freeze time, then these would be my suggestions depending on how deep the dependencies go:
- For projects with pure Python libraries that depend on pretty basic system libraries, which includes PyTorch and possibly ffmpeg, but not Cuda, I'd use virtual environments with pyenv to also freeze the Python version
- For projects with libraries that have more specific native dependencies, like a particular ffmpeg version (I haven't tracked that project closely enough to know how they are about updates), I would use Docker. Docker is not made for desktop use, so it will not do video, and probably sound only if you can do it over the network (https://www.freedesktop.org/wiki/Software/PulseAudio/Documentation/User/Network/). But there is apparently Cuda support (https://www.celantur.com/blog/run-cuda-in-docker-on-linux/), although this obviously still depends on the driver on the host.
- For projects that need to show a UI but do not need Cuda, I'd use VirtualBox
- For projects that need full hardware access, like video, Cuda, etc, I'd make a bootable USB stick. There are ways to provide GPU pass-thru to virtual machines, but I don't think that's worth it for this scenario since it still depends on the host configuration.
Yep, pretty much this.
If it doesn't need native code dependencies like ffmpeg then a Python virtual environment with frozen versions of the dependencies would be enough. However, as this needs external, non-python dependencies, then stuffing it into a VM or a container (Docker ...) is the safest option because if those external libraries change the Python packages that depend on them could get broken.
Tools like VirtualBox and I believe even Docker are able to give access to the raw hardware like the GPU, so it is possible to have CUDA and the matching driver installed inside of the VM, even though the host system is running a different version. As long as the GPU is compatible with the old driver it should work. However, this is unfortunately a rather non-trivial to set up and make play nicely with the host system. Graphic cards and their proprietary (the open source drivers usually work - but are useless for these use cases ...) drivers are notorious for not playing nicely with virtualization of any sort.
And finally, OP, if you need speech recognition - unless you are specifically doing research in this area, did you consider other tools? There are multiple mature and even open source speech recognizers that work well and have both acoustic and language models for various languages and applications, typically using hidden Markov models instead of neural networks. Using a month old university research project is bound to give you trouble, for exactly the reasons I wrote above - it is code meant to publish papers/get a PhD, not anything that is supposed to be "production quality".
E.g.:
- CMU Sphinx: https://cmusphinx.github.io/ (https://cmusphinx.github.io/) (multiple versions, including a lite one that works on small computers like RPi - PocketSphinx)
- Vosk: https://alphacephei.com/vosk/install (https://alphacephei.com/vosk/install)
- Kaldi: https://github.com/kaldi-asr/kaldi (https://github.com/kaldi-asr/kaldi)
They may not be the bleeding edge state of the art like the many recent deep learning based ones but may be far less hassle to set up and use because there are no crazy fast moving dependencies like PyTorch, CUDA or Tensorflow involved.
-
another thing is if the developer dont allow you to do full offline installer download nor letting to install older version, like Python does.
What language are you talking about? Python offers version going back to 2001 right on their website (https://www.python.org/downloads/). You can download them for offline installation if you want. The tool PyEnv (https://github.com/pyenv/pyenv) makes it trivial to install different versions. It will even compile them from source if there's no suitable binary release for your system. But the changes between Python releases are so minor that I don't even pay attention to the version for personal use. My desktop runs Python 3.10, but my data acquisition code runs on Python 3.7. Never ran into a problem as long as I don't use the new features that have been recently introduced, which are generally easy enough to avoid.
Can you give an example of any Python 3.x program that you wrote that does not run in Python 3.10? The only thing I can remember is the Python 2.x to 3.x migration, which was over a decade ago and was well documented with tools to help.
Of course you need to manage dependencies, but that applies to any program that uses a non-trivial number of dependencies. Tools like Poetry and Pipenv are available to take care of this.
-
Can you give an example of any Python 3.x program that you wrote that does not run in Python 3.10? The only thing I can remember is the Python 2.x to 3.x migration, which was over a decade ago and was well documented with tools to help.
As someone who drops into Python occasionally and very begrudgingly, the examples all over the web, frankly, are hit and miss when trying to solve a problem by yourself, as a python novice.
Of course you need to manage dependencies, but that applies to any program that uses a non-trivial number of dependencies. Tools like Poetry and Pipenv are available to take care of this.
At the risk of getting shouted at, if I invest time looking up how to use these depends managers and what not, is the same thing going to happen to these tutes and the tools breaking when Python has again moved on but certain info on the internet hasn't.
I'm not trying to throw shade. I too may find myself in George's shoes some day if I get around to learning Python instead of just using C. Shouldn't these tools be included in the language if it wants to be regarded as a higher level language?
-
Of course you need to manage dependencies, but that applies to any program that uses a non-trivial number of dependencies. Tools like Poetry and Pipenv are available to take care of this.
i dont program in python, but there's lecroy dso hack tool scripted in python that i downloaded here. It relies on some encryption library and another one i cant recall... to add the libraries, i have to type command line in console to install them (after i installed python environment) .. from the time taken, i felt its an online install with silent report, no option no version to choose from. Its just wait few seconds and done.. my thinking is that if those libraries are upgraded with no back compatibility in mind, my hack tool can easily break anytime.. i just need to pray and finger cross they will not major upgrade anytime soon.
-
Of course you need to manage dependencies, but that applies to any program that uses a non-trivial number of dependencies. Tools like Poetry and Pipenv are available to take care of this.
i dont program in python, but there's lecroy dso hack tool scripted in python that i downloaded here. It relies on some encryption library and another one i cant recall... to add the libraries, i have to type command line in console to install them (after i installed python environment) .. from the time taken, i felt its an online install with silent report, no option no version to choose from. Its just wait few seconds and done.. my thinking is that if those libraries are upgraded with no back compatibility in mind, my hack tool can easily break anytime.. i just need to pray and finger cross they will not major upgrade anytime soon.
Well, but that's a complaint you should take with the author of that tool. That's no fault of Python that the author of that thing doesn't do things right for their users.
Can you give an example of any Python 3.x program that you wrote that does not run in Python 3.10? The only thing I can remember is the Python 2.x to 3.x migration, which was over a decade ago and was well documented with tools to help.
As someone who drops into Python occasionally and very begrudgingly, the examples all over the web, frankly, are hit and miss when trying to solve a problem by yourself, as a python novice.
At the risk of getting shouted at, if I invest time looking up how to use these depends managers and what not, is the same thing going to happen to these tutes and the tools breaking when Python has again moved on but certain info on the internet hasn't.
That's sadly nothing specific or particular to Python. Did you try to follow some Javascript or even C++ tutorials? You will find tons of stuff online that is decades old and won't even compile with current tooling ...
That there is outdated information out there, whether in books or online is a fact of life and not only with computing. As always, it helps to search in the canonical sources (e.g. official Python documentation) first and not just grab whatever snippet from StackOverflow showed up on top in Google.
I'm not trying to throw shade. I too may find myself in George's shoes some day if I get around to learning Python instead of just using C. Shouldn't these tools be included in the language if it wants to be regarded as a higher level language?
That has nothing to do with it being a "higher level language". However, even then, those tools are part of the language installers and are shipped with it.
Pip and virtualenv are all you strictly speaking need to manage dependencies in Python - and both come standard with it even though they are not "included in the language" (similarly like make is not part of C yet it comes with the toolchains). And the documentation and best practices on using them have been around for ages.
Tools that have been mentioned above - poetry, pipenv or pyinstaller are only frontends to pip and virtualenv (the first two) that make the process a bit more convenient for the developer. Pyinstaller is one of the possible bundling utilities to build binaries. There are several. But as building binaries is both platform specific and not part of Python's mission (it is still an interpreted language, after all), I am fine with that being an external tool.
In the future it is possible that something like poetry will get bundled with Python installations by default but Python code packaging (not dependency management) is still a bit of an evolving target ("eggs" vs "wheels" vs setup tools. etc).
-
Of course you need to manage dependencies, but that applies to any program that uses a non-trivial number of dependencies. Tools like Poetry and Pipenv are available to take care of this.
i dont program in python, but there's lecroy dso hack tool scripted in python that i downloaded here. It relies on some encryption library and another one i cant recall... to add the libraries, i have to type command line in console to install them (after i installed python environment) .. from the time taken, i felt its an online install with silent report, no option no version to choose from. Its just wait few seconds and done.. my thinking is that if those libraries are upgraded with no back compatibility in mind, my hack tool can easily break anytime.. i just need to pray and finger cross they will not major upgrade anytime soon.
Well, but that's a complaint you should take with the author of that tool. That's no fault of Python that the author of that thing doesn't do things right for their users.
my point is that those libraries should be available for download too so we can make our own version control. so i dont have to blame anybody, not the author, not the library developer not the python.
-
I too may find myself in George's shoes some day if I get around to learning Python instead of just using C. Shouldn't these tools be included in the language if it wants to be regarded as a higher level language?
Don't you have the problem with dependencies in C if you include huge, fast-moving external libraries like GTK, ffmpeg, etc? Does GTK 2.x software still run or compile? I remember plenty of problems with dynamic libraries and OS upgrades requiring rebuilding, and sometimes changing, C programs because the interface had changed. Python programs that only use the standard library (which is quite a bit more comprehensive than libc) are also easy to keep running.
I agree that some sort of dependency management / virtual environment management tool like poetry should be included.
i dont program in python, but there's lecroy dso hack tool scripted in python that i downloaded here. It relies on some encryption library and another one i cant recall... to add the libraries, i have to type command line in console to install them (after i installed python environment) .. from the time taken, i felt its an online install with silent report, no option no version to choose from. Its just wait few seconds and done.. my thinking is that if those libraries are upgraded with no back compatibility in mind, my hack tool can easily break anytime.. i just need to pray and finger cross they will not major upgrade anytime soon.
That key gen distribution was purposely obfuscated because of the intellectual property issues, hence the cumbersome installation without proper documentation. The original post said it was developed for Python 2.7.x with PyCrypto. This is a Python version from a decade ago, but you can still download it from the Python.org website (see the link I posted above). Same with the PyCrypto module. PyCrypto didn't become incompatible, but was discontinued and PyCryptodome came in its place. It's compatible, and you can replace the Crypto import by Cryptodome and make some other changes to get it working with Python 3 (https://pastebin.com/s1SCi9KE). Admittedly those changes are a bit more complicated for this than for most programs, because it does byte-level string manipulation which is one of the things that changed with Python 3 due to the switch to unicode by default. But nothing prevents you from downloading Python 2.7.x and keep using that.
my point is that those libraries should be available for download too so we can make our own version control. so i dont have to blame anybody, not the author, not the library developer not the python.
If I search for PyCrypto with Google, I find this (https://pypi.org/project/pycrypto/) and from there this homepage (https://www.pycrypto.org/), both of which contain download links for the almost-a-decade-old libraries. Which libraries are not available for download?
-
Don't you have the problem with dependencies in C if you include huge, fast-moving external libraries like GTK, ffmpeg, etc? Does GTK 2.x software still run or compile? I remember plenty of problems with dynamic libraries and OS upgrades requiring rebuilding, and sometimes changing, C programs because the interface had changed. Python programs that only use the standard library (which is quite a bit more comprehensive than libc) are also easy to keep running.
I only do very simple console stuff in C, though in the past year I've gotten a lot of help embracing simple, but effective GUI. :)
Yeah, on other platforms (arm?) my own code might well bork cos I'm very lazy with variable types and endiness. Oh well.
Most of the time the source build tools and scripts are quite good that the ./configure script tells you which header file is missing and more often than not, what range of version it was really hoping for. I was using the same code for XMMS for 17 years or something. The distro maintainers are top notch though which helps in that you just bring in the dev packages your exotic source code asks for.
The point is, though, I expect push back from a compiler, but not an interpreter. Or am I being too harsh?
Edit: ./configure not MAKE
-
As others have mentioned, the idea is to "freeze" the environment, such that if you start it a year from now, it is the same as it was then. A "thought exercise" then, on how to do this ...
I would:
- use VirtualBox (whether on Windows/Linux, doesn't matter)
- build, for example, a Windows VM with the current set of updates, and with all your desired software components
- freeze it, with a program like "Deep Freeze, by Faronics"; this would avoid any future updates to any of the Windows pieces (windows OS itself, your software components, etc) inside the VM.
- save the vm disk file into your catalog of years/projects
- additionally, use Macrium Reflect (free windows backup program) to create an image of the entire VM os and software, inside the vm, for a 2nd fallback.
At this point, nothing should change a year later, as Deep Freeze always sets it back to pristine start on reboot. It doesn't allow anything, even windows os updates, to take place. You could, with more effort, replace Deep Freeze with an imaging tool.
As further insurance for the host os that runs the vbox environment, as vbox might change over the years, use macrium (or any imaging tool of your choice) to capture the host os running the vbox environment as well.
You might also want to set aside a PC/Laptop from nearby years for the host os, or sort through vbox changes if you put a newer pc/laptop in place. Saving a pc from every 5 to 10 years ought to do it (complete mobo, cpu/ram, power supply, all internal drives ... don't need the entire pc shell).
Macrium is both a backup tool and an imaging tool, so before you think this is getting too circular, you can substitute any imaging tool, if you think another one has a better way of preserving file formats over the years. But, basically, if Macrium changes, you'll just restore each item in your catalog with the old macrium, image it with the new Macrium, and throw it back in the catalog.
I *think* I've covered all possible changes that might occur, at any point in the chain ... it is a thought exercise.
This process works for me when preserving ancient os's from decades past (dos, os/2, lan manager, netware, etc), and then pulling them out and running again for a bit. Can't yet see why it wouldn't work for this thread's premise as well.
-
Most of the time the source build tools and scripts are quite good that MAKE tells you which header file is missing and more often than not, what range of version it was really hoping for. I was using the same code for XMMS for 17 years or something. The distro maintainers are top notch though which helps in that you just bring in the dev packages your exotic source code asks for.
The point is, though, I expect push back from a compiler, but not an interpreter. Or am I being too harsh?
Linux distributions do package Python packages, for example I installed the PyCryptodome cryptography library for the scripts Mechatrommer mentioned in Ubuntu using apt install python3-pycryptodome. But the more obscure may not be there, and if you want the very latest version, that's probably also not available through your distribution. To be able to guarantee the same version on different operating systems and versions, Python programs will often install dependencies outside the OS. The Python package manager pip and wrappers around pip like pipenv and Poetry can be used to manage this. Obviously this does not apply by Python programs shipped as OS packages, which will depend on other packages just like any programming language.
Compilers and modern interpreters aren't that different, in that both will parse the code in some intermediate representation, but compilers will then translate this to machine code and generate a binary, while interpreters will run this byte code on a virtual machine. So the problems they encounter with language versions (K&R C, ANSI C/C89, C99, etc) and dependencies that need to be present and have a compatible version are actually quite similar.
One thing that is annoying about Python is that modules are installed in a directory for that specific minor version. So if you install Python 3.9, install a library, and then upgrade to Python 3.10 (for example an OS upgrade), that library won't be available in Python 3.10 until you reinstall. That's why I suggested either using a tool like pyenv to keep the Python version in your project independent from the system Python version, or using a tool like pipenv or Poetry that can reinstall all packages with the new Python version with minimal effort.
As others have mentioned, the idea is to "freeze" the environment, such that if you start it a year from now, it is the same as it was then. A "thought exercise" then, on how to do this ...
I would:
- use VirtualBox (whether on Windows/Linux, doesn't matter)
[...]
Cuda is something that makes this a bit more complicated (you have to set up some sort of pass-thru, and hope this path-thru does not rely on compatible driver versions). I would like to add that to ease maintenance you ideally want to freeze as small an area as possible. Just like you wouldn't freeze your entire house to keep some frozen vegetables fresh. I gave some guidelines for that in an earlier post.
-
I *think* I've covered all possible changes that might occur, at any point in the chain ... it is a thought exercise.
As I said above, you can snapshot the daylights out of VBox. This alleviates the inevitable crunch time when you realize the sanctuary is antiquated and want to update the underlying system.
-
Also, you don't have to virtualize. There is timeshift and/or BTRfs, if set up properly can aid in walking back lots of system randomness.
I don't use them though. I'll reinstall the same way I'll get a new car because the ash tray is full.
-
... I'll get a new car because the ash tray is full.
Best not to smoke then :-DD Modern cars might not even have an ash tray anymore >:D
-
i dont mind modern car as long as i know how to open the window and commit the ashes to the air... i dont have time to learn python's mechanics, i'll learn as i encounter problem later. that pycryptodome may not be available in the next 2-5 years maybe they will rename to fancier name such as pycryptotray who knows. anyway thanks alm for the tips and chiming in that lecroy's thread.
-
my point is that those libraries should be available for download too so we can make our own version control. so i dont have to blame anybody, not the author, not the library developer not the python.
Since you ran some commands from the console to install them, they are by definition available for download. Nothing prevents you from backing them up, ideally along with the rest of your system because they may have their own dependencies. Or get their source code so that you can compile them in the future.
-
i dont mind modern car as long as i know how to open the window and commit the ashes to the air... i dont have time to learn python's mechanics, i'll learn as i encounter problem later. that pycryptodome may not be available in the next 2-5 years maybe they will rename to fancier name such as pycryptotray who knows. anyway thanks alm for the tips and chiming in that lecroy's thread.
Which Python module did you find that is not available anymore? I showed you links to download PyCrypto. Links to Python going back to 2001. You seem to imagine problems that don't actually exist.
-
i dont mind modern car as long as i know how to open the window and commit the ashes to the air... i dont have time to learn python's mechanics, i'll learn as i encounter problem later. that pycryptodome may not be available in the next 2-5 years maybe they will rename to fancier name such as pycryptotray who knows. anyway thanks alm for the tips and chiming in that lecroy's thread.
Which Python module did you find that is not available anymore? I showed you links to download PyCrypto. Links to Python going back to 2001. You seem to imagine problems that don't actually exist.
i havent tried yet. maybe later when i do lecroy hack again, the older Python installer is now dormant in my external storage... you mentioned it yourself that the original pyCrypto is not available anymore (replaced with pyCryptoDome?), i'm not sure if its a problem or not, but i'm guessing a possibility (expect the worst so i can prepare at least mentally later) but if its no problem, then its no issue. cheers.
-
you mentioned it yourself that the original pyCrypto is not available anymore (replaced with pyCryptoDome?), i'm not sure if its a problem or not, but i'm guessing a possibility (expect the worst so i can prepare at least mentally later) but if its no problem, then its no issue. cheers.
No, I said that PyCrypto was discontinued: it's no longer maintained or recommended to be used. But it's still available for download. I already gave you the links. You can download it in your archive.
It has a compatible successor PyCryptodome that you can use if you want to run the script in Python 3, like I did. But you can also still download and run Python 2 and PyCrypto.