Linux and Virtual Environments in conda/miniconda/anaconda

[RoGeorge]

I am bamboozled by virtual environments
- there are virtual environments for Python only
- there are virtual environments for other binaries too, not only for python, by using "conda"
- there are also environment variables and chroots mechanisms native to Linux

I noticed that without 'conda (or anaconda) activate', some things apparently are missing, e.g. anaconda-navigator is not found unless an environment that contains anaconda was previously activated by a 'conda activate' command, or if the auto activate option was set.

At this point I don't know if conda environments are just messing with the Linux's $PATH environment variable, or is it some other "environment mechanism" native to Linux that I didn't know about?  Or is it something specific to conda only?

What mechanism is used to create virtual environments with conda, or roughly said how are the virtual environments implemented in conda?

How does conda achieve the isolation between different environments?

Does Linux have native virtual environments capabilities, or something similar?

[janoc]

Careful, Anaconda/conda stuff is not standard to neither Linux nor Python. It is a specific solution developed by Anaconda (the company) for solving Python distribution issues, especially in Windows. In Linux it is typically less relevant because most Linux distros ship recent versions of Python along with other dependencies (numpy, scipy, nodejs, etc.). In Windows using regular stock Python from python.org it has been traditionally fairly painful to install all of that, so Anaconda has made their own, pre-packaged distribution of Python with its own dependency-solving package manager (conda).

What Anaconda virtual environment gives you is a full isolation so you can install e.g. different versions of Python in parallel, including any binary dependencies ("DLLs") those things may need. For example if your install some machine learning stuff (like PyTorch or Tensorflow), it will also install binaries for CUDA, MKL, OpenCV and what not for you, managing the dependencies in such way that the pieces will hopefully work together (unlike pip, which handles only Python dependencies, not external binaries).

Python virtual environments (virtualenv) are totally separate and orthogonal to whatever Anaconda does (even though conda builds on them). These are designed to manage only Python packages and to separate them from each other, so that you can install one version of e.g. Numpy needed by one program side by side with another version required by another program. However, virtualenv does not manage the binary dependencies for you - e.g. if you install PyTorch using pip in your virtual environment, you will need to make sure that a compatible CUDA version is installed yourself. This is usually no problem in Linux because you will want to install the version shipped by your distro anyway to make sure it is compatible with the rest of the system.

I noticed that without 'conda (or anaconda) activate', some things apparently are missing, e.g. anaconda-navigator is not found unless an environment that contains anaconda was previously activated by a 'conda activate' command, or if the auto activate option was set.

At this point I don't know if conda environments are just messing with the Linux's $PATH environment variable, or is it some other "environment mechanism" native to Linux that I didn't know about?  Or is it something specific to conda only?

Yes, 'conda activate' changes the PATH variable (and possibly others, such as LD_LIBRARY_PATH in Linux) in the current shell so that the programs (and libraries) for the activated virtual environment take precedence over whatever else you may have installed. Otherwise you could be attempting to load incompatible (system) libraries and that would cause crashes.

It will set the PATH to load the copy of python from your virtual environment, set up LD_LIBRARY_PATH to load the proper binaries and then set up Python-specific environment variables to tell that python where it needs to look for the python packages (in the directory where you have created the virtual environment instead of the system-wide one).

What mechanism is used to create virtual environments with conda, or roughly said how are the virtual environments implemented in conda?
How does conda achieve the isolation between different environments?

Mostly changing environment variables and then for the python bits the standard virtualenv mechanism (which is mostly relying on env. variables as well).

Does Linux have native virtual environments capabilities, or something similar?

That quite depends on what you mean. Conda, virtualenv and similar tools are Python-specific solutions for Python-specific issues. These work the same on all platforms where Python is used these days.

Linux has native virtualization solutions which are not Python-specific, but that's a very different kettle of fish (and likely overkill for use with Python):

- chroot (creates a separate filesystem hierarchy and switches to it using a system call)
- virtual machines using e.g. Xen or KVM hypervisors (makes an entire virtual computer for you)
- container solutions (e.g. Docker) (a pre-packaged, self contained environment that runs on the host using various tricks like chroot - kinda virtual machine "lite")