Skip to main content

One frequently asked question is "How can I install the Python packages I need on the cluster?".

Sure, you could install packages in your /home directory using pip install --user <some_package>.

This will install packages in $HOME/.local/. However this does not create an isolated environment, which is essential if you intend to use python for different projects or if you rely on specific versions.

conda

We recommend the use of conda virtual environments for your Python projects.

On the cluster, Miniconda 23.11, Python 3.11 (12/20/2023) is available as a module (conda/23.11-py311).

Loading the module​

To load the conda module do :

module load conda/23.11-py311

or even simpler:

ml conda

Upon loading, the following command is run, initializing conda:

source `$CONDA_PREFIX/etc/profile.d/conda.sh`

There is no need to run conda init (which would modify your ~/.bashrc).

However, the base environment is not automatically activated.

Show existing conda environments​

conda env list

shows all visible conda environments. For example,

# conda environments:
#
testenv                  /home/jan.gmys/.conda/envs/testenv
base                     /share/tools/miniconda3/23.11
python311-tensorflow     /share/tools/miniconda3/23.11/envs/python311-tensorflow
python311-tools          /share/tools/miniconda3/23.11/envs/python311-tools
pytorch-gpu              /share/tools/miniconda3/23.11/envs/pytorch-gpu
scikit-learn             /share/tools/miniconda3/23.11/envs/scikit-learn
tf-gpu                   /share/tools/miniconda3/23.11/envs/tf-gpu

Here testenv is a user-created environment and the other environments are shared by all users of the HPC cluster.

Activate an environment​

To activate an environment use conda activate. For instance,

conda activate python311-tensorflow

If the environment was activated successfully, it's name should appear in your command prompt:

(python311-tensorflow) jan.gmys@zeus-2:~$

You can check that python is now the one of the active environment:

(python311-tensorflow) jan.gmys@zeus-2:~$ which python
/share/tools/miniconda3/23.11/envs/python311-tensorflow/bin/python

You can also check that the tensorflow package is available

python -c 'import tensorflow as tf; print(tf.__version__)'

Inspect environments​

You can use the command conda list to see all packages installed in the current environment, or

conda list -n <env-name>

to see the packages installed in a specific environment.

Create new environments​

You can create a new environment with

conda create -n my_new_environment

The default location for this new environment will be $HOME/.conda/envs/my_new_environment.

You can also create a new environment by cloning an existing one

conda create -n my_new_env_cloned --clone existing_env

Installing packages with conda or pip​

You can use conda install to add packages to your environment

conda install functools

or pip install:

pip install abc

Remove packages​

conda remove package_name removes the package package_name from the currently active environment.

conda remove -n env_name package_name removes package_name from the environnement named env_name.

Deactivate an environnement​

conda deactivate

deactivates the current environment.

Remove an environnement​

You can delete your environment my_env with

conda remove -n my_env --all