Computer Documentation Wiki

User Tools

Site Tools

Trace: • nx_client • nextcloud • putty • pdf_tools • matlab_on_cluster • vscode • software_categories • environment_modules • x2go • working_with_python
working_with_python

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
working_with_python [2020/12/22 14:06] – jansenworking_with_python [2025/04/07 15:07] (current) – Modified example for RHEL9 jansen
Line 73: Line 73:
 === Method 1 - subsection Incompatible versions === === Method 1 - subsection Incompatible versions ===
  
-Unfortunately, python's 'user' directory is independent of the operating system version, but most of the compute nodes including the LOFAR cluster, run RedHat Enterprise Linux, which is sufficiently different to cause packages installed on the desktop not to work all the time.+Unfortunately, python's 'user' directory is independent of the operating system version, but most of the compute nodes including VDESK, LOFAR cluster and other compute nodes, run RedHat Enterprise Linux, which is sufficiently different to cause packages installed on the desktop not to work all the time.
  
 In cases like this, it might be necessary to create a separate python user directory structure for those machines: In cases like this, it might be necessary to create a separate python user directory structure for those machines:
Line 79: Line 79:
 Add to your .bashrc something like this: Add to your .bashrc something like this:
   if [ ! -f /etc/fedora-release ]; then   if [ ! -f /etc/fedora-release ]; then
-    export PYTHONUSERBASE=$HOME/.local-rhel7+    export PYTHONUSERBASE=$HOME/.local-rhel9
   fi   fi
 For users of the ''tcsh'' shell, add this to your .tcshrc in stead: For users of the ''tcsh'' shell, add this to your .tcshrc in stead:
   if (! -f /etc/fedora-release) then   if (! -f /etc/fedora-release) then
-    setenv PYTHONUSERBASE $HOME/.local-rhel7+    setenv PYTHONUSERBASE $HOME/.local-rhel9
   endif   endif
  
-And make sure to create that directory ~/.local-rhel7. Now the pip --user commands on RHEL7 machines will install into that newly created directory in stead of the default one used by the desktop systems.+And make sure to create that directory ~/.local-rhel9. Now the pip --user commands on RHEL9 machines will install into that newly created directory in stead of the default one used by the desktop systems.
  
-==== METHOD 2: virtualenv ====+==== METHOD 2: venv ==== 
 + 
 +''venv'' is a tool that creates isolated Python environments; it replaces the obsolete ''virtualenv'' that provided similar functionality for python 2.x. A python environment is essentially  a folder which contains copies of all necessary files needed for a Python project to run. In addition each virtual environment will contain a copy of the utility pip to manage packages. For example, let us suppose you would like to install ''pymatlab'' which is not installed on the departmental workstations, then you could do 
 +<code bash> 
 +$ mkdir /data2/username/venvs  
 +$ python3 -m venv /data2/username/venvs/pymatlab 
 +</code> 
 +to create a virtual environment (folder) called pymatlab (note that this example explicitly creates this in a directory on your local ''/data2'' disk, in order to avoid running out of disk quota in your home directory, which can easily happen since venvs can become rather big). 
 + 
 +In the example, we use ''python3'' as the python for this environment; if there are multiple python versions on the system, and you want to base your venv on a specific version, use that version to create the venv, e.g. ''python3.12 -m venv /data2/username/venvs/pymatlab''. 
 + 
 +The last step before starting to use the newly generated environment is to activate it, that is to prepend its ''/bin'' folder to your $PATH environment variable. This is done by issuing 
 +<code bash> 
 +source /data2/username/pymatlab/bin/activate # bash shells 
 +source /data2/username/pymatlab/bin/activate.csh # c shells 
 +</code> 
 + 
 +To acknowledge the activation of pymatlab, the terminal prompt will be changed to 
 +<code bash> 
 +(pymatlab)username@hostname:~/python_virt_envs/pymatlab$ 
 +</code> 
 +to emphasize that you  are operating in a virtual environment. To install pymatlab (or any other package) locally (in your virtual environment) run pip within that environment 
 +<code bash> 
 +pip install  pymatlab 
 +</code> 
 +Your virtual environment now should have the same core python packages defined globally for all the Observatory or Lorentz Institute  users plus any packages installed in the virtual environment.  
 +Note that you do NOT use ''--user'' on the pip command in this case, since that would install in your ''$PYTHONUSERBASE'' directory (see above) instead of the venv!! 
 + 
 +In any cases, it is advisable you keep a backup of your virtual environment configuration by creating a list of installed packages 
 +<code bash> 
 +pip freeze > packages.dat 
 +</code> 
 +This can help collaborators and fellow developers to reproduce your environment with  
 +<code bash> 
 +pip install -r packages.dat 
 +</code> 
 +When you are done working in a virtual environment deactivate it running  
 +<code bash> 
 +deactivate  
 +</code> 
 +At any time, any virtual environment can be destroyed by removing the corresponding folder from the file system so do not panic if things do not work, just delete your virtual environment and start all over again. 
 + 
 +Note: __System administrators will not be responsible and/or manage users virtual environments__. You are strongly advised you consult the documentation. 
 + 
 +==== METHOD 2: OBSOLETE: virtualenv (python 2.x) ====
 This guide refers to virtualenv version 12.0.7.  This guide refers to virtualenv version 12.0.7. 
  
Line 189: Line 233:
 In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library.  In this example we create a python2 virtual environment in which we will install the latest version of numpy that will use the openBLAS library. 
  
-:!: The procedure and paths below will work on any maris node and on the para cluster. +:!: The procedure and paths below will work on any maris node. 
  
 <code bash> <code bash>
Line 269: Line 313:
 ====== Jupyter Notebooks ====== ====== Jupyter Notebooks ======
 Depending on your operating system (Fedora or RedHat) you might get a different python kernel version as the standard kernel. If you get ''python2'' as the default kernel and only option, but wish the use the ''python3'' kernel you need to add this kernel to you local environment. This can be done by executing: Depending on your operating system (Fedora or RedHat) you might get a different python kernel version as the standard kernel. If you get ''python2'' as the default kernel and only option, but wish the use the ''python3'' kernel you need to add this kernel to you local environment. This can be done by executing:
-    python3 -m ipykernel install –user+    python3 -m ipykernel install --user
 Once this command has run successfully, it will have installed python3 as a jupyter kernel. Once this command has run successfully, it will have installed python3 as a jupyter kernel.
  
 After starting ''jupyter notebook'' you can select ''python3'' as kernel. After starting ''jupyter notebook'' you can select ''python3'' as kernel.
 +
 +If you need to work with several python setups (e.g. the system python3, but also from loaded environment modules), it is easy to assign a name to the generated kernel, e.g:
 +    python3 -m ipykernel install --user --name system-python3
 +
 +
working_with_python.1608645967.txt.gz · Last modified: by jansen

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
CC Attribution-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki