Python on the Teal Cluster

Python on the Teal Cluster


 

Learning Python

http://learnpythonthehardway.org/book/

http://www.codecademy.com/en/tracks/python

Python User Group - Richmond, VA - https://www.meetup.com/PyRVAUserGroup/ 

Showing all available python modules

Three different ways of showing all available python modules

1] pip freeze

2] >>>help("modules")

3]>>>import sys

>>>print sys.modules

pip

pip is a package management tool for python.

pip is installed by default in many linux distributions.  To verify that pip is installed use the command:

which pip

If it is not installed, you need to manually install pip do the following:

http://www.pip-installer.org/en/latest/installing.html

Some python modules are installed by default by the operating system, but there are many more that can be installed. Not all modules are compatible with other modules, so they are not installed by default.
A user can use pip to install modules into a user's home directory without the need for administrator privileges:

pip install --user <module>

modules will be downloaded, compiled, and installed under ~/.local

To show all installed python modules use:

pip freeze

To search for  python modules available for installation use:

pip search <module>

 

pyenv

pyenv allows for multiple version of python to be installed in a users home directory and they are able to switch from one to another.  If a user needs a version of python that is not installed by default, they can use pyenv to any almost any version they wish in their home directory.  They can then use pip that comes with that version to install the modules they want. Read the documentation for pyenv before using these direction.

Documentation

 

Install pyenv

( also installs easy_install and pip )

 

On teal first do:

ssh teal97

(This will put you on a node which can download the archives needed.)

cd
git clone git://github.com/yyuu/pyenv.git .pyenv

 

Create a file to set the correct environmental variables for pyenv

Create a file called ~/enable_pyenv.sh with the following lines:

 

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

 

If a newer version of gcc is required, add the following line to enable_pyenv.sh:

source /opt/rh/devtoolset-7/enable

 

If a newer version of openssl is required, add the following line to enable_pyenv.sh:

source /usr/global/openssl/enable_openssl.sh

 

 

At this point, on the teal cluster, use qrsh to get a shell on a compute node:

qrsh

 

activate these variables with the source command:

 

source ~/enable_pyenv.sh

 

If you want these to be set with every login, add "source ~/enable_pyenv.sh" to your ~/.bash_profile

 

Verify pyenv is installed:

which pyenv

pyenv --version

Show all available python version that can be installed

If you need a version of python3, its recommended to use the latest version of 3.8 since its likely to be used on future servers. But, not all python modules are available for every version of python, research supported version for a desired module.

pyenv install --list

Output should look like below

$ pyenv install --list Available versions: 2.1.3 2.2.3 2.3.7 2.4.0 2.4.1 2.4.2 2.4.3 2.4.4 2.4.5 2.4.6 2.5.0 2.5.1 2.5.2 2.5.3 2.5.4 2.5.5 2.5.6 2.6.6 2.6.7 2.6.8 2.6.9 2.7.0 2.7-dev 2.7.1 2.7.2 2.7.3 2.7.4 2.7.5 2.7.6 2.7.7 2.7.8 2.7.9 2.7.10 2.7.11 2.7.12 2.7.13 2.7.14 2.7.15 2.7.16 3.0.1 3.1.0 3.1.1 3.1.2 3.1.3 3.1.4 3.1.5 3.2.0 3.2.1 3.2.2 3.2.3 3.2.4 3.2.5 3.2.6 3.3.0 3.3.1 3.3.2 3.3.3 3.3.4 3.3.5 3.3.6 3.3.7 3.4.0 3.4-dev 3.4.1 3.4.2 3.4.3 3.4.4 3.4.5 3.4.6 3.4.7 3.4.8 3.4.9 3.4.10 3.5.0 3.5-dev 3.5.1 3.5.2 3.5.3 3.5.4 3.5.5 3.5.6 3.5.7 3.6.0 3.6-dev 3.6.1 3.6.2 3.6.3 3.6.4 3.6.5 3.6.6 3.6.7 3.6.8 3.6.9 3.7.0 3.7-dev 3.7.1 3.7.2 3.7.3 3.7.4 3.8.0b4 3.8-dev 3.9-dev activepython-2.7.14 activepython-3.5.4 activepython-3.6.0 anaconda-1.4.0 anaconda-1.5.0 anaconda-1.5.1 anaconda-1.6.0 anaconda-1.6.1 anaconda-1.7.0 anaconda-1.8.0 anaconda-1.9.0 anaconda-1.9.1 anaconda-1.9.2 anaconda-2.0.0 anaconda-2.0.1 anaconda-2.1.0 anaconda-2.2.0 anaconda-2.3.0 anaconda-2.4.0 anaconda-4.0.0 anaconda2-2.4.0 anaconda2-2.4.1 anaconda2-2.5.0 anaconda2-4.0.0 anaconda2-4.1.0 anaconda2-4.1.1 anaconda2-4.2.0 anaconda2-4.3.0 anaconda2-4.3.1 anaconda2-4.4.0 anaconda2-5.0.0 anaconda2-5.0.1 anaconda2-5.1.0 anaconda2-5.2.0 anaconda2-5.3.0 anaconda2-5.3.1 anaconda2-2018.12 anaconda2-2019.03 anaconda2-2019.07 anaconda3-2.0.0 anaconda3-2.0.1 anaconda3-2.1.0 anaconda3-2.2.0 anaconda3-2.3.0 anaconda3-2.4.0 anaconda3-2.4.1 anaconda3-2.5.0 anaconda3-4.0.0 anaconda3-4.1.0 anaconda3-4.1.1 anaconda3-4.2.0 anaconda3-4.3.0 anaconda3-4.3.1 anaconda3-4.4.0 anaconda3-5.0.0 anaconda3-5.0.1 anaconda3-5.1.0 anaconda3-5.2.0 anaconda3-5.3.0 anaconda3-5.3.1 anaconda3-2018.12 anaconda3-2019.03 anaconda3-2019.07 ironpython-dev ironpython-2.7.4 ironpython-2.7.5 ironpython-2.7.6.3 ironpython-2.7.7 jython-dev jython-2.5.0 jython-2.5-dev jython-2.5.1 jython-2.5.2 jython-2.5.3 jython-2.5.4-rc1 jython-2.7.0 jython-2.7.1 micropython-dev micropython-1.9.3 micropython-1.9.4 micropython-1.10 micropython-1.11 miniconda-latest miniconda-2.2.2 miniconda-3.0.0 miniconda-3.0.4 miniconda-3.0.5 miniconda-3.3.0 miniconda-3.4.2 miniconda-3.7.0 miniconda-3.8.3 miniconda-3.9.1 miniconda-3.10.1 miniconda-3.16.0 miniconda-3.18.3 miniconda2-latest miniconda2-3.18.3 miniconda2-3.19.0 miniconda2-4.0.5 miniconda2-4.1.11 miniconda2-4.3.14 miniconda2-4.3.21 miniconda2-4.3.27 miniconda2-4.3.30 miniconda3-latest miniconda3-2.2.2 miniconda3-3.0.0 miniconda3-3.0.4 miniconda3-3.0.5 miniconda3-3.3.0 miniconda3-3.4.2 miniconda3-3.7.0 miniconda3-3.8.3 miniconda3-3.9.1 miniconda3-3.10.1 miniconda3-3.16.0 miniconda3-3.18.3 miniconda3-3.19.0 miniconda3-4.0.5 miniconda3-4.1.11 miniconda3-4.2.12 miniconda3-4.3.11 miniconda3-4.3.14 miniconda3-4.3.21 miniconda3-4.3.27 miniconda3-4.3.30 pypy-c-jit-latest pypy-c-nojit-latest pypy-dev pypy-stm-2.3 pypy-stm-2.5.1 pypy-1.5-src pypy-1.5 pypy-1.6 pypy-1.7 pypy-1.8 pypy-1.9 pypy-2.0-src pypy-2.0 pypy-2.0.1-src pypy-2.0.1 pypy-2.0.2-src pypy-2.0.2 pypy-2.1-src pypy-2.1 pypy-2.2-src pypy-2.2 pypy-2.2.1-src pypy-2.2.1 pypy-2.3-src pypy-2.3 pypy-2.3.1-src pypy-2.3.1 pypy-2.4.0-src pypy-2.4.0 pypy-2.5.0-src pypy-2.5.0 pypy-2.5.1-src pypy-2.5.1 pypy-2.6.0-src pypy-2.6.0 pypy-2.6.1-src pypy-2.6.1 pypy-4.0.0-src pypy-4.0.0 pypy-4.0.1-src pypy-4.0.1 pypy-5.0.0-src pypy-5.0.0 pypy-5.0.1-src pypy-5.0.1 pypy-5.1-src pypy-5.1 pypy-5.1.1-src pypy-5.1.1 pypy-5.3-src pypy-5.3 pypy-5.3.1-src pypy-5.3.1 pypy-5.4-src pypy-5.4 pypy-5.4.1-src pypy-5.4.1 pypy-5.6.0-src pypy-5.6.0 pypy-5.7.0-src pypy-5.7.0 pypy-5.7.1-src pypy-5.7.1 pypy2-5.3-src pypy2-5.3 pypy2-5.3.1-src pypy2-5.3.1 pypy2-5.4-src pypy2-5.4 pypy2-5.4.1-src pypy2-5.4.1 pypy2-5.6.0-src pypy2-5.6.0 pypy2-5.7.0-src pypy2-5.7.0 pypy2-5.7.1-src pypy2-5.7.1 pypy2.7-5.8.0-src pypy2.7-5.8.0 pypy2.7-5.9.0-src pypy2.7-5.9.0 pypy2.7-5.10.0-src pypy2.7-5.10.0 pypy2.7-6.0.0-src pypy2.7-6.0.0 pypy2.7-7.0.0-src pypy2.7-7.0.0 pypy2.7-7.1.0-src pypy2.7-7.1.0 pypy2.7-7.1.1-src pypy2.7-7.1.1 pypy3-dev pypy3-2.3.1-src pypy3-2.3.1 pypy3-2.4.0-src pypy3-2.4.0 pypy3.3-5.2-alpha1-src pypy3.3-5.2-alpha1 pypy3.3-5.5-alpha-src pypy3.3-5.5-alpha pypy3.5-c-jit-latest pypy3.5-5.7-beta-src pypy3.5-5.7-beta pypy3.5-5.7.1-beta-src pypy3.5-5.7.1-beta pypy3.5-5.8.0-src pypy3.5-5.8.0 pypy3.5-5.9.0-src pypy3.5-5.9.0 pypy3.5-5.10.0-src pypy3.5-5.10.0 pypy3.5-5.10.1-src pypy3.5-5.10.1 pypy3.5-6.0.0-src pypy3.5-6.0.0 pypy3.5-7.0.0-src pypy3.5-7.0.0 pypy3.6-7.0.0-src pypy3.6-7.0.0 pypy3.6-7.1.0-src pypy3.6-7.1.0 pypy3.6-7.1.1-src pypy3.6-7.1.1 pyston-0.5.1 pyston-0.6.0 pyston-0.6.1 stackless-dev stackless-2.7-dev stackless-2.7.2 stackless-2.7.3 stackless-2.7.4 stackless-2.7.5 stackless-2.7.6 stackless-2.7.7 stackless-2.7.8 stackless-2.7.9 stackless-2.7.10 stackless-2.7.11 stackless-2.7.12 stackless-2.7.14 stackless-3.2.2 stackless-3.2.5 stackless-3.3.5 stackless-3.3.7 stackless-3.4-dev stackless-3.4.1 stackless-3.4.2 stackless-3.4.7 stackless-3.5.4

 

There are many versions of python supported by pyenv. Most users want the latest version of 2.7.x or 3.8.x

Note: Python 2.7 reached the end of its life on January 1st, 2020, though some packages still use it. Avoid if possible!

Install Python 2.7.16

pyenv install 2.7.16 -v 2>&1 | tee pyenv-install-2.7.16.log

 

The install takes several minutes to complete after the "Installing ..." message appears.

Installing openssl 1.0.2

This is necessary for Centos 6 systems in order to install python 3.7 or later. This has already been done for teal and is found at /usr/global/openssl. You should not need to do this unless you need a different version of openssl for your project.

On teal first do:

ssh teal97

(This will put you on a node which can download the archives needed.)

 

mkdir -p src/openssl-1.0.2t
cd src/openssl-1.0.2t

wget https://www.openssl.org/source/openssl-1.0.2t.tar.gz -O openssl-1.0.2t.tar.gz

tar -zxf openssl-1.0.2t.tar.gz

cd openssl-1.0.2t

./config --prefix=/usr/global/openssl-1.0.2t --openssldir=/usr/global/openssl-1.0.2t shared zlib 2>&1 | tee config-$(date +%F-%H%M).log

make 2>&1 | tee make-$(date +%F-%H%M).log

make test 2>&1 | tee make-test-$(date +%F-%H%M).log

make install 2>&1 | tee make-install-$(date +%F-%H%M).log

Install Python 3.8.5

On teal first do:

qrsh

 

On servers with Centos 6 (like teal) an updated version of openssl must be used with python 3.8.x, do the following to install python 3.8.x with pyenv:

 

source ~/enable_pyenv.sh

source /usr/global/openssl/enable_openssl.sh

CONFIGURE_OPTS="--with-openssl=/usr/global/openssl/" pyenv install 3.8.5 -v 2>&1 | tee pyenv-install-3.8.5.log

 

The install takes several minutes to complete after the "Installing ..." message appears.

 

Show current python

which python
pyenv versions

python -V

 

Switch to different version of python and show it

pyenv local 2.7.16

python -V

 

Note: when using pip from within the pyenv environment, there is no need to use the --user switch, because you have access to default python directories for version you have installed.

 

When trying to install modules with pip, if the following error comes up:

 

$ pip install numpy
Collecting numpy
/home/username/.pyenv/versions/2.7.14/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
  Could not find a version that satisfies the requirement numpy (from versions: )
No matching distribution found for numpy 

 

You'll need to first install updated version of openssl using the following command:

 

$ pip install --trusted-host pypi.python.org pyOpenSSL ndg-httpsclient pyasn1
Collecting pyOpenSSL
/home/cgchildr/.pyenv/versions/2.7.14/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
  Downloading pyOpenSSL-17.5.0-py2.py3-none-any.whl (53kB)
    100% |████████████████████████████████| 61kB 6.0MB/s
Collecting ndg-httpsclient
  Downloading ndg_httpsclient-0.4.4-py2-none-any.whl
Collecting pyasn1
  Downloading pyasn1-0.4.2-py2.py3-none-any.whl (71kB)
    100% |████████████████████████████████| 71kB 17.8MB/s
Collecting six>=1.5.2 (from pyOpenSSL)
  Downloading six-1.11.0-py2.py3-none-any.whl
Collecting cryptography>=2.1.4 (from pyOpenSSL)
  Downloading cryptography-2.2.2-cp27-cp27mu-manylinux1_x86_64.whl (2.2MB)
    100% |████████████████████████████████| 2.2MB 18.8MB/s
Collecting cffi>=1.7; platform_python_implementation != "PyPy" (from cryptography>=2.1.4->pyOpenSSL)
  Downloading cffi-1.11.5-cp27-cp27mu-manylinux1_x86_64.whl (407kB)
    100% |████████████████████████████████| 409kB 16.3MB/s
Collecting enum34; python_version < "3" (from cryptography>=2.1.4->pyOpenSSL)
  Downloading enum34-1.1.6-py2-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography>=2.1.4->pyOpenSSL)
  Downloading asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
    100% |████████████████████████████████| 102kB 16.9MB/s
Collecting idna>=2.1 (from cryptography>=2.1.4->pyOpenSSL)
  Downloading idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 14.8MB/s
Collecting ipaddress; python_version < "3" (from cryptography>=2.1.4->pyOpenSSL)
  Downloading ipaddress-1.0.19.tar.gz
Collecting pycparser (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=2.1.4->pyOpenSSL)
  Downloading pycparser-2.18.tar.gz (245kB)
    100% |████████████████████████████████| 256kB 18.0MB/s
Installing collected packages: six, pycparser, cffi, enum34, asn1crypto, idna, ipaddress, cryptography, pyOpenSSL, pyasn1, ndg-httpsclient
  Running setup.py install for pycparser ... done
  Running setup.py install for ipaddress ... done
Successfully installed asn1crypto-0.24.0 cffi-1.11.5 cryptography-2.2.2 enum34-1.1.6 idna-2.6 ipaddress-1.0.19 ndg-httpsclient-0.4.4 pyOpenSSL-17.5.0 pyasn1-0.4.2 pycparser-2.18 six-1.11.0 

 

After that, python modules with install normally:

 

$ pip install numpy
Collecting numpy
/home/cgchildr/.pyenv/versions/2.7.14/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
  Downloading numpy-1.14.2-cp27-cp27mu-manylinux1_x86_64.whl (12.1MB)
    100% |████████████████████████████████| 12.1MB 66kB/s
Installing collected packages: numpy
Successfully installed numpy-1.14.2 

 

 

Remove pyenv

 

remove (or move/rename) the following

~/.pyenv

~/.python-version

~/enable_pyenv.sh

 

pipenv

 

researching...

(Thanks to Carlisle Childress for the documentation upon which this was based. Any errrors were added by Brad.)