Python on the Teal Cluster
- 1 Learning Python
- 2 pyenv
- 2.1.1 Documentation
- 2.1.2 Install pyenv
- 2.1.3 Verify pyenv is installed:
- 2.1.4 Show all available python version that can be installed
- 2.1.5 Install Python 2.7.16
- 2.1.6 Installing openssl 1.0.2
- 2.1.7 Install Python 3.8.5
- 2.1.8 Show current python
- 2.1.9
- 2.1.10 Switch to different version of python and show it
- 2.2 Remove pyenv
- 3 pipenv
Learning Python
http://learnpythonthehardway.org/book/
http://www.codecademy.com/en/tracks/python
Python User Group - Richmond, VA - https://www.meetup.com/PyRVAUserGroup/
Showing all available python modules
Three different ways of showing all available python modules
1] pip freeze
2] >>>help("modules")
3]>>>import sys
>>>print sys.modules
http://blog.revathskumar.com/2011/10/python-list-all-packages-installed.html
http://stackoverflow.com/questions/2550667/python-list-of-modules-helpmodules-not-working
pip
pip is a package management tool for python.
pip is installed by default in many linux distributions. To verify that pip is installed use the command:
which pip
If it is not installed, you need to manually install pip do the following:
http://www.pip-installer.org/en/latest/installing.html
Some python modules are installed by default by the operating system, but there are many more that can be installed. Not all modules are compatible with other modules, so they are not installed by default.
A user can use pip to install modules into a user's home directory without the need for administrator privileges:
pip install --user <module>
modules will be downloaded, compiled, and installed under ~/.local
To show all installed python modules use:
pip freeze
To search for python modules available for installation use:
pip search <module>
pyenv
pyenv allows for multiple version of python to be installed in a users home directory and they are able to switch from one to another. If a user needs a version of python that is not installed by default, they can use pyenv to any almost any version they wish in their home directory. They can then use pip that comes with that version to install the modules they want. Read the documentation for pyenv before using these direction.
Documentation
Install pyenv
( also installs easy_install and pip )
On teal first do:
ssh teal97
(This will put you on a node which can download the archives needed.)
cdgit clone git://github.com/yyuu/pyenv.git .pyenv
Create a file to set the correct environmental variables for pyenv
Create a file called ~/enable_pyenv.sh with the following lines:
export PYENV_ROOT="$HOME/.pyenv"export PATH="$PYENV_ROOT/bin:$PATH"eval "$(pyenv init -)"
If a newer version of gcc is required, add the following line to enable_pyenv.sh:
source /opt/rh/devtoolset-7/enable
If a newer version of openssl is required, add the following line to enable_pyenv.sh:
source /usr/global/openssl/enable_openssl.sh
At this point, on the teal cluster, use qrsh to get a shell on a compute node:
qrsh
activate these variables with the source command:
source ~/enable_pyenv.sh
If you want these to be set with every login, add "source ~/enable_pyenv.sh" to your ~/.bash_profile
Verify pyenv is installed:
which pyenv
pyenv --version
Show all available python version that can be installed
If you need a version of python3, its recommended to use the latest version of 3.8 since its likely to be used on future servers. But, not all python modules are available for every version of python, research supported version for a desired module.
pyenv install --list Output should look like below
There are many versions of python supported by pyenv. Most users want the latest version of 2.7.x or 3.8.x
Note: Python 2.7 reached the end of its life on January 1st, 2020, though some packages still use it. Avoid if possible!
Install Python 2.7.16
pyenv install 2.7.16 -v 2>&1 | tee pyenv-install-2.7.16.log
The install takes several minutes to complete after the "Installing ..." message appears.
Installing openssl 1.0.2
This is necessary for Centos 6 systems in order to install python 3.7 or later. This has already been done for teal and is found at /usr/global/openssl. You should not need to do this unless you need a different version of openssl for your project.
On teal first do:
ssh teal97
(This will put you on a node which can download the archives needed.)
mkdir -p src/openssl-1.0.2tcd src/openssl-1.0.2t
wget https://www.openssl.org/source/openssl-1.0.2t.tar.gz -O openssl-1.0.2t.tar.gz
tar -zxf openssl-1.0.2t.tar.gz
cd openssl-1.0.2t
./config --prefix=/usr/global/openssl-1.0.2t --openssldir=/usr/global/openssl-1.0.2t shared zlib 2>&1 | tee config-$(date +%F-%H%M).log
make 2>&1 | tee make-$(date +%F-%H%M).log
make test 2>&1 | tee make-test-$(date +%F-%H%M).log
make install 2>&1 | tee make-install-$(date +%F-%H%M).log
Install Python 3.8.5
On teal first do:
qrsh
On servers with Centos 6 (like teal) an updated version of openssl must be used with python 3.8.x, do the following to install python 3.8.x with pyenv:
source ~/enable_pyenv.sh
source /usr/global/openssl/enable_openssl.sh
CONFIGURE_OPTS="--with-openssl=/usr/global/openssl/" pyenv install 3.8.5 -v 2>&1 | tee pyenv-install-3.8.5.log
The install takes several minutes to complete after the "Installing ..." message appears.
Show current python
which pythonpyenv versions
python -V
Switch to different version of python and show it
pyenv local 2.7.16
python -V
Note: when using pip from within the pyenv environment, there is no need to use the --user switch, because you have access to default python directories for version you have installed.
When trying to install modules with pip, if the following error comes up:
$ pip install numpy Collecting numpy /home/username/.pyenv/versions/2.7.14/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning. SNIMissingWarning Could not find a version that satisfies the requirement numpy (from versions: ) No matching distribution found for numpy
You'll need to first install updated version of openssl using the following command:
$ pip install --trusted-host pypi.python.org pyOpenSSL ndg-httpsclient pyasn1 Collecting pyOpenSSL /home/cgchildr/.pyenv/versions/2.7.14/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning. SNIMissingWarning Downloading pyOpenSSL-17.5.0-py2.py3-none-any.whl (53kB) 100% |████████████████████████████████| 61kB 6.0MB/s Collecting ndg-httpsclient Downloading ndg_httpsclient-0.4.4-py2-none-any.whl Collecting pyasn1 Downloading pyasn1-0.4.2-py2.py3-none-any.whl (71kB) 100% |████████████████████████████████| 71kB 17.8MB/s Collecting six>=1.5.2 (from pyOpenSSL) Downloading six-1.11.0-py2.py3-none-any.whl Collecting cryptography>=2.1.4 (from pyOpenSSL) Downloading cryptography-2.2.2-cp27-cp27mu-manylinux1_x86_64.whl (2.2MB) 100% |████████████████████████████████| 2.2MB 18.8MB/s Collecting cffi>=1.7; platform_python_implementation != "PyPy" (from cryptography>=2.1.4->pyOpenSSL) Downloading cffi-1.11.5-cp27-cp27mu-manylinux1_x86_64.whl (407kB) 100% |████████████████████████████████| 409kB 16.3MB/s Collecting enum34; python_version < "3" (from cryptography>=2.1.4->pyOpenSSL) Downloading enum34-1.1.6-py2-none-any.whl Collecting asn1crypto>=0.21.0 (from cryptography>=2.1.4->pyOpenSSL) Downloading asn1crypto-0.24.0-py2.py3-none-any.whl (101kB) 100% |████████████████████████████████| 102kB 16.9MB/s Collecting idna>=2.1 (from cryptography>=2.1.4->pyOpenSSL) Downloading idna-2.6-py2.py3-none-any.whl (56kB) 100% |████████████████████████████████| 61kB 14.8MB/s Collecting ipaddress; python_version < "3" (from cryptography>=2.1.4->pyOpenSSL) Downloading ipaddress-1.0.19.tar.gz Collecting pycparser (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=2.1.4->pyOpenSSL) Downloading pycparser-2.18.tar.gz (245kB) 100% |████████████████████████████████| 256kB 18.0MB/s Installing collected packages: six, pycparser, cffi, enum34, asn1crypto, idna, ipaddress, cryptography, pyOpenSSL, pyasn1, ndg-httpsclient Running setup.py install for pycparser ... done Running setup.py install for ipaddress ... done Successfully installed asn1crypto-0.24.0 cffi-1.11.5 cryptography-2.2.2 enum34-1.1.6 idna-2.6 ipaddress-1.0.19 ndg-httpsclient-0.4.4 pyOpenSSL-17.5.0 pyasn1-0.4.2 pycparser-2.18 six-1.11.0
After that, python modules with install normally:
$ pip install numpy Collecting numpy /home/cgchildr/.pyenv/versions/2.7.14/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning. SNIMissingWarning Downloading numpy-1.14.2-cp27-cp27mu-manylinux1_x86_64.whl (12.1MB) 100% |████████████████████████████████| 12.1MB 66kB/s Installing collected packages: numpy Successfully installed numpy-1.14.2
Remove pyenv
remove (or move/rename) the following
~/.pyenv
~/.python-version
~/enable_pyenv.sh
pipenv
researching...
(Thanks to Carlisle Childress for the documentation upon which this was based. Any errrors were added by Brad.)