Write at 2019 Jul 02 in notes Python pyenv

I’m not a Python programmer but due to job constraint I end up writing Python code. Being used to Ruby’s RVM, I struggle to get something similar for Python.

This is a process that works for me and I hope it might work for you. The primary tool I used to manage different Python version is pyenv .

Install pyenv

An easy step.

brew install pyenv

Install dependencies

pyenv will compile Python from sources. Thus we need many library. On Linux, these are usually in lib* or *devel package. On Mac, we use homebrew for everything.

brew install zlib openssl

Install a python version

Now that you have pyenv , you can start to install Python for real. Not so quick. If you are on Linux, lots of library is in standard place where python compiler expect. But if you are on Mac, we usually have 2 places:

Xcode command line tools

Homebrew

Homebrew has lot of stuff such as above zlib . Therefore we need to set a few extra environment var to point to these.

# Setup env export FLAGS="${FLAGS} -I$(brew --prefix zlib)/include -I$(brew --prefix readline)/include -I$(brew --prefix openssl)/include -I$(xcrun --show-sdk-path)/usr/include" export LDFLAGS="${LDFLAGS} -L$(brew --prefix zlib)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix openssl)/lib" export CPPFLAGS="${CPPFLAGS} -I$(brew --prefix zlib)/include -I$(brew --prefix readline)/include -I$(brew --prefix openssl)/include -I$(xcrun --show-sdk-path)/usr/include" export PYTHON_CONFIGURE_OPTS=--enable-unicode=ucs2 export PKG_CONFIG_PATH="${PKG_CONFIG_PATH} /usr/local/opt/zlib/lib/pkgconfig"

Now you can compile and install any Python version:

# See what is avaibale pyenv install --list # Install specificed version pyenv install -v 3.7.3

Setup PATH

As always, in order to override default system Python, we prepend our PATH with path to our custom Python.

This is usually done in shell rc file

bash: ~/.bashrc

zsh: ~/.zshrc

Add this block to the end:

export PYENV_ROOT="$HOME/.pyenv" export PATH="$PYENV_ROOT/bin:$PATH" if command -v pyenv 1>/dev/null 2>&1; then eval "$(pyenv init -)" fi

Using pyenv’s Python version

In a given directory, run this

pyenv local 3.7.3

This will also create a file call .python-version in your current directory. Later on, when you cd into this directory again, the right Python version will automatically load for you.

Example

# default system python cuz no `.python-version` file is found ➜ cd ~ ➜ python -V Python 2.7.15 # When I move into this directory, python version changed ➜ cd ~/src/axcoto.com ➜ python -V Python 3.7.3

Install requirements.txt with native dependencies

Similarly to pyenv install , if any packages need native library you will need to set those LDFLAGS , CPPFLAGS before run pip install

Example, if you need geoip package, you will have to do:

brew install libGeoIp export LDFLAGS="${LDFLAGS} -L/usr/local/lib" export CFLAGS="${CFLAGS} -I/usr/local/include"

How do you know where these lib are locate? Most of time they are in /usr/local/lib and /usr/local/include

ls /usr/local/lib/ | grep libGeo libGeoIP.1.dylib libGeoIP.a libGeoIP.dylib

Libraries that don’t located in /usr/local/lib means they aren’t linked by homebrew so they usually in /usr/local/lib/[name] . When you run brew install , it will tell you that the package isn’t linked and the path to it etc. Just pay attention to the brew install output and you should be good.

How does pyenv work

During install, pyenv install python versions into ~/.pyenv/versions

ls -la ~/.pyenv/versions total 0 drwxr-xr-x 4 vinh staff 128 Jul 7 12:57 . drwxr-xr-x 4 vinh staff 128 Jul 7 00:38 .. drwxr-xr-x 6 vinh staff 192 Jul 7 13:00 3.6.7 drwxr-xr-x 6 vinh staff 192 Jul 7 00:38 3.7.3

the script we add to our shell rc file prepend pyenv path to a file call python .

➜ echo $PATH /Users/vinh/.pyenv/shims:/Users/vinh/.pyenv/bin:/Users/vinh/.cargo/bin:/Users/vinh/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Users/vinh/.rvm/bin:/Users/vinh/bin:/Users/vinh/go/bin: ➜ which python /Users/vinh/.pyenv/shims/python

This file isn’t the Python executable but a shell script that try to load and use exec to replace the shell process with the right Python version.

➜ cat /Users/vinh/.pyenv/shims/python #!/usr/bin/env bash set -e [ -n "$PYENV_DEBUG" ] && set -x program="${0##*/}" if [[ "$program" = "python"* ]]; then for arg; do case "$arg" in -c* | -- ) break ;; */* ) if [ -f "$arg" ]; then export PYENV_FILE_ARG="$arg" break fi ;; esac done fi export PYENV_ROOT="/Users/vinh/.pyenv" exec "/usr/local/Cellar/pyenv/1.2.11/libexec/pyenv" exec "$program" "[email protected]"