[PYTHON] Accelerate CI by skipping unnecessary pip install

Thing you want to do

Travis CI testing is slow and I want to speed it up. Looking at the log, pip install is taking a long time. Especially numpy installation is slow. Also, I want to skip the fact that the installed package is processed every time because it is specified in the cache directory.

environment

• I'm using Travis CI
• python 2.7.9 (currently)
• .pip-cache, virtualenv directory is cached

.travis.yml

python:
- 2.7.9
cache:
  apt: true
  directories:
  - "$HOME/.pip-cache/"
  - "$HOME/virtualenv/python2.7.9"
before_install:
- sudo apt-get update
- sudo apt-get install XXX #Required package installation
install:
- pip install -r requirements.txt --download-cache $HOME/.pip-cache
#Omitted below

What i did

Change the install item so that the installation process runs only the difference compared to pip freeze

.travis.yml

install:
- pip freeze | comm -13 - requirements.txt | grep -v '^#' | xargs pip install --download-cache $HOME/.pip-cache

• I tried to use the diff command normally, but it worked on Mac, but it didn't work in Travis environment. It seems to be a bash-only method?
  • diff <(pip freeze | sort) <(cat requirements.txt | sort) | grep '^> ' | sed 's/^> //'
• I've never heard of the comm command this time. Convenient.
• Since comment lines may appear with pip freeze, I put `grep -v'^ #`

Task

The line installed by specifying tag from the git repository will always be a difference because the hash value of the commit is displayed in pip freeze.

• requirements.txt
  • -e git://github.com/Example/hoge.git@tag#egg=hoge
• pip freeze
  • -e git://github.com/Example/hoge.git@ba34106b63048ba3d43140a09bd4f99140609da6#egg=hoge-origin/tag

Well, is it okay because the slowest numpy was skipped? I think.