Saving Python pip dependencies

Recently I learned about the Python tool html5validator tool and used it to run HTML5 validation on local HTML files. You can read more about that in the Offline HTML5 Validator post.

But then I wondered .. “Wouldn’t it be awesome if I run this validation step each time after Hugo generates this site on Netlify, but before it gets deployed?”. That curiosity led me to Netlify’s documentation on Python dependencies, and I learned that Netlify will run pip install to install the dependencies in requirements.txt present in the repository’s base directory.

I followed the pip freeze > requirements.txt step in that documentation but that ended up listing all my pip installed packages in that file! I needed the requirements.txt to include the dependencies only for html5validator. This post was born in my quest to achieve that.

The solution to this problem was to first create a Python virtual environment in my site directory, and then do all the pip operations in there. I learned about this virtual environment step from this Python Pandemonium post.

The venv documentation has a lot of details — I’ll just list the key steps in this post.

1 Create a virtual environment #

cd to your site directory and run the below command create a virtualenv directory named pyenv in there.

python3 -m venv pyenv

2 Activate the virtual environment #

The virtualenv directory pyenv/bin/ will have multiple flavors of shell scripts to activate your virtualenv. I am using activate.csh here as my shell is tcsh 🙄.

source pyenv/bin/activate.csh

Just to emphasize, it is important to use the source command here, and not just call the shell script directly.

Once you activate this virtualenv, you should see something like [pyenv] in the shell prompt.

3 Install the packages using pip #

Install all the project-specific packages. In this case it was just this:

pip install html5validator

The package (and its dependencies) will be installed in pyenv/lib/python3.7/site-packages/. Here, the python3.7/ sub-directory name will match the version of Python you have installed.

4 Generate requirements.txt #

Finally, we run the pip freeze command mentioned in Netlify docs, but with the --local switch:

pip freeze --local > requirements.txt

Here’s the requirements.txt created by that command:

html5validator==0.4.2
PyYAML==6.0

Now, I could have just manually typed html5validator==0.4.2 in a requirements.txt and committed that, and that would work too. But then, I wouldn’t have learned how to create a project-specific pip dependency file 😄.

Netlify deployment #

The html5validator has a dependency on Java 8. Luckily the Netlify environment already comes with that installed. So the only extra setup needed to make this package work on Netlify was to set the PYTHON_VERSION environment variable to 3.8  The version number should be one of the values listed in Netlify’s included software list. .

Summary #

The numbered headings in this post already summarize the steps needed to create a project-specific requirements.txt.

With these in place:

my Netlify deployment script now does HTML5 Validation along with few other checks before this site gets deployed 🎉.

Here’s the full build.sh script.