validator on A Scripter's Notes

Saving Python pip dependencies

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Tue, 14 Jun 2022 00:38:00 -0400

How I generated the Python dependencies file requirements.txt so that Netlify can install and run the HTML5 Validator before deploying this site.

Table of Contents

1 Create a virtual environment
2 Activate the virtual environment
3 Install the packages using pip
4 Generate requirements.txt
Netlify deployment
Summary

Recently I learned about the Python tool html5validator tool and used it to run HTML5 validation on local HTML files. You can read more about that in the Offline HTML5 Validator post.

But then I wondered .. “Wouldn’t it be awesome if I run this validation step each time after Hugo generates this site on Netlify, but before it gets deployed?”. That curiosity led me to Netlify’s documentation on Python dependencies, and I learned that Netlify will run pip install to install the dependencies in requirements.txt present in the repository’s base directory.

I followed the pip freeze > requirements.txt step in that documentation but that ended up listing all my pip installed packages in that file! I needed the requirements.txt to include the dependencies only for html5validator. This post was born in my quest to achieve that.

The solution to this problem was to first create a Python virtual environment in my site directory, and then do all the pip operations in there. I learned about this virtual environment step from this Python Pandemonium post.

The venv documentation has a lot of details — I’ll just list the key steps in this post.

1 Create a virtual environment #

cd to your site directory and run the below command create a virtualenv directory named pyenv in there.

python3 -m venv pyenv

As we are creating this virtual environment just for the sake of creating a requirements.txt, we don’t need to commit this directory to git.

2 Activate the virtual environment #

The virtualenv directory pyenv/bin/ will have multiple flavors of shell scripts to activate your virtualenv. I am using activate.csh here as my shell is tcsh 🙄.

source pyenv/bin/activate.csh

Just to emphasize, it is important to use the source command here, and not just call the shell script directly.

Once you activate this virtualenv, you should see something like [pyenv] in the shell prompt.

3 Install the packages using `pip` #

Install all the project-specific packages. In this case it was just this:

pip install html5validator

The package (and its dependencies) will be installed in pyenv/lib/python3.7/site-packages/. Here, the python3.7/ sub-directory name will match the version of Python you have installed.

4 Generate `requirements.txt` #

Finally, we run the pip freeze command mentioned in Netlify docs, but with the --local switch:

pip freeze --local > requirements.txt

The --local option ensures that globally-installed packages are not listed even if the virtualenv has global access.

Here’s the requirements.txt created by that command:

html5validator==0.4.2
PyYAML==6.0

Code Snippet 1: The requirements.txt for Netlify

Now, I could have just manually typed html5validator==0.4.2 in a requirements.txt and committed that, and that would work too. But then, I wouldn’t have learned how to create a project-specific pip dependency file 😄.

Netlify deployment #

The html5validator has a dependency on Java 8. Luckily the Netlify environment already comes with that installed. So the only extra setup needed to make this package work on Netlify was to set the PYTHON_VERSION environment variable to 3.8 The version number should be one of the values listed in Netlify’s included software list. .

Summary #

The numbered headings in this post already summarize the steps needed to create a project-specific requirements.txt.

With these in place:

✅ requirements.txt committed in site directory root
✅ Netlify environment variable PYTHON_VERSION set to 3.8

my Netlify deployment script now does HTML5 Validation along with few other checks before this site gets deployed 🎉.

Here’s the full build.sh script.

Zero HTML Validation Errors!

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Sun, 05 Jun 2022 17:58:00 -0400

How I fixed my site content and went down from 46 HTML validations errors down to 0!

Table of Contents

1 Avoid duplicate heading id attributes
2 Remove inline <style> elements
3 Ensure that all images have captions or alt attributes
4 Do not have hyperlinks in headings
Validation Ignores
- 5 Ignore errors due to hyperlinks in inline SVG
- 6 Ignore files not expected to serve HTML content
Summary

In my previous HTML5 Validator post, I mentioned:

I was a bit disappointed to see validation errors on my site, but then it wasn’t too bad .. 46 errors.

But they truly say ..

Ignorance is bliss.

So .. once I realized that my site had 46 validation errors, I lost that bliss .. and I couldn’t rest easy — I had to fix them all 😁.

This post summarizes the categories of those errors and how I fixed The fixes mentioned in this post refer to changes in Org mode content. But you should be able to derive equivalent fixes for Markdown too. them all.

1 Avoid duplicate heading `id` attributes #

"file:/public/notes/string-fns-nim-vs-python/index.html":636.34-636.46: error: Duplicate ID "notes".
"file:/public/notes/nim-fmt/index.html":300.34-300.52: error: Duplicate ID "older-issue".
"file:/public/notes/nim-fmt/index.html":306.20-306.33: error: Duplicate ID "floats".
"file:/public/page/6/index.html":29.39-29.54: error: Duplicate ID "fnref:1".

Errors with above kind of signatures were fixed by,

Converting headings to description lists

I had a bunch of generic headings like “Notes” and “Older Issue” in some of my posts. After taking a second look at those, it made more sense to convert those to description lists. So in Org mode, I converted headings like * Notes to description lists - Notes ::.
Setting CUSTOM_ID heading property

For the cases, where the headings needed to be left as so, their IDs were uniquified by setting their CUSTOM_ID property. For example, below fixed the Duplicate ID “floats” errors.
```
 ** Precision
 ..
 *** Floats
+:PROPERTIES:
+:CUSTOM_ID: precision-floats
+:END:
 ..
 ** Type (only for numbers)
 ..
 *** Floats
+:PROPERTIES:
+:CUSTOM_ID: type-floats
+:END:
```
Code Snippet 1: Using CUSTOM_ID property to uniquify heading ID's
Prevent footnote links in post summaries

This issue was due to me not being conscious about how the footnote references work in a post versus on a page outside that post’s context. The issue was caused by footnote references getting into the post summaries parsed by Hugo, which will then show up on the list pages.

The fix was simple — Edit the post summaries so that they don’t contain any footnote references.

2 Remove inline `<style>` elements #

"file:/public/grep-po/index.html":51.139-51.145: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/how-do-i-write-org-mode/index.html":23.194-23.200: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)

Errors with above kind of signatures were fixed by,

Avoiding export of raw <style> elements in the Markdown content

I figured out which functions were responsible for injecting <style> elements in Markdown content and then advised them to stop that. After applying these advises, I lost the in-content rules for CSS classes .org-center and .csl-entry. So I put those rules directly in this website’s CSS.

(defun modi/org-blackfriday-center-block (_center-block contents info)
 (let* ((class "org-center"))
 (format "<div class=\"%s\">%s\n\n%s\n</div>"
 class (org-blackfriday--extra-div-hack info) contents)))
(advice-add 'org-blackfriday-center-block :override #'modi/org-blackfriday-center-block)

(defun modi/org-cite-csl-render-bibliography (bib-str)
 (replace-regexp-in-string "<style>\\.csl-entry[^<]+</style>" "" bib-str))
(advice-add 'org-cite-csl-render-bibliography :filter-return #'modi/org-cite-csl-render-bibliography)

Code Snippet 2: Emacs-Lisp advices to prevent <style> elements in exports

Removing unnecessary micro-styling

I found a single case, where an inline CSS rule was defined in content for CSS class .repr-type for a table. I just removed that without affecting the looks of that rendered table too much.

3 Ensure that all images have captions or `alt` attributes #

"file:/public/hugo-use-goat-code-blocks-for-ascii-diagrams/index.html":24.130-24.255: error: An "img" element must have an "alt" attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.

Errors with above kind of signatures were easily fixed by ensuring that all images had captions Thankfully, there were only two images that were missing captions. . The Hugo figure shortcode adds the caption to the alt attribute if the alt is not specified separately.

As an example, here’s how I fixed the above error:

+ #+name: fig__disproportionate_box_drawing
+ #+caption: Disproportionate box drawing characters
[[file:images/hugo-use-goat-code-blocks-for-ascii-diagrams/ascii-diagram-rendered-in-plain-text-code-block.png]]

Code Snippet 3: A git diff showing addition of caption to an image

4 Do not have hyperlinks in headings #

"file:/public/using-emacs-advice-to-silence-messages-from-functions/index.html":151.687-151.732: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/using-emacs-advice-to-silence-messages-from-functions/index.html":151.748-151.751: error: Stray end tag "a".
"file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: End tag "a" violates nesting rules.
"file:/public/auto-count-100daystooffload-posts/index.html":114.526-114.529: error: Stray end tag "a".

While the hyperlinks in headings work well, they created invalid HTML in the Hugo-generated TOC. So while these errors were created technically because of a bug in Hugo It’s really cool when you end up finding a bug in an upstream project while trying to fix the errors in your own thing 😎. , I wanted to fix these errors on my end as soon as I can.

I reviewed the errors, and this is all it took to get rid of them all:

Remove manually inserting hyperlinks in headings

 show two methods of finding sources of any printed messages.
-***** Using plain-old /grep/ or [[https://github.com/BurntSushi/ripgrep][/rg/]]
+***** Using plain-old /grep/ or /ripgrep (rg)/
 This method is pretty easy (but not robust) to use if the search
 ..
 Org source directory and search for the /"org-babel-exp process .."/
-string ..
+string using [[https://github.com/BurntSushi/ripgrep][~rg~]] ..

Code Snippet 4: Removing hyperlink from a heading

Remove Org Radio links that created links in headings

Here, a Org heading happened to contain the string “Day count”, which was also an an Org Radio link in that post. While ideally that shouldn’t have mattered, I removed that radio link to get around this Hugo bug.
```
 .. /Just may be/. But regardless, I am already enjoying writing once
-again, and it's great to see the <<<Day count>>> (counting up to 100)
+again, and it's great to see the Day count (counting up to 100)
 increase with each new post!
```
Code Snippet 5: Removing an Org Radio link

Validation Ignores #

Above fixes fixed 43 out of 46 errors, but the remaining 3 were unfixable.

5 Ignore errors due to hyperlinks in inline SVG #

"file:/public/notes/plantuml/index.html":114.474-114.678: error: Attribute "title" not allowed on element "a" at this point.

This error was caused by hyperlinks in inline SVG elements. These SVG elements are created by PlantUML. The hyperlinks in SVG feature works great, and as these are generated by PlantUML, I chose to just ignore these errors.

I ignored this error by adding the --ignore-re 'notes/plantuml.*Attribute.*title.*not allowed' switch to the html5validator command.

6 Ignore files not expected to serve HTML content #

"file:/public/googleFOO.html":1.1-1.52: error: Non-space characters found without seeing a doctype first. Expected "<!DOCTYPE html>".

The googleFOO.html file here is not a valid HTML file. It’s a just a cookie file that was used by Google to verify that I own this domain.

This error was masked by adding the --ignore 'googleFOO' switch to the html5validator command.

Summary #

Once I fixed the 43 errors by tweaking the Org mode content, and added those two ignores, I had zero validation errors! 🎉

If you are interested in the fix details, here are the commits.

Offline HTML5 Validator

Kaushal.Modi@fakeEmailToMakeValidatorHappy.com (Kaushal Modi) — Wed, 01 Jun 2022 00:11:00 -0400

Validate your website offline — It’s just one curl command away.

Table of Contents

Using the .jar
- Using pip install
Using pre-compiled binary
Results
Conclusion

I have been using the online HTML5 Validator https://html5.validator.nu/ for few years now. I have a link at the bottom of each post to validate that page’s HTML. For an example, see the html5 validator link at the bottom of this post.

But it didn’t occur to me until now to look for a way to do the same validation offline! Offline validation would be useful so that I can look at any HTML generation problem before I deploy the website. So I looked for a solution online, and of course it’s answered on StackOverflow 😄.

It turns out that same Nu HTML5 Validator project that provides the online validation service, also provides a Java application as well as pre-compiled binaries for Linux, Windows and MacOS for offline use!

To use the Java .jar file, you need to have at least Java 8 installed on your system. But you don’t need to have any version of Java installed if you use the pre-compiled binary instead. See its documentation for more details.

Using the `.jar` #

Requires at least Java 8

Download

Download the latest vnu.jar from the project’s GitHub Releases section.

curl -ROLs https://github.com/validator/validator/releases/download/latest/vnu.jar

Run

Below command runs the validator only on the HTML files in the public/ If you are using Hugo, the hugo command will publish the HTML files in the public/ directory by default. directory. See its Usage documentation for more details.

java -jar vnu.jar --skip-non-html --errors-only public/

For my usecase, if I don’t provide the --skip-non-html --errors-only switches, the output is too noisy.

Using `pip install` #

If you do not want to manually download the .jar, there’s a Python wrapper available to do the same for you: html5validator.

Install

 pip install --user html5validator

Run

 html5validator --root public/

It seems like this Python wrapper implicitly passes --skip-non-html --errors-only to the Java app. So those are not needed when running html5validator. But on the flip side, it needs the --root switch when specifying the directory to run the script on.

Note that you still need to have at least Java 8 installed when running this Python app too, because it downloads and run the same .jar behind the scenes.

Using pre-compiled binary #

If your system doesn’t have the required Java version, you can use the pre-compiled binary instead.

Download & Extract

Download and extract the vnu.<OS>.zip for your OS from the same Releases section. Here, I am showing how to do that on Linux:

curl -ROLs https://github.com/validator/validator/releases/download/latest/vnu.linux.zip
unzip vnu.linux.zip

The extracted binary path will be vnu-runtime-image/bin/vnu.

Run

The run options will be the exact same; just that you will be running the binary directly instead of running through java.

vnu-runtime-image/bin/vnu --skip-non-html --errors-only public/

Results #

<2022-06-05>: This website now has zero validation errors! 🎉 All the errors listed in the collapsed log below are now resolved. See my Zero HTML Validation Errors! post on how I did that.

I was a bit disappointed to see validation errors on my site, but then it wasn’t too bad .. 52 46 errors:

Some I already fixed

These 6 errors were fixed in this commit.

"file:/public/getting-started-with-texlive/index.html":6.2198-6.2206: error: Element "package" not allowed as child of element "li" in this context. (Suppressing further errors from this subtree.)
"file:/public/getting-started-with-texlive/index.html":6.2256-6.2259: error: End tag "li" implied, but there were open elements.
"file:/public/getting-started-with-texlive/index.html":6.2198-6.2206: error: Unclosed element "package".
"file:/public/getting-started-with-texlive/index.html":6.2307-6.2315: error: Element "package" not allowed as child of element "li" in this context. (Suppressing further errors from this subtree.)
"file:/public/getting-started-with-texlive/index.html":6.2316-6.2319: error: End tag "li" implied, but there were open elements.
"file:/public/getting-started-with-texlive/index.html":6.2307-6.2315: error: Unclosed element "package".

Some I can probably fix

 "file:/public/notes/nim/index.html":472.480-472.486: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
 "file:/public/notes/nim/index.html":1359.221-1359.231: error: Duplicate ID "log".
 ..

Some need to be ignored

 "file:/public/google4a938eaf9bbacbcd.html":1.1-1.52: error: Non-space characters found without seeing a doctype first. Expected "<!DOCTYPE html>".
 "file:/public/google4a938eaf9bbacbcd.html":1.1-1.52: error: Element "head" is missing a required instance of child element "title".
 ..

And the rest would be out of my scope to fix

 "file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: Start tag "a" seen but an element of the same type was already open.
 "file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: End tag "a" violates nesting rules.
 "file:/public/auto-count-100daystooffload-posts/index.html":114.526-114.529: error: Stray end tag "a".
 ..

Expand the below drawer if you’d like to see the full log with 46 errors:

Output of running html5validator --root public/

"file:/public/google4a938eaf9bbacbcd.html":1.1-1.52: error: Non-space characters found without seeing a doctype first. Expected "<!DOCTYPE html>".
"file:/public/google4a938eaf9bbacbcd.html":1.1-1.52: error: Element "head" is missing a required instance of child element "title".
"file:/public/notes/plantuml/index.html":114.474-114.678: error: Attribute "title" not allowed on element "a" at this point.
"file:/public/notes/nim-fmt/index.html":300.34-300.52: error: Duplicate ID "older-issue".
"file:/public/notes/nim-fmt/index.html":306.20-306.33: error: Duplicate ID "floats".
"file:/public/notes/nim-fmt/index.html":339.34-339.52: error: Duplicate ID "older-issue".
"file:/public/notes/nim-fmt/index.html":341.320-341.335: error: Duplicate ID "integers".
"file:/public/notes/nim-fmt/index.html":341.494-341.507: error: Duplicate ID "floats".
"file:/public/notes/nim-fmt/index.html":385.34-385.48: error: Duplicate ID "strings".
"file:/public/notes/string-fns-nim-vs-python/index.html":134.34-134.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":239.34-239.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":269.34-269.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":336.34-336.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":513.106-513.118: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":588.106-588.118: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":636.34-636.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":731.34-731.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":797.34-797.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":846.34-846.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":894.34-894.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":920.34-920.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":942.34-942.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":1012.34-1012.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":1074.34-1074.46: error: Duplicate ID "notes".
"file:/public/notes/nim/index.html":472.480-472.486: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/notes/nim/index.html":1359.221-1359.231: error: Duplicate ID "log".
"file:/public/notes/nim/index.html":2364.130-2364.149: error: Duplicate ID "named-tuples".
"file:/public/notes/nim/index.html":2403.149-2403.172: error: Duplicate ID "anonymous-tuples".
"file:/public/notes/nim/index.html":3370.284-3370.303: error: Duplicate ID "installation".
"file:/public/notes/nim/index.html":3410.354-3410.373: error: Duplicate ID "installation".
"file:/public/notes/nim/index.html":5033.34-5033.52: error: Duplicate ID "older-issue".
"file:/public/notes/nim/index.html":5376.34-5376.45: error: Duplicate ID "json".
"file:/public/notes/nim/index.html":6066.64-6066.81: error: Duplicate ID "references".
"file:/public/bits/plantuml-version/index.html":7.37-7.94: error: An "img" element must have an "alt" attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.
"file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: End tag "a" violates nesting rules.
"file:/public/auto-count-100daystooffload-posts/index.html":114.526-114.529: error: Stray end tag "a".
"file:/public/generics-not-exactly-in-systemverilog/index.html":118.232-118.238: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/grep-po/index.html":51.139-51.145: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/how-do-i-write-org-mode/index.html":23.194-23.200: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/hugo-use-goat-code-blocks-for-ascii-diagrams/index.html":24.130-24.255: error: An "img" element must have an "alt" attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.
"file:/public/hugo-modules-getting-started/index.html":6.865-6.871: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/using-emacs-advice-to-silence-messages-from-functions/index.html":151.687-151.732: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/using-emacs-advice-to-silence-messages-from-functions/index.html":151.748-151.751: error: Stray end tag "a".
"file:/public/page/6/index.html":29.39-29.54: error: Duplicate ID "fnref:1".
"file:/public/page/6/index.html":49.169-49.184: error: Duplicate ID "fnref:1".

Conclusion #

It was really easy to download the run the vnu application using Java, the standalone Linux binary and also through the html5validator Python wrapper.

After my quick trials, I think I will use the html5validator approach more because,

It works as I expect will the least number of switches.
I am able to redirect the output using html5validator --root public/ > validate.log. I tried the same using the vnu.jar and Linux compiled vnu binary, but the error log redirection didn’t work with those.

validator on A Scripter's Notes

Saving Python pip dependencies

1 Create a virtual environment #

2 Activate the virtual environment #

3 Install the packages using pip #

4 Generate requirements.txt #

Netlify deployment #

Summary #

Zero HTML Validation Errors!

1 Avoid duplicate heading id attributes #

2 Remove inline <style> elements #

3 Ensure that all images have captions or alt attributes #

4 Do not have hyperlinks in headings #

Validation Ignores #

5 Ignore errors due to hyperlinks in inline SVG #

6 Ignore files not expected to serve HTML content #

Summary #

Offline HTML5 Validator

Using the .jar #

Using pip install #

Using pre-compiled binary #

Results #

Conclusion #

3 Install the packages using `pip` #

4 Generate `requirements.txt` #

1 Avoid duplicate heading `id` attributes #

2 Remove inline `<style>` elements #

3 Ensure that all images have captions or `alt` attributes #

Using the `.jar` #

Using `pip install` #