requirements.txt so
that Netlify can install and run the HTML5 Validator before deploying
this site.Recently I learned about the Python tool html5validator tool and
used it to run HTML5 validation on local HTML files. You can read more
about that in the Offline HTML5 Validator post.
But then I wondered .. “Wouldn’t it be awesome if I run this
validation step each time after Hugo generates this site on Netlify,
but before it gets deployed?”. That curiosity led me to Netlify’s
documentation on Python dependencies, and I learned that Netlify will
run pip install to install the dependencies in requirements.txt
present in the repository’s base directory.
I followed the pip freeze > requirements.txt
step in that documentation but that ended up listing all my pip
installed packages in that file! I needed the requirements.txt to
include the dependencies only for html5validator. This post was born
in my quest to achieve that.
The solution to this problem was to first create a Python virtual
environment in my site directory, and then do all the pip
operations in there. I learned about this virtual environment step
from this Python Pandemonium post.
The venv documentation has a lot of details — I’ll just list the key steps in this post.
cd to your site directory and run the below command create a
virtualenv directory named pyenv in there.
python3 -m venv pyenv
As we are creating this virtual environment just for the sake of
creating a requirements.txt, we don’t need to commit this
directory to git.
The virtualenv directory pyenv/bin/ will have multiple flavors of
shell scripts to activate your virtualenv. I am using
activate.csh here as my shell is tcsh 🙄.
source pyenv/bin/activate.csh
Just to emphasize, it is important to use the source command here,
and not just call the shell script directly.
Once you activate this virtualenv, you should see something like [pyenv] in the shell prompt.
pip #Install all the project-specific packages. In this case it was just this:
pip install html5validator
The package (and its dependencies) will be installed in
pyenv/lib/python3.7/site-packages/. Here, the python3.7/
sub-directory name will match the version of Python you have
installed.
requirements.txt #Finally, we run the pip freeze command mentioned in Netlify docs,
but with the --local switch:
pip freeze --local > requirements.txt
The --local option ensures that globally-installed packages are not
listed even if the virtualenv has global access.
Here’s the requirements.txt created by that command:
html5validator==0.4.2
PyYAML==6.0
requirements.txt for Netlify
Now, I could have just manually typed html5validator==0.4.2 in a
requirements.txt and committed that, and that would work too. But
then, I wouldn’t have learned how to create a project-specific pip
dependency file 😄.
The html5validator has a dependency on Java 8. Luckily the Netlify
environment already comes with that installed. So the only extra setup
needed to make this package work on Netlify was to set the
PYTHON_VERSION environment variable to 3.8
The version number should be one of the values listed in Netlify’s
included software list.
.
The numbered headings in this post already summarize the steps needed
to create a project-specific requirements.txt.
With these in place:
✅ requirements.txt committed in site directory root
✅ Netlify environment variable PYTHON_VERSION set to 3.8
my Netlify deployment script now does HTML5 Validation along with few other checks before this site gets deployed 🎉.
Here’s the full build.sh script.
In my previous HTML5 Validator post, I mentioned:
I was a bit disappointed to see validation errors on my site, but then it wasn’t too bad .. 46 errors.
But they truly say ..
Ignorance is bliss.
So .. once I realized that my site had 46 validation errors, I lost that bliss .. and I couldn’t rest easy — I had to fix them all 😁.
This post summarizes the categories of those errors and how I fixed The fixes mentioned in this post refer to changes in Org mode content. But you should be able to derive equivalent fixes for Markdown too. them all.
id attributes #"file:/public/notes/string-fns-nim-vs-python/index.html":636.34-636.46: error: Duplicate ID "notes".
"file:/public/notes/nim-fmt/index.html":300.34-300.52: error: Duplicate ID "older-issue".
"file:/public/notes/nim-fmt/index.html":306.20-306.33: error: Duplicate ID "floats".
"file:/public/page/6/index.html":29.39-29.54: error: Duplicate ID "fnref:1".
Errors with above kind of signatures were fixed by,
Converting headings to description lists
I had a bunch of generic headings like “Notes” and “Older Issue” in
some of my posts. After taking a second look at those, it made more
sense to convert those to description lists. So in Org mode, I
converted headings like * Notes to
description lists - Notes ::.
Setting CUSTOM_ID heading property
For the cases, where the headings needed to be left as so, their
IDs were uniquified by setting their CUSTOM_ID property. For
example, below fixed the Duplicate ID “floats” errors.
** Precision
..
*** Floats
+:PROPERTIES:
+:CUSTOM_ID: precision-floats
+:END:
..
** Type (only for numbers)
..
*** Floats
+:PROPERTIES:
+:CUSTOM_ID: type-floats
+:END:
CUSTOM_ID property to uniquify heading ID's
Prevent footnote links in post summaries
This issue was due to me not being conscious about how the footnote references work in a post versus on a page outside that post’s context. The issue was caused by footnote references getting into the post summaries parsed by Hugo, which will then show up on the list pages.
The fix was simple — Edit the post summaries so that they don’t contain any footnote references.
<style> elements #"file:/public/grep-po/index.html":51.139-51.145: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/how-do-i-write-org-mode/index.html":23.194-23.200: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
Errors with above kind of signatures were fixed by,
Avoiding export of raw <style> elements in the Markdown content
I figured out which functions were responsible for injecting
<style> elements in Markdown content and then advised them to
stop that. After applying these advises, I lost the in-content
rules for CSS classes .org-center and .csl-entry. So I put
those rules directly in this website’s CSS.
(defun modi/org-blackfriday-center-block (_center-block contents info)
(let* ((class "org-center"))
(format "<div class=\"%s\">%s\n\n%s\n</div>"
class (org-blackfriday--extra-div-hack info) contents)))
(advice-add 'org-blackfriday-center-block :override #'modi/org-blackfriday-center-block)
(defun modi/org-cite-csl-render-bibliography (bib-str)
(replace-regexp-in-string "<style>\\.csl-entry[^<]+</style>" "" bib-str))
(advice-add 'org-cite-csl-render-bibliography :filter-return #'modi/org-cite-csl-render-bibliography)
<style> elements in exports
Removing unnecessary micro-styling
I found a single case, where an inline CSS rule was defined in
content for CSS class .repr-type for a table. I just removed that
without affecting the looks of that rendered table too much.
alt attributes #"file:/public/hugo-use-goat-code-blocks-for-ascii-diagrams/index.html":24.130-24.255: error: An "img" element must have an "alt" attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.
Errors with above kind of signatures were easily fixed by ensuring
that all images had captions
Thankfully, there were only two images that were missing captions.
. The Hugo figure shortcode adds the caption to the alt attribute if
the alt is not specified separately.
As an example, here’s how I fixed the above error:
+ #+name: fig__disproportionate_box_drawing
+ #+caption: Disproportionate box drawing characters
[[file:images/hugo-use-goat-code-blocks-for-ascii-diagrams/ascii-diagram-rendered-in-plain-text-code-block.png]]
git diff showing addition of caption to an image
"file:/public/using-emacs-advice-to-silence-messages-from-functions/index.html":151.687-151.732: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/using-emacs-advice-to-silence-messages-from-functions/index.html":151.748-151.751: error: Stray end tag "a".
"file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: End tag "a" violates nesting rules.
"file:/public/auto-count-100daystooffload-posts/index.html":114.526-114.529: error: Stray end tag "a".
While the hyperlinks in headings work well, they created invalid HTML in the Hugo-generated TOC. So while these errors were created technically because of a bug in Hugo It’s really cool when you end up finding a bug in an upstream project while trying to fix the errors in your own thing 😎. , I wanted to fix these errors on my end as soon as I can.
I reviewed the errors, and this is all it took to get rid of them all:
Remove manually inserting hyperlinks in headings
show two methods of finding sources of any printed messages.
-***** Using plain-old /grep/ or [[https://github.com/BurntSushi/ripgrep][/rg/]]
+***** Using plain-old /grep/ or /ripgrep (rg)/
This method is pretty easy (but not robust) to use if the search
..
Org source directory and search for the /"org-babel-exp process .."/
-string ..
+string using [[https://github.com/BurntSushi/ripgrep][~rg~]] ..
Remove Org Radio links that created links in headings
Here, a Org heading happened to contain the string “Day count”, which was also an an Org Radio link in that post. While ideally that shouldn’t have mattered, I removed that radio link to get around this Hugo bug.
.. /Just may be/. But regardless, I am already enjoying writing once
-again, and it's great to see the <<<Day count>>> (counting up to 100)
+again, and it's great to see the Day count (counting up to 100)
increase with each new post!
Above fixes fixed 43 out of 46 errors, but the remaining 3 were unfixable.
"file:/public/notes/plantuml/index.html":114.474-114.678: error: Attribute "title" not allowed on element "a" at this point.
This error was caused by hyperlinks in inline SVG elements. These SVG elements are created by PlantUML. The hyperlinks in SVG feature works great, and as these are generated by PlantUML, I chose to just ignore these errors.
I ignored this error by adding the --ignore-re 'notes/plantuml.*Attribute.*title.*not allowed' switch to the
html5validator command.
"file:/public/googleFOO.html":1.1-1.52: error: Non-space characters found without seeing a doctype first. Expected "<!DOCTYPE html>".
The googleFOO.html file here is not a valid HTML file. It’s a just a
cookie file that was used by Google to verify that I own this
domain.
This error was masked by adding the --ignore 'googleFOO' switch to
the html5validator command.
Once I fixed the 43 errors by tweaking the Org mode content, and added those two ignores, I had zero validation errors! 🎉
If you are interested in the fix details, here are the commits.
]]>curl command away.I have been using the online HTML5 Validator https://html5.validator.nu/ for few years now. I have a link at the bottom of each post to validate that page’s HTML. For an example, see the html5 validator link at the bottom of this post.
But it didn’t occur to me until now to look for a way to do the same validation offline! Offline validation would be useful so that I can look at any HTML generation problem before I deploy the website. So I looked for a solution online, and of course it’s answered on StackOverflow 😄.
It turns out that same Nu HTML5 Validator project that provides the online validation service, also provides a Java application as well as pre-compiled binaries for Linux, Windows and MacOS for offline use!
To use the Java .jar file, you need to have at least Java 8 installed on your system. But you don’t need to have any version of Java installed if you use the pre-compiled binary instead. See its documentation for more details.
.jar #Requires at least Java 8
vnu.jar from the project’s GitHub
Releases section.
curl -ROLs https://github.com/validator/validator/releases/download/latest/vnu.jar
public/
If you are using Hugo, the hugo command will publish the HTML
files in the public/ directory by default.
directory. See its Usage documentation for more details.
java -jar vnu.jar --skip-non-html --errors-only public/
For my usecase, if I don’t provide the --skip-non-html --errors-only switches, the output is too noisy.
pip install #If you do not want to manually download the .jar, there’s a Python
wrapper available to do the same for you: html5validator.
pip install --user html5validator
html5validator --root public/
--skip-non-html --errors-only to the Java app. So those are not needed when running
html5validator. But on the flip side, it needs the --root switch
when specifying the directory to run the script on.Note that you still need to have at least Java 8 installed when
running this Python app too, because it downloads and run the same
.jar behind the scenes.
If your system doesn’t have the required Java version, you can use the pre-compiled binary instead.
vnu.<OS>.zip for
your OS from the same Releases section. Here, I am showing how
to do that on Linux:
curl -ROLs https://github.com/validator/validator/releases/download/latest/vnu.linux.zip
unzip vnu.linux.zip
The extracted binary path will be vnu-runtime-image/bin/vnu.
java.
vnu-runtime-image/bin/vnu --skip-non-html --errors-only public/
I was a bit disappointed to see validation errors on my site, but then
it wasn’t too bad .. 52 46 errors:
"file:/public/getting-started-with-texlive/index.html":6.2198-6.2206: error: Element "package" not allowed as child of element "li" in this context. (Suppressing further errors from this subtree.)
"file:/public/getting-started-with-texlive/index.html":6.2256-6.2259: error: End tag "li" implied, but there were open elements.
"file:/public/getting-started-with-texlive/index.html":6.2198-6.2206: error: Unclosed element "package".
"file:/public/getting-started-with-texlive/index.html":6.2307-6.2315: error: Element "package" not allowed as child of element "li" in this context. (Suppressing further errors from this subtree.)
"file:/public/getting-started-with-texlive/index.html":6.2316-6.2319: error: End tag "li" implied, but there were open elements.
"file:/public/getting-started-with-texlive/index.html":6.2307-6.2315: error: Unclosed element "package".
"file:/public/notes/nim/index.html":472.480-472.486: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/notes/nim/index.html":1359.221-1359.231: error: Duplicate ID "log".
..
"file:/public/google4a938eaf9bbacbcd.html":1.1-1.52: error: Non-space characters found without seeing a doctype first. Expected "<!DOCTYPE html>".
"file:/public/google4a938eaf9bbacbcd.html":1.1-1.52: error: Element "head" is missing a required instance of child element "title".
..
"file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: End tag "a" violates nesting rules.
"file:/public/auto-count-100daystooffload-posts/index.html":114.526-114.529: error: Stray end tag "a".
..
Expand the below drawer if you’d like to see the full log with 46 errors:
html5validator --root public/"file:/public/google4a938eaf9bbacbcd.html":1.1-1.52: error: Non-space characters found without seeing a doctype first. Expected "<!DOCTYPE html>".
"file:/public/google4a938eaf9bbacbcd.html":1.1-1.52: error: Element "head" is missing a required instance of child element "title".
"file:/public/notes/plantuml/index.html":114.474-114.678: error: Attribute "title" not allowed on element "a" at this point.
"file:/public/notes/nim-fmt/index.html":300.34-300.52: error: Duplicate ID "older-issue".
"file:/public/notes/nim-fmt/index.html":306.20-306.33: error: Duplicate ID "floats".
"file:/public/notes/nim-fmt/index.html":339.34-339.52: error: Duplicate ID "older-issue".
"file:/public/notes/nim-fmt/index.html":341.320-341.335: error: Duplicate ID "integers".
"file:/public/notes/nim-fmt/index.html":341.494-341.507: error: Duplicate ID "floats".
"file:/public/notes/nim-fmt/index.html":385.34-385.48: error: Duplicate ID "strings".
"file:/public/notes/string-fns-nim-vs-python/index.html":134.34-134.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":239.34-239.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":269.34-269.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":336.34-336.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":513.106-513.118: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":588.106-588.118: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":636.34-636.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":731.34-731.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":797.34-797.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":846.34-846.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":894.34-894.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":920.34-920.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":942.34-942.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":1012.34-1012.46: error: Duplicate ID "notes".
"file:/public/notes/string-fns-nim-vs-python/index.html":1074.34-1074.46: error: Duplicate ID "notes".
"file:/public/notes/nim/index.html":472.480-472.486: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/notes/nim/index.html":1359.221-1359.231: error: Duplicate ID "log".
"file:/public/notes/nim/index.html":2364.130-2364.149: error: Duplicate ID "named-tuples".
"file:/public/notes/nim/index.html":2403.149-2403.172: error: Duplicate ID "anonymous-tuples".
"file:/public/notes/nim/index.html":3370.284-3370.303: error: Duplicate ID "installation".
"file:/public/notes/nim/index.html":3410.354-3410.373: error: Duplicate ID "installation".
"file:/public/notes/nim/index.html":5033.34-5033.52: error: Duplicate ID "older-issue".
"file:/public/notes/nim/index.html":5376.34-5376.45: error: Duplicate ID "json".
"file:/public/notes/nim/index.html":6066.64-6066.81: error: Duplicate ID "references".
"file:/public/bits/plantuml-version/index.html":7.37-7.94: error: An "img" element must have an "alt" attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.
"file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/auto-count-100daystooffload-posts/index.html":114.446-114.475: error: End tag "a" violates nesting rules.
"file:/public/auto-count-100daystooffload-posts/index.html":114.526-114.529: error: Stray end tag "a".
"file:/public/generics-not-exactly-in-systemverilog/index.html":118.232-118.238: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/grep-po/index.html":51.139-51.145: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/how-do-i-write-org-mode/index.html":23.194-23.200: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/hugo-use-goat-code-blocks-for-ascii-diagrams/index.html":24.130-24.255: error: An "img" element must have an "alt" attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.
"file:/public/hugo-modules-getting-started/index.html":6.865-6.871: error: Element "style" not allowed as child of element "div" in this context. (Suppressing further errors from this subtree.)
"file:/public/using-emacs-advice-to-silence-messages-from-functions/index.html":151.687-151.732: error: Start tag "a" seen but an element of the same type was already open.
"file:/public/using-emacs-advice-to-silence-messages-from-functions/index.html":151.748-151.751: error: Stray end tag "a".
"file:/public/page/6/index.html":29.39-29.54: error: Duplicate ID "fnref:1".
"file:/public/page/6/index.html":49.169-49.184: error: Duplicate ID "fnref:1".
It was really easy to download the run the vnu application using
Java, the standalone Linux binary and also through the
html5validator Python wrapper.
After my quick trials, I think I will use the html5validator
approach more because,
html5validator --root public/ > validate.log. I tried the same using the vnu.jar and
Linux compiled vnu binary, but the error log redirection didn’t
work with those.Ever wondered how to show LaTeX in HTML or a Hugo blog post exported from Org?
There are 2 ways to do this:
If you don’t mind including the MathJax script, it’s as simple as
typing $\LaTeX$ in Org, which results in \(\LaTeX\).
Similarly \(\TeX\) ($\TeX$) also works, though not $\XeTeX$.
And here’s another way if you don’t want to include MathJax.
#+macro: tex @@html:<span class="tex">T<sub>e</sub>X</span>@@
#+macro: latex @@html:<span class="latex">L<sup>a</sup>T<sub>e</sub>X</span>@@
#+macro: xetex @@html:<span class="xetex">X<sub>Ǝ</sub>T<sub>E</sub>X</span>@@
Add this CSS directly in the page within a #+begin_export html /
#+end_export block, or add that CSS your site’s stylesheet.
<style>
.tex, .latex, .tex sub, .latex sub, .xetex sub {
font-size: 1em;
}
.tex sub, .latex sub, .latex sup, .xetex sub {
text-transform: uppercase;
}
.tex sub, .latex sub, .xetex sub {
vertical-align: -0.5ex;
margin-left: -0.1667em;
margin-right: -0.125em;
}
.latex sup {
font-size: 0.85em;
vertical-align: 0.15em;
margin-left: -0.36em;
margin-right: -0.15em;
}
</style>
- {{{tex}}}
- {{{latex}}}
- {{{xetex}}}
Export that from Org to HTML or Hugo, and you get: