The Making of This Blog

2019-10-15 6:30

New Blog

So, I had wanted to make a blog for quite a while and I’ve finally done it. However, the blogging framework I am using is not a traditional one such as WordPress or Squarespace. It is actually a custom-built framework.

Motivation

For the longest time, I’ve wanted to be able publish or articles in general, but I was very picky with the features I wanted. I basically wanted the flexibility of HTML (i.e. to be able to embed arbitrary content) while still not hand-writing HTML every single time. Although there exist many static site generators (such as hexo, hugo, jekyll), they did not offer all the features I wanted, and also subjected the user to specific structure which could not be customized.

In addition, I highly value the overall transparency of a library, framework, or system that I am working with. By transparency, I mean how well I can understand the system as a whole, as well as being able to interact with all the parts of the system at any given time. The aforementioned blogging frameworks would not be able to offer the same level of transparency as a custom framework.

Features

I wanted a framework that would

Stack

So for this project, I used the following stack:

I used NodeJS because I was already familiar with it, so it would allow me to focus on the actual application.

I found pandoc from a post in Hacker News, and started messing around with it to convert various documents. I decided to use it because it was very easy to use (at first) and had existing wrapper libraries written for it in NodeJS.

I used EJS so that I could write as little HTML as possible, and have the templating engine generate it for me. The use of EJS makes it no longer a purely static site, making this framework a hybrid of sorts.

Markdown provides a strong foundation because it supports embedding raw HTML, and pandoc supports rendering math using mathjax out of the box.

Algorithm

Essentially, there is one build script that:

  1. Reads in a markdown file
  2. Converts it into HTML
  3. Embeds mathjax to typeset the math expressions
  4. If the content has a tikz picture, it skips it and instead uses a 3rd party renderer
  5. Evaluates and renders the EJS
  6. Writes the compiled article to an HTML file

Implementation

All of compilation happens in one file: build.js.

The function that models the process above is buildFile().

function buildFile(filePath, shouldDumpJSON = false) {
    Promise
        .resolve(fs.readFileSync(filePath))                 // 1
        .then((file) => dumpYFM(buildArticleName, file))    // 2
        .then(markdownToJSON)                               // 3
        .then(applyFilters)                                 // 4
        .then(filteredJSON => {                             // 5
            if (shouldDumpJSON) dumpJSON(buildArticleName, filteredJSON);
            return filteredJSON
        })
        .then(JSONtoHTML)                                   // 6
        .then(HTML => fs.writeFileSync(buildPath, HTML))    // 7
        .catch(err => {                                     // 8
            console.error(err);
        })
}

The above code can be read as the following:

  1. Read the file
  2. Dump the file’s metadata (known as YML Front Matter or YFM)
    1. The metadata consists of things such as the article’s title as well as its date of creation
  3. Convert the file to an intermediate JSON representation that we can filter
    1. This is what pandoc uses as a “ground truth” when converting between documents.
  4. Apply filters
    1. Here is where we can do things like skipping math content that contains tikzpictures
  5. Optionally dump the filtered JSON for debugging purposes
  6. Convert the JSON to HTML
  7. Finally, write the built HTML file to the desired path
  8. Handle any errors that come up

Blog Structure

The way that the blog is structured is shown below.

.
├── article-staging
├── articles
├── build-artifacts
│   ├── article-metadata
│   └── json-dumps
└── files
    ├── css
    ├── font
    ├── js
    └── views
        └── processed-articles

Articles are written in the article-staging directory, then moved to the articles directory once they are ready to be published.

Published articles are then built and placed in the files/views/processed-articles directory.

Implementation Problems

There were various problems and roadblocks that I encountered while implementing this framework.

One of the major issues was that early on, I didn’t know how to have pandoc “skip” processing only certain blocks of text, such as a math markup block containing a tikzpicture. This is because pandoc supports rendering LaTeX math expressions out of the box in markdown, using mathjax.

It took me a lot of time and effort (and frustration) to figure out how pandoc actually worked, and what all the flags meant, and the syntax of applying or removing certain features. In my original design, I didn’t want pandoc to render math at all, and simply wanted to forward that responsibility to a third party renderer (tex.s2cms.com). I finally found the tex_math_dollars flag and disabled pandoc’s automatic rendering.

Then, I realized that still wanted mathjax because it allowed a user to actually select the text, as opposed to a much more unsatisfying rendered image that the 3rd party solution offered. This eventually made me have to dive into pandoc’s filtering features, and had me writing a custom pandoc filter to skip math content if it contained a tikzpicture. Getting the setup for this right took way longer than I expected, and was a really finicky process.

Complexity Creep

The pandoc filters and specific math rendering setup did introduce way more complexity than I had imagined, and resulted in the build.js script becoming a bit more unwieldy than was originally desired. In addition, I needed certain core functionality for the blog to work—such as listing all the blog posts on the landing page, sorting posts by date, etc.—which all contributed to increasing the amount of code in the once simple build script.

Design

For the design, I had made it an objective from the start to make the blog’s design plain and simple, mainly drawing inspiration from various brutalist websites (although I do not consider this site to have brutalist design). This was done in order for me to be able to focus on the blog’s content rather than it’s look and feel, which has been distracting to me in the past.

Final Remarks

Overall, I am satisfied with the framework that I have made. Although I underestimated the amount of time it would take to build it, I believe it was a worthwhile effort.

Credit

In addition to various software libraries, this project also made use of