Oz static site generation!

For the last year or so, I've working on Oz. What initially started as a humble fork of Vizard, a Vega-Lite based data visualization wrapper for Clojure, has turned into an increasingly powerful toolkit for working with scientific documents in a number of contexts.

The latest release, [metasoarous/oz "1.6.0-alpha3"], comes with one of the most exciting features yet: Static site generation, complete with live code reloading!

First, a bit about Oz & Vega

Because I haven't really posted about Oz yet, I figure it's worth taking some time to explain what it is, and how it's evolved to this point.

The story starts some years ago, when I came across Vega as the underlying data visualization engine behind Gorilla REPL's plotting utilities. As a card-carrying Clojurist, I instantly liked the declarative, data-driven model. It seemed a bit verbose for day to day data science usage, but I was intrigued to find that their sister project Vega-Lite aimed more squarely at that use case. Unfortunately, what I found didn't match the power of the R-based toolchain I had grown to love (ggplot2). But, it looked like a promising start, and all based on the same underlying Grammar of Graphics.

Fast forward to a year or so ago, Erick Matsen mentioned I take another look at the latest version of Vega-Lite. To my pleasant amazement, Vega-Lite 2.0 ended up being a major rewrite, and had solved many of the design problems which I had stumbled upon during my first encounter years before. Further, I was blown away when I realized the true scope of the interactive features. The Vega team at the Interactive Data Lab have done a bang up job, and have produced a set of tools which give R's toolkit a run for it's money.

If you'd like to learn more about Vega-Lite & Vega in particular, and how they relate to Clojure philosophically, please take a look at this I gave at Amperity about Vega-Lite and Vega:

As Oz evolved, it started getting features like Hiccup & Markdown processing, so that it's easy to combine data visualizations with Clojure data structures and plain text Markdown as appropriate for a broader range of scientific documentation. This set the stage for live file reloading via live-view!, which lets watch a file of Markdown or Hiccup, combined with Vega-Lite and/or Vega, and render the most recent compilation as the file changes. I also took things one step further, and responded to a challenge by Aria to bring the Figwheel experience to the data science workflow, by building a differential live-code reloading experience for Clojure.

All of this combined with the ability to export Hiccup or Markdown with data visualizations to html files, and it became very clear than an intriguing prospect was in site: a static site generation framework with live-code reloading (hot code swapping).

Static site generation

So what's all this buzz about static sites? I thought we liked our sites "live" and "dynamic"!?

Live and dynamic web applications are great. However, many web apps (even "dynamic" ones) can get away with a "static" infrastructure, meaning that all of the assets needed to run the site can be made available as static, pre-compiled files, ready to serve. Because caching and scaling of static content is more or less a solved problem, it's typically dirt cheap or even free to host statically compiled web sites or web applications (including GitHub pages and surge.sh; more on that later).

While deploying static content is great, it's often the case that someone authoring a simple website or blog would want to produce most of their content in an easier to write format, such as Markdown. For this reason, there are static site generation tools which compile Markdown (or something like it) into HTML for you, such as Jekyll.

However, the number of these tools which feature Figwheel-style live code reloading is very small. The number which come pre-packaged with tools for data scientists, such as data visualizations and LaTeX, is smaller still. In fact, please let me know if you find another.

Demonstration

The following video provides a fairly thorough demonstration of these new features:

If you don't want to play a video now, read on a for a super-focused quick start.

A very simple site generated from a single directory of input Markdown or Hiccup files might be as straightforward as:

(require '[oz.core :as oz])

(oz/build!
  [{:from "site-src/pages"
    :to   "build"}])

A more complete example might illustrate the :template-fn functionality, together with our ability to specify different multiple specifications for certain sets of input files:

(defn blog-post-layout [spec]
  [:div
   [:h1 (:title (meta spec))]
   spec])

(oz/build!
  [{:from "site-src/pages"
    :to   "build"}
   {:from "site-src/blog"
    :to   "build/blog"
    :template-fn blog-post-layout}])

This ability to differentiate processing behavior into separate spec entries gives you the expressiveness you need to cover the full range of needs which users of a solid static site generation toolkit should expect.

For a full description of what options are available please see (doc oz/build!).

Taking the features for a spin

This page clearly would not be complete without a gratuitous trial of the visualization and LaTeX functionality, so without further adieu:

This visualization is produced by the following markdown code:

```edn vega-lite
{:data {:values [{:a 1 :b 3 :species "cat"}
                 {:a 3 :b -1 :species "dog"}
                 {:a 5 :b 8 :species "dog"}
                 {:a 2.5 :b 4.5 :species "cat"}]}
 :mark :point
 :encoding {:x {:field :a}
            :y {:field :b}
            :color {:field :species}}}
```

Now for some LaTeX:

$$ e ^ { i \theta } = \cos \theta + i \sin \theta $$

Hosting options

As I said above, once you're ready to deploy, you'll have lots of options available to you:

• GitHub Pages: Free and simple, with git based deploy. There are some restrictions on what kinds of projects you can put up, but for basic personal homepages or project pages, it's a great fit.
• Surge: This is what I use; It's free and super easy to get going, and has quite a few nice features out of the box, and paid plans for extra features.
• AWS: You can also deploy fairly easily directly to AWS with S3 and CloudFront. The process here is a little bit more involved, and there may be some conveniences amiss, but you also end up having more control over the things.

While they may be a bit overkill for basic static sites, you may also want to check out Zeit & Firebase. These so called serverless platforms take static sites to the next level by integrating services which take care of needs which would otherwise require you to write and administer your own server (e.g. database access, user auth, etc.). Surge has some of these features as well, but is more squarely focused on static content.

Conclusion

I'm super excited about this new phase in the development of Oz, and look forward to writing more posts like this for you with it in the near future. Because, in case you haven't guessed it, I'm authoring this post as we speak using this feature, and f#@king loving it.

Still, we're just scratching the surface, and there's lots I have planned for the project. So please stay tuned for developments.