Oz Updates - More flexible output & styling

Feb 7, 2020

I just finished publishing my latest update and am looking for things to dig into now.

One of the things which has come up recently is the need for increased control of styling and other document header information. In fact, at this very moment this site (metasoarous.com) has the Oz logo as its favicon because that's the only thing it knows how to do :grimacing:

So this is the next thing I'll be working on.

As I'm digging in, I've found myself in a bit of fog of design challenges. It's been interesting to see that as Oz grows, it's been a challenge to adapt old design elements to new demands, while also trying to predict what the tooling will need in the future, all while strictly growing and avoiding breakage.

In this particular case, the challenge has been trying to figure out what all of the options for rendering as html should be called, how they should organized etc. In particular, I've been trying to think through how these options will interplay with some of the other functionality I want to tackle, such as static rendering, different options for how visualizations are embedded (live, fully static, a combination of the two) and so on. I've decided to take a bit of a step back for a bit of hammock time and think a bit more about Oz's functionality, and the functionality I hope to have it offer. En route, I've produced this diagram mapping things out:

Feb 26, 2020

Well... I've been in Mexico on vacation for the last couple of weeks, and while I had hoped to have some time to work on this, the beaches & tacos have kept me rather more busy than I anticipated. However, I'll be leaving back for Seattle tomorrow and have a bit of time to myself tonight to sip mezcal and continue hacking on html styling.

Last time I started in on this, I realized it's a somewhat thorny design problem. Taking a bit of time off has been a bit helpful though, and in my background mind I've settled on some particular options for control over styling and layout for now. I've also been using spec as a sketching/thinking tool for solidifying these ideas, and so will present these options in spec form:

(s/def ::title string?)
(s/def ::description string?)
(s/def ::author string?)
(s/def ::keywords (s/coll-of string?))
(s/def ::shortcut-icon-url string?)
         
(s/def ::omit-shortcut-icon? boolean?)
(s/def ::omit-styles? boolean?)
(s/def ::omit-charset? boolean?)
(s/def ::omit-vega-libs? boolean?)

(s/def ::hiccup
  (s/and vector?
         #(let [x (first %)]
            (or (keyword x) (string? x)))))

(s/def ::header-extras ::hiccup)
                       
(s/def ::html-opts
  (s/keys :opt-un [::title ::description ::author ::keywords ::shortcut-icon-url ::omit-shortcut-icon? ::omit-styles? ::omit-charset? ::omit-vega-libs? ::header-extras]))

The options are now accepted as entries in the opts argument of (oz.core/html spec opts), and do their best to play nice with the options which are currently supported as metadata of the spec argument. In case you didn't know, it's possible to specify hiccup document metadata as markdown metadata like so (in following with the Jekyl pattern):

---
title: A great blog post
published-at: Today
tags: [clojure, oz]
---

## Hello world

Cool blog post, bro.

These options will be passed through as metadata on the spec object. Moreover, for options which should affect the html head content, such as [:title :description :author :keywords :tags], this info will get merged into the html-opts for constructing the appropriate html head section. The :tags metadata in particular gets merged in with any keywords which might have been specified explicitly as opts.

Feb 27, 2020

I'm now on an airplane on my way back to Seattle, and have been finishing up my implementation of the above, and testing my assumptions about how everything will work together. In particular, I've gone back and edited out a few of the options out which I'm not 100% committed to exposing yet, but may add back in later in some form or another.

As I've been tinkering around to get things working, I've realized that the live view currently does not provide correct styling if you are operating offline, since it uses the references to the stylesheets and fonts up on ozviz.io. This has me realizing that ideally the live view server would serve these assets from the project resources. Similarly, for output compiled from export, it may be worth looking at how we can embed these assets directly, so that they aren't as dependent on ozviz.io.

Feb 28th, 2020

Today I've been taking another look at my diagram and thinking about how to marry a number of threads of progress I have in mind for Oz's document processing features. In many ways, these threads converge towards a major step up in the abstract conception of what Oz is as a scientific document processing tool.

• code block highlighting/prettifying
• how to flexibly embed visualizations
  • so they can come with pre-rendered images
  • can "come alive" (interactive, etc) in an html context
• data table components
• the ability to extend or customize the base interpretation of Oz hiccup

Of course, this is made all the more challenging by the fact that this spec has to get interpreted in multiple ways in multiple places (to static HTML content, to live render view, etc). My initial stab at this is to generalize the processing of hiccup content with a function like this:

(defn process-form
  "Processes a form according to the given processors map, which maps tag keywords
  to a function for transforming the form."
  [[tag & _ :as form] processors]
  (let [tag (keyword tag)]
    (if-let [processor (get processors tag)]
      (processor form)
      form)))

(defn process
  [spec
   processors]
  (clojure.walk/prewalk
    (fn [form]
      (cond
        ;; If we see a function, call it with the args in form, so that you can use fns as components, like Reagent
        (and (vector? form) (fn? (first form))) 
        (apply-fn-component form)
        ;; apply processor function if applicable
        (vector? form)
        (process-form form processors)
        ;; Else, assume hiccup and leave form alone
        :else form))
    spec))

(process
  [:div
   [:h1 "Hello"]
   [:foo "What it be like?"]]
  {:foo
   (fn [form]
     (into ["BAR!!!"] (rest form)))})
;; => [:div [:h1 "Hello"] ["BAR!!!" "What it be like?"]]

This is a pretty flexible general purpose approach to processing content, and gives us ways of exposing and overriding defaults as necessary. Some questions remain about how this will get used in different contexts, and how using with a combination of static and live views:

• Generally, how will these options "bubble up" to higher level parts of the Oz API?
• Will different functions have their own processing defaults?
• Is there a better way of organizing or "registering" processing functions for application in different contexts?

These are some pretty big implications to work through as far as how this impacts the overall design and functionality of Oz, and so I'm going to take some more time to think through the various use cases and make sure everything makes sense.

Conclusions

This month has been a bit challenging time wise. Still, I've managed to improve on Oz's static HTML output flexibility and styling. The good news is that I've been able to secure a bigger chunk of dedicated time to work on the project this month, and am optimistic that I'll be able to get a lot accomplished in the coming weeks!

Please stay tuned :-)