Oz Updates - Honing in on API updates

Picking up from my latest update, I've been reviewing the changes proposed in the last post. Along the way, I cleaned up the API diagram:

Note that:

• The green arrows represent new API functions.
• Targets with a * represent side effects.
• There's a bit of double-speak going on with filename vs static-files*, but the graph made more sense this way.

Specifically regarding the new API functions:

• compile aims to be a new general purpose conversion/compilation function, based on multimethods (more on this below).
• wrap-html is for taking a hiccup document and wrapping it in an [:html ...] tag with all of the goodies needed for compiling to html.
  • This comes with a slew of new options for controlling the styling/details of the output, as relate to headers and such.
• embed takes hiccup which might contain Vega-Lite/Vega visualizations and embeds the corresponding viz blocks as html which can be live rendered.
  • This will eventually come with options for how you want to embed; as a live interactive viz, a static image, or both (live viz replaces static once ready).

More on compile

I think the biggest of these breakthroughs is the compile function. I've had a sense for some time now that the way we think about taking one thing and getting another was in need of some massive overhaul. We already have a html function for rendering hiccup &/or Vega-Lite/Vega as html. And we're looking at having support for static viz compilation and a pdf mode. With all of that going on, I didn't want to start tacking on new API function left and right without thinking about the right way to organize it all.

Here's a map of all the conversions we need to be able to handle:

To handle all of this sanely, I propose a multimethod system where compilers can be registered based on :to & :from entries in an opts map. The default multimethod implementation would first convert the input data to :hiccup, and then convert that hiccup to whatever the intended output format is. This gives us a flexible system where we only have to specify conversions to/from hiccup for any given output type we might want to add in the future, but when appropriate, we can override this when we have a way of implementing it more efficiently.

Note that this all also has to play nice with export!, which is going to get a bit of a makeover in the process, allowing you to export :to a particular format. The default implementation will be to use compile with the appropriate :from and :to settings. However, in cases where we'd just be calling out to the vega-cli, we can override to call out to the CLI directly.

Options

Across the API, I've been reviewing all of the options, and adding specs for them. I'm taking the time to do this now to ensure that the overlapping functionality in compile, export!, vega-cli feels internally consistent, and to avoid any conflicts in intended meaning. This is particularly important with a number of functions which expect to be shuttling data back and forth between each other, such as compile and export! as defined above.

Crazy idea

One of the crazier ideas that has popped up though (and one on which I'd like feedback), is that I'd like to make these functions accept options in two different styles:

(export! data filename opts)
(export! data filename & {:as opts})

The story here is this: When I inhereited forked Oz (from Vizard), it's API used the & {:as opts} variant, and not wanting to break the API for folks who wanted to switch, I stuck with this. At the time, the only real function in the API was the view! function. Given that this was primarily intended for REPL usage, I wasn't too concerned about this.

As Oz grew, I made the decision to stick with the existing pattern, to maintain consistency in the API. However, as more and more of the functionality has come to expand beyond the focus of REPL tooling, this has started to irk me. I like being able to pass option maps as a single argument, as it makes it easier to compose/compute these options and pass in. As mentioned, I didn't want to make the API inconsistent, and I certainly didn't want to break any of it.

The thought occurred to me that I could support both options by defining like this

(defn export!
  ([data filename] (export! data filename {}))
  ([data filename opts]
   ( ...))
  ([data filename opt-k opt-val & {:as more-opts}]
   (export! data filename (assoc more-opts opt-k opt-val))))

Having not seen this in the wild much, I kept telling myself it was crazy. But maybe it's crazy like a fox!

And so, with compile and friends (see above) en route, I'd like some feedback from the community about whether they think this is a good idea or not. With that in mind, I invite you to contribute to this advisory twitter poll!

https://twitter.com/metasoarous/status/1240085028064727040?s=20

Please leave a vote, and if you have any more specific/nuanced thoughts or concerns, please let me know in a comment there! And thank you in advance for the feedback!

Pull requests!

I've been very fortunate this last couple of weeks to have a number of pull requests come in. Some of them have been simple README fixes, others fixes for usage with Shadow-CLJS (which I'm considering switching to at some point). Thanks to everyone who has submitted one of these, or even just submitted an issue or comments on an issue letting me know of things that need attention. It's wonderful to have such a helpful community of users :-)

Conclusion

It may feel like things have been moving slowly, but there's been a lot of progress in speccing things out, and hammocking my way through some of the core design problems that Oz faces. And with a number of pull requests having made it in the last few weeks, please expect a release soon which captures all of this great work.

Thanks for reading, and please stay tuned!