Published at: 2020-01-16 3:50PM
Tags: clojure, oz
My first funded task for Oz has been to mint new cljsjs releases of the various Vega libraries. The latest Vega-Lite release (
4.0.x) in particular comes with quite an impressive set of new features! These include several useful transforms (density plots, regression, loess and quantiles), responsive width/height, image marks, and interactive legends (to name just a few)!
As I've begun to dig in, I've decided to keep a sort of running log of my development activities to keep folks up to date on what I've been working on. Unfortunately, as you'll see below, the process this time around has been somewhat fraught.
Cljsjs has always felt a bit challenging to work with for me. For a family of libraries like Vega, it can be rather cumbersome getting all the right versions, checksums, Google Closure externs all organized into a set of
build.boot files, each of which may refer to the versions and such of other
build.boot files, and each of which in turn needs it's own PR for review, etc.
A while ago I started working on a script for automating this process. It's a beautiful (read: terrifying) bit of bash, of which I'm very proud. In general, it's made things quite a bit more manageable for me, but it still seems to take a bit of expertise to wield.
This time around, not only did I hit some of the usual turbulence of working with this system, I also hit a number of miscellaneous issues which have kept me from getting the release out (yet!). However, I'm taking this as an opportunity to smooth out this part of the process more generally so that it'll be easier for myself and others in the future. What I'd ultimately like to do is empower developers to quickly run this script themselves when they need an updated version of the Vega libs, instead of necessarily having to wait for me to mint a new release of Oz.
Below is the day by day account of my journey thus far.
Right off the bat I've hit a snag and found that I was initially unable to mint a new release of the
cljsjs libraries. This is a result of a bug in
boot-cljsjs when running with newer Javas (specifically,
There are a few different ways this can be gotten around for now:
[javax.xml.bind/jaxb-api "2.3.0"]to the dependencies list in each
I'm opting for the latter for now.
# First install jabba curl -sL https://github.com/shyiko/jabba/raw/master/install.sh | bash && . ~/.jabba/jabba.sh # Install 1.8 jabba install email@example.com # Now using Java 8 for the rest of this session jabba use firstname.lastname@example.org
Unfortunately, this didn't do the trick :-/ I couldn't get
jabba to actually switch between the different versions of I installed, even after installing a few!
However, I remember having found another tool,
jenv, which was supposed to be able to switch between different versions of Java, but couldn't install versions on its own. The
jabba tool did appear to be doing this successfully, so I wondered if I might be able to get them to work together.
After following the install instructions for
jenv on its webpage (simply installing through homebrew didn't work, FWIW), I was able to run
# point jenv to the installation jenv add $(jabba which email@example.com) # set this shell to use the specified java jenv shell 1.8
This time it worked!
% java -version openjdk version "1.8.0_40" OpenJDK Runtime Environment (build 1.8.0_40-b25) OpenJDK 64-Bit Server VM (build 25.40-b25, mixed mode)
However... this still didn't get
oz/bin/update-cljsjs.sh script (my tooling for quickly updating vega cljsjs releases) working correctly! This time I got:
I let Deraen know about the above issues with
boot-cljsjs, and he suggested that the
javax dependencies should have been removed in cljs
1.10.238. I tried adding this dependency to the
(set-env! :dependencies [...]) vector, but unfortunately I got this error:
Could not transfer artifact org.clojure:clojurescript:pom:1.10.597 from/to clojars (https://repo.clojars.org/): java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
I tried instead adding a dependency on
[javax.xml.bind/jaxb-api "2.3.0"]. Since I already had this locally cached on my machine, I still ended up with the same
trustAnchors parameter error, but this time at the step in the
boot script where it tries to download the JS assets.
At this point, I became suspicious that something was wrong with the Java installed by
jabba. Sure enough, switching back to the system Java, I was able to get past this particular issue by adding the updating cljs dependency.
However, I'm not out of the hot water yet. While my build process was able to get through the necessary work for Vega, it choked as soon as it started working on Vega-Lite :-/
It seems that the Vega-Lite github archive assets are no longer structured the same way, breaking the assumptions in the
build.boot script. I'm not sure why I had been retrieving this particular asset from github instead of https://unpkg.com, as with the other assets, but I decided to take this as an opportunity to make this more uniform.
After making the above changes and running the script again, it failed on trying to commit changes for the vega package, because it had already gotten through that step on the last go around. In the past, I think I've just commented out the portions of the script which do the committing. This time however, I decided to add some logic to check whether there are actually any changes to commit before trying to commit, to hopefully make the script a bit more robust.
This looks something like this:
git add . VEGA_UPDATES=$(git diff --cached) if [ ! -z VEGA_UPDATES ] then git commit -m "[vega] update vega version to $v_version-$v_build_version" fi
We can use this same
VEGA_UPDATES variable later to print out more useful info about which branches need to be pushed/PRd.
In fact, we can do much better. We can use version (etc) diff information to decide whether to go through the whole install process at all! This may have to wait for another time however.
Last night was trying to wrap up these changes In particular, I was trying to fix some of the issues with trying to run the script with uncommitted changes in one of the
cljsjs/packages directories (which I had due to trying to get around the Java version issues, and from switching to
As luck would have it, my computer died right in the middle of this process, even though my battery was at ~20% (it would seem that my battery is starting to bite the dust). In any case, when I fired things up this morning, I found that a number of files had been corrupted, both some of the
build.boot files, and some of the internal git files. So basically everything is going great right now.
Anyway, what I'm trying to do now with my script is get it to stash local changes, update master from origin, and then reapply the stash so that each of the package update PRs are based on the correct/common set of changes (if you don't think through this right, some of the branches/PRs end up with the changes they're supposed to, but not others). However, because the git internals relating to the stash were corrupted from the dead battery, I had to clean up like this:
rm .git/logs/refs/stash rm -r .git/refs/stash
That got me half of the way there. The script was now able to stash the changes, but when it tried to applied them, I got
error: object file .git/objects/fd/f6aaec6770c1eae3a9de24c790b83cee576fae is empty fatal: loose object fdf6aaec6770c1eae3a9de24c790b83cee576fae (stored in .git/objects/fd/f6aaec6770c1eae3a9de24c790b83cee576fae) is corrupt
This StackOverflow answer got me past the issue. Basically, I had to reclone the project for its uncorrupted
.git repository, and move my changes in there.
Unfortunately though, as I continued to debug, I realized that the stash/apply process I came up with only does so much good. If you get through one package (e.g. Vega), but not a subsequent (e.g. Vega-Lite), you've got necessary changes in the former that are now hidden in a branch when you rerun. Again, this is all part in parcel with the difficulty of trying to automate/orchestrate changes around git, and there are a lot of possible states one can get in which need to be accounted for.
Continued working on script changes to support submission of cljsjs pull requests. Having a problem where when vega-lite is building it's not finding the vega library its expecting. Will have to look into this in greater depth later.
It's been a few days now since I've been able to spend time on the project by I've cleared out the remainder of the week. My goal is to finish getting the script back and working, and hopefully more robust for myself and other users who wish to assist with the cljsjs update process.