Julia Environment

This notebook describes and creates the default Julia environments in Nextjournal. It is based on the Bash environment. Check out the showcase if you want to see what the environment contains. To see how it’s built, see setup.

Showcase

These packages are included in Nextjournal's Julia 1.2 environment.

]st
0.8s
Julia Test (Julia)
Julia 1.3.1

System Packages and Basics

A wide variety of support libraries are installed, as well as gcc v7 and ImageMagick.

Version

"1.0.10"
of the SoftGlobalScope package is available—this adjusts scope-handling to be more like Julia 0.6, providing a more REPL-like interface.

Julia packages are installed normally, using Pkg in a Julia cell. Please refer to the Julia section of Installing Software and Packages for more detailed information.

Plotting

The default environment comes with GR v

"0.44.0"
, PlotlyBase
"0.3.0"
, and PlotlyJS
"0.13.0"
, as well as version
"0.28.4"
of the Plots framework. Makie version
"0.9.17"
is also installed.

Plots

The JuliaPlots framework provides a unified interface to multiple graphical backends. The default backend is GR, which efficiently generates static plots and animations.

using Plots; gr()
# define the Lorenz attractor
mutable struct Lorenz
    dt; σ; ρ; β; x; y; z
end
function step!(l::Lorenz)
    dx = l.σ*(l.y - l.x)       ; l.x += l.dt * dx
    dy = l.x*(l.ρ - l.z) - l.y ; l.y += l.dt * dy
    dz = l.x*l.y - l.β*l.z     ; l.z += l.dt * dz
end
attractor = 
  Lorenz((dt = 0.02, σ = 10., ρ = 28., β = 8//3, x = 1., y = 1., z = 1.)...)
# initialize a 3D plot with 1 empty series
plt = plot3d(1, xlim=(-25,25), ylim=(-25,25), zlim=(0,50),
             title = "Lorenz Attractor", marker = 2)
# build an animated gif by pushing new points to the plot, saving every 10th frame
@gif for i=1:1500
    step!(attractor)
    push!(plt, attractor.x, attractor.y, attractor.z)
end every 10
40.1s
Julia Test (Julia)
Julia 1.3.1

Switching to the Plotly backend adds some interactivity to the output.

plotly(lw=3)
x = -100:100
Plots.plot(x, 100x.^2)
Plots.plot!(x, x.^3 - x.^2)
10.7s
Julia Test (Julia)
Julia 1.3.1

Makie

The WGLMakie package can generate beautiful 3D visualizations.

using WGLMakie, AbstractPlotting, LinearAlgebra
n = 20
set_theme!(resolution = (600, 500))
f   = (x,y,z) -> x*exp(cos(y)*z)
∇f  = (x,y,z) -> Point3f0(exp(cos(y)*z), -sin(y)*z*x*exp(cos(y)*z), x*cos(y)*exp(cos(y)*z))
∇ˢf = (x,y,z) -> ∇f(x,y,z) - Point3f0(x,y,z)*dot(Point3f0(x,y,z), ∇f(x,y,z))
θ = [0;(0.5:n-0.5)/n;1]
φ = [(0:2n-2)*2/(2n-1);2]
x = [cospi(φ)*sinpi(θ) for θ in θ, φ in φ]
y = [sinpi(φ)*sinpi(θ) for θ in θ, φ in φ]
z = [cospi(θ) for θ in θ, φ in φ]
pts = vec(Point3f0.(x, y, z))
∇ˢF = vec(∇ˢf.(x, y, z)) .* 0.1f0
AbstractPlotting.surface(x, y, z)
AbstractPlotting.arrows!(pts, ∇ˢF, arrowsize = 0.03, linecolor = (:white, 0.6), linewidth = 3)
107.0s
Julia Test (Julia)
Julia 1.3.1

VegaLite

VegaLite can also provide interactive plotting.

using VegaLite, VegaDatasets
vega_plot = dataset("cars") |>
@vlplot(
    :point,
    x = :Horsepower,
    y = :Miles_per_Gallon,
    color = :Origin,
    width = 650,
    height = 400
) |> VegaLite.interactive()
16.4s
Julia Test (Julia)
Julia 1.3.1

Data Structures

Several data-related packages are installed in the default environment.

  • For handling their relative datatypes and I/O for associated files, we install a variety of packages, including JLD v

    "0.9.1"
    , CSV v
    "0.5.22"
    , HDF5 v
    "0.12.5"
    , and JSON v
    "0.21.0"
    .

  • The FileIO v

    "1.2.1"
    framework provides a unified interface for loading files of many types via multiple backends, including ImageMagick.

  • DataFrames v

    "0.9.1"
    defines and handles objects similar to those found in R and the Python pandas toolkit, with a comparable interface.

File Handling

The HDF5 package provides a Julia interface to the HDF5 library. It is also used by the JLD and MAT packages. Let's look at some example temperature data.

NEONDSTowerTemperatureData.hdf5
using HDF5
# Some methods to traverse the first path in an h5 file.
dd(node::HDF5File) = dd(node[names(node)[1]])
dd(node::HDF5Group) = dd(node[names(node)[1]])
dd(node::HDF5Dataset) = node
dspath = h5open(
NEONDSTowerTemperatureData.hdf5
) do h5
  name(dd(h5))
end
print("First path: $dspath.")
data = h5read(
NEONDSTowerTemperatureData.hdf5
, dspath)
using Plots
Plots.plot([x.data[1] for x in data],[y.data[3] for y in data],
  xrotation=45,legend=:none)
26.5s
Julia Test (Julia)
Julia 1.3.1

The JLD package provides a type-preserving way to save Julia objects to file.

using JLD
struct testData
  x::Int64
  y::String
end
foo = testData(7,"test")
save("/tmp/blah.jld", "foo", foo)
load("/tmp/blah.jld")
5.4s
Julia Test (Julia)
Julia 1.3.1
Dict{String,Any} with 1 entry: "foo" => testData(7, "test")

FileIO

FileIO provides a unified set of methods to access data in files: query(), load(), and save(), as well as loadstreaming() and savestreaming() for large files. The functions can automatically recognize files using headers or extensions, but you can also provide format information as below, where we load the Iris Dataset from a CSV file.

iris.csv
using FileIO
load(File(format"CSV",
iris.csv
))
6.7s
Julia Test (Julia)
Julia 1.3.1

DataFrames

DataFrame objects represent tabular data as a set of vectors.

using DataFrames
df = DataFrame(A = 1:5, B = 1:2:10, C = ["a","b","c","d","q"])
5.3s
Julia Test (Julia)
Julia 1.3.1
ABC
11a
23b
35c
47d
59q
5 items

Column names are referenced via accesing the fields.

df.A .+ df.B
1.2s
Julia Test (Julia)
Julia 1.3.1
5-element Array{Int64,1}: 2 5 8 11 14

JSON

Import and export JSON using the JSON package, which is always loaded on Nextjournal. In the example below, a Julia data structure input results in JSON output. The change from nothing to null is a clear indicator.

json(["foo", Dict("bar" => ("baz", nothing, 1.0, 2))])
0.6s
Julia Test (Julia)
Julia 1.3.1
"[\"foo\",{\"bar\":[\"baz\",null,1.0,2]}]"

Setup

Julia 1.x

Build a Minimal Julia 1.x Environment

We'll base our environment off of our Minimal Python image, which has a small conda Python setup. Note that the Julia version is set as an environment variable on the runtime.

Minimal Julia 1.3.1
Minimal Julia 1.3.1 (Bash)
exporting environment
Type: Nextjournal
Environment:
Machine Type:
Environment Variables:
PATH/usr/local/julia/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
JULIA_PATH/usr/local/julia
JULIA_VERSION1.3.1
CONDA_JL_HOME/opt/conda
Download this environment as a Docker image from:

The exact version we're installing is

"1.3.1"
. Here we download the archive and signatures, verify, and save. Cell is locked to prevent redownload.

tarArch="x86_64"
dirArch="x64"
BASEURL="https://julialang-s3.julialang.org/bin"
tarfile="julia-${JULIA_VERSION}-linux-${tarArch}.tar.gz"
dir="${dirArch}/${JULIA_VERSION%[.-]*}"
tarurl="${BASEURL}/linux/${dir}/${tarfile}"
sigfile="${tarfile}.asc"
sigurl="${tarurl}.asc"
shafile="julia-${JULIA_VERSION}.sha256"
shaurl="${BASEURL}/checksums/${shafile}"
keyfile="juliareleases.asc"
keyurl="https://julialang.org/${keyfile}"
wget -q --show-progress --progress=bar:force \
  $tarurl $sigurl $shaurl $keyurl
sha256sum -c --ignore-missing $shafile
export GNUPGHOME="$(mktemp -d)"
gpg --import $keyfile
gpg --batch --verify $sigfile $tarfile
gpgconf --kill all
cp $tarfile /results/
rm -rf "$GNUPGHOME" $tarfile $sigfile $shafile $keyfile
7.5s
Minimal Julia 1.3.1 (Bash)
julia-1.3.1-linux-x86_64.tar.gz

Install from saved archive.

mkdir -p "$JULIA_PATH"
tar -xzf 
julia-1.3.1-linux-x86_64.tar.gz
-C "$JULIA_PATH" --strip-components 1
3.8s
Minimal Julia 1.3.1 (Bash)

And verify it runs.

julia -v
0.6s
Minimal Julia 1.3.1 (Bash)

The JSON package is required to run on Nextjournal.

julia -e 'using Pkg; pkg"up; add JSON; precompile; test JSON"'
49.4s
Minimal Julia 1.3.1 (Bash)

Install Conda.jl, setting it to use the existing Conda installation.

julia -e 'using Pkg; pkg"add Conda; build Conda; precompile"'
9.1s
Minimal Julia 1.3.1 (Bash)

Check size.

du -hsx /
5.9s
Minimal Julia 1.3.1 (Bash)

Build the Default Julia 1.x Environment

We'll add a number of packages as well as the libraries and support programs required by them. The major packages installed are JuliaPlots along with the GR and PlotlyJS backends, WGLMakie, and the DataFrames, CSV, and HDF5 data-handling packages.

First we'll check that the minimal Julia env works.

"$VERSION"
0.3s
Julia 1.x versionJulia 1.3.1 (Julia)
Minimal Julia 1.3.1
"1.3.1"

Install a few system tools and libraries.

apt-get -qq update
DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends \
  imagemagick libhdf5-dev hdf5-tools mesa-utils \
  libxt6 libxrender1 libgl1-mesa-glx libqt5widgets5 `# for GR` \
  libhttp-parser2.7.1 `# for PlotlyJS`
apt-get clean
rm -r /var/lib/apt/lists/* # Clear package list so it isn't stale
32.0s
Julia 1.3.1 (Bash in Julia)
Minimal Julia 1.3.1

Next the Julia package installs.

]up; add SoftGlobalScope DataFrames JLD CSV CSVFiles Netpbm NRRD MeshIO HDF5 MAT FileIO JSExpr CSSUtil StatsBase StatsPlots Observables Interact WebSockets HTTP Blink WebIO PlotlyBase PlotlyJS RecipesBase GR Plots ImageCore ImageShow ImageMagick Colors BenchmarkTools FixedPointNumbers IJulia WGLMakie AbstractPlotting JSCall
140.4s
Julia 1.3.1 (Julia)
Minimal Julia 1.3.1

VegaLite currently requires a little custom NodeJS setup. Should no longer be required on the release after v1.0.0.

]dev VegaLite; add VegaDatasets
34.2s
Julia 1.3.1 (Julia)
Minimal Julia 1.3.1

Install NodeJS from binaries to get a recent version of npm that supports Python 3.

NODE_FN="node-v${NJS_VERSION}-linux-x64"
NODE_URL="https://nodejs.org/dist/v${NJS_VERSION}/${NODE_FN}.tar.xz"
cd /opt
curl $NODE_URL | tar -Jx
ln -s $NODE_FN node
npm config set python /opt/conda/bin/python
npm config set unsafe-perm true # required because we are root
2.7s
Julia 1.3.1 (Bash in Julia)
Minimal Julia 1.3.1

Install to VegaLite.jl's directory. Pkg.dev pulls master, checkout 1.0.0.

cd /root/.julia/dev/VegaLite
git checkout tags/v1.0.0
cd deps
npm install --scripts-prepend-node-path=true \
  --production --no-package-lock --no-optional
9.9s
Julia 1.3.1 (Bash in Julia)
Minimal Julia 1.3.1

Downgrade JSONSchema for VegaLite 1.0.0.

]add JSONSchema@0.1
1.5s
Julia 1.3.1 (Julia)
Minimal Julia 1.3.1
]resolve; up
3.9s
Julia 1.3.1 (Julia)
Minimal Julia 1.3.1

Build all packages.

]build
45.0s
Julia 1.3.1 (Julia)
Minimal Julia 1.3.1

Precompile any qualifying packages.

]precompile
548.1s
Julia 1.3.1 (Julia)
Minimal Julia 1.3.1

Finally, add font defaults for JuliaPlots and Makie. These named Code Listings will be mounted as files to the runtime's filesystem, and saved with the environment.

PLOTS_DEFAULTS = Dict(:fontfamily => "Open Sans")
.juliarc.jl
Julia
Attributes(font = "Open Sans", resolution = (600, 500))
theme.jl
Julia

Makie caches some fonts on first using, so we make sure that they're already cached.

using AbstractPlotting, WGLMakie
for res in (AbstractPlotting.High, AbstractPlotting.Low)
  AbstractPlotting.set_glyph_resolution!(res)
  AbstractPlotting.cached_load();
end
39.5s
Julia 1.3.1 (Julia)
Minimal Julia 1.3.1

Test Default 1.x Env

using SoftGlobalScope, DataFrames, JLD, CSV, CSVFiles, Netpbm, NRRD, MeshIO, HDF5, MAT, FileIO, JSExpr, CSSUtil, StatsBase, StatsPlots, Observables, Interact, WebSockets, HTTP, Blink, WebIO, PlotlyBase, PlotlyJS, RecipesBase, GR, Plots, ImageCore, ImageShow, ImageMagick, Colors, BenchmarkTools, FixedPointNumbers, IJulia, WGLMakie, AbstractPlotting, VegaLite, VegaDatasets
IJulia.verbose
0.2s
Julia Test (Julia)
Julia 1.3.1
false
"$(Pkg.installed()["SoftGlobalScope"])"
2.1s
SoftGlobalScope versionJulia Test (Julia)
Julia 1.3.1
"1.0.10"
"$(Pkg.installed()["GR"])"
0.2s
GR versionJulia Test (Julia)
Julia 1.3.1
"0.44.0"
"$(Pkg.installed()["PlotlyBase"])"
0.2s
PlotlyBase versionJulia Test (Julia)
Julia 1.3.1
"0.3.0"
"$(Pkg.installed()["PlotlyJS"])"
0.3s
PlotlyJS VersionJulia Test (Julia)
Julia 1.3.1
"0.13.0"
"$(Pkg.installed()["Plots"])"
0.2s
Plots versionJulia Test (Julia)
Julia 1.3.1
"0.28.4"
"$(Pkg.installed()["AbstractPlotting"])"
0.2s
Makie versionJulia Test (Julia)
Julia 1.3.1
"0.9.17"
"$(Pkg.installed()["JLD"])"
0.2s
JLD versionJulia Test (Julia)
Julia 1.3.1
"0.9.1"
"$(Pkg.installed()["CSV"])"
0.2s
CSV versionJulia Test (Julia)
Julia 1.3.1
"0.5.22"
"$(Pkg.installed()["HDF5"])"
0.3s
HDF5 versionJulia Test (Julia)
Julia 1.3.1
"0.12.5"
"$(Pkg.installed()["JSON"])"
0.2s
JSON versionJulia Test (Julia)
Julia 1.3.1
"0.21.0"
"$(Pkg.installed()["FileIO"])"
0.2s
FileIO versionJulia Test (Julia)
Julia 1.3.1
"1.2.1"
"$(Pkg.installed()["DataFrames"])"
0.2s
DataFrames versionJulia Test (Julia)
Julia 1.3.1
"0.20.0"

Appendix

Since the site hosting the static build packages can be slow, we'll download FFmpeg here and save it for repeated uses. Current version is 4.1.3 (released 1 Apr 2019, downloaded 1 May 2019).

TARFILE="ffmpeg-release-amd64-static.tar.xz"
wget --progress=dot:giga \
  https://johnvansickle.com/ffmpeg/releases/${TARFILE}
wget https://johnvansickle.com/ffmpeg/releases/${TARFILE}.md5
md5sum -c ${TARFILE}.md5
cp $TARFILE /results/
8.8s
Download FFmpeg (Bash)
ffmpeg-release-amd64-static.tar.xz
Runtimes (4)