I’ve just about wrapped up a set of improvements to storage for Juju 2.3, the next “minor” release. If you’re already using, or have been planning to use Juju’s storage support, read on.

Dynamic storage management

Juju charms can specify storage requirements: the number of filesystems or block devices its application requires. For example, the PostgreSQL charm requires a filesystem on which to store the database. If you don’t tell Juju otherwise, the storage will go onto the root filesystem, but you can also tell Juju to provide the charm with cloud storage (Amazon EBS, OpenStack Cinder, etc.)

One of the missing pieces that users have been asking for is the ability to manage the lifecycle of storage independently of applications and units, and to reuse existing storage. In Juju 2.3, when you remove an application or unit, the storage attached to the unit(s) will (if possible) be detached, rather than destroyed, and will remain in the model. You can then either remove the storage using juju remove-storage, or attach it to a new unit using the new juju attach-storage command, or the --attach-storage flag added to juju deploy and juju add-unit. To complement juju attach-storage, there is also a new juju detach-storage command.

So to illustrate, you can now deploy PostgreSQL with cloud storage, then remove the application, and redeploy (e.g. with more RAM), using the same storage.

juju deploy postgresql --storage=10G
…
juju remove-application postgresql
…
juju deploy postgresql --constraints mem=16G --attach-storage pgdata/0

We’re still working on giving you commands to remove storage from the model without destroying it, and then import it into a new model (possibly new controller). This is required for disaster recovery. Whether this makes it for 2.3 depends on prioritisation; if it doesn’t make it for 2.3, it shouldn’t be far behind.

LXD Storage Provider

One thing that we hadn’t planned for 2.3, but we did manage to get done, is a LXD storage provider. LXD has recently added its own storage management API, and Juju 2.3 will have a storage provider that uses it. I originally implemented the Juju side of things as a bit of a hack, behind a feature flag, in order to speed up the development of the aforementioned attach/detach changes. The LXD storage API turned out to be very straight forward to build on, so we decided to release the Juju changes into the wild in case it’s of use to others. Particularly if you’re developing or testing charms that use storage, this should be useful.

Using the LXD storage provider is as simple as:

juju deploy postgresql --storage=10G,lxd

Each storage pool using the “lxd” storage provider will create an associated storage in LXD. When you create a storage pool in Juju, you need to specify two configuration attributes:

• the LXD storage pool name, as the “lxd-pool” attribute
• the LXD storage driver, as the “driver” attribute

You can also define driver-specific attributes, which will be passed through to the LXD storage driver verbatim.

Juju predefines a “lxd-zfs” pool, with the following attributes:

• lxd-pool=juju-zfs
• driver=zfs
• zfs.pool_name=juju-zfs

If you deploy an application with storage using the lxd-zfs pool, Juju will create a LXD storage pool called “juju-zfs” with the “zfs” driver, and ZFS pool called “juju-zfs”. To find out more about the LXD storage driver options, see the LXD storage docs.

In the Juju 2.1 release, I made a couple of small changes to better support CentOS servers.

The first change was to support “manual provisioning” of CentOS machines. Manual provisioning is when you point Juju at a machine, and Juju connects to the machine over SSH and sets it up with a Juju agent. To do this, Juju needs to run a small shell script to discover the OS version and hardware characteristics of the machine. With a minor change to that script, you can now manually provision CentOS machines.

The second change is to support CentOS LXD images. A small change was needed in the Juju code to support the “centos7” OS version, and alter the way we handle local LXD image aliases. If an image exists locally with the expected alias (e.g. “juju/centos7/amd64”), then we’ll use that and skip looking in the remote image sources. This also improves container startup time when you live in a faraway land like me. Altering Juju is not quite enough though, as there are no existing CentOS images that Juju can use.

Juju (mostly) requires cloud-init to be present on the machines it starts, so that it can inject Juju-specific configuration and scripts to run on startup. Unforunately, there are no CentOS LXD images that have cloud-init already. To work around this, I wrote a standalone Go program to transform the linuxcontainers.org CentOS image: github.com/axw/juju-lxd-centos-image-builder. Eventually we hope to have pre-canned CentOS LXD images available to Juju, but for now you can use this program to prepare an image for Juju. Run it from the LXD host, and Juju will be able to use the resulting image.

Some time ago I created a snap for llgo. Snaps are a relatively new packaging format; at the time I created the initial snap for llgo, it was only possible to create them confined by default. In particular, access to the user’s filesystem was blocked by default, which is unhelpful for a tool like a compiler. Because of that, the initial snap did not expose the llgo compiler, only the llgoi interpreter.

Now there is the concept of “classic snaps”, which use the snap packaging format but do not confine the application. The resulting package is much more like what you would get from a Debian package, but without the arcana.

I have released a new llgo classic snap, exposing the llgoi interpreter, and the llgo compiler and “go” tool wrapper. To install the llgo snap, run:

sudo snap install --classic llgo

The exposed “llgo” command is an alias for the go tool wrapper, so you can just run:

$ llgo install <package>
$ llgo run /path/to/source.go
...

as you would with the standard go tool. The llgo compiler itself can be invoked with the “llgo.compiler” command:

$ llgo.compiler -emit-llvm -S -o -o main.go
; ModuleID = 'main'
source_filename = "main"
target datalayout ...

The llgoi interpreter is still available, but as it is part of the same snap it is no longer confined:

$ llgo.llgoi
(llgo) import "fmt"
(llgo) fmt.Println("hello, world")
hello, world
13
<nil>

Juju is an application modelling tool, enabling “model-driven operations”. I won’t go into detail about what Juju is in this blog post, so if you’re new to Juju I suggest clicking on the link and reading a bit more.

Juju is a distributed application, with a “controller” cluster that manages cloud resources (machines, networks, volumes, etc.), and applications that use those resources. The controller cluster is currently based on top of MongoDB, utilising replica sets for data replication and leadership election.

As well as the controller cluster, Juju agents run on every virtual machine that the controller manages. The controllers, and those agents, each run many fine-grained, but dedicated “workers”. For example, each agent runs a worker to detect block devices and publish that information to the controller cluster; each controller runs a worker to maintain the replica sets in MongoDB.

Many things can go wrong in a distributed system. Network partitions can cause system-wide failures. Bad actors (badly written; less often, malicious) may starve others of resources. Failure to release memory or file handles leads to exhaustion, causing a DoS. Juju has seen its fair share of each of these problems.

To combat such issues, we have recently added Prometheus monitoring to Juju. As of Juju 2.1, Juju controllers and agents will export Prometheus metrics. There are two ways to get at them:

• (on controllers) an HTTPS endpoint, https://…:17070/introspection/metrics.
• (on Linux agents) an abstract domain socket, @jujud-machine-<machine-ID>

Configuring Prometheus to scrape Juju controllers

To configure Prometheus to scrape metrics from Juju controllers, you will need to add a new scrape target to Prometheus. The metrics endpoint requires authorisation, so you will need to configure a user and password for Prometheus to use:

$ juju add-user prometheus
$ juju change-user-password prometheus
new password: <password>
type new password again: <password>

For this new “prometheus” user to be able to access the metrics endpoint, you must grant the user read access to the controller model:

$ juju grant prometheus read controller

This gives the prometheus user just enough permission to read information on the controller, without allowing it to make changes, which would not be ideal for a monitoring application.

Juju serves the metrics over HTTPS, currently with no option of degrading to HTTP. You can configure your Prometheus to skip validation, or you can store the controller’s CA certificate in a file for Prometheus to verify the server’s certificate against:

$ juju controller-config ca-cert > /path/to/juju-ca.crt

We can now add a scrape target to Prometheus. Modify prometheus.yml, adding the following scrape target:

scrape_configs:
  job_name: juju
    metrics_path: /introspection/metrics
    scheme: https
    static_configs:
      targets: ['<controller-address>:17070']
    basic_auth:
      username: user-prometheus
      password: <password>
    tls_config:
      ca_file: /path/to/juju-ca.crt

Configuring Prometheus to scrape Juju agents

To expose the metrics of agents, you can deploy the juju-introspection charm onto that agent’s machine. For example, on machine 1, you would run:

juju deploy ~axwalk/juju-introspection --to 1

The metrics of that agent can then be obtained via:

http://<machine-1-address>:19090/agents/machine-1/metrics

Note that this is not an officially supported charm. The code for it is available at:

• https://github.com/axw/juju-introspection-charm
• https://github.com/axw/juju-introspection-proxy

I have finally gotten around to moving off Blogger, and over to using Hugo. The old blog posts have been imported, and are also accessible at my old Blogger site.

Availability Zones in Juju

juju add-machine zone=us-east-1b

juju deploy -n 3 mongodb

$ juju status mongodb | grep instance-id
instance-id: i-7a6d2b50
instance-id: i-ff1562d4
instance-id: i-627f0a30

$ ec2-describe-availability-zones
AVAILABILITYZONE us-east-1a available us-east-1
AVAILABILITYZONE us-east-1b available us-east-1
AVAILABILITYZONE us-east-1d available us-east-1

$ ec2-describe-instance-status i-7a6d2b50 i-ff1562d4 i-627f0a30 | grep i-
INSTANCE i-627f0a30 us-east-1d running 16 ok ok active
INSTANCE i-ff1562d4 us-east-1a running 16 ok ok active
INSTANCE i-7a6d2b50 us-east-1b running 16 ok ok active

(Note: the ec2-* commands are available in the ec2-api-tools package.)

Hello there!

I’ve been busy hacking on llgo again. In case you’re new here: llgo is a Go frontend for LLVM that I’ve been working on for the past ~2 years on and off. It’s been quite a while since I last wrote; there has been a bunch of new work since, so I have some things to talk about at last.

A few months ago, I started working on rewriting swathes of llgo’s internals to base it on go.tools/ssa. LLVM uses an SSA representation, which made the process fairly straightforward. Basing llgo on go.tools/ssa gives me much higher confidence in the quality of the output; it also presented a good opportunity to clean up llgo’s source itself, which I have begun, but certainly not finished. llgo is now able to compile all packages in the standard library, except those that require cgo (net, os/user, runtime/cgo).

llgo now works something like this:

1. Go source is scanned and parsed by go/ast and go/parser, producing an AST;
2. The AST is fed into go/types, type-checking;
3. The output of go/types is passed onto go.tools/ssa, which generates the SSA form;
4. llgo translates the go.tools/ssa SSA form into an LLVM module,
5. llgo-build links the LLVM modules for a program together and translates to an executable.

• Translating Phi nodes requires a bit of finessing, to ensure processing of the Phi or the edges is not order-sensitive. This was dealt with by generating placeholder values for instructions that haven’t yet been visited, and then replacing them later.
• ssa.Index is emitted for indexing into arrays. If an array is in a register, then indexing it means extracting a value; in LLVM, an array element extraction requires a constant index. This is currently kludged by storing to a temporary alloca, and using the getelementptr LLVM instruction. Hopefully I’m missing something and this is easily fixed.
• The Recover block is not dominated by the entry block, so it may not be valid for it to refer to the Alloc instructions for parameters and results. To deal with this, I generate a prologue block that contains the param/result Allocs; the prologue block conditionally jumps to either the recover or entry block, depending on panic/recover control flow. Alan has agreed to do something along these lines in go.tools/ssa.
• The ssa.Next instruction required some assumptions to be made about block ordering and instruction placement, in order to be able to translate string-range using Phi nodes. Recent changes to go.tools/ssa exposed the dominator tree, making it possible to do away with the assumption now.

• Interfaces are now represented like in gc: empty interfaces with the runtime type & data, non-empty interfaces with an “itab” and data. Russ Cox wrote an article about the interface representation back in 2009.
• Panic/recover (and defer, by consequence) are now using setjmp/longjmp. I had been using exceptions, but it was rightly pointed out to me that this wouldn’t work unless there were a way of doing non-call exceptions in LLVM (which has not been implemented). The setjmp/longjmp approach incurs a cost for every function that may defer or recover, but it works without modifications to LLVM. Perhaps this will be revisited in the future.
• go/types/typemap is now used for mapping types.Types to runtime type descriptors and LLVM types. Runtime type descriptors are now generated more completely, and more correctly. Identical type descriptors will now be merged at link-time.
• llgo no longer generates conditional branching for calls to non-global functions, when comparing structs, or in map iteration. Apart from producing better code, this makes it much simpler to work with go.tools/ssa, which has its own idea about how the SSA basic blocks relate to one another.

• A custom importer/exporter, thanks to Fredrik Ehnbom. The importer side is disabled at the moment, due to an apparent bug in go.tools/ssa.
• Debugger support, thanks again to Fredrik Ehnbom. I haven’t reenabled it since the move to go.tools/ssa. I’ll get onto that real soon now, because debugging without it can be tiresome.
• llgo-build can now take a “-test” flag that causes llgo-build to compile the test Go files, yet again thanks to Fredrik Ehnbom. This is currently reliant on the binary importer being enabled, so it won’t work out of the box until that bug is fixed.
• Shifts now generate correct values for shifts greater than the width of the lhs operand.
• Signed integer conversions now sign-extend correctly.
• bytes.Compare now works as it should (-1, 0, 1, not <0, 0, >0). “llgo-build -test bytes”: PASS
• llgo-build can now take a “-run” flag that causes llgo-build to execute and then dispose of the resulting binary.
• Type strings are propagated to LLVM types, making the IR more legible, thanks to Travis Cline.

• Move to using libgo. Ideally the gc runtime would be rewritten in Go already, but that’s not going to happen just yet. The compiler and linker are due to be rewritten in Go soon, which is a lot of work as it is.
• Finish off runtime type descriptor generation (notably, type algorithms).
• Get PNaCl support working again. This should be pretty close, but requires the binary importer to be enabled.
• Implement cgo support.
• Implement bounds checking, nil pointer checks, etc.
• Get garbage collection working. There’s Pull Request #108, but this is perpetuating the problem that is llgo’s custom runtime. Since GC is fairly invasive, I don’t want to go tying llgo to that runtime any more than it is currently. I expect this will have to wait until libgo is integrated.
• Escape analysis. This is a must-have, but not immediately necessarily. The implementation should be based on go.tools/ssa, interfacing with the exporter/importer to record/consume information about external functions.
• Make use of go.tools/ssa/ssautil/Switches. This is an optimisation, so again, not immediately necessary.

Ahoy there, mateys!

It’s been three months since our last correspondence. Apologies for the negligence. I’ve been busy, as usual, but it’s more self-inflicted than usual. I’ve taken up a new role at Canonical, working on Juju. I’m really excited about Juju (both the concept and realisation), and the fact that it’s written in Go is icing on the cake. Working remotely is taking some getting used to, but so far it’s been pretty swell. Anyway, you didn’t come here to read about that, did you?

I’m still working on llgo in the background, quietly prodding it along towards the 0.1 milestone. There’s just one big ticket item left, and that’s partially done now: channels. I’ve just finished porting the basics of channels from gc’s standard library to llgo’s runtime. That doesn’t include select, which is entirely missing. When that’s done, I’ll be content to release 0.1.

So what’s new since last time?

• There’s a new llgo-build tool, which takes the pain out of building packages and programs with llgo and the LLVM toolchain. Just run “llgo-build <package>”, and you’ll either build and install a package, or build a program in the working directory. There’s no freshness checking, so you’re currently required to manually build all dependencies before building a program.
• Simplified building against PNaCl: llgo-dist now accepts a “-pepper” option, which points to a NaCl SDK.
• Implemented support for map literals.
• Implemented complex number arithmetic.
• Implemented channels (apart from anything select-related)
• Numerous bug fixes.

Hi folks,

(For those of you coming from HN/Twitter/elsewhere, this is a post about llgo. llgo is an LLVM frontend for the Go programming language).

In my last post I mentioned that work had began on moving to Go 1.1 compatibility; this has been my primary focus since then. Since Go 1.1 is now released (woohoo!), I’ve gone ahead and pulled all the changes back into the master branch on GitHub. If you want to play around, you can do the following:

1. Get Go 1.1.
2. Get Clang and LLVM (I’ve tested with 3.2, Ubuntu x86-64). Make sure llvm-config is in your $PATH.
3. Run “go get github.com/axw/llgo/cmd/llgo-dist”
4. Run “llgo-dist”. This will install llgo into $GOBIN, and build the runtime.

• Method sets are handled properly now (or at least not completely wrong like before). This means you can use a embedded types’ methods to satisfy an interface.
• “return” requirements are now checked by go/types
• cap() is now implemented for slices.
• llgo-dist now builds against the LLVM static libraries (if available) by default now, with an option for building against the shared libraries.

Oh my, it’s been a while.

In my previous post I wrote about llgo and PNaCl. I haven’t had much time to play with PNaCl recently, but I have been prodding llgo along. In February, my wife gave birth to our son, Jeremy, so naturally I’ve been busy. But anyway, let’s talk about what has been happening in llgo. Quick, while he’s sleeping!

Feature-wise, there’s nothing terribly exciting going on. Without getting too boring, what’s new is:

• A new “go1.1” branch in the Git repository. The go1.1 branch aims to make llgo compatible with the Go tip, and will replace the master branch when Go 1.1 is released.
• Removed llgo/types (a fork of the old exp/types package), and moved to go/types.
• Updated runtime type representations to match those from gc’s tip (thanks to minux for initiating this effort).
• Updated to use architecture-specific size for “int” (same as uintptr).
• Changed function representation to be a pair of pointers, to avoid trampolines/runtime code generation for closures. The rationale is the same as for rsc’s proposal for Go 1.1; using runtime code generation limits the environments that Go can run in (e.g. PNaCl).
• A slew of bug fixes and minor enhancements.