awilkins.id.au

Coming in Juju 2.3: storage improvements

Thu, 13 Jul 2017 15:41:31 +0800

I’ve just about wrapped up a set of improvements to storage for Juju 2.3, the next “minor” release. If you’re already using, or have been planning to use Juju’s storage support, read on.

Dynamic storage management

Juju charms can specify storage requirements: the number of filesystems or block devices its application requires. For example, the PostgreSQL charm requires a filesystem on which to store the database. If you don’t tell Juju otherwise, the storage will go onto the root filesystem, but you can also tell Juju to provide the charm with cloud storage (Amazon EBS, OpenStack Cinder, etc.)

One of the missing pieces that users have been asking for is the ability to manage the lifecycle of storage independently of applications and units, and to reuse existing storage. In Juju 2.3, when you remove an application or unit, the storage attached to the unit(s) will (if possible) be detached, rather than destroyed, and will remain in the model. You can then either remove the storage using juju remove-storage, or attach it to a new unit using the new juju attach-storage command, or the --attach-storage flag added to juju deploy and juju add-unit. To complement juju attach-storage, there is also a new juju detach-storage command.

So to illustrate, you can now deploy PostgreSQL with cloud storage, then remove the application, and redeploy (e.g. with more RAM), using the same storage.

juju deploy postgresql --storage=10G
…
juju remove-application postgresql
…
juju deploy postgresql --constraints mem=16G --attach-storage pgdata/0

We’re still working on giving you commands to remove storage from the model without destroying it, and then import it into a new model (possibly new controller). This is required for disaster recovery. Whether this makes it for 2.3 depends on prioritisation; if it doesn’t make it for 2.3, it shouldn’t be far behind.

LXD Storage Provider

One thing that we hadn’t planned for 2.3, but we did manage to get done, is a LXD storage provider. LXD has recently added its own storage management API, and Juju 2.3 will have a storage provider that uses it. I originally implemented the Juju side of things as a bit of a hack, behind a feature flag, in order to speed up the development of the aforementioned attach/detach changes. The LXD storage API turned out to be very straight forward to build on, so we decided to release the Juju changes into the wild in case it’s of use to others. Particularly if you’re developing or testing charms that use storage, this should be useful.

Using the LXD storage provider is as simple as:

juju deploy postgresql --storage=10G,lxd

Each storage pool using the “lxd” storage provider will create an associated storage in LXD. When you create a storage pool in Juju, you need to specify two configuration attributes:

the LXD storage pool name, as the “lxd-pool” attribute
the LXD storage driver, as the “driver” attribute

You can also define driver-specific attributes, which will be passed through to the LXD storage driver verbatim.

Juju predefines a “lxd-zfs” pool, with the following attributes:

lxd-pool=juju-zfs
driver=zfs
zfs.pool_name=juju-zfs

If you deploy an application with storage using the lxd-zfs pool, Juju will create a LXD storage pool called “juju-zfs” with the “zfs” driver, and ZFS pool called “juju-zfs”. To find out more about the LXD storage driver options, see the LXD storage docs.

Juju 2.1 and CentOS

Thu, 23 Feb 2017 11:50:00 +0800

In the Juju 2.1 release, I made a couple of small changes to better support CentOS servers.

The first change was to support “manual provisioning” of CentOS machines. Manual provisioning is when you point Juju at a machine, and Juju connects to the machine over SSH and sets it up with a Juju agent. To do this, Juju needs to run a small shell script to discover the OS version and hardware characteristics of the machine. With a minor change to that script, you can now manually provision CentOS machines.

The second change is to support CentOS LXD images. A small change was needed in the Juju code to support the “centos7” OS version, and alter the way we handle local LXD image aliases. If an image exists locally with the expected alias (e.g. “juju/centos7/amd64”), then we’ll use that and skip looking in the remote image sources. This also improves container startup time when you live in a faraway land like me. Altering Juju is not quite enough though, as there are no existing CentOS images that Juju can use.

Juju (mostly) requires cloud-init to be present on the machines it starts, so that it can inject Juju-specific configuration and scripts to run on startup. Unforunately, there are no CentOS LXD images that have cloud-init already. To work around this, I wrote a standalone Go program to transform the linuxcontainers.org CentOS image: github.com/axw/juju-lxd-centos-image-builder. Eventually we hope to have pre-canned CentOS LXD images available to Juju, but for now you can use this program to prepare an image for Juju. Run it from the LXD host, and Juju will be able to use the resulting image.

llgo: classic snap

Wed, 15 Feb 2017 08:00:37 +0800

Some time ago I created a snap for llgo. Snaps are a relatively new packaging format; at the time I created the initial snap for llgo, it was only possible to create them confined by default. In particular, access to the user’s filesystem was blocked by default, which is unhelpful for a tool like a compiler. Because of that, the initial snap did not expose the llgo compiler, only the llgoi interpreter.

Now there is the concept of “classic snaps”, which use the snap packaging format but do not confine the application. The resulting package is much more like what you would get from a Debian package, but without the arcana.

I have released a new llgo classic snap, exposing the llgoi interpreter, and the llgo compiler and “go” tool wrapper. To install the llgo snap, run:

sudo snap install --classic llgo

The exposed “llgo” command is an alias for the go tool wrapper, so you can just run:

$ llgo install <package>
$ llgo run /path/to/source.go
...

as you would with the standard go tool. The llgo compiler itself can be invoked with the “llgo.compiler” command:

$ llgo.compiler -emit-llvm -S -o -o main.go
; ModuleID = 'main'
source_filename = "main"
target datalayout ...

The llgoi interpreter is still available, but as it is part of the same snap it is no longer confined:

$ llgo.llgoi
(llgo) import "fmt"
(llgo) fmt.Println("hello, world")
hello, world
13
<nil>

New in Juju 2.1: Prometheus Metrics

Mon, 06 Feb 2017 17:52:06 +0800

Juju is an application modelling tool, enabling “model-driven operations”. I won’t go into detail about what Juju is in this blog post, so if you’re new to Juju I suggest clicking on the link and reading a bit more.

Juju is a distributed application, with a “controller” cluster that manages cloud resources (machines, networks, volumes, etc.), and applications that use those resources. The controller cluster is currently based on top of MongoDB, utilising replica sets for data replication and leadership election.

As well as the controller cluster, Juju agents run on every virtual machine that the controller manages. The controllers, and those agents, each run many fine-grained, but dedicated “workers”. For example, each agent runs a worker to detect block devices and publish that information to the controller cluster; each controller runs a worker to maintain the replica sets in MongoDB.

Many things can go wrong in a distributed system. Network partitions can cause system-wide failures. Bad actors (badly written; less often, malicious) may starve others of resources. Failure to release memory or file handles leads to exhaustion, causing a DoS. Juju has seen its fair share of each of these problems.

To combat such issues, we have recently added Prometheus monitoring to Juju. As of Juju 2.1, Juju controllers and agents will export Prometheus metrics. There are two ways to get at them:

(on controllers) an HTTPS endpoint, https://…:17070/introspection/metrics.
(on Linux agents) an abstract domain socket, @jujud-machine-<machine-ID>

Configuring Prometheus to scrape Juju controllers

To configure Prometheus to scrape metrics from Juju controllers, you will need to add a new scrape target to Prometheus. The metrics endpoint requires authorisation, so you will need to configure a user and password for Prometheus to use:

$ juju add-user prometheus
$ juju change-user-password prometheus
new password: <password>
type new password again: <password>

For this new “prometheus” user to be able to access the metrics endpoint, you must grant the user read access to the controller model:

$ juju grant prometheus read controller

This gives the prometheus user just enough permission to read information on the controller, without allowing it to make changes, which would not be ideal for a monitoring application.

Juju serves the metrics over HTTPS, currently with no option of degrading to HTTP. You can configure your Prometheus to skip validation, or you can store the controller’s CA certificate in a file for Prometheus to verify the server’s certificate against:

$ juju controller-config ca-cert > /path/to/juju-ca.crt

We can now add a scrape target to Prometheus. Modify prometheus.yml, adding the following scrape target:

scrape_configs:
  job_name: juju
    metrics_path: /introspection/metrics
    scheme: https
    static_configs:
      targets: ['<controller-address>:17070']
    basic_auth:
      username: user-prometheus
      password: <password>
    tls_config:
      ca_file: /path/to/juju-ca.crt

Configuring Prometheus to scrape Juju agents

To expose the metrics of agents, you can deploy the juju-introspection charm onto that agent’s machine. For example, on machine 1, you would run:

juju deploy ~axwalk/juju-introspection --to 1

The metrics of that agent can then be obtained via:

http://<machine-1-address>:19090/agents/machine-1/metrics

Note that this is not an officially supported charm. The code for it is available at:

Old blog posts

Sat, 31 Dec 2016 15:30:35 +0800

I have finally gotten around to moving off Blogger, and over to using Hugo. The old blog posts have been imported, and are also accessible at my old Blogger site.

Availability Zones in Juju

Tue, 19 Aug 2014 10:49:00 +0000

You would be forgiven for thinking that I’d fallen off the face of the earth, considering how long it has been since I last wrote. I’ve been busy with my day job, moving into a new house; life in general. Work on llgo has been progressing, mostly due to Peter Collingbourne. I’ll have more to say about llgo’s progress in future posts.

This post is about some of the work I’ve done on Juju recently. Well, semi-recently; this post has been sitting in my drafts for a little while, waiting for the new 1.20.5 release to be announced.

Availability Zones in Juju

One of the major focuses of the Juju 1.20 release has been around high availability (HA). There are two sides to this: high availability of Juju itself, and high availability of your deployed services. We’ll leave the “Juju itself” side for another day, and talk about HA charms/services.

Until now, if you deployed a service via a charm with Juju, your cloud instance containing the service unit would be allocated wherever the cloud provider decided best. Most cloud providers split their compute services up into geographic regions (“us-east-1” in Amazon EC2, “US West” in Microsoft Azure, etc.). Some providers also break those regions down into “availability zones” (though the actual term may vary between providers, we use the term availability zone to describe the concept). An availability zone is essentially an isolated subset of a region.

If you’re developing an application that demands high availability, then you probably want to make sure your application is spread across availability zones. Some providers will guarantee a service level agreement (SLA) if you do this, such as on Microsoft Azure. Provided that you allocate at least two VMs to a “Cloud Service” on Azure, then you’re guaranteed a 99.95% uptime under the SLA and you get reimbursed if the guarantee isn’t met.

In Juju 1.20, there are two options for distributing your service units across availability zones: explicit (akin to machine placement) and automatic. So far we have enabled explicit availability zone placement in the Amazon EC2 and OpenStack (Havana onwards) providers, with support for the MAAS provider on the horizon. To add a new machine to a specific availability zone, use the “zone=” placement directive as below:

juju add-machine zone=us-east-1b

As well as support for explicit zone placement, we’ve implemented automatic spreading of services units across availability zones for Amazon EC2, OpenStack and Microsoft Azure. When cloud instances are provisioned, they will be allocated to an availability zone according to the density of the availability zone population for related instances. Two cloud instances are considered related if they both contain units of a common service, or if they are both Juju state servers.

To illustrate automatic spread, consider the mongodb charm. You’re going to use MongoDB as the datastore for your application, and you want to make sure the datastore is highly available; to do that, you’ll want to create a MongoDB replica set. It’s trivial to do this with the mongodb Juju charm:

juju deploy -n 3 mongodb

Wait a little while and you’ll have a 3-node MongoDB replica set. If a node happens to disappear, then the replica set will rejig itself so that there is a master (if the master was in fact lost) and everything should continue to work. If all the nodes go away, then you’re in trouble. This is where you want to go a step further and ensure your nodes are distributed across availability zones for greater resilience to failure. As of Juju 1.20, that “juju deploy” you just did handles that all for you: your 3 nodes will be uniformly spread across availability zones in the environment. If you add units to the service, they will also be spread across the zones according to how many other units of the service are in the zones. Let’s see what Juju did…

$ juju status mongodb | grep instance-id
instance-id: i-7a6d2b50
instance-id: i-ff1562d4
instance-id: i-627f0a30

$ ec2-describe-availability-zones
AVAILABILITYZONE us-east-1a available us-east-1
AVAILABILITYZONE us-east-1b available us-east-1
AVAILABILITYZONE us-east-1d available us-east-1

$ ec2-describe-instance-status i-7a6d2b50 i-ff1562d4 i-627f0a30 | grep i-
INSTANCE i-627f0a30 us-east-1d running 16 ok ok active
INSTANCE i-ff1562d4 us-east-1a running 16 ok ok active
INSTANCE i-7a6d2b50 us-east-1b running 16 ok ok active

(Note: the ec2-* commands are available in the ec2-api-tools package.)

Juju has distributed the mongodb units so that there is one in each zone, so if one zone is impaired the others will be unaffected. If we add a unit, it will go into one of the zones with the fewest mongodb units.

Explicit placement is currently only supported by Juju’s Amazon EC2 and OpenStack providers, but automatic spread is also supported by the Microsoft Azure provider. Due to the way that Microsoft Azure ties together availability zones and load balancing, it is currently necessary to forego “density” (i.e. explicit machine placement) in order to support automatic spread. If you are upgrading an existing environment to 1.20, then automatic spread will not be enabled. Newly created environments enable spread (and disable placement) by default, with an option to disable (availability-sets-enabled=false in environments.yaml).

Enjoy.

llgo on ssa

Mon, 06 Jan 2014 22:02:00 +0000

Hello there!

I’ve been busy hacking on llgo again. In case you’re new here: llgo is a Go frontend for LLVM that I’ve been working on for the past ~2 years on and off. It’s been quite a while since I last wrote; there has been a bunch of new work since, so I have some things to talk about at last.

A few months ago, I started working on rewriting swathes of llgo’s internals to base it on go.tools/ssa. LLVM uses an SSA representation, which made the process fairly straightforward. Basing llgo on go.tools/ssa gives me much higher confidence in the quality of the output; it also presented a good opportunity to clean up llgo’s source itself, which I have begun, but certainly not finished. llgo is now able to compile all packages in the standard library, except those that require cgo (net, os/user, runtime/cgo).

llgo now works something like this:

Go source is scanned and parsed by go/ast and go/parser, producing an AST;
The AST is fed into go/types, type-checking;
The output of go/types is passed onto go.tools/ssa, which generates the SSA form;
llgo translates the go.tools/ssa SSA form into an LLVM module,
llgo-build links the LLVM modules for a program together and translates to an executable.

go.tools/ssa supports translating a whole program to SSA form, but llgo works in the traditional way: packages are translated one at a time. Whole program optimisation is enabled by linking the LLVM modules together, prior to any translation to machine code.

There were a few bits that I stumbled on, when rewriting. Alan Donovan, the author of go.tools/ssa, was kind enough to give me some assistance along the way. Anyway, the main issues I had were:

Translating Phi nodes requires a bit of finessing, to ensure processing of the Phi or the edges is not order-sensitive. This was dealt with by generating placeholder values for instructions that haven’t yet been visited, and then replacing them later.
ssa.Index is emitted for indexing into arrays. If an array is in a register, then indexing it means extracting a value; in LLVM, an array element extraction requires a constant index. This is currently kludged by storing to a temporary alloca, and using the getelementptr LLVM instruction. Hopefully I’m missing something and this is easily fixed.
The Recover block is not dominated by the entry block, so it may not be valid for it to refer to the Alloc instructions for parameters and results. To deal with this, I generate a prologue block that contains the param/result Allocs; the prologue block conditionally jumps to either the recover or entry block, depending on panic/recover control flow. Alan has agreed to do something along these lines in go.tools/ssa.
The ssa.Next instruction required some assumptions to be made about block ordering and instruction placement, in order to be able to translate string-range using Phi nodes. Recent changes to go.tools/ssa exposed the dominator tree, making it possible to do away with the assumption now.

Various significant changes have been made during the course of the migration to go.tools/ssa:

Interfaces are now represented like in gc: empty interfaces with the runtime type & data, non-empty interfaces with an “itab” and data. Russ Cox wrote an article about the interface representation back in 2009.
Panic/recover (and defer, by consequence) are now using setjmp/longjmp. I had been using exceptions, but it was rightly pointed out to me that this wouldn’t work unless there were a way of doing non-call exceptions in LLVM (which has not been implemented). The setjmp/longjmp approach incurs a cost for every function that may defer or recover, but it works without modifications to LLVM. Perhaps this will be revisited in the future.
go/types/typemap is now used for mapping types.Types to runtime type descriptors and LLVM types. Runtime type descriptors are now generated more completely, and more correctly. Identical type descriptors will now be merged at link-time.
llgo no longer generates conditional branching for calls to non-global functions, when comparing structs, or in map iteration. Apart from producing better code, this makes it much simpler to work with go.tools/ssa, which has its own idea about how the SSA basic blocks relate to one another.

There have also been miscellaneous bug fixes, and improvements, not directly related to the move to go.tools/ssa. Some highlights:

A custom importer/exporter, thanks to Fredrik Ehnbom. The importer side is disabled at the moment, due to an apparent bug in go.tools/ssa.
Debugger support, thanks again to Fredrik Ehnbom. I haven’t reenabled it since the move to go.tools/ssa. I’ll get onto that real soon now, because debugging without it can be tiresome.
llgo-build can now take a “-test” flag that causes llgo-build to compile the test Go files, yet again thanks to Fredrik Ehnbom. This is currently reliant on the binary importer being enabled, so it won’t work out of the box until that bug is fixed.
Shifts now generate correct values for shifts greater than the width of the lhs operand.
Signed integer conversions now sign-extend correctly.
bytes.Compare now works as it should (-1, 0, 1, not <0, 0, >0). “llgo-build -test bytes”: PASS
llgo-build can now take a “-run” flag that causes llgo-build to execute and then dispose of the resulting binary.
Type strings are propagated to LLVM types, making the IR more legible, thanks to Travis Cline.

I think that’s everything. I have various things I’d like to tackle now, but not enough time to do it all at once. If you’re interested in helping out then there’s plenty to do, including:

Move to using libgo. Ideally the gc runtime would be rewritten in Go already, but that’s not going to happen just yet. The compiler and linker are due to be rewritten in Go soon, which is a lot of work as it is.
Finish off runtime type descriptor generation (notably, type algorithms).
Get PNaCl support working again. This should be pretty close, but requires the binary importer to be enabled.
Implement cgo support.
Implement bounds checking, nil pointer checks, etc.
Get garbage collection working. There’s Pull Request #108, but this is perpetuating the problem that is llgo’s custom runtime. Since GC is fairly invasive, I don’t want to go tying llgo to that runtime any more than it is currently. I expect this will have to wait until libgo is integrated.
Escape analysis. This is a must-have, but not immediately necessarily. The implementation should be based on go.tools/ssa, interfacing with the exporter/importer to record/consume information about external functions.
Make use of go.tools/ssa/ssautil/Switches. This is an optimisation, so again, not immediately necessary.

If you want to have a play around, then grab LLVM and Go, and then:

go get github.com/axw/llgo/cmd/llgo-dist && llgo-dist
llgo-build <some/package> or llgo-build file1.go, file2.go, …

Let me know how you get on with that.

Here’s hoping 2014 can be a productive year for llgo. Happy new year.

Cheers,
Andrew

llgo update #14

Fri, 16 Aug 2013 22:55:00 +0000

Ahoy there, mateys!

It’s been three months since our last correspondence. Apologies for the negligence. I’ve been busy, as usual, but it’s more self-inflicted than usual. I’ve taken up a new role at Canonical, working on Juju. I’m really excited about Juju (both the concept and realisation), and the fact that it’s written in Go is icing on the cake. Working remotely is taking some getting used to, but so far it’s been pretty swell. Anyway, you didn’t come here to read about that, did you?

I’m still working on llgo in the background, quietly prodding it along towards the 0.1 milestone. There’s just one big ticket item left, and that’s partially done now: channels. I’ve just finished porting the basics of channels from gc’s standard library to llgo’s runtime. That doesn’t include select, which is entirely missing. When that’s done, I’ll be content to release 0.1.

So what’s new since last time?

There’s a new llgo-build tool, which takes the pain out of building packages and programs with llgo and the LLVM toolchain. Just run “llgo-build <package>”, and you’ll either build and install a package, or build a program in the working directory. There’s no freshness checking, so you’re currently required to manually build all dependencies before building a program.
Simplified building against PNaCl: llgo-dist now accepts a “-pepper” option, which points to a NaCl SDK.
Implemented support for map literals.
Implemented complex number arithmetic.
Implemented channels (apart from anything select-related)
Numerous bug fixes.

In my previous post I talked about having implemented panic/recover, and having implemented them in terms of DWARF exception handling. Well, it looks like PNaCl isn’t going to support that, at least initially, so a setjmp/longjmp version is likely inevitable now.

I also said I would be working on a temporary for of cmd/go. I gave up on that, after hitting a few stumbling blocks. I figured it was more important to actually get the compiler and runtime working than get bogged down in the tooling, hence the simpler llgo-build tool.

That’s about it! “Feature complete” is getting closer, though lots of things still don’t work very nicely. Still no garbage collection, no proper escape analysis, etc. Those will come in time.

For now, though… I think I might go catch up on some sleep.

llgo on Go 1.1

Sat, 18 May 2013 21:28:00 +0000

Hi folks,

(For those of you coming from HN/Twitter/elsewhere, this is a post about llgo. llgo is an LLVM frontend for the Go programming language).

In my last post I mentioned that work had began on moving to Go 1.1 compatibility; this has been my primary focus since then. Since Go 1.1 is now released (woohoo!), I’ve gone ahead and pulled all the changes back into the master branch on GitHub. If you want to play around, you can do the following:

Get Go 1.1.
Get Clang and LLVM (I’ve tested with 3.2, Ubuntu x86-64). Make sure llvm-config is in your $PATH.
Run “go get github.com/axw/llgo/cmd/llgo-dist”
Run “llgo-dist”. This will install llgo into $GOBIN, and build the runtime.

The biggest new feature would have to be: defer, panic and recover (I’m lumping them together as they’re closely related). I’ve implemented them on top of LLVM’s exception handling support. The panic and recover functions are currently tied to DWARF exception handling, though it’s simple enough that it should be feasible to use setjmp/longjmp on platforms where DWARF exception handling isn’t viable.

Aside from that, there’s some new bits and bobs:

Method sets are handled properly now (or at least not completely wrong like before). This means you can use a embedded types’ methods to satisfy an interface.
“return” requirements are now checked by go/types
cap() is now implemented for slices.
llgo-dist now builds against the LLVM static libraries (if available) by default now, with an option for building against the shared libraries.

I’ll be working on a temporary fork of cmd/go to build programs with llgo, while a long-term solution is figured out. I’d also like to get PNaCl integration working again, given that its release is nigh.

That’s all for now.

llgo update #12

Fri, 01 Mar 2013 21:50:00 +0000

Oh my, it’s been a while.

In my previous post I wrote about llgo and PNaCl. I haven’t had much time to play with PNaCl recently, but I have been prodding llgo along. In February, my wife gave birth to our son, Jeremy, so naturally I’ve been busy. But anyway, let’s talk about what has been happening in llgo. Quick, while he’s sleeping!

Feature-wise, there’s nothing terribly exciting going on. Without getting too boring, what’s new is:

A new “go1.1” branch in the Git repository. The go1.1 branch aims to make llgo compatible with the Go tip, and will replace the master branch when Go 1.1 is released.
Removed llgo/types (a fork of the old exp/types package), and moved to go/types.
Updated runtime type representations to match those from gc’s tip (thanks to minux for initiating this effort).
Updated to use architecture-specific size for “int” (same as uintptr).
Changed function representation to be a pair of pointers, to avoid trampolines/runtime code generation for closures. The rationale is the same as for rsc’s proposal for Go 1.1; using runtime code generation limits the environments that Go can run in (e.g. PNaCl).
A slew of bug fixes and minor enhancements.

The go/types change in particular was not a small one, but llgo came out much better at the end. As of the most recent go/types commits, llgo now passes all of its tests in the go1.1 branch. Now I can get back to implementing features again.

That’s about all there is to report. It has been suggested that I set up some milestones in the GitHub project; I will spend a bit of time coming up with what I think are the bare essentials for a 0.1 release, and what would constitute future releases and so on.

One last thing: there’s a new(ish) llgo-dev mailing list. If you want to get involved, or just lurk, come and join the party.

Until next time.

Go in the Browser: llgo does PNaCl

Sun, 09 Dec 2012 12:45:00 +0000

Last week I briefly reported on Google+ that I had written a Go-based Native Client module, built it with llgo, and successfully loaded it into Google Chrome. I’d like to expand on this a little now, and describe how to build and run it.

Before your start…

If you want to want to try this out yourself, then you’ll need to grab yourself a copy of the Native Client SDK. I’ve only tested this on Ubuntu Linux 12.10 (x86-64), so if you’re trying this out on a different OS/arch you may need to alter the instructions.

Anyway, grab the SDK according to the instructions on the page I linked to above. Be sure to get the devevelopment/unstable branch, by updating with the “pepper_canary” target:

$ cd nacl_sdk; ./naclsdk update pepper_canary

This is not a small download, so go and brew some tea, or just read on to see where we’re going with this.

The anatomy of a PNaCl module

By now I guess you probably know whatNative Client is, but if you don’t, I suggest you take a moment to read about it on the Google Developers (https://developers.google.com/native-client/) site.What may not be so well known is PNaCl, the next evolution of Native Client. PNaCl (pronounced pinnacle), is short for PortableNative Client, and is based on LLVM.

Developers continue to write their code the same as in traditional NaCl, but now it is compiled to LLVM bitcode; PNaCl restricts usage to a portable subset of bitcode so that it can then be translated to native x86, x86-64, or ARM machine code. To compile C or C++ modules to PNaCl/LLVM bitcode, one uses the pnacl-clang compiler provided with the Native Client SDK.

To make use of Native Client, one develops a module, which is an executable, that can be loaded into Google Chrome (or Chromium). A module implements certain functions specified in the Pepper API (PPAPI), which is the API that interfaces your module with the browser. One of the functions is PPP_InitializeModule, and another is PPP_GetInterface. The former provides a function pointer to the module for calling back into the browser; the latter is invoked to interrogate the module for interfacesthat it implements.

**Anacl/ppapi package for Go**

Since llgo speaks LLVM, it should be feasible to write PNaCl modules in Go, right? Right! So I set about doing this last week, and found that it was fairly easy to do. I have written a demo module which you can find here: https://github.com/axw/llgo/tree/master/pkg/nacl/ppapi, which I later intend to morph into a reusable Go package, with a proper API. I have made a lot of shortcuts, and the code is not particularly idiomatic Go; bear in mind that llgo is still quite immature, and that this is mostly a proof of concept.

Most of the code in the package is scaffolding; the example module is mostly defined in example.go, some also in ppapi.go. At the top of example.go, we instantiate a pppInstance1_1, which is a structure which defines the “Instance” interface. This interface is used to communicate the lifecycle of an instance of the module; when a module is loaded in a web page, then this interface is invoked. We care about when a module instance is created, and when it is attached to a view (i.e. the area of the page which contains the module). Note that when I say interface, I mean a PPAPI interface, not a Go interface. Later, I hope to have modules implement Go interfaces, and hide the translation to PPAPI interfaces.

The example is contrived, and quite simple; it demonstrates the use of the Graphics2D interface, which, as the name suggests, enables a module to perform 2D graphics operations. The demo simply draws repeating rectangles of different colours, animated by regularly updating the graphics context and shifting the pixels on each iteration. I would have used the standard “image” Go package, but unfortunately llgo is currently having trouble compiling it. I’ll look into that soon.

Building llgo

Alright, how do we build this thing? We’re going to do the following things:

Build llgo, and related tools.
Compile the PNaCl-module Go code into an LLVM module.
Link the llgo runtime into the module.
Link the ppapi library from the Native Client SDK into the module.
Translate the module into a native executable.*

The final step is currently necessary, but eventually Chrome/Chromium will perform the translation in the browser.

Let’s begin by building the llgo-disttool. This will be used to build the llgo compiler, runtime, and linker. More on each of those in a moment. Go ahead and build llgo-dist:

$ go get github.com/axw/llgo/cmd/llgo-dist

The llgo-dist tool takes two options: -llvm-config, and -triple. The former is the path to the llvm-configtool, and defaults to simply “llvm-config” (i.e. find it using PATH). The latter is the LLVM target triple used for compiling the runtime package (and other core packages, like syscall). The Native Client SDK contains an llvm-config and the shared library that we need to link with to use LLVM’s C API.

As I said above, I’m running on Linux x86-64, so for my case, the llvm-config tool can be found in:

$ nacl_sdk/pepper_canary/toolchain/linux_x86_pnacl/host_x8664/bin/llvm-config

At this point, you should put the “host<arch>/bin” directory in your PATH, and the “host_<arch>/lib” directory in your LD_LIBRARY_PATH, as llgo currently requires it, and I refer to executables without their full paths in some cases.

The Native Client SDK creates shared libraries with the target armv7-none-linux-gnueabi, so we’ll do the same. Let’s go ahead and build llgo now.

$ llgo-dist -triple=armv7-none-linux-gnueabi -llvm-config=nacl_sdk/pepper_canary/toolchain/linux_x86_pnacl/host_x86_64/bin/llvm-config

We now have a compiler, linker, and runtime. As an aside, on my laptop it took about 2.5s to build, which is great! The gc toolchain is a wonderful thing. You can safely ignore the warning about “different data layouts” when llgo-dist compiles the syscall package, as we will not be using the syscall package in our example.

Building the example

Now, let’s compile the PNaCl module:

$ llgo -c -o main.o -triple=armv7-none-linux-gnueabi llgo/pkg/nacl/ppapi/.go llgo/testdata/programs/nacl/example.go

This creates a file called “main.o”, which contains the LLVM bitcode for the module. Next, we’ll link in the runtime. Eventually, I hope that the “go” tool will be able to support llgo (I have hacked mine up to do this), but for now you’re going to have to do this manually.

$ llgo-link -o main.o main.o $GOPATH/pkg/llgo/armv7-none-linux-gnueabi/runtime.a

Now we have a module with the runtime linked in. The llgo runtime defines things like functions for appending to slices, manipulating maps, etc. Later, it will contain a more sophisticated memory allocator, a garbage collector runtime, and a goroutine scheduler.

We can’t translate this to a native executable yet, because it lacks an entry point. In a PNaCl module, the entry point is defined in a shared library called libppapi_stub.a,which is included by the libppapi.a linker script. We can link this in using pnacl-clang, like so:

$ pnacl-clang -o main.pexe main.o -lppapi

This creates a portable executable (.pexe), an executable still in LLVM bitcode form. As I mentioned earlier, this will eventually be the finished product, ready to load into Chrome/Chromium. For now, we need to run a final step to create the native machine code executable:

$ pnacl-translate -arch x86-64 -o main_x86_64.nexe main.pexe

That’s it. If you want to load this in an x86 or ARM system, you’ll also need to translate the pexe to an x86 and/or ARM nexe. Now we can run it.

Loading the PNaCl module into Chrome

I’m not sure at what point all the necessary parts became available in Chrome/Chromium, so I’ll just say what I’m running: I have added the Google Chrome PPA, and installed google-chrome-beta. This is currently at version 24.0.1312.35 beta.

By default, Chrome only allows Native Client modules to load from the Chrome Web Store, but you can override this by mucking about in about:flags. Load up Chrome, go to about:flags, enable “Native Client”, and restart Chrome so the change takes effect. Curiously, there’s a “Portable Native Client” flag; it may be that the translator is already inside Chrome, but I’m not aware of how to use it.

To simplify matters, I’m going to hijack the hello_world example in the Native Client SDK. If you want to start from scratch, refer to the Native Client SDK documentation. So anyway we’ll build the hello_world example, then replace the executable with our own one.

$ cd nacl_sdk/examples/hello_world

$ make pnacl/Release/hello_world.nmf

$ cp <path/to/main_x86_64.nexe> pnacl/Release/hello_world_x86_64.nexe

Now start an HTTP server to serve this application (inside the hello_world directory):

$ python -m SimpleHTTPServer

Serving HTTP on 0.0.0.0 port 8000 …

Finally, navigate to the following location:

http://localhost:8000/index_pnacl_Release.html

Behold, animated bars! Obviously the example is awfully simplistic, but the I wanted to get this out so others can start playing with it. I’m not really in the business of fancy graphics, so I’ll leave more impressive demos to others.

Next Steps

I’ll keep dabbling with this, but my more immediate goals are to complete llgo’s general functionality. As wonderful as all of this is, it’s no good if the compiler doesn’t work correctly. Anyway, once I do get some more time for this, I intend to:

Clean up nacl/ppapi, providing an external API.
Update llgo-link to transform a “main” function into a global constructor (i.e. an “init” function) when compiling for PNaCl.
Update llgo-link to link in libppapi_stub.a when compiling for PNaCl, so we don’t need to use pnacl-clang. Ideally we should be able to “go build”, and have that immediately ready to be loaded into Chrome.
Get the image package to build, and update nacl/ppapi to use it.
Implement syscall for PNaCl. This will probably involve calling standard POSIX C functions, like read, write, mmap, etc. Native Client code is heavily sandboxed, but provides familiar POSIX APIs to do things like file I/O.

If you play around with this and produce something interesting, please let me know.

That’s all for now – have fun!

llgo update #10: "hello, world!" redux

Sun, 25 Nov 2012 14:28:00 +0000

It’s about time for another progress update on llgo. I’ve made decent progress recently, so let’s go through what’s new.

Highlights

I’ve been refactoring bits of code and fixing bugs aplenty, so there is a mass of noise in the git commits. In terms of new function, the news is that we now have:

Type switches.
Type assertions.
Labeled statements; goto, labeled break and continue.
The llgo-dist command; more on this below.
String conversions: to/from byte slices; from rune/int.
String range. I’m sure the implementation could be improved.
Implemented sync/atomic using LLVM atomic operations intrinsics.
Various changes to enable linking multiple packages (e.g. exported symbols are now prefixed with their package path).
Additional support for floats (thanks to spate); partial support for complex numbers.
”…args” calls to variadic functions (including slice append).
A self-contained runtime package. I have cloned (and slightly modified in some cases) the Go portion of the runtime package from gc, and combined it with the runtime code I had already written for llgo.
Bridge code for the math package, which mostly just redirects the exported functions to the internal, pure-Go implementations.
System calls (Linux/AMD64 only so far).
Closures; more below.

llgo-dist

I have begun implementing a command that takes care of building llgo, its runtime, and in the future any other tools that might be considered part of llgo (e.g. an in-development linker). This tool will set up the cgo flags given the path to an “llvm-config” program, and build gollvm.

reflect, fmt, oh my!

Last week, I mentioned on Google+ that I managed to get the reflect package working. At least enough of it to get the fmt package to work. At least enough of the fmt package to get fmt.Println(“Hello, world!”) to work… Yep, the holy grail of programming examples now compiles, links, and runs, using llgo. This demonstrates the following things work:

Compilation of the following packages: errors, io, math, os, reflect, runtime, strconv, sync, sync/atomic, syscall, time, unicode/utf8, unsafe.
Package imports (still using the gcimporter from exp/types.)
Linking multiple compiled packages using llvm-link.
Interfaces and reflection (fmt.Println uses reflection to determine the underlying type).
System calls (fmt.Println will eventually issue a system call to write to the stdout file).

Closures

Yes indeed, we now have closures. The code is pretty hackish, so I expect it’s not very solid. I have implemented them using LLVM’s trampoline intrinsics. Essentially you provide LLVM with a function that takes N parameters, give it a block of (executable) memory and an argument to bind, and it fills in the block with function code for a function with N-1 parameters (the Nth one being bound).

Unfortunately I have found that the closures are not playing nicely with lli/JIT, which means the closure unit test I have written fails. If I compile it with llc/gcc, though, it works just fine. So either I’ve done something subtly stupid, or the JIT is clobbering something it shouldn’t. As far as I got with debugging was finding that the bound argument value is wrong when the function is entered.

I expect I’ll probably replace this implementation for a couple of reasons:

Portability: I’d rather avoid platform-specific code like this. For one thing, the PNaCl ABI calls out trampoline intrinsics as being unsupported.
Testability: I should investigate the problems I observed with lli/JIT further, and I’m loath to change implementation to support tests, it is a real problem. I rely heavily on tests to make sure I haven’t broken anything.

Until I find out that using trampolines has a marked benefit to performance in real programs, I intend to replace the current implementation with one that uses a pair of pointers for functions. The bound argument will stored in one pointer, and the function pointer in another. This has implications for all function calls, though it should be simple to achieve good performance in most cases.

What’s next?

Haven’t figured this one out yet. I have been meaning to play more with PNaCl, so I might take some time now to do that. I expect I’ll be slowing down development considerably early 2013, as (a) we’re knocking down our place and rebuilding, and (b) my second child is on the way. I hope to have llgo in a better state for contributions by then, so others can pick up the slack.

I expect in the near future I’ll start playing with clang/cgo integration, as I start playing with PNaCl. I’ll write back when I have something to demonstrate.

Until then.

llgo update, milestone

Sun, 09 Sep 2012 18:26:00 +0000

In between gallivanting in Sydney, working, and organising to have a new house built, I’ve squeezed in a little bit of work on llgo. If you’ve been following along on Github, you’ll have seen that things have progressed a bit since last time I wrote.

Aside from a slew of bug fixes and trivialities, llgo now implements:

Slice operations (make, append, slice expressions). I’ve only implemented single-element appends so far, i.e. No append(s, a, b, c, …) or (s, abc…) yet.
Named results in functions.
Maps - creation, indexing, assignment, and deletion. The underlying implementation is just a dumb linked-list at this point in time. I’ll implement it as a hash map in the future, when there aren’t more important things to implement.
Range statements for arrays, slices and maps. I haven’t done strings yet, simply because it requires a bit more thought into iterating through strings runes-at-a-time. I don’t expect it’ll be too much work.
Branch statements, except for goto. You can now break, continue, and fallthrough.
String indexing, and slicing.
Function literals. Once upon a time these were working, but they haven’t been for a while. Now they are again. Note that this does not include support for closures at this stage, so usefulness is pretty limited.

Early on in the development of llgo, I decided that rather than implementing the compiler by going through the specification one item at a time, I’d drive the development by attempting to compile a real program. For this, I chose maketables, a program from the unicode standard library package. As of today, llgo can successfully compile the program. That is, it compiles that specific file, maketables.go. It doesn’t yet compile all of its dependencies, and it certainly doesn’t link or produce a usable program.

So now I’ll be working towards getting all of the dependencies compiling, then linking. In the interest of seeing usable progress, I think I might now take a bottom-up approach and start focusing on the core libraries, like runtime and syscall. I’ll report back when I have something interesting to say.

gocov, llgo update

Sat, 21 Jul 2012 22:09:00 +0000

I guess it’s time for a quick update. I’m not very diligent with this blogging thing; too busy having fun, programming. Sorry about that!

Introducing gocov

A couple of weeks ago I announced gocov, a coverage testing tool for the Go programming language. I wrote gocov to quickly get an idea of how broadly tested packages are (namely exp/types, which I’m working on in the background). The tool itself is written in Go, and works by source instrumentation/transformation. Currently gocov only does statement coverage.

Using gocov is relatively simple (if I do say so myself). First, you install gocov by running:

go get github.com/axw/gocov/gocov

This will install the gocov tool into your $GOPATH/bin directory. Once you have it installed, you can test a package (i.e. run its tests, and generate coverage data), by running:

gocov test <path/to/package>

Under the covers, this will run “go test <path/to/package>”, after having gone through the process of instrumenting the source. Once the tests are complete, gocov will output the coverage information as a JSON structure to stdout. So you might want to pipe that output somewhere…

Once you’ve got the coverage information, you’ll probably want to view it. So there are two other gocov commands: report, and annotate. The report command will generate a text report of the coverage of all the functions in the coverage information provided to it. For example:

gocov test github.com/axw/llgo/types | gocov report

… will generate a report that looks something like:

…
types/exportdata.go      readGopackHeader              69.23% (⁹⁄₁₃)
types/gcimporter.go      gcParser.expect               66.67% (⁴⁄₆)
types/gcimporter.go      gcParser.expectKeyword        66.67% (²⁄₃)
…

The annotate command will print out the source for a specified function, along with an annotation for each line that was missed. For example:

gocov test github.com/axw/llgo/types | gocov annotate - types.gcParser.expectKeyword

… will output the following:

266             func (p gcParser) expectKeyword(keyword string) {
267                     lit := p.expect(scanner.Ident)
268                     if lit != keyword {
269 MISS                        p.errorf(“expected keyword %s, got %q”, keyword, lit)
270                     }
271             }

As is often the case when I write software, I wrote gocov for my own needs; as such it’s not terribly featureful, only doing what I’ve needed thus far. If you would like to add a feature (maybe HTML output, or branch coverage), feel free to send a pull request on the Github repository, and I’ll take a gander.

Anyway, I hope it’s of use to people. But not too many people, I don’t have time to fix all of my crappy code! (Just kidding, I have no life.)

Update on llgo: interface comparisons, exp/types
I don’t have a lot to report on this front, as I’ve been doing various other things, like that stuff up there, but I can share a couple of bits of mildly interesting news.

I’ve been working a little on the runtime for llgo, and I’m proud to say there’s now an initial implementation of interface comparison in the runtime. This involved filling in the algorithm table for runtime types, implementing the runtime equality function (runtime.memequal), and implementing a runtime function (runtime.compareI2I) to extract and call it. It probably doesn’t sound exciting when put like that, but this is something of a milestone.

By the way, if you want to actually use the runtime, you can do it like this:

Compile your program with llgo, storing the bitcode in file x.ll.

Compile llgo/runtime/.go with llgo, storing the bitcode in file y.ll.
Link the two together, using llvm-link: llvm-link -o z.ll x.ll y.ll

And you’re done. The resultant module, housed in z.ll, contains your program and the llgo runtime. Now you can concatenate strings and compare interfaces to your heart’s content. Eventually llgo will contain an integrated linker, which will rewrite symbol names according to package paths.

Finally, on exp/types: I submitted my first two CL’s. Some of my ideas for exp/types were ill thought out, so the first was rejected (fairly), and the second needs some rework. I’ll be writing up a design proposal document at some stage, to better document my rationale for changes. Anyway, I’ll keep plugging away…

Ade!

Unit-testing llgo's runtime

Sun, 03 Jun 2012 15:53:00 +0000

It’s been a while since I last wrote, primarily because I’ve been moving house and was without Internet at home during the process. It’s back now, but now I have Diablo III to contend with.

In my previous post I mentioned that I would create a new branch for working on the llgo runtime. I haven’t done that yet, though I haven’t broken the build either. Rather, I’ve introduced conditional compilation to gollvm for builds against LLVM’s trunk where unreleased functionality is required, e.g. LinkModules. This isn’t currently being used in llgo-proper, so I’ve gotten away without branching so far.

The tag for building gollvm with unreleased functions is “llvmsvn”, so to build gollvm with LLVM’s trunk, including the LinkModules function, do the following:

curl https://raw.github.com/axw/gollvm/master/install.sh -tags llvmsvn | sh

So I didn’t break “the build”, meaning you can still build gollvm/llgo without also building LLVM from source. I did, however, break the llgo unit tests, as they are using the new LinkModules function. If you want to run the unit tests without building LLVM from source, then you can comment out the call to llvm.LinkModules in llgo/utils_test.go; of course, you should expect failures due to the runtime not being linked in, but that doesn’t involve all tests.

What else is new?

I announced on golang-dev a couple of weeks ago that I intend to work on getting exp/types up to snuff. I’ve moved some of the type construction code out of llgo-proper into llgo/types (a fork of exp/types), and eliminated most of the llgo-specific stuff from llgo/types. I’ll need to set aside some time soon to learn how to use Mercurial and create some changelists.

A few weeks ago I started playing with llgo and PNaCl, to see how hard it would be to get something running in Chrome. It works (with the IR Translator/external sandbox anyway), but then llgo doesn’t really do much at the moment.

That’s all for now.