It’s been a while. I’ve implemented bits and pieces of the Go language: constants (though not properly handling arbitrary precision numbers at this stage), structs, functions (with and without receivers, as declarations and as literals), and goroutines. Much of the implementation is simplistic, not covering all bases. I want to get something vaguely useful working before I go down the path to completeness.

I intended to wait until I had something interesting to show off, but unfortunately I’ve hit a snag with LLVM.

Debug information might sound luxurious, but it’s how I’m intending to encode information about exported symbols, so I can implement imports. The gc compiler creates a header in archives that lists all of this information. LLVM provides a standard set of metadata for describing debug information, using DWARF descriptors.

So I figured I could build on the LLVM metadata, extending it to describe Go-specific information where necessary. The benefit here, aside from reusing an existing well-defined format, is that I’d get DWARF debugging information in my output binaries, something I’d eventually want anyway. Unfortunately it appears that there’s no C API for generating named metadata.

I’ll have a look at extending the LLVM C API now. Down the rabbit hole…

Update - 7 January 2012
I submitted a patch to LLVM to add support for adding named metadata to a module, which has been accepted. This will, presumably, be part of LLVM 3.1.

For a while now I’ve been lamenting the lack of progress in systems programming languages. I work primarily in C++ (professionally, anyway), a language which leaves a lot to be desired. The end result, assuming you know what you’re doing, is likely to be fast code. But getting there can be a tedious and precarious journey. Don’t get me wrong, I’m not a C++ hater. I just think we can do better.

So for the past few I’ve been storing away ideas for writing my own language. (Groan, another one.) The vast majority of languages these days seem to be JVM based, or some other VM such as the CLR. This is fine for many applications, but for a lot of systems programming you typically want to be much closer to the hardware. What I want is something with the ease and rapidity of development of, say, Python, with the power of a lower-level language such as C/C++.



The Easy: Ideas

So what are those ideas I’d stored away then? Well to be honest, many of them boil down to syntactic sugar (syntactic caramel?), so here are the more functional ones:

• Implicit Interfaces. The idea is that classes should be able to implement an interface without explicitly saying so, just by implementing the methods defined in the interface. Poor-man’s duck-typing if you will.
• Link Time Optimisation (LTO) should be the norm. Say you write a function in a library which does everything and the kitchen sink. If the calling program only uses a function from your library in a certain way, then the function should be optimised for that usage.
• Pure Functions. This kind of fits under LTO. I want simpler meta-programming: it should be possible to mark functions as being “pure”, which means (in my terminology) that the function either has no side-effects, or affects only temporary variables. Calls to these functions could then be evaluated at compile time, e.g. to calculate complex mathematical expressions. I guess this just comes under “fancy optimiser”?
• Pattern Matching. I haven’t really fleshed this one out, but I think it would be handy to be able to match functions not just based on signature, but based on the values of the arguments.
• No preprocessor, no headers. When code is distributed, the modules should be self describing, such as with Java’s compiled .class files. This would eliminate problems such as compiling against an old version of an interface, and linking against a new, incompatible implementation.
• For the love of God, no garbage collection. What can I say? I like my RAII.
• I’m not really sure what title to give this one, but I wanted to eliminate the problems you get with linking objects together that have been compiled with different options in C++ (e.g. with/out RTTI, exceptions, or perhaps against an entirely different STL/standard library.)

I’ve been on annual leave this week, so I’ve taken the opportunity to do some work on cmonster. I’ve added preliminary support for source-to-source translation by introducing a wrapper for Clang’s “Rewriter” API. My fingers have been moving furiously so it’s all a bit rough, but it does work.

The API flow is:

1. Parse translation unit, returning an Abstract Syntax Tree (AST).
2. Walk AST to find bits of interest.
3. Insert/replace/erase text in the original source, using the location stored in each declaration/statement/token.

int main(int argc, const char *argv[])
{
    if (argc % 2 == 0)
    {
        return 1;
    }
    else
    {
        return 0;
    }
}

int main(int argc, const char *argv[])
{
    const char *function = “main”;
    printf(“Entering %s:%s:%d\n”, function,
           FILE, LINE);
    if (argc % 2 == 0)
    {
        printf(“Leaving %s:%s:%d\n”, function,
               FILE, LINE);
        return 1;
    }
    else
    {
        printf(“Leaving %s:%s:%d\n”, function,
               FILE, LINE);
        return 0;
    }
    return 0;
}

#include <stdio.h>
int main(int argc, const char *argv[])
{
    printf(“Entering main at line 2\n”);
    if (argc % 2 == 0)
    {
        printf(“Returning from main at line 6\n”);
        return 1;
    }
    else
    {
        printf(“Returning from main at line 10\n”);
        return 0;
    }
}

I’ve been quietly beavering away at cmonster, and thought I should share an update.

Note: the changes I describe below are not yet part of a cmonster release. I’ll release something once I’ve stabilised the API and tested it more thoroughly. In the mean time you can pull the source from github.

In the last month or so I have been adding C/C++ parsing capabilities to cmonster, by exposing the underlying Clang parser API. I’ve been wanting a simple, scriptable framework for analysing and manipulating C++ source for a few years now. The reason for wanting such a thing is so that I, and others, can more rapidly develop tools for writing better C++, and eliminate some of the drudgery. I’ve only just made a start, but cmonster now provides an API for parsing a C/C++ source file, returning a Python object to inspect the result.

So what’s cmonster looking like now? We now have a preprocessor and a parser interface, the former now being incorporated into the latter. The parser interface will parse a single source file, and return its Abstract Syntax Tree (AST). As is typical with parsers, there are many classes involved to describe each declaration, statement, type, etc. So I’ve added Cython into the mix to speed up the process of defining Python equivalents for each of these classes.

Unfortunately Cython does not yet support PEP 384 (Py_LIMITED_API), so at the moment cmonster is back to requiring the full Python API, and thus must be rebuilt for each new release. I’ve had a tinker with Cython to get its output to compile with Py_LIMITED_API, and hope to provide a patch in the near future.

What’s next? Once I get the AST classes mapped out, I intend to introduce a source-to-source translation layer. I’m not entirely sure how this’ll work yet, but I think ideally you’d just modify the AST and call some function to rewrite the main source file. Elements of the AST outside of the main source file would be immutable. That’s the hope, but it may end up being something a little more crude, using Clang’s “Rewriter” interface directly to replace source ranges with some arbitrary string. I expect this will be a ways off yet, though.

I’ve been procrastinating on cmonster. I have some nasty architectural decisions to make, and I keep putting it off. In the mean time I’ve been working on a new little tool called “Mr. Hooker” (or just mrhooker).

Introducing Mr. Hooker

The idea behind mrhooker is very simple: I wanted to be able to write LD_PRELOAD hooks in Python. If you’re not familiar with LD_PRELOAD, it’s a mechanism employed by various UNIX and UNIX-like operating systems for “preloading” some specified code in a shared library. You can use this to provide your own version of native functions, including those in standard libraries such as libc.

Anyway, I occasionally find the need for an LD_PRELOAD library to change the behaviour of a program that I can’t easily recompile. Often these libraries will be throw-away, so it might end up taking just as long to write the LD_PRELOAD library. So I wrote mrhooker to simplify this.

It turns out there’s very little to do, since Cython (and friends) do most of the hard work. Cython is a programming language that extends Python to simplify building Python extensions. It also has an interface for building these extensions on-the-fly. So mrhooker doesn’t need to do much - it takes a .pyx (Pyrex/Cython source) and compiles it to a shared library using Cython. Mrhooker takes this, and some common code, and loads it into a child process using LD_PRELOAD.

Example - Hooking BSD Sockets


Let’s look at an example of how to use mrhooker. Hooks are defined as external functions in a Cython script. Say we want to hook the BSD sockets “send” function. First we’d find the signature of send (man 2 send), which is:

ssize_t send(int sockfd, const void *buf, size_t len, int flags);

cdef extern ssize_t send(int sockfd, char *buf, size_t len, int flags) with gil:
    …

cdef extern ssize_t send(int sockfd, char buf, size_t len, int flags) with gil:
    print “====> send(%r, %r, %r, %r)” % (sockfd, buf[:len], len, flags)
    real_send = dlsym(RTLD_NEXT, “send”)
    if real_send:
        with nogil:
            res = (<ssize_t()(int, void, size_t, int) nogil>real_send)(
                sockfd, buf, len, flags)
        return res
    else:
        return -1

# Import stuff from <dlfcn.h>
cdef extern from “dlfcn.h”:
    void dlsym(void, char)
    void* RTLD_NEXT

mrhooker <script.pyx> <command>

$ mkdir ~/.mrhooker
$ echo [default] > ~/.mrhooker/mrhooker.config

A couple of months ago I wrote about my foray into the world of Google App Engine. More recently, I’d gotten the itch again, and had some ideas of how to fix the problems I found when attempting to get Pushy to work in Google App Engine.

The root of most of the problems is that Google App Engine is stateless in nature. Server instances can be spun up or spun down without notice, and so we can’t store complex state, which Pushy really requires. So a couple of weeks ago I set to investigating implementing a server-initiated RPC mechanism that is asynchronous and (mostly) stateless.

How would it work? Well, earlier this year I read that ProtoRPC was released, which brought RPC services to Google App Engine. In our case, Google App Engine is the client, and is calling the agent - but we can at least reuse the API to minimise dependencies and hopefully simplify the mechanism. Okay, so we have a ProtoRPC service running on a remote machine, consumed by our Google App Engine application. How do they talk?

One thing I wanted to avoid was the need for polling, as that’s both slow and expensive. Slow in that there will necessarily be delays between polls, and expensive in that unnecessary polls will burn CPU cycles in Google App Engine, which aren’t free. Long-polling isn’t possible, either, since HTTP requests are limited to 30 seconds of processing time. If you read my last post, you probably already know what I’m going to say: we’ll use XMPP.

What’s XMPP? That’s the Extensible Messaging and Presence Protocol, which is the protocol underlying Jabber. It is also the primary protocol that Google Talk is built on. It’s an XML-based, client-server protocol, so peers do not talk directly to each other. It’s also asynchronous. So let’s look at the picture so far…

• The client (agent) and server (GAE application) talk to each other via XMPP.
• The agent serves a ProtoRPC service, and the GAE application will consume it.

• Allows users to define function macros in Python, inline.
• Allows users to define a callback for locating #include files, when the file can not be found in the specified include directories.
• Provides a Python API for iterating over tokens in the output stream.

py_def(factorial(n))
    import math
    return str(math.factorial(int(str(n))))
py_end

A way of detecting (or at least configuring) pre-defined macros and include paths for a target compiler/preprocessor. A standalone C preprocessor isn't worth much. It needs to act like or delegate to a real preprocessor, such as GCC.

A #pragma to define Python macros in source, or perhaps if I'm feeling adventurous, something like #pydefine.

A simple command line interface with the look and feel of a standard C preprocessor

A month or so ago I was brainstorming ideas related to Google App Engine (GAE), as I had been wanting a reason to play with it for a while. One idea that stuck was connecting a remote Python process to GAE via Pushy, so we could either control GAE or GAE could control the remote process. I’m still working on the C/Python Preprocessor thingy, but I took a break from that this weekend to look into the possibility of a GAE Pushy transport.

So yesterday morning I signed up for an account, and started tinkering. I had been reading the docs already, and I figured there were a few possible approaches:

• The obvious: use HTTP. This has one major drawback in that it is inherently synchronous and wholly driven by the client. Moreover, GAE only allows request handlers around 30s to complete, so no kind of fancy long-polling is possible here.
• Channel API. This sounds like the right kind of thing to use, but it’s aimed at interacting with Javascript in a webpage.
• XMPP. Huh? Isn’t that for instant messaging? Exactly. The client (remote Python process) and server (GAE) are peers in XMPP, and either one can initiate sending messages to the other. Let’s look into this a bit more…

I’ve been coming across more problems with Boost Wave. The current ones blocking me are:

• A lack of an efficient way to conditionally disable a macro. The “context policy” provides hooks for handling macro expansions, and its return value is meant to control whether the expansion takes place. It doesn’t work. I’ll write up a bug report when I get some time.
• Wave isn’t very forgiving about integer under/overflow. For example, the GNU C library’s header “/usr/include/bits/wchar.h” has the following tidbit to determine the sign of wide characters, which Boost Wave barfs on:

#elif L’\0’ - 1 > 0