March 9, 2011

Fabric and Pushy, together at last

A workmate of mine recently brought Fabric to my attention. If you’re not familiar with Fabric, it is a command-line utility for doing sys-admin type things over SSH. There’s an obvious overlap with Pushy there, so I immediately thought, “Could Pushy be used to back Fabric? What are the benefits, what are the downsides?”


A couple of fairly significant benefits:
  • Pushy does more than just SSH, so Fabric could conceivably be made to support additional transports by using Pushy (which uses Paramiko), rather than using Paramiko directly.
  • Pushy provides access to the entire Python standard library, which is largely platform independent. So you could do things like determine the operating system name without using “uname”, which is a *NIX thing. That’s a trivialisation, but you get the idea I’m sure.
One big fat con:
  • Pushy absolutely requires Python on the remote system (as well as SSH, of course, but Fabric requires that anyway.) So requiring Pushy would mean that Fabric would be restricted to working only with remote machines that have Python installed. Probably a safe bet in general, but not ideal.
How about using Pushy if Python is available, and just failing gracefully if it doesn’t? This turns out to be really easy to do, since Fabric and Pushy both use Paramiko. So I wrote a Fabric “operation” to import a remote Python module. Under the covers, this piggy-backs a Pushy connection over the existing Paramiko connection created by Fabric. I’ll bring this to the attention of the Fabric developers, but I thought I’d just paste it here for now.

First an example of how one might use the “remote_import” operation. Pass it a module name, and you’ll get back a reference to the remote module. You can then use the module as you would use the module as if you had done a plain old “import ”.

fabfile.py

from fabric_pushy import remote_import

def get_platform():
    platform = remote_import(“platform”)
    print platform.platform()

You just execute your fabfile as per usual, and the “remote_import” operation will create a Pushy connection to each host, import the remote Python interpreter’s standard platform module, and call its platform method to determine its platform name. Easy like Sunday morning…
    $ fab -H localhost get_platform
    [localhost] Executing task ‘get_platform’
    Linux-2.6.35-27-generic-i686-with-Ubuntu-10.10-maverick

    Done.
    Disconnecting from localhost… done.

fabric_pushy.py

from fabric.state import env, default_channel
from fabric.network import needs_host

import pushy
import pushy.transport
from pushy.transport.ssh import WrappedChannelFile

class FabricPopen(pushy.transport.BaseTransport):
    “””
    Pushy transport for Fabric, piggy-backing the Paramiko SSH connection
    managed by Fabric.
    “””

    def init(self, command, address):
        pushy.transport.BaseTransport.init(self, address)

        # Join arguments into a string
        args = command
        for i in range(len(args)):
            if “ ” in args[i]:
                args[i] = “‘%s’” % args[i]
        command = “ “.join(args)

        self.channel = default_channel()
        self.channel.exec_command(command)
        self.stdin  = WrappedChannelFile(self.channel.makefile(“wb”), 1)
        self.stdout = WrappedChannelFile(self.channel.makefile(“rb”), 0)
        self.stderr = self.__channel.makefile_stderr(“rb”)

    def del(self):
        self.close()

    def close(self):
        if hasattr(self, “stdin”):
            self.stdin.close()
            self.stdout.close()
            self.stderr.close()
        self.__channel.close()

# Add a “fabric” transport”, which piggy-backs the existing SSH connection, but
# otherwise operates the same way as the built-in Paramiko transport.
class pseudo_module:
    Popen = FabricPopen
pushy.transports[“fabric”] = pseudo_module

###############################################################################

# Pushy connection cache
connections = {}

@needs_host
def remote_import(name, python=“python”):
    “””
    A Fabric operation for importing and returning a reference to a remote
    Python package.
    “””

    if (env.host_string, python) in connections:
        conn = connections[(env.host_string, python)]
    else:
        conn = pushy.connect(“fabric:“, python=python)
        connections[(env.host_string, python)] = conn
    m = getattr(conn.modules, name)
   if “.” in name:
        for p in name.split(“.”)[1:]:
            m = getattr(m, p)
    return m