solid Python queueing?

OK, message queueing has become insufferably trendy. You don’t need to tell me, I’ve known it’s the bees knees for 4 years now ;)

The only problem is, there doesn’t seem to be a good queue broker written in Python. They’re in Java, Perl, more Perl, or Erlang, but a solid, reliable, persistent queueing backend in Python is nowhere to be found, as far as I can see. Work is a mainly-Python shop, and while we can deploy other languages to our production, staging and test grids easily enough, it’s a lot easier to do developer-desktop testing if we had an all-Python queue backend.

Am I missing one?

This entry was posted in Uncategorized and tagged , , , , , , , . Bookmark the permalink. Both comments and trackbacks are currently closed.

8 Comments

  1. Brian White
    Posted October 15, 2008 at 17:45 | Permalink

    I know it’s not Python, but beanstalkd (http://xph.us/software/beanstalkd/) is pretty nice — and zero configuration makes it tolerable on a desktop machine.

    There’s a Python client (http://code.google.com/p/pybeanstalk/), I didn’t find it particularly pythonic though but the protocol is easy enough if you want to roll your own (very similar to memcached).

    Downsides; it’s C not Python, no persistence — but it’s on the way.

  2. jose
    Posted October 15, 2008 at 19:00 | Permalink

    have a look at msgbus:

    http://code.google.com/p/msgbus/

    “implements scalable server push via HTTP multipart/x-mixed-replace for local or remote messaging, and a minimal (optional) webserver.”

    scales nicely.

  3. Augusto Becciu
    Posted October 15, 2008 at 19:56 | Permalink

    I’ve been using MorbidQ (http://www.morbidq.com/) for development. It’s light and simple to use on the desktop.

  4. Posted October 16, 2008 at 08:35 | Permalink

    RabbitMQ is very easy to set up and has several good Python clients. Also recently someone has released an integration with twisted (alpha quality but worth a look). Cheers, alexis

  5. Posted October 16, 2008 at 11:31 | Permalink

    Thanks for the comments! Very helpful.

    @Brian: yeah, I saw beanstalkd — but the lack of persistence is bad news for some of our use cases. also, I forgot to mention, some of the developer desktops are running XP :( so beanstalkd is out on that count too.

    @jose: msgbus looks interesting! Another Dug Song project… Dunno if it’d be a good candidate for this though due to the use of libevent, see above re XP, but worth keeping an eye on.

    @Augusto: I think that’s the best option. Good to hear you’re using it, and it’s working for you; I was wondering if it had got beyond the “toy project” stage.

    OK, here’s what I’m thinking.

    MorbidQ for cheap, simple, built-in developer-desktop-based test MQ; no persistence. The lack of persistence is ok in this scenario; we wipe down the queues between test cases anyway.

    Then RabbitMQ on deployment, with full persistence and the other nice stuff. Since both use STOMP, that should be doable… is anyone using the experimental rabbitmq-stomp plugin? http://hg.rabbitmq.com/rabbitmq-stomp/

    (Alternatively, I may have to check out having my own i14y facade to shim over AMQP/STOMP. hopefully they’re close enough conceptually that this is ok.)

  6. Posted October 16, 2008 at 14:17 | Permalink

    Coincidentally I just noticed this RabbitMQ google tech talk in my inbox: http://www.youtube.com/watch?v=ZQogoEVXBSA

  7. Posted October 17, 2008 at 13:39 | Permalink

    A bunch of us are evaluating RabbitMQ + python at the moment and so far its looking good (much better than ActiveMQ anyway). One big issue for us is we need .net interoperability so RabbitMQ’s .net client is a big win.

    We’re looking at using stomp initially, since we’re using that already, but I expect we’ll switch to AMQP at some point.

    A note on the rabbitmq-stomp plugin: get the latest mercurial version, there are a lot of fixes which make it work better with more stomp implementations.

  8. Posted October 17, 2008 at 17:00 | Permalink

    Call me a Windows fan boy, but there is already a message queueing service built into Windows XP, that exposes its interface to .NET, so presumely Python can see it aswell? So if you are already using XP for development……

    http://www.microsoft.com/windowsserver2003/technologies/msmq/whatsnew.mspx