solid Python queueing?
OK, message queueing has become insufferably trendy. You don’t need to tell me, I’ve known it’s the bees knees for 4 years now ;)
The only problem is, there doesn’t seem to be a good queue broker written in Python. They’re in Java, Perl, more Perl, or Erlang, but a solid, reliable, persistent queueing backend in Python is nowhere to be found, as far as I can see. Work is a mainly-Python shop, and while we can deploy other languages to our production, staging and test grids easily enough, it’s a lot easier to do developer-desktop testing if we had an all-Python queue backend.
Am I missing one?
Tags: activemq, languages, messaging, mq, putplace, python, queueing, rabbitmq
Brian White said,
October 15, 2008 @ 5:45 pm
I know it’s not Python, but beanstalkd (http://xph.us/software/beanstalkd/) is pretty nice — and zero configuration makes it tolerable on a desktop machine.
There’s a Python client (http://code.google.com/p/pybeanstalk/), I didn’t find it particularly pythonic though but the protocol is easy enough if you want to roll your own (very similar to memcached).
Downsides; it’s C not Python, no persistence — but it’s on the way.
jose said,
October 15, 2008 @ 7:00 pm
have a look at msgbus:
http://code.google.com/p/msgbus/
“implements scalable server push via HTTP multipart/x-mixed-replace for local or remote messaging, and a minimal (optional) webserver.”
scales nicely.
Augusto Becciu said,
October 15, 2008 @ 7:56 pm
I’ve been using MorbidQ (http://www.morbidq.com/) for development. It’s light and simple to use on the desktop.
alexis said,
October 16, 2008 @ 8:35 am
RabbitMQ is very easy to set up and has several good Python clients. Also recently someone has released an integration with twisted (alpha quality but worth a look). Cheers, alexis
Justin said,
October 16, 2008 @ 11:31 am
Thanks for the comments! Very helpful.
@Brian: yeah, I saw beanstalkd — but the lack of persistence is bad news for some of our use cases. also, I forgot to mention, some of the developer desktops are running XP :( so beanstalkd is out on that count too.
@jose: msgbus looks interesting! Another Dug Song project… Dunno if it’d be a good candidate for this though due to the use of libevent, see above re XP, but worth keeping an eye on.
@Augusto: I think that’s the best option. Good to hear you’re using it, and it’s working for you; I was wondering if it had got beyond the “toy project” stage.
OK, here’s what I’m thinking.
MorbidQ for cheap, simple, built-in developer-desktop-based test MQ; no persistence. The lack of persistence is ok in this scenario; we wipe down the queues between test cases anyway.
Then RabbitMQ on deployment, with full persistence and the other nice stuff. Since both use STOMP, that should be doable… is anyone using the experimental rabbitmq-stomp plugin? http://hg.rabbitmq.com/rabbitmq-stomp/
(Alternatively, I may have to check out having my own i14y facade to shim over AMQP/STOMP. hopefully they’re close enough conceptually that this is ok.)
Pádraig Brady said,
October 16, 2008 @ 2:17 pm
Coincidentally I just noticed this RabbitMQ google tech talk in my inbox: http://www.youtube.com/watch?v=ZQogoEVXBSA
Michael Twomey said,
October 17, 2008 @ 1:39 pm
A bunch of us are evaluating RabbitMQ + python at the moment and so far its looking good (much better than ActiveMQ anyway). One big issue for us is we need .net interoperability so RabbitMQ’s .net client is a big win.
We’re looking at using stomp initially, since we’re using that already, but I expect we’ll switch to AMQP at some point.
A note on the rabbitmq-stomp plugin: get the latest mercurial version, there are a lot of fixes which make it work better with more stomp implementations.
Conor Delaney said,
October 17, 2008 @ 5:00 pm
Call me a Windows fan boy, but there is already a message queueing service built into Windows XP, that exposes its interface to .NET, so presumely Python can see it aswell? So if you are already using XP for development……
http://www.microsoft.com/windowsserver2003/technologies/msmq/whatsnew.mspx