taint.org: Justin Mason’s Weblog

A while back, I linkblogged about “iotop”, a very useful top-like UNIX utility to show which processes are initiating the most I/O bandwidth.

Teodor Milkov left a comment which is well worth noting, though:

Definitely iotop is a step in the right direction.

Unfortunately it’s still hard to tell who’s wasting most disk IO in too many situations.

Suppose you have two processes - dd and mysqld.

dd is doing massive linear IO and its throughput is 10MB/s. Let’s say dd reads from a slow USB drive and it’s limited to 10MB/s because of the slow reads from the USB.

At the same time MySQL is doing a lot of very small but random IO. A modern SATA 7200 rpm disk drive is only capable of about 90 IO operations per second (IOPS).

So ultimately most of the disk time would be occupied by the mysqld. Still iotop would show dd as the bigger IO user.

He goes into more detail on his blog. Fundamentally, iotop works based on what the Linux kernel offers for per-process I/O accounting, which is I/O bandwidth per second, not I/O operations per second. Most contemporary storage in desktops and low-end server equipment is IOPS-bound (’A modern 7200 rpm SATA drive is only capable of about 90 IOPS’). Good point! Here’s hoping a future change to the Linux per-process I/O API allows measurement of IOPS as well…

Tags: disk, iops, iotop, kernel, linux, measurement, performance, storage

Introducing Karmic Koala, Ubuntu 9.10:

What if you want to build an EC2-style cloud of your own? Of all the trees in the wood, a Koala’s favourite leaf is Eucalyptus. The Eucalyptus project, from UCSB, enables you to create an EC2-style cloud in your own data center, on your own hardware. It’s no coincidence that Eucalyptus has just been uploaded to universe and will be part of Jaunty - during the Karmic cycle we expect to make those clouds dance, with dynamically growing and shrinking resource allocations depending on your needs.

A savvy Koala knows that the best way to conserve energy is to go to sleep, and these days even servers can suspend and resume, so imagine if we could make it possible to build a cloud computing facility that drops its energy use virtually to zero by napping in the midday heat, and waking up when there’s work to be done. No need to drink at the energy fountain when there’s nothing going on. If we get all of this right, our Koala will help take the edge off the bear market.

AWESOME — exactly where the Linux server needs to go. Eucalyptus is the future of server farms. Really looking forward to this…

Tags: ec2, eucalyptus, karmic-koala, linux, ubuntu, virtualization

Hey lazyweb! Long time, no write.

I’m wondering what setup people use to deal with the following situation. Upstairs, I have an Ubuntu 8.04 server with 71GB of MP3s. Downstairs, I have a stereo system. In between the two is a wireless network. How can I listen to the music downstairs, without simply copying the lot (or subsets thereof) onto a local disk on some appliance down there?

Currently, I’m using a VNC client on a Nokia 770 to control a JuK window on the server. This works great, believe it or not! KDE 3 can be coaxed into providing a fantastic UI for a small touchscreen. This then uses Pulseaudio to transmit the sound output using the ESD protocol over TCP to the ESD server on the N770, and the N770 plays back the sound.

Until a few months ago, this worked great. However, something (either hardware changes, network topology changes, or an upgrade to Ubuntu 8.04 on the server) has resulted in effective bitrates between the server and the N770 dropping frequently — hence the audio drops out or changes pitch, rendering it unlistenable :(

I’ve tried using UPNP servers (specifically mediatomb, ushare, and Twonkymedia), with the built-in Media Streamer app on the N770. All fail. MP3s cut off near the end, M3U playlists aren’t supported, and sometimes Media Streamer just locks up. In addition it’s pretty messy trying to get the UPNP servers to notice changes to the MP3 collection.

I’ve also tried using Squeezecenter (nee Slimserver), but the MP3 stream playback support on the N770 is pretty atrocious; there are audible decoding artifacts.

So — anyone got a suggestion? Even something involving iTunes might be helpful — as long as it can at least preserve the Linux server. I’m unlikely to host the full MP3 collection on anything else…

Tags: home, lazyweb, linux, mp3, music, networking

Here’s a nice little (totally subjective!) story for Linux users.

At home, I have a HP Laserjet 1018 printer; it’s a dinky little USB laser. When I was setting up my Mac running OSX, I attempted to use it.

A common refrain from Mac users is that MacOS X just works — attempt to get something working, and the Mac will do the right thing with little friction, compared to the Linux situation which will involve complex config file editing and what-not. If this experience is anything to go by, that’s not entirely the case anymore. In fact, the exact opposite applied; when I plugged the printer into the Linux box and ran System -> Administration -> Printing -> New Printer, it “just worked” and I wound up with a working network printer within seconds. No such luck with OSX. Some googling revealed the problem:

• “I kinda found what I need to install, but I can’t understand what to do“
• A thread at macosxhints.com
• And at macrumors.com
• And at discussions.apple.com

In summary, the LJ1018 is just not supported on MacOS X. In order to get it working you need to install a third-party port of the Linux printing components foo2zjs, Foomatic, and Ghostscript, ported to MacOS X, and then get busy with the config file editing and undocumented tweaking and what-not. Ouch.

So there you go. Linux: it just works! ;)

(By the way, I was able to work around it by printing from the Mac to the Linux print server in Postscript; the CUPS print server will transcode PS to the native format.)

Tags: hardware, linux, macosx, support, theswitch

So, that OSX thing. I’m afraid I’ve given up on the switch; I’m back on Linux. :(

I got the keyboard mapping working, but Focus-Follows-Mouse and the couple of window-management hotkeys I rely on were impossible to work around.

Focus-Follows-Mouse is emulated by iTerm, but every time you switch to an X11 app or to Firefox, a click is required. This app-specific behaviour is jarring and inconsistent.

For some reason, the window-management hotkeys had a tendency to break, or to be disabled by other hotkeys or apps. I never figured out exactly why.

In addition, OSX has a built-in tendency to hibernate once the laptop’s lid is closed. I wanted to disable this, for a number of reasons; most importantly, I tend to leave the laptop closed, leaning beside a chair in the TV room, while I’m at work, but there’s frequently something I want to SSH in for. I tried Caffeine.app to avoid this, but it failed entirely on my hardware. InsomniaX generally works, but for some reason it tends to turn itself off occasionally for rather random reasons (such as switching to battery power, no matter how briefly, then back again). This was the final straw.

So just over a week ago, I installed Ubuntu on the MacBook Pro, following the documentation on the Ubuntu Wiki. Everything worked!

The Wiki’s suggestions were a little hairy to configure — but then, the OSX experience had been, if anything, less easy. Plus, I know my way around a Linux /etc.

On the Linux side, the Avant Window Navigator is truly excellent, and rivals the Dock nicely, and the Baghira kwin theme gives a pretty good OSX sheen to KDE 3. It’s not quite as pretty as OSX, but I’m happy to lose some prettiness for better usability.

Regarding the interface — the current version of the Linux Synaptics driver supports multi-touch (Apple’s patents be damned, seemingly), and all the nice multi-touch tricks supported by most OSX apps work with it too. I’m still working out the optimum settings for this, but it’s very configurable, and quite open.

It’s fantastic ;) I feel like I’m home again. Sorry, Mac people.

(image: CC-licensed, thanks to Dr Craig)

Tags: apple, linux, macos, theswitch

• Memeorandum Colors: Visualizing Political Bias with Greasemonkey : nice hack
  (tags: hacks greasemonkey ui us-politics election)

• Xen/VPS hosting reviews : good list from Russell Coker. Gandi’s product is amazingly cheap at EUR6 for super-low-end nodes (UPDATE: that’s because the page is wrong! in reality it’s EUR10+VAT)
  (tags: gandi xen vps linux hosting reviews)

• Make-Believe Maverick : ‘A closer look at the life and career of John McCain reveals a disturbing record of recklessness and dishonesty.’ amazingly vicious hatchet-job on John McCain from _Rolling Stone_
  (tags: rolling-stone articles toread vicious john-mccain expose maverick tim-dickinson)

• Spam Volumes Plummet After Atrivo Shutdown : MessageLabs indicates a ’significant drop’ in spam/botnet activity, due to loss of C&C networks located at Atrivo
  (tags: messagelabs atrivo spam botnets cutwail mega-d securityfix)

• Marshal reckon the Storm botnet has dried up : ‘Spam originating from the Storm botnet suddenly dried up in mid-September. Since that time we have not detected a single Storm spam in our traps.’
  (tags: storm spam botnets malware marshal)

• Django test fixtures : handy; load initial test data into the database from JSON files. must start using these
  (tags: fixtures testing django web-apps sql json)

Tags: articles, atrivo, botnets, cutwail, django, election, expose, fixtures, gandi, greasemonkey, hacks, hosting, john-mccain, json, linux, malware, marshal, maverick, mega-d, messagelabs, reviews, rolling-stone, securityfix, spam, sql, storm, testing, tim-dickinson, toread, ui, us-politics, vicious, vps, web-apps, xen

Well, some bits of this are easy: here’s a MacOS X version of GVim and Vim, which works nicely, is easy to install, and is simply vim/gvim. Great stuff!

But some bits are harder. Remember I was complaining about that silly ± / § key in the top corner of UK/Irish MacBook Pro keyboards? Some investigation reveals that I’m far from alone in this:

‘it fucks up application switching’

‘I hate my MacBook Pro’

a forum post looking for help

another forum post

There are a number of apps that offer key remapping, but for no apparent reason they limit themselves to “popular” remappings only, such as swapping the Control and Caps-Lock keys etc. I presume this is because that was easy to code ;)

The one that does work fully is Ukelele. Watch out though — it comes with a raft of caveats. It’s buggy, at least dealing with my MBP keyboard under OSX 10.5.5; the “Copy Key” functionality doesn’t work, and you need to start using a key mapping file from the Ukelele package, not a system one or one you’ve downloaded, otherwise it’ll silently produce an output file that doesn’t recognise any keys at all. On top of this, each time you make changes, you need to log out and log back in again for them to try them out. (Small mercies: at least you don’t need to do a full reboot, I suppose.)

I’m not impressed by this whole keyboard issue. If you look at photos of the US MacBook Pro keyboard, it’s clear that it doesn’t have the stunted tetris-style Enter and Left-Shift keys that the UK/Irish one does. It also has the tilde key in the normal place, the top left, instead of some bizarre symbol that isn’t even used in this keyboard’s locale, and as Ash Searle noted, when you’re a developer, the # is a hell of a lot more useful than the £ symbol. They’ve basically screwed with a good US keyboard design to bodge in a few extra keys they needed to deal with the tricky European corner cases.

All that would be relatively minor, however, if I could remap the keys to suit my tastes — but it was pretty damn tricky to do that. Key remapping needs to be an easy feature!

I’m still working on the fixed key layout file, but I may post it here once it’s finished to save other Googlers the bother…

Update:: here’s the fixed key layout file:

Irish Fixed.keylayout

Save that to ~/Library/Keyboard Layouts/ , then open System Preferences -> International, select Input Menu, and choose Irish Fixed from the list, and ensure “Show input menu in the menu bar” is on. Close that window, then select “Irish Fixed” from the input menu left of the clock on the menu bar. Log out, and log back in again, and the keys should be sane…

(thanks to Sonic Julez for the MBP key image)

Tags: apple, keyboards, linux, mac, macbook, osx, switch

As previously noted, I’ve just bought myself a nice shiny MacBook Pro, to replace an old reliable 5-year-old Thinkpad T40, which ran Linux.

Initially, I was contemplating installing Linux on this one too, and dual-booting. But right now, I’ve decided to give MacOS X a go — why not? I find it’s worthwhile updating aspects of my quotidian computing environment every now and again, and it seems everyone’s doing it. ;) I’ll log my experience on this blog as I go along.

(Worth noting that this isn’t my first Mac; back in 1990, I was the proud owner of a free Macintosh Plus for a year, courtesy of TCD’s “Project Mac” collaboration with Apple Ireland. I wrote a great Mandelbrot Set explorer app.)

First off, the good news: the hardware is very nice indeed. It’s light in weight, esp. compared to my T61p work laptop, the screen clarity is fantastic, and the CPU fairly zooms along — unsurprisingly, given that the T40 was 5 years old.

In addition, the multi-touch touchpad is wonderful; I’m looking forward to lots more multi-touch features.

Unfortunately, some of the other hardware design decisions were pretty wonky. By default it’s quite tricky to keep the laptop running with the lid closed – it seems a decision was made to use passive cooling via the keyboard, so once the lid is closed, that heat cannot escape, causing overheating. There’s a third-party extension I can install to allow it anyway, but it’s festooned with warnings to overclock the fan speed to make up for it… ugh. Since I need the ability to be able to remotely login to my laptop from work if I should happen to forget something, or to kick off a long transfer before I come home, this means I have to leave the laptop open permanently, which I didn’t want to do.

In addition, I initially thought my brightness control was broken, since the laptop screen fluctuates in brightness continually. Turns out this is a feature, responding to ambient light — a poorly-documented one, but at least it’s easy to turn off in System Preferences once you know it’s there.

(Unfortunately, a lot of MacOS seems to consist of poorly-documented features that are hidden “for my own good”. The concept of switching seems to involve me abdicating a good deal of what I’d consider adult control of the machine, to the cult of Steve Who Knows Better. This is taking some getting used to.)

On to the software… what’s getting my goat right now are as follows:

Inability to remap keys (CapsLock key, the useless “+-” key, a lack of “spare” keys for scripted actions)

Up in the top left corner of “international” MacBook keyboards, there’s a useless key with a “+-” and double-S symbol on it. I don’t think I’ve ever typed those symbols in my entire life. I want a ~ there, since that’s where the ~ key lives, but for some reason, MacOS doesn’t include keyboard-remapping functionality to the same level as X11’s wonderful “xmodmap”. It seems this third-party app might allow me to do that, or maybe something called ‘KeyRemap4Macbook’?

This Tao Of Mac HOWTO seems helpful on how to support the “Home”/”End” keys, for external keyboard use.

Focus Follows Mouse

This is a frequent complaint among UNIX-to-Mac switchers. It seems that some apps do a hacky version of it, but then you’ve got this inconsistent thing where you lose track of which apps will automatically pick up focus (Terminal, iTerm) and which ones need a click first (Firefox, indeed everything else). Unfortunately, it seems an app called CodeTek VirtualDesktop would have fixed it, but seems to have been abandoned. :(

Programmable Hotkeys

I use a few hotkeys to do quick window-control actions without involving the mouse; in particular, F1 brings a window to the front, F2 pushes it to the back, F12 minimizes a window, Ctrl-Alt-LeftArrow moves a window half a screen left, and Ctrl-Alt-RightArrow moves a window half a screen to the right. Those are pretty simple, but effective.

This collection of Applescript files, in conjunction with Quicksilver, look like I may be able to do something similar on the Mac. Here’s hoping. LifeHacker suggests that the default for minimize is Cmd-M, so that’s what I need to remap from, at least…

This is a big issue — Dan Kulp had a lot of hot-key-related woes, and wound up going back to Linux as a result. Evan reported the same. I like the idea of MacOS, but my tendonitis-afflicted wrists need their little shortcuts; I’m not willing to compromise on avoiding mouse usage in this way.

(by the way, in order to get F1/F2/F12 back, check the “Use the F1-F12 keys to control software features” box in the Keyboard control panel. Thanks to this page for that tip; it has a few other good tips for UNIX switchers, too.)

Upgrades and Software

So, there’s two main contenders for the “apt-get for Mac” throne — Fink vs MacPorts. Fink takes the Debian approach of downloading binary packages, while MacPorts compiles them from source, BSD/Gentoo-style, on your machine. Since I’m not looking at the source, or picking build parameters, or auditing the code for security issues there and then, I don’t see the need to build it — Fink wins.

One thing though — the installer for Fink informed me that I needed to run “Repair Permissions”, which took a while, and found some things that had somehow already been modified from their system defaults, I’m not sure why. This left me slightly mystified. I then was later told that this is now considered ‘voodoo’. wtf.

Mind you, Daring Fireball suggests that the Mac software update are so poorly implemented that they require essentially rebooting in single-user mode, which sounds frankly terrifying. I hope that’s not the case.

BTW, it’s worth noting that IMO, AWN is as nice as — possibly nicer than — the Dock. ;)

Anyway, that’s post #1 in a series. Let’s see how I get on from here. (thanks to Aman, Craig and Paddy for various tips so far!)

Tags: apple, linux, macos, osx, switching

• Linux x86_64 frozen by heavy I/O on Dell PowerEdge 2950 : starting to think we may be running into this on our build machine; annoying. bookmarking for future reference
  (tags: poweredge dell hardware linux drivers performance sysadmin)

• Wikipedia:Articles for deletion/Deletionpedia : hahaha. WP deletion gnomes argued that Deletionpedia should not have an entry due to non-notability, just 24 minutes after that entry was created
  (tags: wp wikipedia funny deletion processes bureaucracy deletionpedia)

• Atrivo/InterCage depeered : the ISP’s AS (AS27595) is now offline, due apparently to coordinated lobbying of its upstreams
  (tags: atrivo isps abuse intercage spam malware hosting)

Tags: abuse, atrivo, bureaucracy, deletion, deletionpedia, dell, drivers, funny, hardware, hosting, intercage, isps, linux, malware, performance, poweredge, processes, spam, sysadmin, wikipedia, wp

• THE FOURTH QUADRANT: A MAP OF THE LIMITS OF STATISTICS By Nassim Nicholas Taleb : ‘Statistics can fool you. In fact it is fooling your government right now. It can even bankrupt the system (let’s face it: use of probabilistic methods for the estimation of risks did just blow up the banking system).’ (via Gary Stock)
  (tags: banking probabilistic-methods probability statistics investment black-swans nassim-nicholas-taleb via:gstock essays the-edge)

• International Expert Group - Report - The Innovation Partnership : ‘the findings and recommendations of the International Expert Group on Biotechnology, Innovation and Intellectual Property’. Very anti-Bayh-Dole and the “old IP” patent-everything regime as it pertains to biotech. great stuff (via Techdirt)
  (tags: via:techdirt bayh-dole ip patents biotech canada reports)

• Greg Kroah-Hartman rips Canonical a new one : over allegations that they do not contribute enough development effort to the Linux ecosystem; in all major components, they push a truly miniscule amount of patch code upstream
  (tags: canonical linux greg-kroah-hartman code open-source free-software distros packaging upstream debian)

Tags: banking, bayh-dole, biotech, black-swans, canada, canonical, code, debian, distros, essays, free-software, greg-kroah-hartman, investment, ip, linux, nassim-nicholas-taleb, open-source, packaging, patents, probabilistic-methods, probability, reports, statistics, the-edge, upstream, via:gstock, via:techdirt

• Groklaw - Anonymous Speech in Email Upheld in Spammer Case : Groklaw goes into the detail of how and why the Virginia anti-spam law could be overturned. Ugh. I strongly believe that spam = UBE, not UCE, and political spam is still spam, so this is particularly disappointing for me
  (tags: ube uce spam law legal virginia jeremy-jaynes groklaw anti-spam)

• VMware server tweaks : ugh. quite a lot of voodoo here, need to investigate to see if any of these improve performance on our little build farm vmware server
  (tags: vmware performance linux tweaks kernel)

• consumer tips in response to the XL Airlines collapse : tour operator/airline went bust, leaving its customers well in the lurch. Those who booked flights directly on their website, using a debit card, have lost their money. Most travel insurance doesn’t cover airline collapse. Moral: use a credit card
  (tags: credit-cards safety consumer xl-airlines travel-agents atol flights travel)

Tags: anti-spam, atol, consumer, credit-cards, flights, groklaw, jeremy-jaynes, kernel, law, legal, linux, performance, safety, spam, travel, travel-agents, tweaks, ube, uce, virginia, vmware, xl-airlines

• Techdirt: How Patents Have Harmed University Research : the majority of university technology transfer offices have never made money, according to this. mind-boggling
  (tags: bayh-dole patents ip universities academia tech-transfer techdirt)

• VTun - Virtual Tunnels over TCP/IP networks : ‘The easiest way to create Virtual Tunnels over TCP/IP networks with traffic shaping, compression and encryption’. looks like it. UDP-based, no Windows support, but in Ubuntu’s “universe” apt repository
  (tags: vtun tunneling linux unix networking ip tcp udp security)

• Perl Best Admin Practices : good advice to those of us running systems built on perl. Every interpreted language needs a document like this
  (tags: perl wiki advice sysadmin best-practices guidelines fhs unix)

Tags: academia, advice, bayh-dole, best-practices, fhs, guidelines, ip, linux, networking, patents, perl, security, sysadmin, tcp, tech-transfer, techdirt, tunneling, udp, universities, unix, vtun, wiki

• Ubuntu and Debian AMIs for Amazon EC2 : they look well-maintained
  (tags: amis debian ubuntu ec2 aws)

• Making an AMI use EBS as the root filesystem : using some nasty pivot_root hackery
  (tags: root filesystems aws ec2 amazon ebs disks linux)

• Richard Hulse: Clearing the Cache on Matrix : IPC::DirQueue used in the field as part of a cache-invalidation system for a NZ radio station
  (tags: ipc-dirqueue cache-revalidation users user-stories)

Tags: amazon, amis, aws, cache-revalidation, debian, disks, ebs, ec2, filesystems, ipc-dirqueue, linux, root, ubuntu, user-stories, users

• Four Tweaks for Using Linux with Solid State Drives : good tips (via Jeremy)
  (tags: for:hughescr linux ssd disks optimization performance tmpfs kernel)

• VW Should Bring Back The Microbus And Make It Electric : and a pony! (via fergusb)
  (tags: want electric-vehicles cars volkswagen techcrunch over-ambitious and-a-pony)

Tags: and-a-pony, cars, disks, electric-vehicles, for:hughescr, kernel, linux, optimization, over-ambitious, performance, ssd, techcrunch, tmpfs, volkswagen, want

• non-PC devices increasing browser share : .5 - 1.5% of visitors to Warner music sites are now coming from games consoles and smartphones. bad news for Flash sites (via Torrez)
  (tags: flash web browsers via:torrez warner-music os)

• full(4): always full device : ‘Writes to the /dev/full device will fail with an ENOSPC error. This can be used to test how a program handles disk-full errors.’ - that’s nifty. I can’t believe I’m still finding useful new UNIX features after 18 years
  (tags: devices unix linux testing disks errors edge-cases enospc manual-pages)

Tags: browsers, devices, disks, edge-cases, enospc, errors, flash, linux, manual-pages, os, testing, unix, via:torrez, warner-music, web

• The Trifecta of FAIL : ah, the hazards of monkey-patching core classes illustrated perfectly; a Ruby point-release upgrade broke Rails (via chromatic)
  (tags: ruby rails monkeypatching ouch programming coding oo subclassing apis)

• best intranet form ever : ‘The software my employer uses for booking holidays has recently been “upgraded” and we now need to specify an absence reason.’ there are several hundred reasons, including ‘Abortion’, ‘Stroke’, ‘Warts’, ‘Dementia’, ‘Rectal Problems’, ‘Manic Depression’ and, um, ‘Wax’. best of all, this is for booking time off in advance…
  (tags: intranet funny inept hr wtf daily-wtf bureaucracy pto holidays)

• Sup : ‘a console-based email client for people with a lot of email [..] The goal of Sup is to become the email client of choice for nerds everywhere.’ Looks like they’ve nicked a few ideas from GMail, too (via Luis)
  (tags: via:tieguy console linux unix sup mail mail-readers ui apps)

Tags: apis, apps, bureaucracy, coding, console, daily-wtf, funny, holidays, hr, inept, intranet, linux, mail, mail-readers, monkeypatching, oo, ouch, programming, pto, rails, ruby, subclassing, sup, ui, unix, via:tieguy, wtf

• Hacking Mifare Transport Cards : notable mainly because Schneier calls MIFARE Classic’s crypto ‘terrible’ and ‘kindergarten cryptography’. ‘Anyone with any security experience would be embarrassed to put his name to the design.’ ZING
  (tags: ouch mifare cracks security zing crypto-cards smartcards oyster nxp)

• some good AWS tips : e.g. “upload files to S3 in lexically-sorted order”, apparently it’s faster. who knew!? (Sorry Joshua, gave you a bad tip y’day in that case)
  (tags: amazon aws s3 sqs sdb web-services hosting for:joshua)

• The Coreflood Report : fascinating Joe Stewart post-mortem of a server run by a Russian malware group targeting online banking; one apparent Miami-based victim was defrauded of $90k, and it appears that the group would have had access to a combined $2.5m in all victims’ accounts
  (tags: coreflood trojans joe-stewart secureworks malware banking online-banking autoproxy joe-lopez)

• using “data=writeback” on your Ubuntu filesystems : aha! This is the root cause of the crippling Firefox 3/VIM slowness; Ubuntu and Debian use a crappy ext3 option which sacrifices speed for correctness, by effectively turning every fsync() into a sync(). Here’s how to disable it
  (tags: fsync sync unix linux ext3 ubuntu debian firefox vim grub annoying sysadmin)

Tags: amazon, annoying, autoproxy, aws, banking, coreflood, cracks, crypto-cards, debian, ext3, firefox, for:joshua, fsync, grub, hosting, joe-lopez, joe-stewart, linux, malware, mifare, nxp, online-banking, ouch, oyster, s3, sdb, secureworks, security, smartcards, sqs, sync, sysadmin, trojans, ubuntu, unix, vim, web-services, zing

Del.icio.us 2.0 goes live yay! I’ve been waiting for this for yonks

10 years of Boards.ie massive ~50GB RDF/XML dump, for open crunching, to generate interesting “SIOC Semantic Web” apps

Postmaster.comcast.net how to get mail delivered successfully to Comcast, the usual stuff

Why we’ll never replace SMTP ‘The reason that e-mail is uniquely useful is that you can exchange mail with people you don’t already know. The reason that spam exists is that you can exchange mail with people you don’t already know.’ +1

“Bikes-for-Billboards” scheme exposes major planning flaws ‘what was initially hailed as “free bikes” has become one of the biggest planning controversies to hit Dublin in years.’ No shit. 70% of sites are on the Northside, rather than the richer Southside; and each bike will cost over EUR300k in ad revenue!

Rob Enderle’s page on Wikipedia detailing this analyst’s hilariously wrong pro-SCO, anti-Apple/Linux predictions over the years. John Gruber: ‘the only way it would be worthwhile for reporters to [quote him] would be if they were willing to describe him as “almost always utterly wrong”‘

Tags: ads, advertising, analysts, anti-spam, apple, bikes, billboards, blogging, boards.ie, comcast, cycling, del.icio.us, dublin, email, enderle-group, funny, fussp, guidelines, history, internet, ireland, isps, jc-decaux, john-gruber, Links, linux, mail, microsoft, oh-dear, planning, postmaster, releases, rob-enderle, scandals, sco, semantic-web, sioc, smtp, spam, tech, urban, via:archiseek, wikipedia

ZSFA — I Want The Mutt Of Feed Readers Zed recommends Newsbeuter. must take a look

We Want A Dead Simple Web Tablet For $200. Help Us Build It. having worked on a project to do just this, believe me, this is doomed. DOOMED

Science Clouds ‘compute cycles in the cloud for scientific communities .. allows you to provision customized compute nodes .. that you have full control over using a leasing model based on the Amazon’s EC2 service.’ Wonder if they’d like to give SA some time ;)

Tags: appliances, apps, atom, cloud-computing, ec2, feeds, grids, hardware, linux, news, newsbeuter, oh-dear, open-source, rss, scaling, science, software, spamassassin, tablets, techcrunch, virtualization, web, zed-shaw

O2 Leaking Customer Photos (updated) the JBoss/Tomcat install leaks the “secret” URLs through it’s default status page. this is the 3rd helping of FAIL for O2’s web team; 2 previous occasions in the last year exposed customer data through “secret” URL manipulation

Avant Window Navigator “a ‘dock-like’ (cough) navigator bar for the Linux desktop” (via Danny, again!)

trickle ‘user-space bandwidth shaper’, ie. like nice(1) for network bandwidth (via Danny)

RFC 5218 - What Makes For a Successful Protocol? ‘Based on case studies, this document identifies some of the factors influencing success and failure of protocol designs.’ (via spicylinks)

Tags: bandwidth, collaboration, cool, dock, fail, gnome, http, jboss, kde, linux, mms, mod_jk, net, networking, nice, o2, protocols, rfcs, security, software, standards, tcp, tomcat, traffic-shaping, ui, urls, via:danny, via:spicylinks, web

Great article from LWN.net regarding the Debian OpenSSL vulnerability:

It is in the best interests of everyone, distributions, projects, and users, for changes made downstream to make their way back upstream. In order for that to work, there must be a commitment by downstream entities — typically distributions, but sometimes users — to push their changes upstream. By the same token, projects must actively encourage that kind of activity by helping patch proposals and proposers along. First and foremost, of course, it must be absolutely clear where such communications should take place.

Another recently reported security vulnerability also came about because of a lack of cooperation between the project and distributions. It is vital, especially for core system security packages like OpenSSH and OpenSSL, that upstream and downstream work very closely together. Any changes made in these packages need to be scrutinized carefully by the project team before being released as part of a distribution’s package. It is one thing to let some kind of ill-advised patch be made to a game or even an office application package that many use; SSH and SSL form the basis for many of the tools used to protect systems from attackers, so they need to be held to a higher standard.

+1.

Tags: debian, linux, lwn, openssl, packaging, security

I’ve just fixed a bug on my linux desktop which had been annoying me for a while. Since there seems to be little online written about it, here’s a blog post to help future Googlers.

Here’s the symptoms: while you’re running VMWare, your Vim editing sessions freeze up for 20 seconds or so, roughly every 5 minutes. The editor is entirely hung.

If you strace -p the process ID before the hang occurs, you’ll see something like this:

select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
_llseek(7, 4096, [4096], SEEK_SET)      = 0
write(7, “tp\21\0\377\0\0\0\2\0\0\0|\0\0\0\1\0\0\0\1\0\0\0\6\0\0″…, 4096) = 4096
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost -isig -icanon -echo …}) = 0
select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
_llseek(7, 20480, [20480], SEEK_SET)    = 0
write(7, “ad\0\0\245\4\0\0\341\5\0\0\0\20\0\0J\0\0\0\250\17\0\0\247″…, 4096) = 4096
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost -isig -icanon -echo …}) = 0
select(6, [0 3 5], NULL, [0 3], {0, 0}) = 0 (Timeout)
fsync(

In other words, the hung process is sitting in an fsync() call, attempting to flush changed data for the current file to disk.

Investigation threw up the following: a kerneltrap thread about disk activity, poor responsiveness with Firefox 3.0b3 on linux, and a VIM bug report regarding this feature interfering with laptop-mode and spun-down hard disks.

VMWare must be issuing lots of unsynced I/O, so when Vim issues its fsync() or sync() call, it needs to wait for the VMWare I/O to complete before it can return — even though the machine is otherwise idle. A bit of a Linux kernel (or specifically, ext3) misfeature, it seems.

Synthesising details from those threads comes up with this fix: edit your ~/.vimrc and add the following lines –

set swapsync=
set nofsync

This will inhibit use of both fsync() and sync() by Vim, and the problem is avoided nicely.

Update: one of the Firefox developers discusses how this affects FF 3.0.

Tags: fsync, hangs, linux, swapsync, sync, vim, vmware

For a while now, I’ve been using various hacks to play music from my Linux laptop, holding my main music collection, to client systems which drive the speakers.

Previously, I used this setup to play via my MythTV box. Nowadays, however, my TV isn’t in the room where I want to listen to music. Instead, I have my Nokia 770 hooked up to the speakers; this plays the BBC Radio 4 RealAudio streams nicely, and also the laptop’s MP3 collection using a uPnP AV MediaServer.

I specifically use TwonkyMedia right now, playing back via the N770’s Media Streamer app. (That works pretty well – uPnP AV is one of those standards plagued with incompatibilities, but TwonkyMedia and Media Streamer seem to be a reliable combination.)

However, TwonkyMedia sometimes fails to notice updates of the library, and nothing has quite as good a music-player user interface as JuK, the KDE music player and organiser app, so a way to play directly from the laptop instead of via uPnP would be nice…

A weekend’s hacking reveals that this is pretty easily done nowadays, thanks to some cool features in pulseaudio, the current standard sound server on Ubuntu gutsy, and the Esound server running on the N770.

Unfortunately, the N770 doesn’t (yet) support pulseaudio directly, otherwise we could use its seriously cool support for RTP multicast streams. Still, we can hack something up using the venerable “esd” protocol (again!) Here’s how to set it up…

On the N770:

You need to fix the N770’s “esd” sound server to allow public connections. Set up your wifi network’s DHCP server to give the N770 a static IP address. Log in over SSH, or fire up an xterm. Run the following:

mv /usr/bin/esd /usr/bin/esd.real

cat > /usr/bin/esd <<EOM
#!/bin/sh
exec /usr/bin/esd.real -tcp -public -promiscuous -port 5678 $*
EOM

chmod 755 /usr/bin/esd
/etc/init.d/esd restart

On the server:

Download this file, and save it as n770.pa. Edit it, and change server=n770:5678 on the fourth line to use the IP address or hostname of your Nokia 770 instead of n770. Then run:

cp n770.pa ~/.n770.pa

cat > ~/bin/sound_n770 <<EOM
#!/bin/sh
pulseaudio -k; pulseaudio -nF $HOME/.n770.pa &
EOM

cat > ~/bin/sound_here <<EOM
#!/bin/sh
pulseaudio -k; pulseaudio &
EOM

chmod 755 ~/bin/sound_here ~/bin/sound_n770

Now you just need to run ‘~/bin/sound_n770′ to redirect sound playback to the N770, and ‘~/bin/sound_here’ to reset back to laptop speaker output, for the entire desktop environment. Nifty!

Update: it appears that things may work more reliably if you add “rate=22050″ at the end of the “load-module module-esound-sink” line — this halves the bitrate of the network stream, which copes better with harsh wifi network conditions. The n770.pa file above now includes this.

Tags: esd, hacks, linux, n770, nokia, pulseaudio, sound

Screen real estate, that is — 3600×1050 pixels of it:

(That’s a Samsung SyncMaster 225bw226bw connected to a Thinkpad T61p running Ubuntu Gutsy, if you’re curious.)

Tags: desk, linux, putplace, samsung, work

Mike Culver, Amazon’s “Web Services Evangelist”, will be in Dublin next week to evangelize about the goodness that is Amazon S3, EC2, SQS and so on. It seems he’ll be talking at the following locations:

• in the Auditorium of the Digital Exchange, Crane Street, Dublin 8 on Tuesday October 30th, 3-5pm; here’s a flyer the Amazonites have been passing around. (upcoming.org page)

• according to Damien, later that evening, he’s in the Westin Hotel on Westmoreland St., D2, starting at 7pm; note, it seems you need to book places at this, see Damien’s post.

• and again at the Irish Linux User’s Group on Thursday November 1st at 19:30 in the Irish Computer Society in Dublin (map).

I guess these are all going to be same talk, bar the Q&A ;)

There was some kind of an ICTE get-together mooted for Friday 2nd.

Also, the ILUG annual general meeting is scheduled on the following Saturday, 3rd November, also at the ICS. Gareth Eason notes ‘we’re hoping to start at 3pm sharp, with talks from Dave Wilson (HEAnet), Frank Duignan, John Looney (Google), and others, followed by a relaxing wind-down in the Schoolhouse pub later on.’ (upcoming.org page)

Hopefully I’ll get to at least one of the AWS talks (probably the Digital Exchange one) and the ILUG AGM… busy week!

Tags: amazon, aws, dublin, ec2, icte, ilug, ireland, linux, mike-culver, s3, social, talks

I haven’t paid a whole lot of attention to the BBC’s “iPlayer” project, since, as a non-UK resident, I’m not allowed to use it anyway. But this interview at Groklaw with Mark Taylor, President of the UK Open Source Consortium, was really quite eye-opening. Here’s some choice snippets.

On the management team’s Microsoft links:

The iPlayer is not what it claimed to be, it is built top-to-bottom on a Microsoft-only stack. The BBC management team who are responsible for the iPlayer are a checklist of senior employees from Microsoft who were involved with Windows Media. A gentleman called Erik Huggers who’s responsible for the iPlayer project in the BBC, his immediately previous job was director at Microsoft for Europe, Middle East & Africa responsible for Windows Media. He presided over the division of Windows Media when it was the subject of the European Commission’s antitrust case. He was the senior director responsible. He’s now shown up responsible for the iPlayer project.

On their attempts to bullshit the BBC Trust on the cross-platform issue:

In the consultations that the BBC Trust made, there were 10,000 responses from the public. And the overwhelming majority of them, over 80% — which is an unheard-of figure in these kind of things — said, we don’t like the platform. We don’t like it being single-platform. So it’s a big issue. And the BBC Trust said to us, “Why the vehemence? Why have people reacted this way?” And I explained the ‘Auntie’ analogy. It’s people don’t expect that from the BBC. It’s got this huge history of integrity, doing the right thing, standing up to bullies. (laughter) They’ve done this for a very long time. And people find that it’s surprising. And they said, “Yeah, but,” you know, the BBC guys said, “Well, trust us. This is going to be cross-platform.” And we said, “Well, how? It’s completely single-platform.” They say that, but we haven’t been able to find anyone who’s been able to explain how they’re going to achieve that at the moment, even though they’re entirely locked into one single platform.

(aside: MS did this at one point with Internet Explorer — remember, there was some mystery team in Germany that supposedly had IE ported to Solaris, hence it therefore qualified as ‘cross-platform’.)

On the architecture of the product:

Q: it’s a Verisign Kontiki architecture, it’s peer-to-peer, and in fact one of the more worrying aspects is that you have no control over your node. It loads at boot time under Windows, the BBC can use as much of your bandwidth as they please (laughter), in fact I think OFCOM … made some kind of estimate as to how many hundreds of millions of pounds that would cost everyone [...]. There is a hidden directory called “My Deliveries” which pre-caches large preview files, it phones home to the Microsoft DRM servers of course, it logs all the iPlayer activity and errors with identifiers in an unencrypted file. Now, does this assessment agree with what you’ve looked at?

Mark Taylor: Yes.

Q: What are the privacy implications for an implementation like this?

Mark Taylor: Well, just briefly going back to the assessment thing, yes it does log precisely RSS and stuff like that and more importantly, anyone technically informed who’s had a look at it — even more importantly, the user’s assessment as well and — frankly horrified if you go and spend some time in the BBC iPlayer forums, it’s eye-opening to see the sheer horror of the users, some of them technically not — you know, relatively early-stage users — but when it gets explained to them by some of the longer-using users of it, it’s concentrated misery. (laughter)

[...]

it’s a remarkable thing with them as well, there’s a lot of pain going on in the user forums, and some of the main technical support questions in there are “how do I remove Kontiki from my computer?” See, it’s not just while iPlayer is running that Kontiki is going, it’s booted up. When the machine boots up, it runs in the background, and it’s eating people’s bandwidth all the time. (laughter) In the UK we still have massive amounts of people who’ve got bandwidth capping from their ISPs and we’ve got poor users on the online forums saying, “Well, my internet connection has just finished, my ISP tells me I’ve used up all of my bandwidth.”

Q: It uses up their quota, but they can’t throttle it, they can’t reduce it –

Mark Taylor: No, they can’t throttle it. [...] It’s malware as well as spyware.

And to top this off, there’s a (frankly insane) budget of UKP 130,000,000 to build this — that’s $266,000,000 — for something that could be built better by just hiring the guys behind UKNova and simply negotiating with the rights-holders directly.

Holy crap. Talk about a technical disaster masquerading as a solution to a business problem…

Tags: bbc, bbc-trust, beeb, drm, iplayer, linux, microsoft, open-source, pain

A couple of weeks back, there was quite a bit of buzz in the Irish blogosphere and elsewhere about the Nokia 770; prices for new N770s had dropped from $290ish to a very reasonable $140 / EUR130-ish price-point. I, along with a good few others, bought one.

I bought mine through Expansys, with a free 1GB RS-MMC memory card. They’ve sold out and no longer have any N770s listed; however, Buy.com still seem to have them in stock, so if you’re interested, you can probably still pick one up. (It seems Nokia is trying to sell off their remaining N770 stock, cheap, with plans to drop support for the software platform. I’m fine with this, but it may put other buyers off.)

I’ve now been using it for a while, and am still happy. ;) Here are my recommended top apps:

Slimserver. Originally designed to operate as the backend software for the Squeezebox thin-client MP3 player, this has a fantastic UI built for the N770, and its MP3 stream output works perfectly on the tablet.

This is by far the neatest way to get at a 6000-song music library without a laptop; there was some talk in the GNOME community of making a decent DAAP client, but so far there’s no working results there that I could find. :(

maemo-mapper. This is a fantastic mapping app for the tablet; it presents map tiles downloaded from OpenStreetMap or Google Maps in an N770-optimized format, with the usual nice draggable UI. Bonus: it’ll work offline, so you can follow a route while online, then take the tablet along to help navigate.

Tip: once you start maemo-mapper, click the “Download…” button in the “Repository Manager” and it’ll download details for the 5 most useful map repositories, including Google and Virtual Earth.

FBReader. A very nice document reader; much nicer than trying to read long HTML pages in the builtin web browser, especially since it allows you to turn the device on its side.

In general, the Opera Mini browser works fine; be sure to enable Javascript and set up a swap file on the RS-MMC card first. It does all the basic HTML and rudimentary AJAX; Google Calendar is a no-go, but GMail and even Google Maps works adequately, modulo minor bugs. Plain Old HTML sites like Wikipedia, IMDB and so on all work great.

As long as you’re realistic about the platform, it won’t disappoint — video requires custom transcoding, for example, and proprietary apps like Flash and RealPlayer lag behind their desktop equivalents, but as far as I can tell that’s the case for every embedded platform. (Since I spent a couple of years developing such a platform, I’m quite comfortable with this.)

A really really nifty thing about the N770 is that it’s now entirely hackable — within 30 minutes of powering on, I was able to get a terminal window open with a root prompt, and was adding ext3 partitions to the RS-MMC card. Apps are installed using “apt-get”. The terminal even has word-completion system optimized for the UNIX command-line - nice ;)

This SomethingAwful thread contains plenty more good tips. I’m happy I bought it — so many of these gadgets can wind up as an overpriced door-stop, but this is easily worth what I paid for it.

Update: this thread at InternetTabletTalk seems pretty chock-full of good advice, too.

Tags: gadgets, linux, n770, nokia, software

I’ve been running into problems recently (since several months ago at least), with USB hardware on my Thinkpad T40 running Ubuntu Hoary Dapper; in particular, every time I plug in my iPod or one of my USB hard disks nowadays, I get this:

[5008549.187000] usb 4-3: USB disconnect, address 14
[5008550.143000] usb 4-3: new high speed USB device using ehci_hcd and address 18
[5008552.643000] usb 4-3: new high speed USB device using ehci_hcd and address 27
[5008557.393000] usb 4-3: new high speed USB device using ehci_hcd and address 43
[5008557.893000] usb 4-3: new high speed USB device using ehci_hcd and address 44
[5008558.643000] usb 4-3: new high speed USB device using ehci_hcd and address 46
[5008558.895000] ehci_hcd 0000:00:1d.7: port 3 reset error -110
[5008558.896000] hub 4-0:1.0: hub_port_status failed (err = -32)
[5008559.893000] usb 4-3: new high speed USB device using ehci_hcd and address 48
[5008562.643000] usb 4-3: new high speed USB device using ehci_hcd and address 58
[5008563.143000] usb 4-3: new high speed USB device using ehci_hcd and address 59
[5008563.643000] usb 4-3: new high speed USB device using ehci_hcd and address 60
[5008570.143000] usb 4-3: new high speed USB device using ehci_hcd and address 85

This repeats ad infinitum until the USB device is disconnected.

I had this down as a hardware issue (since it started happening just after warranty expiration ;), but some accidental googling revealed several other cases – and a workaround:

sudo modprobe -r ehci-hcd

Run that repeatedly, each time replugging the device and monitoring dmesg via watch -n 1 ‘dmesg | tail’ in a window, until the device is finally recognised as a USB hard disk. It generally seems to take 3 or 4 attempts, in my experience.

This LKML thread suggests hardware changes can cause it, but this hardware hasn’t changed in years. Annoying.

Anyway, this is ongoing. This tip seems to help, but it might be just treating a symptom, I don’t know — just posting for google and posterity… and to moan, of course :(

Tags: bugs, disks, ehci-hcd, hardware, ipod, linux, storage, usb

I took the plunge over the weekend, and live-upgraded the new ‘Dapper Drake’ Ubuntu release — ouch. Here’s the two key lessons I learned:

• Don’t run “grub-install” in a misremembered attempt to update the current GRUB boot menu ‘menu.lst’ file with the new kernel; sadly, this will quietly remove important details from your old menu.lst, such as “initrd” lines, rendering those kernels unbootable. Moral: ensure brain is in gear before meddling with MBRs!

• If you’re a Kubuntu user, watch out. Ensure you run apt-get install ubuntu-base ubuntu-desktop — bringing the entirety of GNOME up to date — as well as apt-get install kubuntu-desktop after the upgrade; it appears that some part of a new hotplugging subsystem is not included as a dependency of kubuntu-desktop. Failure to do this results in an inability to use USB/hotpluggable devices, including internal devices like the Synaptics touchpad. No pointer devices (mice or touchpads) means no X server at boot, which is always a little annoying.

Some day I’ll just do things the right way, and do a fresh-from-CD install instead. Ah well. The good stuff: the new kernel, or possibly Xorg, is proving to be a lot speedier — window updates are noticeably smoother; and the new Ubuntu GNOME theme is similarly tasty.

Tags: dapper, gnome, grub, kde, kubuntu, linux, software, ubuntu, upgrades

Cory Doctorow at Boing Boing links to an article at TechCrunch that lists Better and Cheaper Online File Storage as a product that needs to be made. However, Ben Laurie does the sums on online storage as a useful backup medium, and found them not exactly compelling (e.g. 100GB of data will take 75 days to upload over an 128Kbps link).

I tend to agree. An online host isn’t great as a backup host, since, in my experience, there are two types of backups required:

• The important small files (for example: encrypted password lists, my address book, my ~/bin directory)
• The massive big filesets (for example: MP3s, photos)

The first kind of fileset is amenable to an online backup-storage service, at first glance. However — in my opinion you’re better off going the whole hog for these files, and using the distributed, versioned backup method of putting it in a good networked revision control system, and checking it out everywhere, so you can also make changes and check in from any host; otherwise, you face the perils of syncing up a single backup from multiple “writers”, without conflicts. So far, none of the online file storage services offer SVN as an access method, so a shell account at a colo server still seems more useful on that count.

The second kind of fileset, as Ben notes, will take donkey’s years to upload and sync as a backup mechanism; and the economics are hardly compelling for the service provider.

I think I prefer Brad Templeton’s idea to deal with large-data backups –

I propose a software RAID-5, done over a LAN with 3 to 5 drives scattered over several machines on the LAN.

Slow as hell, of course, having to read and write your data out over the LAN even at 100mbits. Gigabit would obviously be better. But what is it we have that’s taking up all this disk space ? it?s video, music and photos. Things which, if just being played back, don?t need to be accessed very fast. If you’re not editing video or music, in particular, you can handle having it on a very slow device. (Photos are a bigger issue, as they do sometimes need fast access when building thumbnails etc.)

This could even be done among neighbours over 802.11g, with suitable encryption. In theory.

As a commenter notes, Linux has support for this already, in the form of software RAID and the network block device.

So: take an external IDE enclosure, add a GumStix board running Linux with software RAID, LVM, and nbd, and add wifi. Then add DAV, SMB and NFS export of the disk, and some decent UI code to organise the volumes into a single exported RAID volume (hopefully automatically!), and it’d be a pretty compelling product, in my opinion!

(hey Craig! I said GumStix! ;)

Tags: backups, disks, gumstix, hardware, linux, lvm, nbd, product, raid

The NY Times story on “life hacking” is a pretty good one, and an excellent intro for anyone who hasn’t been religiously reading the changing transcripts of Danny O’Brien’s talk and so on.

This line:

Mann has embarked on a 12-step-like triage: he canceled his Netflix account, trimmed his instant-messaging “buddy list” so only close friends can contact him and set his e-mail program to bother him only once an hour.

Reminded me of something I ran into recently.

Last month, I switched from Sawfish, the venerable UNIX window manager, to GNOME’s Metacity, which is the new(ish) GNOME standard window manager. (I was tired of some long-standing Sawfish crashes, and didn’t want to be the last Sawfish user on the planet, which was seeming increasingly likely.)

One interesting UI change is that application windows no longer ‘pop up’ — if an app wants to notify you of some important change, it instead can only cause its taskbar button to subtly pulse in the corner of your screen.

Initially, this threw me for a loop, and I rudely (albeit accidentally) ignored my friends on IM and suchlike. But I quickly got the hang of glancing at the taskbar once in a while when I wasn’t concentrating on a task; it’s now second nature, and has significantly reduced the number of interruptions I find myself experiencing in a typical day.

BTW, in passing: switching WMs is a big deal, user interface-wise. One of the key gating factors, for me, was a feature I use to control windows without laying hands on the dreaded rodent — namely, a ‘move window to screen corner’ keyboard shortcut. This patch implements it for Metacity.

I implemented this last year for KWin, too, to resounding disapproval and bitchy comments about how I’m using the mouse all wrong. Meh. I fully expect the Metacity maintainers to throw it out, likewise, leaving me hand-patching WMs for a while yet ;)

Update, Nov 2006: they applied it! yay.

Tags: desktop, gnome, jm-software, kde, linux, patches, software, unix, usability, window-managers

Urban Dead is a new browser-based MMORPG that’s become popular recently. I’m not planning to talk about the game itself, at least not until I’ve played it a bit!, but there’s something worth noting here — a cheat called Groove Theory:

Groove Theory was a cheat for Urban Dead that claimed to exploit an apparent lack [sic] of a random number generator in the game, [so] that performing an action exactly eight seconds after a successful action would also be successful.

Kevan, the Urban Dead developer, confirmed that Groove Theory did indeed work, and made this comment after fixing the bug:

There is some pattern to the random numbers, playing around with them; “srand(time())” actually brings back some pretty terrible patterns, and an eight-second wait will catch some of these.

So — here’s my guess as to how this happened.

It appears that Urban Dead is implemented as a CGI script. I’ll speculate that somewhere near the top of the script, there’s a line of code along the lines of srand(time()), as Kevan mentioned. With a sufficiently fast network connection, and a sufficiently unloaded server, you can be reasonably sure that hitting “Refresh” will cause that srand call to be executed on the server within a fraction of a second of your button-press. In other words, through careful timing, the remote user can force the pseudo-random-number generator used to implement rand() into a desired set of states!

As this perl script demonstrates, the output from perl’s rand() is perfectly periodic in its low bits on a modern Linux machine, if constantly reseeded using srand() — the demo script’s output decrements from 3 to 0 by 1 every 2 seconds, then repeats the cycle, indefinitely.

I don’t know if Urban Dead is a perl script, PHP, or whatever; but whatever language it’s written in, I’d guess that language uses the same PRNG implementation as perl is using on my Linux box.

As it turns out, this PRNG failing is pretty well-documented in the manual page for rand(3):

on older rand() implementations, and on current implementations on different systems, the lower-order bits are much less random than the higher-order bits. Do not use this function in applications intended to be portable when good randomness is needed.

That manual page also quotes Numerical Recipes in C: The Art of Scientific Computing (William H. Press, Brian P. Flannery, Saul A. Teukolsky, William T. Vetterling; New York: Cambridge University Press, 1992 (2nd ed., p. 277)) as noting:

“If you want to generate a random integer between 1 and 10, you should always do it by using high-order bits, as in

j=1+(int) (10.0*rand()/(RAND_MAX+1.0));

and never by anything resembling

j=1+(rand() % 10);

(which uses lower-order bits).”

I think Groove Theory demonstrates this nicely!

Update: I need to be clearer here.

Most of the Groove Theory issue is caused by the repeated use of srand(). If the script could be seeded once, instead of at every request, or if the seed data came from a secure, non-predictable source like /dev/random, things would be a lot safer.

However, the behaviour of rand() is still an issue though, due to how it’s implemented. The classic UNIX rand() uses the srand() seed directly, to entirely replace the linear congruential PRNG’s state; on top of that, the arithmentic used means that the low-order bits have an extremely obvious, repeating, simple pattern, mapping directly to that seed’s value. This is what gives Groove Theory its practicability by a human, without computer aid; with a more complex algorithm, it’d still be guessable with the aid of a computer, but with the simple PRNG, it’s guessable, unaided.

Update 2: as noted in a comment, Linux glibc’s rand(3) is apparently quite good at producing decent numbers. However, perl’s rand() function doesn’t use that; it uses drand48(), which in glibc is still a linear congruential PRNG and displays the ‘low randomness in low-order bits’ behaviour.

Tags: algorithms, code, games, linux, prng, randomness, security, software, unix

On saturday, I spent a little time trying to work out how to give Steve Jobs my money; more accurately, I wanted to get some way to buy music from the iTunes Music Store from my Linux desktop, and this isn’t as easy as it really should be, because the official iTMS is a mess of proprietary Mac- and Windows-only DRM-laden badness.

Here’s a quick walkthrough of how this went:

• install iTunes in my VMWare Windows install
• sign up for iTMS, and give Apple all my personal info, including super-s3kr1t card verification codes, eek
• buy a song
• find the DRM’d file in the filesystem; it’s an .m4p file, and xine doesn’t seem to like it
• do some googling for ‘iTunes DRM remove linux’; that leads to Jon Lech Johansen’s JusteTune
• download and run JusteTune installer
• get obscure hexadecimal error code dialog. hmm! what could that mean?
• download and run .NET runtime, link on JusteTune page
• rerun JusteTune — it works this time
• select Account -> Authorize, enter login info
• drag and drop file — it’s decrypted!

So, that yields a decrypted AAC file, which I can play on Linux using xine. That’s the hard part done!

However, I want to play my purchases in JuK, the very nice iTunes-style music player app for KDE.

While the gstreamer audio framework supports playback of AAC files with the gstreamer0.8-faad package (’sudo apt-get install gstreamer0.8-faad’), JuK itself can’t find the file or read its metadata, so it doesn’t show up in the music collection as playable. I don’t want to go hacking code from CVS into my desktop’s music player — possibly the most essential app on the desktop — so transcoding them to MP3 seems to be the best option.

Somebody’s already been here before, though — that’s one of the benefits of being a late adopter! Here’s a script to convert .m4a files to .mp3 using the ‘faad’ tool (’sudo apt-get install faad’).

During this work, I came across Jon Lech Johansen’s latest masterwork — SharpMusique, a fully operational native Linux interface to the iTMS. Building on Ubuntu Hoary was a simple matter of tar xvfz, configure, make, sudo make install, and it works great — and automatically de-DRMs the files on the fly as it downloads them! Now that’s the way to enjoy the iTMS on Linux, at least until Apple’s engineers break it again.

Update, May 2006: Apple’s engineers broke it. Thanks Wilfredo ;)

End result: a brand new, complete, high-quality copy of Dengue Fever’s new album, Escape From Dragon House. Previously I’d only had a couple of tracks off this, so I’m now a happy camper, music-wise.

BTW, I was also considering trying out the new Yahoo! Music Store, but it too uses fascist DRM tricks and is platform-limited, and I’m not sure how breakable it is. On top of that, the prospect of not being able to try it out before handing over credit-card details put me off. As far as I can see, I can’t even look up the albums offered before subscribing. All combined, I’ll stick with iTMS for now.

Tags: apple, drm, internet, itunes, linux, software

Linux: PingWales’ round-up of UKUUG Linux 2005 Day 3 includes this snippet:

As well as running (Virtual Machines), Xen allows them to be migrated on the fly. If a physical system is overloaded, or showing signs of failure, a virtual machine can be migrated to a spare node. This process takes time, but causes very little interruption to service. The machine state is first copied in its entirety, then the changes are copied repeatedly until there are a small enough number than the machine can be stopped, the remaining changes copied and the new version started. This usually provides a service interruption of under 100ms - a small enough jitter that people playing Quake 3 on a server in a virtual machine did not notice when it was moved to a different node.

Now that is cool.

Tags: linux, software, virtual-machines, xen

Politics: EDRI-gram notes that the Firenze Linux User Group’s server was tampered with last month at its ISP colo:

On Monday 27 June 2005, two members of FLUG (Firenze Linux User Group) visited the data centre of Dada S.p.a., in Milan, where the community server of the group is physically housed, in order to move it to another provider.

When the server was put out of the rack, however, it was discovered that the upper lid of the server case was half-opened. At a closer inspection, it was also discovered that the case lid was scratched, as if it had been put out and reinserted into the rack. Worse, the CD-ROM cable was missing, as were the screws that kept the hard disks in place.

What is particularly worrying is that the server hosted an anonymous remailer, whose keys and anonymity capabilities could have been compromised. Considering what happened to Autistici/Inventati server - which hosted another anonymous remailer - this possibility is not so far fetched. This begs the question whether a co-ordinated attempt at intercepting anonymous/private communications on the Internet has been ongoing in the past weeks and months.

Bizarre goings-on.

Tags: case, edri-gram, firenze, group, lid, linux, politics, rack, remailer, server, user

Hardware: After a few weeks running OpenWRT on a Linksys WRT54G, here’s a status report.

Things that the new WRT54G running OpenWRT does a whole lot better than the Netgear MR814:

• Baseline: obviously it doesn’t DDoS the University of Wisconsin, and it doesn’t lose the internet connection regularly, as noted in that prior post. I knew that, so those are not really new wins, though.
• It’s quite noticeably faster. I’ve seen it spike to double the old throughput rates, and it’s solid, too; less deviation in those rates.
• It doesn’t break my work VPN. I wasn’t sure if it was the MR814 that was doing this, requiring an average of about 20 reconnects per day – now, I know it for a fact. I’ve had to reconnect my VPN connection about 4 times over the past week.
• It doesn’t break the Gigafast UIC-741 USB wifi dongle I’m using on the MythTV box. Previously that would periodically disappear from the HAN. Again, I had this pegged as an issue with the driver for that; removing the MR814 from the equation has solved it, too, and it’s now running with 100% uptime so far.
• It does traffic shaping with Wondershaper, so I can use interactive SSH, VNC, or remote desktop while downloading, even if it’s another machine on the HAN doing the download.
• It’s running linux — ssh’ing in, using ifconfig, and vi’ing shell scripts on my router is very, very nice.

Man, that MR814 was a piece of crud. ;) I can’t recommend OpenWRT enough…

Tags: bugs, ddos, hackability, linksys, linux, netgear, ntp, open-source, openwrt, router

Linux: the MythTV hacking continues (infrequently). Here’s the latest – a way to play music from my laptop, with sound output via the Mythbox.

Tags: laptop, linux, music, mythbox, mythtv, output, sound, way

Linux: Linux sound is still a mess. Due to the ever-changing ’sound server of the week’ system used to decide how an app should output sound, it’s perfectly possible to have 3 apps on your desktop happily making noise at the same time, while another app complains about requiring exclusive access to /dev/dsp — or worse, hangs silently while it attempts to grab an exclusive lock on the device.

This page gives a reasonably good guide to getting software mixing working across (virtually) all apps, using ALSA software mixing and esd.

However, some cases are still very kludgy — in particular, JuK, the excellent KDE mp3 jukebox app, has a tendency to play poorly with others, requiring playback via no less than two sound servers — artsd and esd — to work correctly in the above setup. In addition, the support for mp3 files in artsd is buggy — it’s frequently unable to open certain mp3s, depending on how they were encoded.

Well, good news — the current release of JuK now supports direct playback from GStreamer via ALSA. Here’s how. By adding these lines:

[GStreamerPlayer]
SinkName=alsasink

to ~/.kde/share/config/jukrc, you can skip sending JuK mp3 playback via 2 sound servers, and just play directly to the hardware from the mp3 player. An improvement! Not quite optimal, and certainly not user-friendly — but getting there…

Tags: alsa, app, artsd, esd, juk, kde, linux, playback, software, sound

Freevo: so I’m planning to build myself a PVR, of the home-built, running Linux with mythTV or Freevo, mini-ITX variety.

So far I’m still at the hardware planning stages, but the price looks good – around $455 (plus shipping) for a working, thoroughly hackable, silent, set-top PVR system.

(Silence is a key aim here — last thing I want is something noisy taking over the room. But silence typically seems to cost the dollars, once you get into Shuttle gear and the like.)

If anyone wants to follow along, or provide some tips — I’m going to track progress (very slowly) on this wiki page. Like all wiki pages, it’s editable — although you’ll need to create an account to edit pages there (sorry, anti-spam measure).

BTW, lately, there’s been a lot of talk about using a Mac mini as a media center. So I took a quick look — but wow, it’s pricey! $499 + $329 for an EyeTV 200 tuner? Dude, that’s over 800 dollars, not include shipping or sales tax. Given whatever extras turn out to be appropriate, I wouldn’t be surprised if it hits double the mini-ITX’s price.

Tags: freevo, itx, linux, mythtv, planning, price, pvr, silence, variety, wiki

Tech: January 24th, besides being the date the first Apple Macintosh went on sale, is supposedly the day of maximal post-xmas misery. Well, it certainly was for me today.

I decided to power on my old desktop to set it up as a back-room fileserver, and twiddled the partition table accordingly to nuke a few unused Windows partitions and maximise usable space.

Somehow or other, some component of my system decided that it would henceforth be non-bootable. It seems some BIOSes don’t like partition tables where a high-numbered logical partition have a lower starting sector than a boot logical partition, or something… GRUB just errored out with an obscure ‘Error 17′, which apparently means that it couldn’t find its boot partition any more.

OK, so I needed a boot disk. But I had 1 laptop with a CD/DVD drive but no floppy drive, and a desktop with a floppy drive but no CD drive (due to hardware failure)… and the original linux boot floppy was long gone, seeing as I’d hardly booted this machine in the duration of two house moves. Argh.

A dinky little Cruzer mini 128MB USB flash drive saved the day. (R)ecovery (I)s (P)ossible is a tiny Linux distro that fits into 27MB, well inside the USB drive’s limits; it has an exceptionally helpful and detailed README detailing exactly what needs to be done to create a bootable USB flash drive from its ISO image, using just the generic linux toolchain.

Together with fdisk and parted’s ‘rescue a lost partition’ mode, I was able to get the mangled partition table back into shape, mount the boot disk, change the fstab and grub configuration file, and reboot into a working system. phew!

Many thanks to Kent Robotti, who’s done a great job with RIP.

On the other hard — no thanks to whoever came up with the arcane rules behind the IDE partition table… argh.

Tags: boot, day, desktop, disk, drive, grub, linux, partition, table, tech, usb