taint.org: Justin Mason’s Weblog

TV: I’ve taken a little time to throw up my PVR build log.

If you’re hacking on one yourself, or curious about what it takes, or just like reading cut-and-pasted UNIX command lines — go take a look!

Tags: command, look, pvr, reading, time, tv, unix

UNIX: a quick recap of a good tip combo picked up from ILUG recently. To paraphrase Conor Wynne’s original question:

What’s the best way to set up a secure connection between two hosts, possibly over the internet, using SSH, suitable for use from cron so that it can run via crontab without entering authentication manually?

Barry O’Donovan replied:

I suggested ssh keys without passphrases … in
http://www.barryodonovan.com/publications/lg/104/ and it includes instructions. … You can invoke rsync over ssh and specify a specific key with:

rsync -a -e ’ssh -i /home/username/.ssh/id_rsa-serverbackup’

Colm MacCárthaigh followed up with:

You can restrict what commands an ssh account can run in the ssh public key. This is how some of our more important projects (like Debian, FreshRPMS, and a few more) push us updates. The key looks like (jm: all on one line, no space between ‘no-pty,’ and ‘command’):

no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty, command=”/home/ximian/rsync-ximian-nolog &”
ssh-dss keydata username@blah

So, create a passwordless public key like so, and just change the command to whatver rsync runs.

Combined, that’s a useful tip — I knew about the ssh command restriction technique, but being able to use a specific single-purpose key from the ssh client is very useful.

(updated: mbp mailed to note some missing quotes in Barry’s command above; they’d been eaten by WebMake. drat.)

Tags: combo, command, home, ilug, key, recap, rsync, ssh, tip, unix, username

UNIX: another useful tip. Bash supports a wide variety of command line editing tricks; you have the usual GUIish editing (backspace, insert new characters, delete, blah blah) through the GNU Readline library, and in addition to that you have the traditional csh-style history expansion (like ‘!!’ to refer to the previous command typed).

The latter are great, but they won’t actually be expanded until you hit Enter and run the command line. That can be inconvenient, resulting in the user being forced to reach for the rodent for some cut’n'paste instead.

Here’s a handy trick — add this line to ~/.inputrc (creating the file if necessary):

Control-x: shell-expand-line

Start a new bash shell. Now, if you type CTRL-X during command line entry, any shell metacharacters will be expanded on the current command line. For example:

% echo Hello world
Hello world

% echo Hi !$       (press CTRL-X)
           (current command line expands to:)
% echo Hi world

There’s a few more commands supported, but none of them are really quite as useful as shell-expand-line.

Update: ‘Smylers’ wrote to point me at this UKUUG talk from 2003 which discusses .inputrc expansions, and provides some insanely useful tips.

In particular, Magic Space clearly knocks this tip into a cocked hat, by performing the expansion on the fly as you type the command, with no additional keypresses — amazing! Bonus: it works if you use Emacs-mode line editing as well as Vi-mode.

I strongly recommend reading that paper — lots of other good tips there.

Tags: bash, blah, command, editing, expansion, inputrc, line, shell, type, unix, world

Tech: So Danny O’Brien’s ‘Life Hacks’ talk is one of the most worthwhile reflections on productivity (and productivity technology) I’ve heard. (Cory Doctorow’s transcript from NotCon 2004, video from ETCon.)

There’s a couple of things I wanted to write about it, so I’ll do them in separate blog entries.

(First off, I’d love to see Ward Cunningham’s ‘cluster files by time’ hack, it sounds very useful. But that’s not what I wanted to write about ;)

People don’t extract stuff from big complex apps using OLE and so on; it’s brittle, and undocumented. Instead they write little command-line scriptlets. Sometimes they do little bits of ‘open this URL in a new window’ OLE-type stuff to use in a pipeline, but that’s about it. And fundamentally, they pipe.

This ties into the post that reminded me to write about it — Diego Doval’s atomflow, which is essentially a small set of command-line apps for Atom storage. Diego notes:

Now, here’s what’s interesting. I have of course been using pipes for years. And yet the power and simplicity of this approach had simply not occurred to me at all. I have been so focused on end-user products for so long that my thoughts naturally move to complex uber-systems that do everything in an integrated way. But that is overkill in this case.

Exactly! He’s not the only one to get that recently — MS and Google are two very high-profile organisations that have picked up the insight; it’s the Egypt way.

There’s fundamentally a breakage point where shrink-wrapped GUI apps cannot do everything you want done, and you have to start developing code yourself — and the best APIs for that, after 30 years, has been the command-line and pipe metaphor.

(Also, complex uber-apps are what people think is needed — however, that’s just a UI scheme that’s prevailing at the moment. Bear in mind that anyone using the web today uses a command line every day. A command line will not necessarily confuse users.)

Tying back into the Life Hacks stuff — one thing that hasn’t yet been done properly as a command-line-and-pipe tool, though, is web-scraping. Right now, if you scrape, you’ve got to do either (a) lots of munging in a single big fat script of your own devising, if you’re lucky using something like WWW::Mechanize (which is excellent!); (b) use a scraping app like sitescooper; or (c) get hacky with a shell script that runs wget and greps bits of output out in a really brittle way.

I’ve been considering a ‘next-generation sitescooper’ a little bit occasionally over the past year, and I think the best way to do it is to split its functionality up into individual scripts/perl modules:

• one to download files, maintaining a cache, taking likely freshness into account, and dealing with crappy HTTP/HTTPS wierdness like cookies, logins and redirects;
• one to diff HTML;
• one to lobotomise (ie. simplify) HTML;
• one to scrape out the ‘good bits’ using sitescooper-style regions

Tie those into HTML Tidy and XMLStarlet, and you have an excellent command-line scraping framework.

Still haven’t got any time to do all that though. :(

Tags: brien, command, everything, hacks, life, line, pipe, productivity, stuff, tech, way

UNIX: I’ve just made the first change to my core bash configuration in years, to add -b to the set command-line. It triggered some thinking about when the last one was.

It turns out, that apart from writing scripts and aliases frequently, I haven’t changed my commandline UI in any respect, since about 2 years ago. By contrast, I’ve been hacking about with GUI settings continually, new desktop backgrounds, themes, colours, etc. Odd!

Anyway, here’s the tip — it’s very handy, I find.

I changed to using a 2-line prompt, with the first line containing the time and the full working directory, in a ‘magic’ cut-and-pasteable format:

        : exit=0 Thu Jun 24 17:55:29 PDT 2004; cd /home/jm/DL
        : jm 1203...; 

Note that the prompt starts with “:”, which means that bash/sh will ignore the line until it hits “;”. The end result is that the entire line evaluates to “cd /home/jm/DL” when pasted. Hey presto, cd’ing several terminals to the same dir just involves triple-clicking in one, and middle-button-pasting into the others. nifty! Similarly, the second line has a little bit of prompt, but that snippet will be ignored when cut and pasted.

Having the exit status of the last command (bash var: $?) is useful too. The code:

  do_prompt () {
    echo ": exit=$? `date`; cd $PWD"
  }
  PROMPT_COMMAND='do_prompt $?'   # executed before every prompt
  do_prompt 0                     # set up first prompt
  PS1=": `whoami` \!"
  PS2="... >>; "            # continuation prompt
  PS1="$PS1...; "

Tags: bash, change, command, commandline, configuration, core, exit, home, line, respect, unix

Software: OK, one of my current UNIX pet peeves, perfectly illustrated by the new RPMs for KDE 3.2.

  : jm 1015...; sudo rpm -Uvh *.rpm 
  Password:
  error: Failed dependencies:
      libiw.so.26 is needed by kdenetwork-3.2.0-0.1

I don’t have a wireless card in this machine.

WHY does kdenetwork, a network configuration applet, link with a shared library component of the wireless-tools package? Why is this not simply a shell script, or even an optional binary command? Have the UNIX desktop environments forgotten all about the UNIX way in their rush to implement ‘components’? To quote Doug McIlroy :

This is the Unix philosophy. Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

(my emphasis added.)

Hint: if you don’t intend to call some third-party code over and over
again several times a second — in other words, so that performance is essential — you do not need to link against it as a shared library. Calling it as a command, with fork and exec, will work just fine and avoids this kind of ‘DLL hell’.

A related issue is how this emphasis on binary or component ABIs impacts scriptability and plugins. Ever since Netscape came up with their plugins, we’ve had this new model that third-party application extensibility meant linking shared libraries into the app (with ABI issues), or calling out to components over distributed-object transports like CORBA or MCOP (with API issues), instead of the traditional ‘helper app’ model.

As a result, generally, when I install a new version of Mozilla, I have to try and remember what plugins I had in the last one, track them down, download the latest version to work around ABI changes, and hope they work in this version of the browser.

Inevitably, they don’t — I haven’t found a working Java plugin in over a year. On the other hand, I can always click on a .ram link to listen to a RealAudio stream, because it doesn’t really matter if the browser and realplayer were built with different compilers in the ‘helper app’ case.

In addition, and paradoxically, scriptability is becoming less of an option in the modern UNIX GUI apps. Let’s say I want to be able to do the kind of thing Windows has had for years with it’s ‘Send To’ menu; put a simple shell script into an ‘actions’ directory, and it’ll appear in the right-click context menu, so that I can right-click on a file and select ‘Run frobnicator’ to frobnicate it. (Similar is possible from MS Internet Explorer.)

Is it possible in Firebird? Not a hope. But you can write an extension – 100KB of undocumented Javascript. Great.

In fairness, the file managers have the right idea — GNOME’s Nautilus does support this nicely, and so does Konqueror. But there’s an ongoing tendency to adopt the ABI dynamic-linking model, or the distributed-object model, in places where it’s just not necessary, and a simple UNIX pipe or command API — the ‘helper app’ model — would work beautifully.

hmm. </rant> ;)

Tags: command, component, emphasis, kdenetwork, library, link, rpm, software, unix, version, write

Ben mails on this Yahoo! link: ‘U.S. Central Command released these digitally enhanced photographs of what former Iraqi dictator Saddam Hussein might look like’. As he notes, let’s face it, it’s crying out for the SomethingAwful Photoshop Phriday treatment. However, I’m too cheap to have an SA forum account. Anyone got one and care to suggest it?

Tags: central, command, dictator, face, hussein, link, photoshop, saddam, somethingawful, yahoo