Perl: I’ve been writing a few convenience web-scrapers recently using WWW::Mechanize, with great success.
So the latest development, HTTP::Recorder, looks very nifty too:
HTTP::Recorder is a browser-independent recorder that records interactions with web sites and produces scripts for automated playback. Recorder produces WWW::Mechanize scripts by default (see WWW::Mechanize by Andy Lester), but provides functionality to use your own custom logger.
… Simply speaking, HTTP::Recorder removes a great deal of the tedium from writing scripts for web automation. If you’re like me, you’d rather spend your time writing code that’s interesting and challenging, rather than digging through HTML files, looking for the names of forms an fields, so that you can write your automation scripts. HTTP::Recorder records what you do as you do it, so that you can focus on the things you care about.
No SSL support yet, though, as far as I can see, but for simple scraping — or as a good starting point for a more complex Mechanize script — it looks like it’ll work great.
Comments closed