Author: Justin

Maciej Ceglowski asks for a massive surveillance program to defeat COVID-19.

However, as I mentioned on twitter — there IS an alternative, privacy-preserving approach, which is what is being done in Singapore with their TraceTogether app.

In summary, everyone carries a phone running an app which has an anonymized a random ID, scans local Bluetooth periodically for other people’s apps with their random IDs, and records them locally (not uploading to a server). If you find out you have COVID-19 you then trigger an upload of your contact history to a central server. That server then broadcasts out the list of IDs, and everyone you’ve been in contact with will then get a ping on their app to get tested, self-isolate, etc.

No central surveillance, no creepy big brother watching your location.

My pinboard has a few more write-ups on basically the same idea from various other places, including MIT. This is similar to what China’s app does, but (as far as I can tell) with more privacy.

It looks like the Singaporean government digital services team behind TraceTogether is putting together an open source version, at Bluetrace.io.

IMO we have to do this or we will never get out of COVID-19 lockdown before 2021. I am massively in favour of adopting this approach in Ireland and across the world.

2 Comments

Here’s a quick tip for people using Huawei or Honor phones.

Huawei recently released EMUI version 9.1.0.326 as an OTA update, which I applied once it was offered as an upgrade option.

Once I installed that OS upgrade, however, I noticed that whenever I listened to music or podcasts using a Bluetooth headset or stereo speakers, there was a new and very noticeable ‘echoing’ effect on the audio.

It appears this was due to the addition of Huawei Histen, a 3D audio/equaliser feature, which apparently will add 3D audio effects when listening on wired headphones of various varieties — however this is supposed to be disabled on Bluetooth devices.

I spent several days fruitlessly googling how to disable Histen, but with no luck. Eventually, through trial and error, I discovered a workaround — simply plug in a pair of wired headphones, go into Settings -> Sounds -> Huawei Histen sound effects, and choose “Natural sound”. Hey presto, next time you use Bluetooth headphones, it should no longer have the echo.

1 Comment

I came across this cocktail in Pals, in Catalonia, in 30 degree heat, a few weeks back — I saw it on the menu at the cafe in the square of the old town, and had to give it a go. It’s incredible. Basically, it’s lager mixed with a lemon granita — like a beer slushy. Nothing is better at thirst quenching on a hot day, and best of all it’s quite low in alcohol so no worries about lorrying into it during the daytime :)

This year at Groovefest, our yearly get together/mini-festival, I got to serve up a few, with great results — they were quite popular. So here’s the recipe!

First off, a day or two in advance, make a batch of lemon granita. I based mine on this recipe which I’ll copy here just in case the original goes away:

Lemon Granita

Serves: about 8

Ingredients:

• 3-4 lemons
• 1L water
• 150g of sugar

Method:

• Zest the lemons and set the zest aside. Juice the lemons until you have 150ml juice (you may not need all of them).

• Add the water and sugar to a large pan and bring to the boil. Reduce to a simmer and cook for 2 minutes, stirring to dissolve the sugar.

• Add the lemon juice and zest, remove from the heat and cover. Set aside to cool for 20 minutes.

• Strain the mixture into 2 containers that will fit in your freezer and leave to cool to room temperature.

• Freeze until the mixture is partially frozen, which should take several hours. (I just left them overnight)

• Remove the granita from the freezer and leave at room temperature until you can break it into chunks with a large spoon or fork.

• Either transfer to a blender or food processor and blitz, or break it up with a fork. It doesn’t need to be perfectly smooth and snowy — a slushy texture is just right for this drink.

• Store in the freezer. Take out 30 minutes before serving and break it up again with a fork.

Clara Con Limón Granizado

To serve: half-fill a half-pint glass with the lemon granita. Pour the beer on top to fill the glass. Stir once or twice to mix. Enjoy!

Comments closed

It’s been a while since I wrote a long-form blog post here, but this post on the Swrve Engineering blog is worth a read; it describes how we use SSD caching on our EC2 instances to greatly improve EBS throughput.

Comments closed

Man sues RMV after driver’s license mistakenly revoked by automated anti-terror false positive:

John H. Gass hadn’t had a traffic ticket in years, so the Natick resident was surprised this spring when he received a letter from the Massachusetts Registry of Motor Vehicles informing him to cease driving because his license had been revoked. […] After frantic calls and a hearing with Registry officials, Gass learned the problem: An antiterrorism computerized facial recognition system that scans a database of millions of state driver’s license images had picked his as a possible fraud. “We send out 1,500 suspension letters every day,” said Registrar Rachel Kaprielian. […] “There are mistakes that can be made.”

See also this New Scientist story. This story notes that the system’s pretty widespread:

Massachusetts bought the system with a $1.5 million grant from the Department of Homeland Security. At least 34 states use such systems, which law enforcement officials say help prevent identity theft and ID fraud.

In my opinion, this kind of thing — trial by inaccurate, false-positive-prone algorithm, is one of the most worrying things about the post-PRISM world.

When we created SpamAssassin, we were well aware of the risk of automated misclassification. Any machine-learning classifier will always make mistakes. The key is to carefully calibrate the expected false-positive/false-negative ratio so that the negative side-effects of a misclassification corresponds to the expected rate.

These anti-terrorism machine learning systems are calibrated to catch as many potential cases as possible, but by aiming to reduce false negatives to this degree, they become wildly prone to false positives. And when they’re applied as a dragnet across all citizens’ interactions with the state — or even in the case of PRISM, all citizens’ interactions that can be surveilled en masse — it’s going to create buckets of bureaucratic false-positive horror stories, as random innocent citizens are incorrectly tagged as criminals due to software bugs and poor calibration.

Comments closed

So, after just over 3 and a half years, I’m leaving Amazon.

It’s been great fun — I can honestly say, even with my code being used by hundreds of millions of users in SpamAssassin and elsewhere, I hadn’t really had to come to grips with the distributed systems problems that an Amazon-scale service involves.

During my time at Amazon, I’ve had the pleasure of building out a brand-new, groundbreaking innovative internal service, from scratch to its current status where it’s deployed in production datacenters worldwide. It’s a low-latency service, used to monitor Amazon’s internal networks using massive quantities of measurement data and machine learning algorithms. It’s really very nifty, and I’m quite proud of what we’ve achieved. I was lucky to work closely with some very smart people during this, too — Amazon has some top-notch engineers.

But time to move on! In a week’s time, I’ll be joining Swrve to work on the server-side architecture of their system. Swrve have a very interesting product, extending the A/B-testing model into gaming, and a great team; and it’ll be nice to get back into startup-land once again, for a welcome change. (It’s not all roses working for a big company. ;) I’m looking forward to it. Who knows, I may even start blogging here again…

Pity about losing those 12 phone tool icons though!

5 Comments

Reminder to Dublin-based readers — next week, Amazon (my employers) will be putting on Under the Hood at Amazon, billed as ‘A night of Beer, Pizza and Cloud Computing for Software Developers’. I’ll be speaking at it.

It’s partially a recruiting event, but even if you’re not looking for a new job, please come along. It’s also useful for us to talk about some details of what we’ve been doing in Dublin, since we’ve been operating to date with a pretty low profile, and in reality there’s some very interesting stuff going on here… particularly the product I’ll be talking about, naturally.

Also, there’ll be free beer and some Kindles to be won ;)

It’s next Thursday night, in our offices in Kilmainham. More info on this Facebook page.

Comments closed

I’m uncomfortable voting for David Norris for President. Here’s why.

In November last year, he was a key voice in a Senate debate on the topic of “Protection of Intellectual Property Rights”, where he quoted heavily from the flawed judgement by Mr. Justice Peter Charleton in the Warner, Universal, Sony BMG and EMI vs UPC case. (There are allegations that he called the debate after speaking to Paul McGuinness (U2’s manager) and Niall Stokes (of Hot Press).)

In the debate, Norris quotes Mr Justice Charleton, saying:

‘In failing to provide legislative provision for blocking, diverting and interrupting internet copyright theft, Ireland is not yet fully in compliance with its obligations under European law.’ Norris then says: ‘Irish law could be brought into alignment with the intention of the European directive through a simple statutory instrument.’ [1]

Now, let me clarify my position — I’m in favour of some means of resolving the level of piracy of music and movies which is widespread nowadays, and I believe there’s a mutually agreeable way to do this. But what Norris and Mr Justice Charleton propose is not it. Here are the problems as I see them.

It Lets The Internet Filtering Genie Out Of The Bottle

The big one.

The problem is that any infrastructure for ‘blocking, diverting and interrupting internet copyright theft’ is effectively infrastructure for ‘blocking, diverting and interrupting’ any communication on the net. We have to be very careful about how this is permitted, as it’ll very quickly suffer “feature creep” and become a general-purpose censorship system — the Great Firewall Of Ireland. As Damien Mulley put it:

‘first they’ll start with the Pirate Bay. Then comes Mininova, IsoHunt, then comes YouTube (they have dodgy stuff, right?), how long before we have Boards.ie because someone quoted a newspaper article or a section of a book? And don’t think they’ll stop there too, any site that links to The Pirate Bay and the others on the hate list will probably be added to the list too…’

In Australia, the anti-child-porn filtering system was quickly used to block gambling websites, gay and straight porn sites, political parties, Wikipedia entries, Christian sites, Wikileaks, and a dentist; in Thailand, a similar system was used to block criticism of the royal family.

Will It Help? I Don’t Think So

Norris:

‘As long as Irish law is deficient, Mr. Justice Charleton has found that all creative Irish industries are losing money.’

This is quite a hilariously overblown and sweeping statement. ALL creative Irish industries? What qualifies as a ‘creative’ industry? I suspect some in this country have been involved in industrial acts of creation that made money. ;)

While they’re not Irish, the well-known indie label Beggar’s Banquet has gone on the record as stating the opposite where the current music situation is concerned —

“There’s fewer gatekeepers now. We don’t have to knock on a TV station’s door or a radio station’s door and it’s made us far more competitive. […] There’s a wide highway in front of us we can go speeding down, and it wasn’t there even two years ago. It means the majors are looking at a world where only 35 Gold Albums a year are certified compared to ten times that recently. But going above Gold in the US is not a problem for us.”

So it appears a ‘creative’ industry (albeit in the UK) is finding things not quite so bad.

Norris again:

‘the facts were established in the judgment of Mr. Justice Charleton in which he stated: “Between 2005 and 2009 the recording companies experienced a reduction of 40% in the Irish market for the legal sale of recorded music.” That is a devastating blow. […] He went on to state: “Some 675,000 people are likely to be engaged in some form of illegal downloading from time to time.”’

Without quite lining up one statement with the other, this reinforces the impression that the only reason the recording companies have seen these drops in revenues is due to internet-borne piracy. However, quoting the brilliant Mumblin’ Deaf Ro on the topic of lies, damn lies, and music biz statistics:

‘The drop in the value of Irish retail music sales was 11.7% between 2008 and 2009, which is significantly less than the 18% overall drop in retail sales for the economy that year. Digital album sales have increased by 30% since 2007 both in terms of volume and market value.’

So in other words, between 2008 and 2009, Irish retail music sales outperformed the retail sales economy as a whole!

In addition, Ro provides the following BPI figures for UK market volumes over the 2005-2009 period:

    Year  Albums  Singles
    2005  159.0m   47.9m
    2006  154.7m   66.9m
    2007  138.1m   86.6m
    2008  133.6m  115.1m
    2009  128.9m  152.7m

It’s clear that singles sales went through the roof, more than tripling. Album sales did drop however, but nowhere near by 40% — and this coincided with the general drop in the prevailing global economy around that time. He also notes that digital sales in the UK went through the roof globally on a number of metrics in 2009.

While this does not provide figures for the Irish market, I’m at a loss as to how it could be radically different — Irish and UK consumers have pretty similar musical tastes and consumption habits, I would guess.

Here’s a theory: perhaps the issue could be that “Irish” music sales are associated with bricks-and-mortar music shops selling the physical product, whereas digital music sales are associated with online services based outside Ireland, and an Irish buyer buying an album at 7digital.co.uk, or on iTunes, isn’t counted as an “Irish retail sale”? Could the problem be that we don’t have any significant Irish shops selling music online, I wonder?

Bricks-and-mortar music shops, such as ex-Senator Donie Cassidy’s “Celtic Note” (who coincidentally was quite vociferous in that Seanad debate), are indeed hurting in this new model of music consumption — and that’s a problem. But given that good, working digital music sales systems are in operation, it doesn’t necessarily appear to be due to massive volumes of internet-borne piracy, going by these figures.

Essentially, internet piracy is a convenient bogeyman, especially for the technophobic old guard, but may have little bearing on the current woes of the Irish record industry and bricks-and-mortar music shops.

(Update: a couple of days after this was posted, a pair of economists at the LSE have said basically the same thing.)

Audible Magic Won’t Work For Long Anyway

Audible Magic, which Norris suggests is IRMA’s favoured filtering system, received the following verdict from the EFF back in 2004:

‘Should Audible Magic’s technology be widely adopted, it is likely that P2P file-sharing applications would be revised to implement encryption. Accordingly, network administrators will want to ask Audible Magic tough questions before investing in the company’s technology, lest the investment be rendered worthless by the next P2P “upgrade.”‘

Naturally, encryption is widespread nowadays, so this may already be the case.

Internet Censorship Harms Our Global Image

As Adrian Weckler points out:

‘do we really want to send out the message that, digitally, we’re the new France? Come to think of it, do we want to tell Google, Facebook, Apple and Twitter that, digitally, we’re the new Britain?’

Right now, more than ever, we need to put out an image that we’re ready to do business on our end of the internet. Mandatory censorship systems don’t exactly support this.

In Summary

So in summary, I would hope to see a more balanced approach to the issue from Norris. Most of the problematic statements in his speech were directly sourced from Mr. Justice Charleton’s flawed judgement, but some critical thinking would be vital, I would have thought. The fact that this was lacking, particularly given the allegations of heavy music-biz lobbying beforehand, leaves me feeling less inclined to vote for him than I would have been before, particularly since I haven’t heard any clarification on these issues.

([1]: Funnily enough, an SI similar to this was nearly sneaked through a couple of weeks ago, according to reports.)

1 Comment

It’s pretty common for apps to require “configuration” — external files which can contain settings to customise their behaviour. Ideally, apps shouldn’t require configuration, and this is always a good aim. But in some situations, it’s unavoidable.

In the abstract, it may seem attractive to use a fully-fledged programming language as the language to express configuration in. However, I think this is not a good idea. Here are some reasons why configuration files should not be expressed in a programming language (and yes, I include “Ruby without parentheses” in that bucket):

Provability

If a configuration language is Turing-incomplete, configuration files written in it can be validated “offline”, ie. without executing the program it configures. All programming languages are, by definition, Turing-complete, meaning that the program must be executed in full before its configuration can be considered valid.

Offline validation is a useful feature for operational usability, as we’ve found with “spamassassin –lint”.

Security

Some configuration settings may be insecure in certain circumstances; for example, in SpamAssassin, we allow certain classes of settings like whitelist/blacklists to be set in a users ~/.spamassassin/user_prefs file, while disallowing rule definitions (which can cause poor performance if poorly written).

If your configuration file is simply an evaluated chunk of code, it becomes more difficult to protect against an attacker introspecting the interpreter and overriding the security limitations. It’s not impossible, since you can, for instance, use a sandboxed interpreter, but this is typically not particularly easy to implement.

Usability

Here’s a rather hairy configuration file I’ve concocted.

    #! /usr/bin/somelanguage
    !$ app.status load html
    !c = []
    ;c['sources'] = < >
    ;c['sources'].append(
        NewConfigurationThingy("foo_bar",
            baz="flargle"))
    ;c['builders'] = < >
    ;c['bots'] = < >
    !$ app.steps load source, shell
    ;bf_mc_generic = factory.SomethingFactory( <
        woo(source.SVN, svnurl="http://example.com/foo/bar"),
        woo(shell.Configure, command="/bar/baz start"),
        woo(shell.Test, command="/bar/baz test"),
        woo(shell.Configure, command="/bar/baz stop")
        > );
    ;b1 = < "name": "mc-fast", "slavename": "mc-fast",
                 "builddir": "mc-fast", "factory": ;bf_mc_generic >
    ;c['builders'].append(;b1)
    ;SomethingOrOther = ;c

This isn’t actually entirely concocted from thin air — it’s actually bits of our BuildBot configuration file, from before we switched to using Hudson. I’ve replaced the familiar Python syntax with deliberately-unfamiliar made-up syntax, to emulate the user experience I had attempting to configure BuildBot with no pre-existing Python knowledge. ;)

Compare with this re-stating of the same configuration data in a simplified, “configuration-oriented” imaginary DSL:

add_source NewConfigurationThingy foo_bar baz=flargle

buildfactory bf_mc_generic source.SVN http://example.com/foo/bar
buildfactory bf_mc_generic shell.Configure /bar/baz start
buildfactory bf_mc_generic shell.Test /bar/baz test
buildfactory bf_mc_generic shell.Configure /bar/baz stop

add_builder name=mc-fast slavename=mc-fast
     builddir=mc-fast factory=bf_mc_generic

Essentially, I’ve extracted the useful configuration data from the hairy example, discarded the symbology used to indicate types, function calls, data structure construction, and let the configuration domain knowledge imply what’s necessary. Not only is this easier to comprehend for the casual reader, it also reduces the risk of syntax errors, by simply minimising the number of syntactical components.

See Also

The Wikipedia page on DSLs is quite good on the topic, with a succinct list of pros and cons.

This StackOverflow thread has some good comments — I particularly like this point:

When you need your application to be very “configurable” in ways that you cannot imagine today, then what you really need is a plugins system. You need to develop your application in a way that someone else can code a new plugin and hook it into your application in the future.

+1.

This seems to be a controversial topic — as you can see, that page has people on both sides of the issue. Maybe it fundamentally comes down to a matter of taste. Anyway — my $.02.

Update: discussions elsewhere: HackerNews

Another Update, 2012-04-06: Robey Pointer wrote a post called Why Config?, in which he describes a Scala-based configuration language in use at Twitter, which uses Scala’s runtime code evaluation, and a Scala trait, to express configuration succinctly in a Scala source file and load it at runtime. The downside? It’s a Scala source file, executed at runtime, containing configuration. :(

However, this comment in the comments section is worth a read:

At Netli (now part of Akamai) we had a configuration framework very similar in spirit and appearance to Configgy. It was in early 2000-s, we open sourced it since. (http://ncnf.sourceforge.net/). It would provide on-the-fly reload for the C-based programs (the ncnf if a C library). It also had some perks like attribute inheritance and a concept of block references. Most importantly though, it contained a separate schema language and a validator to allow configuration be checked before pushing in production. At Netli we used it to configure 1200 services on over 400 hardware boxes, the configuration becoming about 20+mb in length (assembled from several pieces by the CPP, then M4 templating library).

Naturally, it wasn’t Netli’s first attempt at doing configuration. One of the first attempts failed since it was Turing-complete. That approach was to specify the configuration as a Perl data specification. In a very short time the lure of unused expressiveness of such Turing-complete environment prevailed and people started to write for-loops around data pieces and doing other tricks to remove redundancy from the configuration. It turned out to be a disaster in the end, with configuration becoming unmaintainable and flaky.

One principle I got out out of that exercise is that configuration shall not be Turing-complete. We’ve got burned specifically by that property far too many times. Yet I do agree with you that a validation facility is a must-have, which is something not usually part of the simple text-based frameworks. C-based NCNF had it almost from the very beginning though, and it proved to be a very useful harness.

+1. There’s lots more info on that system at this post at lionet.livejournal.com.

Another Update, 2017-05-09: casio_juarez on Twitter:

Dev: I'll use a declarative language for config this time.
6 months later: Let's add variables.
12 mos: And conditionals.
18 mos: Fuck.

— 0x0DEADA55 (@casio_juarez) May 8, 2017

Also related: The Configuration Complexity Clock.

(Image credit: Turn The Dial by VERY URGENT Photography)

15 Comments

Happy new year! Or maybe not. Doh.

Over a year ago, Lee Maguire noticed that a contributed SpamAssassin rule, FH_DATE_PAST_20XX, was naively written — simply to match any date in the year 2010 or later — and would start to false-positive on all mail in 14 months. We made the trivial fix to avoid this (for at least 10 years, by which point the rule would have obsoleted itself through normal means), and I committed it to SVN.

Problem solved, right? Nope. I’d committed to trunk, but in a moment of inattention had forgotten to backport the fix to the stable release branch, 3.2.x, as well. Nobody else noticed the mistake, and several months later, boom:

• LWN
• Slashdot
• Heise
• jwz
• Spam Resource
• Mike Cardwell

Bugger.

Annoyingly, the GA had assigned this rule 3.5 points in the 3.2.0 rescoring run. This meant that the effective default threshold had been lowered from 5.0 points to 1.5, which produced a 2% false positive rate during the first 13 hours of the new year.

After that point, the fix was pushed to the sa-update channel, and anyone who runs sa-update regularly (as they should!) was brought back to normal filtering behaviour.

The rule is superfluous anyway, since it overlaps with a better-written “eval” rule, DATE_IN_FUTURE_96_XX. Accordingly, most likely scenario is that it’ll be removed.

Personally, I see a few lessons from this:

• Obviously, I need to pay more attention. This is easier said than done though, since SpamAssassin has nothing to do with my day job anymore; it’s a spare-time thing nowadays, and that’s a rare resource, unfortunately. :( But still, a chastening result, and I’m very sorry for my part in this screwup.

• We need more active committers on Apache SpamAssassin. If we’d had more eyes, the fact that I’d forgotten to backport the fix might have been spotted. we’re definitely in a better situation now in this regard than we were 6 months ago, so that’s good.

• IMO, this is a good demonstration of how too many simple rules are risky; without careful vetting and moderation, it’s easy for a bad one to slip past. Perhaps we need to move more towards a DNSBL/network-rule driven approach, although this has its downsides too. Still thinking about this.

• It’d be good to fix the GA so that it wouldn’t assign such high points to simple rules like this, without some indication that a human has vetted them and believes them trustworthy.

Daryl posted a good comment on /.:

Clearly we dropped the ball on this one. As far as I know it’s our first big rule screw up in the project’s 10 years. If you’re going to screw up you might as well do it well.

+1 to that!

And to everyone who had to clean up the fallout and spend a holiday recovering lost mails from spam folders… sorry :(

4 Comments

Found here:

On Wednesday 20 May 2009, speaking at a parliamentary Justice Committee debating his new blasphemy law, Dermot Ahern joked that people were making blasphemous comments about him, and he compared his own purity to that of the baby Jesus.

So we have a Justice Minister joking about himself being blasphemed, at a parliamentary Justice Committee discussing his own blasphemy law, that could make his own jokes illegal.

In honour of this Ministerial revelation, we have founded the Church of Dermotology. We believe God sent Dermot Ahern to save Ireland from rational thinking. Our sacred symbol is the Star of Dermot.

Our sacred beliefs are quite similar to those of other religions.

• We believe ice cream wafers are literally the body of Dermot Ahern.
• We believe Dermot Ahern created the universe on Wed 20 may 2009.
• We’re sometimes not sure whether Dermot Ahern really exists.
• We believe it is blasphemous to publish an image of Dermot Ahern.
• We refuse to gather sticks on the Sabbath, which is Wednesday.
• We wear magic underpants that protect us from fire and bullets.
• We are outraged whenever anybody insults our sacred beliefs.
• We fervently support Dermot Ahern’s proposed blasphemy law.
• If it is passed, we will be regularly outraged, and will take test cases.

Like Scientologists, Dermotologists offer a free personality test. Question one: are you vulnerable? Question two: have you money? If you answer yes to either of these questions, you’re in.

After you join, check out the campaign against the Irish blasphemy law at blasphemy.ie.

1 Comment