ooh, Lascaux 4 is finally opening:
St-Cyr added: “It’s impossible for anyone to see the original now, but this is the next best thing. What is lost in not having the real thing is balanced by the fact people can see so much more of the detail of the wonderful paintings and engravings.”
Johanson said it’s possible to use an RFID “gate antenna” — two electronic readers spanning a doorway, similar to the anti-theft gates in retail stores — to scan the credit cards of people passing through. With enough high-powered gates installed at key doorways in a city or across the country, someone could collect comprehensive information on people’s movements, buying habits and social patterns. “These days you can buy a $500 antenna to mount in doorways that can read every card that goes through it,” Johanson said.Amazingly, these seem to be rife with holes — they still use the legacy EMV protocol, do not require online verification with backend systems, and allow replay attacks. A Journal.ie article today claims that attackers are sniffing EMV data, then replaying it against card readers in shops in Dublin, which while it may not be true, the attack certainly seems viable…
rather dramatic differences
Donald Trump’s media strategy as a form of Surkovian control via post-truth ‘destabilised perception’, through deliberate flooding with fake news:
By attacking the very notion of shared reality, the president-elect is making normal democratic politics impossible. When the truth is little more than an arbitrary personal decision, there is no common ground to be reached and no incentive to look for it. To men like Surkov, that is exactly as it should be. Government policy should not be set through democratic oversight; instead, the government should “manage” democracy, ensuring that people can express themselves without having any influence over the machinations of the state. According to a 2011 openDemocracy article by Richard Sakwa, a professor of Russian and European politics at the University of Kent, Surkov is “considered the main architect of what is colloquially known as ‘managed democracy,’ the administrative management of party and electoral politics.” “Surkov’s philosophy is that there is no real freedom in the world, and that all democracies are managed democracies, so the key to success is to influence people, to give them the illusion that they are free, whereas in fact they are managed,” writes Sakwa. “In his view, the only freedom is ‘artistic freedom.’”
remove RFID from a payment card with a single drilled hole
Nice comparison of a counting Bloom filter and a Cuckoo Filter, implemented in Python:
This post provides an update by exploring Cuckoo filters, a new probabilistic data structure that improves upon the standard Bloom filter. The Cuckoo filter provides a few advantages: 1) it enables dynamic deletion and addition of items 2) it can be easily implemented compared to Bloom filter variants with similar capabilities, and 3) for similar space constraints, the Cuckoo filter provides lower false positives, particularly at lower capacities. We provide a python implementation of the Cuckoo filter here, and compare it to a counting Bloom filter (a Bloom filter variant).
Football Manager includes what is effectively a parallel universe, so they modelled the effects of Brexit on the UK Premier League: ‘In my own current “save”, Brexit kicked in at the end of season three. Unfortunately I got one of the hard options, where all non-homegrown players are now going through a work permit system, albeit one that’s slightly relaxed. It means I can no longer bring in that 19-year-old Italian keeper I’d been eyeing up as one for the future. Instead I have to wait for him to break into the Italian squad, and play 30% of their fixtures over the next two years. Then he’ll be mine. Meanwhile, my TV revenue has just dropped by a few million. Let’s hope that doesn’t continue, or I won’t even be able to afford him.’
It was recently discovered that some surprising operations on Rust’s standard hash table types could go quadratic.Quite a nice unexpected accidental detour into O(n^2)
This is intriguing — using Jupyter notebooks to embody data analysis work, and ensure it’s reproducible, which brings better rigour similarly to how unit tests improve coding. I must try this.
Reproducibility makes data science at Stripe feel like working on GitHub, where anyone can obtain and extend others’ work. Instead of islands of analysis, we share our research in a central repository of knowledge. This makes it dramatically easier for anyone on our team to work with our data science research, encouraging independent exploration. We approach our analyses with the same rigor we apply to production code: our reports feel more like finished products, research is fleshed out and easy to understand, and there are clear programmatic steps from start to finish for every analysis.
neat — aggregation of histograms for Datadog statsd
auditd -> go-audit -> elasticsearch at Slack
Eir ship vulnerable firmware images AGAIN. ffs
Amazing virtuoso performance — be sure to scroll up all the way to Chapter 1
good call — new EMR feature
LMAX’ approach to acceptance/system-testing time-dependent code. We are doing something similar in Swrve too, so finding that LMAX have taken a similar approach is a great indicator
scumbags. Attempting to pass off their pissy beer under alternative names to con consumers into buying it! ‘There will be no sanctions against Heineken for passing off non-craft beer as “locally produced”, the Food Safety Authority of Ireland (FSAI) has said. The FSAI and HSE launched a joint investigation last month after it emerged that Heineken Ireland had sold some of its products, including Foster’s lager, under craft-type names such as Blasket Blonde and Beanntrai Bru. Two well-known stouts, Beamish and Murphy’s, were also sold under craft-type names by the international brewing giant. C&C, a Tipperary-based drinks company, was also investigated after it admitted selling its Clonmel 1650 lager under a different name, Pana Cork, in Cork.’
great, I’ve looked for this so many times. Only tricky limit I can spot is the 300 tps limit, and it’s US-East/US-West only for now
good intro to Airflow usage preso
by John Allspaw, Morgan Evans and Daniel Schauenberg; the Etsy blameless postmortem style crystallized into a detailed 27-page PDF ebook
‘bike-shedding’, or needless arguing about trivial issues, actually dates back to 1957 as C. Northcote Parkinson’s ‘law of triviality’
simple usage of Docker, blue/green deploys, and AWS ALBs
ICRs are the perfect material for blackmail, which makes them valuable in a way that traditional telephone records are not. And where potentially large sums of money are involved, corruption is sure to follow. Even if ICR databases are secured with the best available technology, they are still vulnerable to subversion by individuals whose jobs give them ready access. This is no theoretical risk. Just one day ago, it emerged that corrupt insiders at offshore call centres used by Australian telecoms were offering to sell phone records, home addresses, and other private details of customers. Significantly, the price requested was more if the target was an Australian “VIP, politician, police [or] celebrity.”
a low-cost online vendor in Ireland, recommended by @irldexter on ITS (along with webdoctor.ie): ‘For basic consultations I halved the cost €55 to engage a GP with https://www.webdoctor.ie/ down to €25 (for limited domains) and after paying €8.48 and €9.48 respectively for a Ventolin inhaler, I now get them for €3.50 at http://www.purepharmacy.ie/ (closer to mainland EU costs). I also benchmarked my parents medicine costs which worked out 40% cheaper too.’
“The scale of the challenge here remains depressing,” says the report. “It has never been viable to build apartment blocks in the vast majority of this country.” […] The report notes that the rise in living costs of almost three quarters in less than five years is “a symptom of strong demand for housing” as economic recovery continues and the population grows. “But there is nothing inevitable about housing costs rising with demand,” it says. “That only happens when supply fails to respond, and the complete absence of any meaningful level of construction over the past five years is a systemic failure in desperate need of policy solutions. “There is no more urgent task facing the Minister for Housing, his department and advisers, and the Housing Agency, than understanding why the costs of building, and building apartments in particular, is so dramatically out of line with our own incomes and indeed with the cost in other countries.”
I’m not remotely interested in shockingly good graphics, in murder simulators, in guns and knives and swords. I’m not that interested in adrenaline. My own life is thrilling enough. There is enough fear and hatred in the world to get my heart pounding. My Facebook feed and Twitter feed are enough for that. Walking outside in summer clothing is enough for that. I’m interested in care, in characters, in creation, in finding a path forward inside games that helps me find my path forward in life. I am interested in compassion and understanding. I’m interested in connecting. As Miranda July said, “all I ever wanted to know is how other people are making it through life.” I want to make games that help other people understand life. We are all overwhelmed with shock, with information, with change. The degree of interactivity in our lives is amazing and wonderful and I wouldn’t exchange it for anything, but it is also shocking and overwhelming and it’s causing us to dig in and try to find some peace by shutting each other out. On all sides of the political spectrum we’ve stopped listening to each other and I fear we are all leaning toward fascist thinking. We should be using this medium to help us adapt to our new, interactive lives. This is how we become relevant.
“Any financial loss that results from this fraudulent activity will be borne by the bank,” Mr Higgins said. “Customers are not at financial risk.”Well, that would be surprising….
Hooray for nuclear power. (via Ossian Smyth)
Ivan’s Childhood, Andrei Rublev, Solaris, The Mirror, and Stalker — all viewable for free on YouTube thanks to Mosfilm. quality not great though….
This page contains lecture notes and other course materials for various algorithms classes I have taught at the University of Illinois, Urbana-Champaign. The notes are numbered in the order I cover the material in a typical undergraduate class, wtih notes on more advanced material (indicated by the symbol ?) interspersed appropriately. […] In addition to the algorithms notes I have been maintaining since 1999, this page also contains new notes on “Models of Computation”, which cover a small subset of the material normally taught in undergraduate courses in formal languages and automata. I wrote these notes for a new junior-level course on “Algorithms and Models of Computation” that Lenny Pitt and I developed, which is now required for all undergraduate computer science and computer engineering majors at UIUC.Via Tony Finch
“I started the site for a easy way to make money,” said a 17-year-old who runs a site [from Veles] with four other people. “In Macedonia the economy is very weak and teenagers are not allowed to work, so we need to find creative ways to make some money. I’m a musician but I can’t afford music gear. Here in Macedonia the revenue from a small site is enough to afford many things.”
‘an antagonistic GSM base station [disguised] in the form of an innocuous office printer. It brings the covert design practice of disguising cellular infrastructure as other things – like trees and lamp-posts – indoors, while mimicking technology used by police and intelligence agencies to surveil mobile phone users.’
wow, Docker Swarm looks like a turkey right now if performance is important. Only “host” gives reasonably perf numbers
Subreddit devoted to becoming a software developer in Ireland, with a decent wiki
In short, the answer to the question “is this what it would look like if I was there?” is almost always no, but that is true of every photograph. The photos taken from space cameras are no more fake or false than the photos taken from any camera. Like all photos they are a visual interpretation using color to display data. Most space photos have information online about how they were created, what filters were used, and all kinds of interesting details about processing. The discussion about whether a space photo is real or fake is meaningless. There’s no distinction between photoshopped and not. It’s a nuanced view but the nature of the situation demands it.
LOL as DST bug uncovers spurious automated noise complaints:
In January last year the airport unearthed a scheme whereby campaigners were using automated software to generate complaints against the airport. Officials caught out the set-up when the two anti-Heathrow enthusiasts forgot to take into account the hour going back in October, and began complaining about flights that had not yet taken off or arrived.
Well, this is amazingly awful:
The Guardian claims to have further details of the kind of tell-tale signs that Admiral’s algorithmic analysis would have looked out for in Facebook posts. Good traits include “writing in short concrete sentences, using lists, and arranging to meet friends at a set time and place, rather than just ‘tonight’.” On the other hand, “evidence that the Facebook user might be overconfident—such as the use of exclamation marks and the frequent use of ‘always’ or ‘never’ rather than ‘maybe’—will count against them.”The future is shitty.
An improved hashing algorithm called optimistic cuckoo hashing, and a CLOCK-based eviction algorithm that works in tandem with it. They are evaluated in the context of Memcached, where combined they give up to a 30% memory usage reduction and up to a 3x improvement in queries per second as compared to the default Memcached implementation on read-heavy workloads with small objects (as is typified by Facebook workloads).
the coming incompatibility nightmare of USB-C cabling
Elasticache now supports sharding
The square root staffing law is a rule of thumb derived from queueing theory, useful for getting an estimate of the capacity you might need to serve an increased amount of traffic.
See also https://www.percona.com/blog/2016/02/05/measuring-docker-cpu-network-overhead/ for the CPU/Network equivalent. The good news is that nowadays it’s virtually 0 when the correct settings are used
Luis Villa: ‘Is the React license elegant? No. Should you be worried about using it? Probably not. If anything, Facebook’s attempt to give users an explicit patent license should probably be seen as a good faith gesture that builds some confidence in their ecosystem. But yeah, don’t use it if your company intends to invest heavily in React and also sue Facebook over unrelated patents. That… would be dumb. :)’
Writeup of their Consul-based service discovery system, a bit similar to smartstack. Good description of the production problems that they saw with Consul too, and also they figured out that strong consistency isn’t actually what you want in a service discovery system ;) HN comments are good too: https://news.ycombinator.com/item?id=12840803
Kalina Bontcheva leads the EU-funded PHEME project working to compute the veracity of social media content. She said reducing the amount of human oversight for Trending heightens the likelihood of failures, and of the algorithm being fooled by people trying to game it. “I think people are always going to try and outsmart these algorithms — we’ve seen this with search engine optimization,” she said. “I’m sure that once in a while there is going to be a very high-profile failure.” Less human oversight means more reliance on the algorithm, which creates a new set of concerns, according to Kate Starbird, an assistant professor at the University of Washington who has been using machine learning and other technology to evaluate the accuracy of rumors and information during events such as the Boston bombings. “[Facebook is] making an assumption that we’re more comfortable with a machine being biased than with a human being biased, because people don’t understand machines as well,” she said.
Crockford chose not to version [the] JSON definition: ‘Probably the boldest design decision I made was to not put a version number on JSON so there is no mechanism for revising it. We are stuck with JSON: whatever it is in its current form, that’s it.’ Yet JSON is defined in at least six different documents.“Boldest”. ffs. :facepalm:
We can’t easily fix the already broken devices, we can’t easily stop more broken devices from being shipped and we can’t easily guarantee that we can fix future devices that end up broken. The only solution I see working at all is to require ISPs to cut people off, and that’s going to involve a great deal of pain. The harsh reality is that this is almost certainly just the tip of the iceberg, and things are going to get much worse before they get any better.
This is shocking:
At the end of the panel on artificial intelligence, a young black woman asked [Sebastian Thrun, CEO of the education startup Udacity, who is best known for founding Google X] whether bias in machine learning “could perpetuate structural inequality at a velocity much greater than perhaps humans can.” She offered the example of criminal justice, where “you have a machine learning tool that can identify criminals, and criminals may disproportionately be black because of other issues that have nothing to do with the intrinsic nature of these people, so the machine learns that black people are criminals, and that’s not necessarily the outcome that I think we want.” In his reply, Thrun made it sound like her concern was one about political correctness, not unconscious bias. “Statistically what the machines do pick up are patterns and sometimes we don’t like these patterns. Sometimes they’re not politically correct,” Thrun said. “When we apply machine learning methods sometimes the truth we learn really surprises us, to be honest, and I think it’s good to have a dialogue about this.”“the truth”! Jesus. We are fucked
Using a proxy, remove “securityQuestion0” and “securityQuestion1” from the post data.Massive facepalm.
amazing architectural-oddities Tumblr (via Present and Correct)
I’ve had the privilege of experiencing a few different management levels (responsibilities? jobs?) at Etsy since I’ve joined. At each stage, I felt like the job of being a manager totally changed. What I did day-to-day changed, what was hard about it changed, how I measured my own success changed, and though I feel like the experiences built on one another, it continues to be an enormous shift in brainpower each time the gig changes a bit. Given how intangible (and often hidden) management work can be, I’ve outlined some highlights of what my work has been like as a manager over the last four years. (Obvious, major caveat: this is just my experience, and there’s lots in here that is unique to this particular work environment, hierarchy, requirements, and challenges!)
Cyril Connolly, the 50s critic and writer, said that the greatest enemy of creativity is the pram in the hall, but I think that was completely wrong. It was the enemy of a certain kind of dilettante life that he aspired to, the man of letters, but for the real novelist the pram in the hall is the greatest ally – it brings you up sharp and you realise what reality is all about. My children were a huge inspiration for me. Watching three young minds creating their separate worlds was a very enriching experience.
holy crap this is going to be a serious problem
may need this, depending on kiddie preferences this year ;)
Excellent post on team voice comms tactics. Many tips here
Love the ‘decade of’ dig at FB and Amazon — ‘we were doing it first’ ;) Great details on how Google have built out and improved their DC networking. Includes a hint that they now use DCTCP (datacenter-optimized TCP congestion control) on their internal hosts….
pretty amazing. full 60FPS, 2560×1600, everything on Epic quality, streaming from Azure, for $2 per hour
oh god this is funny. Louise Mensch and various UKIPpers fall for transparent pisstake involving “taking Article 50 out of the ring binder and shredding it. It now goes straight from 49 to 51” etc.
What makes an image NSFW, according to Yahoo? I explore this question with a clever new visualization techniqueDeep Dream applied to an NSFW classifier. This is a bit NSFW, as it happens
stack trace leads with a symbol called “_thereIsNoSadnessLikeTheDeathOfOptimism”
interesting presentation describing how to architect Airflow ETL setups; see also https://gtoonstra.github.io/etl-with-airflow/principles.html
‘Generally, a very small number of people account for a disproportionately high share of the total number of noise complaints. In 2015, for example, 6,852 of the 8,760 complaints submitted to Ronald Reagan Washington National Airport originated from one residence in the affluent Foxhall neighborhood of northwest Washington, DC. The residents of that particular house called Reagan National to express irritation about aircraft noise an average of almost 19 times per day during 2015.’ Somebody needs help.
good advice on how to ETL
the Women on Web approach is backed by a column in New Scientist: ‘It’s also safer than many other medicines that we are allowed to buy from pharmacies without a prescription, such as Viagra in the UK. So why can’t women get abortion pills from pharmacies and manage the process themselves at home if they choose? It might sound radical but it’s already widespread in countries where abortion is illegal, with women buying the pills from online pharmacies. While some countries, such as Poland, are trying to tighten their already strict abortion laws, the advent of mail-order abortion pills means the law is becoming almost irrelevant.’
This is amazing — what a wonderland! For instance:
Six dollars for: a GSM chipset, a CPU, an LCD screen, a battery, a PCB, a metal housing, a molded silicone watch band, a MicroUSB cable, and a box. And the labor to assemble and test all of that.
The UK’s version of the POD database project was used by the Home Office to track immigrants for various reasons — in other words, exactly the reasons why parents will choose not to provide that data
One “outlier” voter—a 19-year old black Trump supporter—was weighted so heavily that it shifted the whole poll significantly. Stats fail
this is a nice zero-config scaling story — good work Kafka Streams
‘a Mexican cerveza preparada made with beer, lime juice, and assorted sauces, spices, and peppers. It is served in a chilled, salt-rimmed glass. There are numerous variations of this beverage throughout Mexico and Latin America.’
a variation on the mimosa, with an IPA and grapefruit juice — sounds like Brewdog’s Elvis Juice
‘This crisp, refreshing beer and ginger beer cocktail is a Cuban favorite, and it’s super-simple to make.’
via Saladin Ahmed — the scientist who coined the term abandoned it as useless years ago:
The concept of the alpha wolf is well ingrained in the popular wolf literature at least partly because of my book “The Wolf: Ecology and Behavior of an Endangered Species,” written in 1968, published in 1970, republished in paperback in 1981, and currently still in print, despite my numerous pleas to the publisher to stop publishing it. Although most of the book’s info is still accurate, much is outdated. We have learned more about wolves in the last 40 years then in all of previous history. One of the outdated pieces of information is the concept of the alpha wolf. “Alpha” implies competing with others and becoming top dog by winning a contest or battle. However, most wolves who lead packs achieved their position simply by mating and producing pups, which then became their pack. In other words they are merely breeders, or parents, and that’s all we call them today, the “breeding male,” “breeding female,” or “male parent,” “female parent,” or the “adult male” or “adult female.” In the rare packs that include more than one breeding animal, the “dominant breeder” can be called that, and any breeding daughter can be called a “subordinate breeder.”
20km of private monorail owned and operated by the electricity grid, used to monitor and inspect one of the longest 400kV circuits in Europe
ugh, 213ms mean response overhead
Full AirBnB property management service in Dublin
According to the bankruptcy documents, one of the assets listed is ‘a chair worth 2,902,119JPY, or roughly $28,000USD.’
“Chinese internet giant Tencent” have been granted a USPTO patent for drawing a box on a screen.
This is the new fear — that FF/FG will accidentally and stupidly disengage Ireland from the EU as a side effect of trying to keep the UK happy and cross-border trade intact
Score 1 “My management culture is fucked up” point for each of the following: We have an unlimited vacation policy; We don’t do regular 1:1s, but we have open office hours/are super available if anyone wants to chat; We don’t have a process for interviewing, we just hire awesome people when we meet them; We super care about diversity, but we don’t want to lower the bar so we just hire the best person for the job even if it means diversity suffers; We don’t have defined levels and career paths for our employees, we’re a really flat org; We don’t have formal managers for every staff member, everyone just gets their work done; We don’t have, like, HR HR, but our recruiter/office manager/only female employee is super good if you want someone to talk to; We don’t do performance improvement plans for employees that are struggling. We just have a super honest conversation about how they aren’t a good fit and fire them; We would have some hard explaining to do if our salary list accidentally became public.
Great font factoid: ‘The name “Noto” comes from the little squares that show when a font is not supported by a computer. This are often referred to as “tofu”, because of their shape, therefore the font is short for No Tofu.’
Specifically, the following 3 classes of errors were implicated in 92% of the major production outages in this study and could have been caught with simple code review:
Error handlers that ignore errors (or just contain a log statement); error handlers with “TODO” or “FIXME” in the comment; and error handlers that catch an abstract exception type (e.g. Exception or Throwable in Java) and then take drastic action such as aborting the system.(Interestingly, the latter was a particular favourite approach of some misplaced “fail fast”/”crash-only software design” dogma in Amazon. I wasn’t a fan)
An excellent explanation of what is going on in the UK right now. What a nightmare:
Finally there are the self-styled buccaneers of the free-trade seas. Boris Johnson would probably cast himself as Sir Walter Raleigh – polymath, wordsmith, adventurer. That leaves Liam Fox to play Sir Francis Drake, looking for domestic glory in global circumnavigation but seen from abroad as a pirate. This is all myth and fantasy, of course. But parties have always been sustained by internal mythologies, and the task of exiting the EU is so complicated and fraught with danger that fantasy becomes a necessary comfort. As one former minister says of the puritan choristers: “They have spent their lives working towards this dream. Of course they don’t want to accept that it’s a nightmare.” Tory pro-Europeans are in the impossible position of using rational argument against faith. If they counsel compromise on migration or the single market, they are accused of talking Britain down or trying to refight the referendum. They have few reinforcements across the political water. Labour is a shambles. The Lib Dems are puny in parliament. Scotland has its own distinct politics, and in Nicola Sturgeon its own remainian queen with her own independence agenda. The Tories do not speak for all of England, but in the absence of credible opposition they feel as if they do, and will act accordingly. To those millions who did not vote to leave the EU, the message is clear: you are free to pray for whatever you like. Your antique rites will be tolerated. But do not expect your concerns to be represented in the court of Queen Theresa. Be humble instead. Swallow your doubts and take a pew in the reformed national church of Brexit.
Martin Fowler’s take on the 4 kinds of tech debt
This is a great blog post, spot on:
You can’t just go “dudes it’s faster” and jump off a cliff. This shit is basic. Test real production workloads. Have a rollback plan. (Not for *10 days* … try a month or two.)The only thing I’d nitpick on is that it’s all very well to say “buy my book” or “come see me talk at Blahcon”, but a good blog post or webpage would be thousands of times more useful.
Excellent talk. I love this analogy for ML applied to real-world data which affects people:
Treating the world as software promotes fantasies of control. And the best kind of control is control without responsibility. Our unique position as authors of software used by millions gives us power, but we don’t accept that this should make us accountable. We’re programmers—who else is going to write the software that runs the world? To put it plainly, we are surprised that people seem to get mad at us for trying to help. Fortunately we are smart people and have found a way out of this predicament. Instead of relying on algorithms, which we can be accused of manipulating for our benefit, we have turned to machine learning, an ingenious way of disclaiming responsibility for anything. Machine learning is like money laundering for bias. It’s a clean, mathematical apparatus that gives the status quo the aura of logical inevitability. The numbers don’t lie.Particularly apposite today given Y Combinator’s revelation that they use an AI bot to help ‘sift admission applications’, and don’t know what criteria it’s using: https://twitter.com/aprjoy/status/783032128653107200
So assuming the mission continues well, in 2014 the Rosetta Probe will land on Comet 67P/Churyumov-Gerasimenko, where it will measure the comet’s molecular composition. Then it will remain at rest as the comet orbits the sun for hundreds of millions of years. So somewhere in the solar system, where it is safe but hard to reach, a backup sample of human languages is stored, in case we need one.As jwz says: ‘The Rosetta Disc is now safely installed on 67P/Churyumov-Gerasimenko.’
Some tantalising discussion on twitter of an Airflow + AMI + ASG workflow for ML packaging: ‘We build models using Airflow. We deploy new models as AMIs where each AMI is model + scoring code. The AMI is hence a version of code + model at a point in time : #immutable_infrastructure. It’s natural for Airflow to build & deploy the model+code with each Airflow DAG Run corresponding to a versioned AMI. if there’s a problem, we can simply roll back to the previous AMI & identify the problematic model building Dag run. Since we use ASGs, Airflow can execute a rolling deploy of new AMIs. We could also have it do a validation & ASG rollback of the AMI if validation fails. Airflow is being used for reliable Model build+validation+deployment.’