Recently we had the national census debacle, quickly badged on Twitter as #censusfail. How cruel and biting Twitter can be brought about by the need for brevity. There was an ominous feeling in the air the day of the census with the task presaged by fears of data loss, privacy concerns and overly long data retention. The census became a political football well before the day we were supposed to complete it. Scott Ludlam, Greens Senator, was a prime mover in the scaremongering and there was encouragement for the public not to file a census return, or at the very least to not fill it out completely. Strange given how much many of the naysayers freely share on their Facebook page. Foolhardy too given this is the data upon which our nation plans.

What the census was for Australia was an opportunity to demonstrate that at a national level we could pull off a digital solution. We are, after all, in transition from resource country and on a pivot to a creative and innovative one. Innovation nowadays is synonymous with technology especially digital technology so it was in some way a test to see how far along our transformation of the economy we have come. Not very far if the experience of the night was anything to go by. The facts of the census day are now forever emblazoned in our nation’s history filed under the ‘cock-ups’ section. However for my international readership let me give you a potted history. National census day set down for 9^th August where most people were down to log-in to the government’s census website and fill out the form. Things go well until about 7pm and the system crashes. Takes a number of days to get it up and running again with a myriad of excuses and absurd assertions made in between time. Egg on face for government, necks on the line in the Australian Bureau of Statistics (ABS) and Sherlock out at the moment looking for IBM’s Watson who seemed to go missing on the night. Pretty much egg on face all round.

So the first point of concern is we failed at the first hurdle when it came to providing a bug-free hassle-free experience in loading our data on-line. This does not bode well for transforming our economy into one of a highly digital and innovative nature. Secondly the thinking under-pinning the census completion seemed to completely flawed from a statistical perspective. This is extremely worrying given the Australian Bureau of Statistics had carriage of the census project. As I understand it the ABS had established a hot rate capability of 1 million hits per hour. With 24 hours to fill it out and 15 million citizens due to complete it this would appear to have enough margin to take the traffic. The only flaw here is that on-line completion would never be a flat-rate across the day but would rather cluster around certain periods. One would expect that the real pattern would look something like an errrr…normal distribution (aka bell shaped curve or Gaussian distribution). The irony here is that the normal distribution is meat and two veg to a statistician. I don’t think the technology therefore was necessarily at fault but rather our lack of understanding of basic statistics and the patterns of on-line behaviour of the general populace. Very concerning indeed. If we can’t read basic data how can we hope to deal with big data?

I found the concerns about a hack (the initial excuse for the site imploding) hard to fathom. We seem overly eager to blame a hack with its inherent overseas bogeymen sentiments all too often nowadays. If we want to be a smart nation then maybe putting some of our apparent intellectual and academic heft into suitable encryption to fend off such hacks would be a good first project to get underway. The underlying presumption in blaming the overseas hacks, most often sheeted back to the Chinese Government, is that in the spy versus spy thrust and counter-thrust of espionage etc. is that we aren’t winning the battle. If the overseas hackers are better than us what hope for our  economy transitioning into a digital tour de force when a few Chinese students in Beijing can bring down our census website?

What did emerge on the night of the census site meltdown was a pretty funny Twitter exchange as lots of people vented their frustration. Much of it in good humour I must admit and I did my best to add my own pithy contribution. I commented at work that if it was Ticketek selling the much anticipated Adele tickets for a tour in 2017 you could bet your bottom dollar that they would get it right. Where commerce is involved the private sector just gets it right. Aren’t I a clever chap!

Well imagine my surprise to find that Optus – a very commercially focused and a tech-savvy subsidiary of equally tech-savvy Singtel – cocked up the launch of its much vaunted English Premier League (EPL) coverage in a way that makes the census look like a momentary buffering blip. So IT stuff ups are not the sole domain of a ‘hapless’ public sector after all. This notion of the private does best and the public lags well behind seems to emanate from a particularly dualistic mindset (subject of a future blog). We need to transform our social and technological policy thinking from one of the government hasn’t got a clue, to one where we appreciate the complexity of both the technology and stakeholder components in equal measure.

While schadenfreude is great to feed a voracious twitter it is essentially a self-defeating emotion that posits no solution. It’s a lazy approach to any issue. The smarts are in the solution not just the problem identification. We need to take a collective deep breath and accept that the best endeavours are being put into IT projects both large and small. The private sector has no monopoly on IT functioning smoothly – Telstra can attest to that. Let’s just hope Ticketek have their servers well and truly denial of service proof for the upcoming Adele concert tickets.