Big Data fast.

If you are in any way involved in the creation, curation, conveyance, or management of knowledge, then Big Data to you is not just another tech fad. Big Data brings the meaning of knowledge to a new frontier. For centuries knowledge has been 'bound' in scrolls, manuscripts, books, libraries, microfiches and even in footnotes. Bound physically, socially, and politically too. Each technological innovation has scaled our reach and has ultimately changed the existing power structures around it. Today, we are riding a tectonic shift--knowledge is now a synchronous "network of connections" bringing with it challenges and opportunities that we have never encountered before. We are moving beyond tables, rows and columns into a world of ambient data. Big Data enabled by an internet of things and events brings us into a new realm of complexity. So what's new here that we didn't already know?

Before now, innovations to the architecture of knowledge were at the mercy of time and happenstance--the printing press and radio, for example. Today we are in a position to shape that architecture as we move through it; shape it through the technologies we develop and how we choose to apply them. Big and Open Data are our first forays into this new environment and they are moving fast--very fast.

What are we going to do with this? How do we deal with the accelerated pace of change? How can we plan around this? Is there some stewardship? These are the overarching questions to be addressed. This time we really are our tools; we are the medium.

Artificial intelligence meets automation--future perfect?

Artificial intelligence, robotics and automation are heading to a tipping point as massive data grows exponentially and analysis improves.
Deloitte University report explains all the domains it is influencing and how it will revolutionise how we behave and work.
The primary forms and areas of influence:

Deciders are information systems that automate decision making: finance, health care, public sector.

Doers, such as collaborative robots, automate physical tasks: automotive, distribution and fulfilment systems

Movers use sensors and artificial intelligence to automate transportation: aerospace and automotive

The Digital Humanities: Big Data and Literary Criticism

Data techniques used to tackle literary criticism? A growing controversy in the digital humanities.

Yes. I've been waiting for this one. A very interesting article from Fast Company addressing a growing controversy in the digital humanities: Should We Teach Literature Students How to Analyse Texts Algorithmically. Machine learning and data visualisation surveying the canon!

So many professorial 'man hours' have been spent on minutiae such as the use of the singular 'their' in Jane Austen or the occurrences of 'yes' in Ulysses. Now the power algorithms are tackling the life works of authors and whole swathes of Google Scholar. What are they generating? Very unusual and novel correlations and hypotheses that academics couldn't hope to find in research journals. The algorithms do not understand what they are examining but that is not their job. Their job is to sift through massive data and find meaningful patterns.

Historical documents and sociological data have been the focus and now it is literature. Really, I can't wait to see what we find in Finnegans Wake!
See full article...

Galileo, Hubble, and Big Data

How do you picture the dynamic of Big Data? People use terms such as "game changer" a "new paradigm," "tectonic shift." Personally, I need a very big metaphor to capture the idea. I think it is futile to measure big data in terms of "potential annual value" as McKinsey does or by petabytes and zettabytes as others do.

When people ask me: What is Big Data anyway? I avoid mentioning tables, rows, columns; an internet of things; passive data or anything else in our lexicon.

I have found an interesting metaphor that is both profound and simple at the same time: Galileo's dark drawings of disks in space compared with Hubble's gargantuan worlds is a good metaphor for what is beginning to happen today with our explosion of Big Data. Comparing Galileo's 'universe' in the 1630s and our 'small' world as seen through Hubble's telescope conveys well the dizzying velocity, variety and volume of incoming data.

Galileo peered through a primitive telescope and saw blurry orbs in the sky from which he made rough drawings and great calculations that changed a world. He could not fully foresee where it would lead.

Likewise we know there are potentially both great and frightening things behind the veil of Big Data and ponder where it will lead.

Have a look at the images below and see if the comparison resonates with you. It's a visual metaphor only.

When Galileo drew these orbs, the data that made the Hubble picture below was around too--he just couldn't see it. 

Big Data is our new Hubble, magnifying our view of life, if only to tell us how small we are.

Looking at the pictures above brings us to the next blog: data visualization. When you get beyond cherry picking data, cognitive bias, confirmation bias, and all the rest (see previous blog), how do you visualize the data for optimal expression? Next time.

Five Deadly Sins of Data Science

1. Cherry picking
2. Confirmation bias
3. Data selection bias
4. Narrative fallacy
5. Cognitive bias
6. (Adding one more): Not sharing the raw data AND the algorithms used.

From the Data Science Association via Information Week, for more

Big Data and Civil Rights

Big Data and Civil Rights

 A great article by Alistair Croll on Civil Rights issues around Big Data--very provocative.

Google is Becoming the Logfile of Our Digital Lives

See Alistair Croll's piece on Solve for Interesting. Great analysis on how Netflix killed Blockbuster and how Google (being the logfile of our behaviour) might just kill Netflix. See linked text below.


Google has all the data on user actions, desires, purchases etc. and is using it to chop up the Long Tail of content. Watch out Netflix!

Strata 2013: Kate Crawford, "Algorithmic Illusions: Hidden Biases of Big...

One of the best presentations I've seen on the implications of Big Data. Also, it is one of the rare instances these days where someone is good enough to present with just a few slides. If you are wondering what happened to great public speaking--here it is, Kate Crawford at Strata

Did I mention that it is by a real technology leader who is a woman? I particularly like her insights into epistemology and Big Data.

Ten Types of Innovation: The Discipline of Building Breakthroughs

OK. If you didn't get enough out of the 3-part innovation model in my previous post on Innovation, try this 10-part one from Doblin :)

Facebook phone next week?

Facebook phone next week!?

But does anyone really want one? Gigaom speculates on the walled garden.

Hadoop according to Hortonworks

Here is a great little paper released by Hortonworks thoroughly explaining Big Data and Hadoop for a lay business audience. Good to use for content in a prez. to non technical.
It the oil refinery analogy all the way:)

Digital Dust--Big Data

A slim timeline for the evolution of Big Data. Great quote from Hillary Clinton: "Once you start measuring problems, people are more inclined to take action to fix them because nobody wants to end up at the bottom of a list of rankings."