Stacey Higginbotham penned a post at GigaOm entitled “Systems to handle big data might be this generation’s moon landing.” Since it was posted on April 1, I asked myself if this was an April Fool’s Day joke as the comparison seemed a little farfetched; but then I realized the analogy was not about the emotional significance of the first moon landing but about the complexity of the problem Big Data is trying to solve.
Coincidentally, on the same day the above post appeared, I started re-watching “Can We Do This?” the first episode from the HBO series “From the Earth to the Moon.” It was well understood in the early 1960s that when President John F. Kennedy set the goal of placing a man on the moon by the end of the decade, the technology required to do just that had not been invented yet.
An appreciation of the burden for Big Data can be seen by looking at just one scientific project alone, the Dome Project. Higginbotham notes that IBM and CERN are working on a radio telescope that can see back 13 billion years in time. This one project is expected to generate a few exabytes of data per day for each one square kilometer of antennae, with an annual storage requirement of between 300 and 1,500 petabytes. Given that most of us don’t even know what an exabyte or petabyte is, that’s a lot of data.
The implications for the enterprise network are profound. We know that moving data is costly and that it is getting less and less expensive to let data live in the cloud. The implication for the enterprise network is that some day in the future, a single query by a single employee might generate exabytes or petabytes of network traffic, all for an analysis that might be related to satisfying a single customer query. What if every customer interaction in the future was akin to a moon shot in terms of data generation and analysis? If your head spins just to think about it, you’re not alone.
