This is the best time to be living at the intersection of the media and data warehousing industries. The media industry is undergoing a tremendous transformation that is increasing demand for terabyte-scale data analysis, and the data warehousing industry is experiencing a renaissance of innovation. What fun!
Ad targeting, content targeting, attribution analysis, yield optimization, click fraud analysis, website optimization, ad inventory forecasting, SEM and PPC arbitrage…these are just a few examples of the types of solutions that create competitive advantage for digital media firms today. And according to Hal Varian, Google’s Chief Economist, people that understand how to implement these solutions are not only scarce – they are sexy:
Click here to view on YouTube.
But what makes data warehousing sexy? Engineering elegance.
My favorite definition of elegance is the Hacker Slang version: “Combining simplicity, power, and a certain ineffable grace of design.” Wikipedia also offers a perspective:
- “In engineering, a solution may be considered elegant if it uses a non-obvious method to produce a solution which is highly effective and simple.”
- “The proof of a mathematical theorem is considered to have mathematical elegance if it is surprisingly simple yet effective and constructive.”
- “A computer program or algorithm is elegant if it uses a small amount of intuitive code to great effect.”
Note the common theme of simplicity…it’s an important criterion, because there’s nothing special in making something complex – anybody can do that.
While there is innovation occurring across the entire data warehousing industry, I am particularly familiar with Netezza (disclosure: they are my employer), so I will use them as an example. In the data warehousing world, one of the ways that engineering elegance manifests itself is in the form of a superior price/performance ratio. Netezza has just announced the new TwinFin appliance that once again shatters traditional price/performance benchmarks by cutting prices to under $20K per terabyte of user data. This represents an immediate 10-15X increase in price-performance, based on a 3X cut in price/terabyte and a 3-5X improvement in mixed workload performance.
Solving the most complex analytical problems on the largest possible data sets while utilizing the smallest and least costly amount of infrastructure is hard. Making it look simple is sexy.
Let me cite a specific example of how engineering elegance creates value for digital media firms…
Traditional approaches to terabyte-scale data analysis require a trade off between query performance and storage density. Improving one or the other is relatively straightforward – but it is very difficult to improve both at the same time. Netezza is leveraging massively parallel processing (MPP) and field-programmable gate array (FPGA) technology to increase both query performance and storage density by ~4x. Netezza’s offerings have been able to achieve this goal – and exceed Moore’s Law by a significant margin every year since inception – because of engineering elegance. Netezza delivers an elegant system architecture that takes what historically has been three independent technology silos – database technology, server technology, and storage technology – and re-integrates them, opening up opportunities to innovate across that entire technology stack. The image below comes from a more detailed set of slides available here: PDF.
Netezza’s approach to engineering elegance produces two qualities that are critically important to the digital media industry:
- Extreme analytic computing efficiency
- Simplicity of deployment and operation
Analytic computing efficiency is important because as digital media data volumes grow, the brute-force approach of filling up a data center with a ton of gear becomes less and less attractive. More efficient approaches are required. And simplicity of deployment and operation are important because, as Hal Varian describes in the video above, the digital media industry is sorely lacking the analytic talent required to implement the solutions that create competitive advantage. Simplicity at the data warehousing tier allows digital media firms to invest their scarce resources on innovating at the application tier in order to differentiate their offerings from the competition.
Go ahead, get your sexy on.
{ 0 comments… add one now }
{ 1 trackback }