One of the most overlooked challenges in big data is not a technical one, but an organizational one. Although capturing terabytes of data can end up being both difficult and costly, there’s an ongoing and hidden cost in how your team interacts with the data.
Data is not useful if it is not actionable, and insight is pointless if it doesn’t lead to impact. Therefore, it is incredibly important to find ways for the people that actually make decisions, regardless of scale, to interact with the data on their own so that the people best equipped to make an impact are directly plugged into your engine for insight.
At the typical “data-driven” organization there is a layer of complexity and obfuscation between these decision makers and the data. The formats of the data is ultimately determined by engineers and will have a profound and long-lasting effect on what types of questions can be meaningfully asked. Once the data has been captured, actually querying it for new insights and bringing new questions and perspectives to the data is a technical hurdle that few can overcome.
“Data Scientist (noun): Person who is better at statistics than any software engineer and better at software engineering than any statistician.”
The result is a reliance on Data Scientist, the new painfully trendy professional that will cost you a minor fortune and act as a human keyboard between your decision makers and your data.
But data scientists don't come cheap. With entry level salaries at around 130k/year in Silicon Valley, these human keyboards cost as much as the engineers that actually build your product. And why do we need them?
Without pointing any fingers, I lifted this sentence from the website of a company that is claiming to innovate in the field of analytics and big data:
“With its unmatched speed and familiar ANSI SQL interface, [product] is a powerful drop-in solution to fix application performance issues.”
Familiar ANSI SQL interface! How’s that for a sentence? How many of the people best suited to make decisions at your company are even proficient in SQL?
Let’s be honest with ourselves: If someone doesn’t know how to SQL, then SQL is not their interface. The data scientist is their interface, a philosophical remnant of an uninspired time where we hired specialists to operate our computers, rather than buying computers to increase the productivity of all of our teammates. I submit to you that the data scientist is not some trendy and desirable vision of the future, but rather an anachronistic throwback to times where computers scared people.
The challenge is not how to fit terabytes of data into your data warehouse, but how to fit the data into your culture. Check out Traintracks.io to find out how we are changing the way companies interact with their data.