An article published in Computer Weekly states that more than 50% of data analysts in the UK anticipate to quit their current job in 2020. The main difficulties they cite were:
- a lack of management support (29% of answers)
- bureaucracy that hinders data science projects (44%)
- a lack of data tools (28%)
Those oldies but goodies
It has been a long way since organizations begun to take care of their data. Data management and data analysis are already well implanted in business processes:
- Reporting is everywhere, from marketing and sales to project management,
- Value analysis, return on investment calculation are the daily bread of operational managers, accountants, and CFO,
- Business planning, sales forecasts are updated every quarter, sometimes every months,
- Web marketing, content marketing, search engine optimization are technical, quantified, tooled process,
- ...
Reporting is as old as IT. Spreadsheet and accounting software was among the first applications shipped with (central) computer systems more than thirty years ago. Excel is the tip of the iceberg, the swiss army knife of many operational managers. But Excel is not alone. Behind the scene, it queries data stores, RDBMS, AS400, applications as diverse as those ugly but efficient VT100 terminal, CSV produced by batch processing or web services. And when it comes to advanced reporting or statistical analysis, a herd of applications, from Business Object, Crystal Report or SAS pops up with their experts, their ecosystem and their specificities.
On top of this stack of software, the organization relies on people who, more than not, know more than an ounce of their business. They master the key drivers of the business, in an empirical and intuitive way. It takes more than a few line of python or a linear regression with SciKit to give more accurate a forecast than they do.
Data analysis must be an integral part of your business processes
Talking about data is neither innovative nor attractive per se. In most companies, everything is already quantified. To succeed, you must offer something different and at least two things:
- A seamless integration in the organization; forget the idea of the data lab, with bright and skilled analysts, that don't speak with the businness; draw a muddy track, where data analysts get their hands dirty with the business, and operational managers get headaches trying to compute a classification of the clients
- An innovative, disruptive point of view on the data; try to mix data that is siloed in your existing organization; add external sources that your team don't think about in the first place; try new tools, try to understand why data is skewed, work on the margins, the 5% on the left or on the right, the outliers.
You should feed the data analyst change process with the existing, ritual reporting and its well-established methodologies, such as Balanced Scorecards. This legacy gives you the dos and donts. It is built upon plain old good principles:
- Strategic alignment : data analyst must contribute to strategic goals
- Business relevance : data analyst must provide new insights, relevant to operational stakeholders, and significantly improved results compared to traditional way of running the business. 5% of improvement is only noise about nothing, 30% of improvement is a good bet,...
- Operational achievability : the results must be able to be produced operationally, reliably, at costs compatible with the constraints of the trade, and be implemented within a known and acceptable timeframe.
If you come from the software industry, you may recognize that these principles are similar to SMART principle (Specific / Measurable / Achievable / Relevant / Timeboxed).
Business expectations are high
Results advertised by major stakeholders of the big data market, from beating Go champions to detecting a needle in a haystack have risen the bar of business expectations regarding AI and big data. Many people think that image processing or text semantic analysis or speech processing are available off the shelf. In fact, going from proof of concept to production still requires a huge amount of skills, hard work, fine tuning, try and error. To meet these goals, you need:
- Specific and well defined objectives
- Access to appropriate tools, storage space and processing power
- a skilled team to take advantage of these resources
You may consider a cloud strategy, with the inherent risk to get locked in the cloud, because many tools and resources are, sometimes only, available as a service offered by cloud providers. Otherwise, you should take this resource constraint when prioritizing your goals.
If you stay away from deep learning technologies, many objectives are still attainable with limited resources :
- Exploiting existing unstructured data
- Enriching data with external sources
- Using more advanced statistical tools, classification, clustering, dimensional analysis, unused visualization tools
The story of a successful data analysis strategy
A company was looking for a solution to improve the quality of its customer risk assessment process.
Historically, The risk managers were working with a spreadsheet software to process customer information to score their risk. To score a portfolio of customers, they picked up a sample of the customer accounts database, partly at random, or based on their intuition or their experience.
The head of the Risk Department wanted to promote advanced statistical tools and to evolve from a sample to an exhaustive analysis of risks.
The project was phased as follows:
- Define the strategical output: two goals were set, first, full coverage of customers database to evaluate risks, second, adding new advanced statistical tools to the team toolbox
- Drive the change: by replacing the legacy spreadsheet application by a new and efficient data analysis and data visualization tool that embarks a guided methodology, the project enforced new ways of wworking among the risk management team.
- Train the risk managers : risk managers were trained to get familiar with the new application. Training sessions were customized to include real examples from their day-to-day work.
- Support in change : a small team of skilled data analysts was enrolled to support the risk managers, to help them in getting up to speed with the new application and to identify, gather and publish good practices, tips, and scripts of interest to the whole team.
- Enrich the data analysis with new methods and new data sources : a collective effort was launched to rethink the data analysis process, to deal with unstructured data and to capitalize on shared historical databases
With our business partner Datalearning, we assisted this client in its transformation.
Concluding thoughts
Hiring a team of data analysts doesn't lead to a data-driven organization. Instilling a data-driven culture requires to embark business stakeholders in the project, to train them to use new tools and to support them by providing pre-implemented real-life use cases and a way - technical or organizational - to share their experiences of data analysis with their peers.
In the future, data analysis skills may be integrated in training objectives so that many people in the organization, accountant, risk manager, production manager, or sales manager are able to produce the data analysis they need.
