Author: Nicholas Varner, Big Data Senior Solution Architect
If your company stores large amounts of data, it surely would benefit from implementing a BI solution. BI solutions, when implemented properly, provide your business with meaningful and accurate insights by retrieving the data you want in a timely manner. That is all fine and good in theory, but where should your business begin? Can you report off of your current billing source, ad server, or any other source of data? What is the best way to store data and what tools should we use to produce reports? How do we know the data we are getting is accurate? Does our solution scale? The questions can and will go on and on.
One of the main goals of a BI project is to provide clarity and insights to what is happening with the people, process, products, and services of your business.
Let’s focus our efforts for this blog post, though, on human resources. What kind of contractors and in-house resources are required to implement a solution? A BI solution contains some core competencies:
1. ETL. This stands for Extract-Transform-Load. The ETL process evaluates source systems including billing, purchase transactions, billing transactions, inventory, shipments, and so on. Typically, these source systems are extracted from their respective sources, transformed for architecture, storage, and reporting requirements, and finally loaded into an Operational Data Store, Data Warehouse, and Data Marts.
An ETL architect has a big job and is highly technical. This is the engineer behind the project. How do you move data from one place where you cannot properly report it or compare it to other data sources to a data storage location where your reporting team and management can make sense of it all. ETL is the back room of a BI project and visitors are not welcome here. ETL architects design data models, not the business. If your business does not get the architecture right, you have no chance to create accurate reports for your business’ decision makers to use and your business will make decisions based on incorrect reports. The effects are disastrous. Needless to say, then, the ETL architect should have a great deal of experience and should be able to pinpoint pitfalls in architecture and explain why the architecture accurately portrays your data and will meet your reporting requirements. The ETL architect should not be an in-house resource 99.9% of the time.
ETL engineers are resources that follow the ETL architect’s data model and can execute it. Many ETL jobs with time can be automated, however, resources need to be available to ensure the data is current and ensure that required fixes to the ETL process are addressed quickly so you have access to accurate and current data. ETL engineers are usually outside resources or are hired by your business. Performing ETL is a technical responsibility and these engineers should have experience behind them, and ideally have experience with the ETL tool that your company elects to use. More on tools in an upcoming blog post.
2. Data Governance. Your business should deputize someone with clout to partner with BI experts (usually external to your business) to lead Data Governance. Just like your business audits its accounting books, Data Governance is in charge of making sure data is accurate, within an acceptable deviation, but also can explain what factors account for any discrepancies. The team lead must hold the entire team accountable for accurate, reliable, timely data, while documenting key data models, key processes, and keep track of failures and data discrepancies so there is a way to track deviations for when the business comes back and asks, “why?”. Data Governance personnel should have experience evaluating large amounts of data and pinpointing holes in processes, data collection, and challenging ETL engineers and reporting teams to ensure that reports are meaningful and that key stakeholders fully understand the data. Data Governance should work with the team to ensure that data is defined and understood by all. A Data Governance lead should be respected throughout the business and be able to work with the business to standardize terminology and business calculations. Consultants should usually not lead this type of discussion because it will be more difficult for an outsider to force the business to conform and unify key business terms, processes, and calculations. To put it bluntly, if senior management does not make standardizing terminology and calculations a priority, you will never have the type of meaningful results you are looking for in a BI solution. Make it a priority and you can use it as a way to simplify, standardize, and improve your business by providing transparency of key business terms and calculations.
3. Project Management. A Project Manager can be someone in-house, or a consultant, that knows how to get the right decision makers in the right room and can track a variety of issues, in various parts of the business, while being able to speak with technical proficiency to Engineers and explaining complex problems to management in a simple way. A Project Manager usually can present ideas to management to show the team’s progress. A Project Manager is an excellent communicator that is usually busy meeting with stakeholders and the BI team, thus, serving as a liaison between the BI team and the business.
4. Reporting. Doing all the aforementioned is all well and good, but is largely an exercise in futility if you cannot visualize the data and slice and dice data by a myriad of filters (geography, customers, vendors, sales orders, purchase orders, date, and so on). Report developers could be in-house resources or consultants depending upon your current staff’s abilities and availability. Whoever you ultimately put in charge of reporting should understand how to use the reporting tool you decide to use (proprietary tools or one your team builds). Your reporting team must understand SQL, what is included/excluded in the data warehouse and data marts, understand the data model, ETL job time lines, understand what standard reports are necessary, what ad-hoc reporting is necessary, what filters are necessary, understand any limitations of the data and the reporting structure, and convey the data in a smart aesthetically manner.
5. Stakeholders. Stakeholders are the key decision makers. This is who we do it all for. These can be CEOs, CFOs, Directors of departments, Business Analysts, Financial personnel. If these stake holders cannot successfully get the answers they need at the right time, the BI team’s effort can a real waste of resources. It’s important to get feedback from stakeholders consistently to ensure that the solution is meeting their needs and determine how/where it should expand.
6. Training. This could be an extension of the Reporting team. Trainers would obviously benefit from any technical BI knowledge, however, they do not necessarily need to be extremely technical resources. They just need to know how to manipulate the data using the tools that business users are provided with. Patience is not required, but surely it does not hurt.
This post provided a high-level view of the players associated in a BI deployment. However, all these teams need tools to do their jobs. In the next post, we will examine the types of tools that each of these players requires to do their jobs.