Advanced Analytics Model Review and Validation

Tuesday, May 9, 2017

Princeton Consultants performs a Quality Assurance service that helps clients understand if best practices are used in the deployment of their predictive analytics or optimization models. Based on years of experience deploying advanced analytics in operational systems that run 24/7, our analysis often uncovers areas for improvement by suggesting new modeling and algorithmic approaches. Practitioner executives gain an understanding of how well their team uses industry best practices. Business sponsors gain more confidence in the solutions provided by their teams of analytics practitioners, and those practitioners improve their skills for future projects.

As part of our review process, we ask the question that is often the most difficult to answer: "What is a correct model?" Behind that question is the fundamental issue of how to define a model to be "correct." A model’s correctness may be measured by solution quality, algorithmic speed and other criteria, but most importantly by how well the model is meeting the business’s needs. We find that many analytics practitioners have a vague definition of "correct" when it comes to their model, so we emphasize formulating an appropriate definition for a client.

Data integration presents another area of focus. We have seen clients embed cost values directly into the code describing a model, without realizing that those cost assumptions change over time, and that the cost value should be parameterized as external data. Other clients built systems with assumptions of data flow between various systems—not realizing that the business changed the way some of those systems were used, thus changing the meaning of the values in the data stream. When precision and accuracy of data are not considered, the downstream effect can cause numerical issues for different algorithms.

We also look at the integration of the advanced analytics solution into the business process. How are solutions published? Is there a review process where people review the answers? How often are the models recalibrated, considering the changing nature of data and the business? If there are small changes in the data, does the output of the model change drastically, impacting business decisions?

These vital questions often can’t be answered internally for a variety of reasons. The original modeler may no longer be available. Expertise and documentation may be absent. Models in question may not have been examined for many years, while the business process and software environment changed. Moreover, there may not be a formal quality assurance process in place for advanced analytics. As an independent third party, we can identify problems and recommend improvements more promptly and cost-effectively than could a team of co-workers.

Based on our experience in creating different systems that embed advanced analytics, our service helps provide assurance that models are deployed to deliver maximum value. We would welcome a conversation to explore how we can benefit your organization’s advanced analytics practice. Feel free to email me directly.