Princeton Consultants has implemented software systems embedding mathematical optimization for many years. We recently documented our methodology to verify that the implementation is satisfactory. We term this the Princeton Optimization Implementation Verification Methodology (POIVM). Through this cyclical 4-phase methodology, diagrammed below, we verify the robustness and reliability of optimization-based software systems, while enabling ongoing improvements to the underlying models:
The Instrument phase of the POIVM process creates agreed-upon Optimization Algorithm Metrics (OAMs) to measure the quality and computational performance of the underlying optimization application. These metrics are defined across the spectrum of business dimensions. The benefits of these metrics allow all decisions about the optimization system, the algorithmic strategies, and administrative policies to be evaluated using a uniform, systematic, and data-driven approach.
The Analyze & Stress phase of the POIVM process consists of three parts. First, a stochastic computational framework is used to simulate the business process that exercises the underlying optimization engine, and OAM values are computed for the current implementation of the engine. The results of the computation are analyzed to determine points of failure and areas of underperformance of the metrics. These results can then be compared to prior results, and summary reports are created across the different business dimensions. Summaries include various descriptive and dispersion statistics of the metrics, allowing analysis of average behavior as well as the behavior of outliers. Finally, the optimization algorithms and the computational results are analyzed to determine new potential failure points and places where the OAMs achieve values that could deviate from average values. This analysis is done using an adversarial approach, where the understanding of the algorithms leads to new test cases that stress the algorithms. This phase is repeated multiple times to fully stress test the current implementation.
The Improve phase of the POIVM process leads to ideas for improvement in the underlying optimization engine. The results from the Analyze & Stress phase are used as a springboard for ideas to improve the performance of the optimizer, both along the dimensions of the OAMs used to measure performance, as well as in performance time. Ideas must be identified, researched, implemented, and then tested again by repeating the Analyze & Stress phase. It is only possible to identify new ideas by having a deep understanding of the underlying algorithms coupled with the methodology used in the Analyze & Stress phase to identify potential points of weakness. Each idea is considered with respect to how well the idea improves the overall solution quality (especially model robustness) and efficiency of the optimization solution.
The Stabilize phase is used to determine release points of the software that can be integrated into other systems for integration testing. Each of these release points contains a version of the optimization software that is deemed production-ready. In the spirit of agile software development practices that promote continuous delivery processes, the Stabilize phase represents the point where the optimization engine has been tested and verified to be ready for production.
When clients engage us for our Advanced Analytics Model Review and Validation service, we adapt the Princeton Optimization Implementation Verification Methodology to their specific situation. This allows us to determine how success of the optimization based application should be measured, the conditions that could affect those measures, and ways to improve the application.