At an e-commerce giant, customer orders are assigned throughout the day to fulfillment centers and third parties across the United States while minimizing shipment costs, observing limits on inventory, reducing split orders, and balancing workload across the fulfillment centers. The company’s data science team created an order fulfillment optimization proof of concept in Python that would significantly improve the operation. To prepare for deployment, the company’s software engineering team converted the code into Java, leveraging the Gurobi Optimizer, and created a solution that integrated with real-time data. Because of the critical nature of the operation, company leaders sought assurance that the solution would be successfully implemented and maintained 24 hours a day, 7 days a week, and yield the desired results.
Obstacles to Implementation
Across industries, the majority of optimization solutions fall short of implementation into production because the original mathematical model may contain errors or fail to observe best design practices, and/or the model cannot be translated and integrated into existing IT systems. For the e-commerce company, the data science and software engineering teams, because of their different areas of expertise, found it difficult to plan and prepare for solution deployment at scale. Furthermore, the team members lacked experience with the Gurobi Optimizer that powered the model.
Phased Implementation with Third-Party Review
The company’s leaders retained Princeton Consultants for an expert third-party review and validation of the mathematical optimization solution prior to a rollout of the solution. The company’s software engineering team had established a framework to record test cases and understand the model/data interaction. The rollout plan called for moving the application from a shadow mode to a pilot test used for several hours per day, and finally to production.
A Princeton Consultants team assisted the phased implementation, documenting best practices to support the application and to establish processes to manage potential future failures. The team delivered actionable technical recommendations for performance improvements; model design and development best practices and collaboration guidelines; ecosystem design improvements and other hardening recommendations; and triage and support strategies for 24x7 production models.
Model Review and Recommendations
To assess the model, the Princeton Consultants team followed its established methodology for optimization model review and validation. First, the team interviewed model designers and technical Subject Matter Experts (SMEs) to understand the model’s purpose and its use in the business process. The team reviewed the existing Java and Python model code to understand how the model is formulated and run in batch mode. As an early deliverable, the team documented the model’s mathematical formulation (e.g., how the business requirements were modeled as mathematical equations) as implemented in the Python code—this document was important for the solution rollout and it will be an important reference tool for the company in the future, to be updated when appropriate.
Technical analysis began with the investigation of methods that can be adjusted in the model that will improve performance, including potential parameter changes, objective and constraint rewrites, and removal of variable upper/lower bounds. This was followed by evaluation of the model wrapper code and the development lifecycle, including model update handoffs from the company’s data science team to the software engineering team. Ideas for model performance improvements were tested and evaluated against data sets exported by the software engineering team and compared to baseline performance metrics using the Gurobi solver.
The Princeton Consultants team recommended model, code, and process improvements. Suggested near-term fixes were to adjust Gurobi parameter values to the existing model to improve performance, reliability and robustness of production runs, and to redesign the allocation penalty function to improve performance while still meeting business needs.
Regarding structural changes to the mathematical model, the hierarchical objective approach was recommended. Each component of the objective should be solved separately, eliminating the need to balance costs and penalties, and providing a truer implementation of business priorities.
Lastly, the team recommended changes in development and support processes. Based partly on the identification of constraints and variables included in the Java model that were not included in the Python model, the Princeton Consultants team recommended a detailed comparison of the two codebases. The company’s data science team was advised to maintain the formulation document of the model as the source of truth and to update its mathematical description to ensure alignment on how the model is updated to meet business objectives. Application design and architecture recommendations addressed batch run solution analysis and alerts; the evaluation of solution feasibility when batch run times run out; and taking advantage of data sparsity when appropriate.
The optimization solution was successfully deployed and has greatly improved the e-commerce company’s order fulfillment. The software engineering and data science teams are progressing through Princeton Consultants’ recommendations.
The Model Design Matters
Model design is as much an art as a science, and there are often alternative approaches to meet the same business requirements. Design choices can significantly impact solution performance, stability, and maintainability. For solution deployment and maintenance, establishing and coordinating the appropriate workflow between different teams—in this case the data science and software engineering teams—can be challenging.
Deep experience in mathematical optimization modeling is typically required for a rigorous review and reformulation prior to deployment. Effective tuning of Gurobi parameters requires extensive testing and significant Gurobi expertise. Princeton Consultants’ Optimization Practice team members have decades of combined experience in end-to-end optimization modeling and deployment and have established a methodology for third-party model review and improvement. Princeton Consultants is a Gurobi Premier Partner and is also a partner with other major commercial optimization solver companies.
Expert Quality Assurance in Optimization
Quality assurance is a prerequisite for traditional operational systems, but typical software QA and testing methodologies are challenged to accommodate the special needs of optimization applications. Princeton Consultants blends technical and management consulting expertise to conduct a rigorous review and validation that helps analytics teams contribute even further to lines of business and their competitive advantage, giving business leaders more confidence in their solutions. To discuss optimization model review at your organization, email us to set up a call.