A few years ago, I served on an INFORMS committee responsible for reviewing the questions that appear on the INFORMS Certified Analytics Practitioner exam. An interesting challenge for INFORMS is to take an item bank of questions, and choose a subset of the questions to appear on the exam that meet certain conditions related to the balance of topics and difficulties of the questions. This challenge doesn't just exist for INFORMS; it also occurs in many industries that use exam procedures to certify specialists. It even occurs for the SAT used in the college admissions process.
Princeton Consultants was contacted by a leading provider of assessment software systems—the software that allows organizations to create, administer and score examinations— who needed to incorporate new business requirements into an existing implementation. The core feature in question was an automated assembly process based on mathematical optimization that managed a large number of constraints to assemble exams from a data bank of questions.
Years ago, the firm retained a consultant to build a model, powered by IBM ILOG CPLEX, to construct exams using a combination of heuristics and mixed integer programming techniques. The firm needed new constraints for the 2017 exam of its client, but the consultant was no longer available and in-house resources were insufficient. The firm’s leaders saw the opportunity to update and improve the model, and retained Princeton Consultants.
Using the existing model, our team analyzed a dataset consisting of over 1,000 former production runs. The users, who are psychometricians, had reported occasional failures in the previous implementation that would interrupt their work. The team’s analysis identified the causes of these failures and provided a resolution path.
We translated the existing model from C# to OPL, which made it more accessible to in-house staff and therefore more easily customizable in the future. The OPL model was embedded in the existing C# code. To accommodate the new business requirements, a new model was built that was more robust and reliable. The new model was tested against the existing dataset, and the new requirements were tested by our client, including a test phase with user acceptance support.
As a result, the firm’s psychometricians can assemble exams with more complex constraints that meet the evolving business requirements. The firm is positioned to increase and accelerate optimization-driven customization—a key differentiator for its signature software system. The firm has retained Princeton Consultants for additional improvements to the model as the business requirements change. We are now encouraging them to leverage cloud computing for their application.
Further Reference
Psychometric Society: https://www.psychometricsociety.org/content/what-psychometrics
van der Linden, W. J. (2005). Linear models for optimal test design. New York:
Springer-Verlag. (http://www.springer.com/us/book/9780387202723)
Modeling with Optimization Programming Language (OPL): https://www-01.ibm.com/software/commerce/optimization/modeling/
Methods for Embedding IBM ILOG CPLEX: http://www-01.ibm.com/support/docview.wss?uid=swg21400017&aid=1