10 Tactics to Manage Decision-Scoring Project Risks in AI and Optimization

Tuesday, February 2, 2021

Following is a lightly edited excerpt from Steve’s presentation for “New Frontiers of Analytics Practice” at the INFORMS Annual Meeting in November. For his introductory discussion of fuzzy objectives, see the previous post.


1. We start by creating cost functions around tangible economics of outlays and revenues because the pre-AI objective has often been described solely in economic terms, with service as a constraint. So, for example, instead of minimizing miles, we might say to the business executive, “The more drivers costs, the more fuel they use and the more of their time they use—but what if they stop at a place and arrive early, is there some time spent? Are you paying the driver by the hour, too?” The answer will likely be yes, so we should really address the driver cost, not merely miles. If we get the dollars right, later we can proxy in some of the soft, non-dollars cost.

2. Make sure to nail feasibility before starting to optimize. Don’t assume that given rules are ironclad. If business executives say, “We never go over time” or “We always start right on time,” we must question if that is true. We can generate a wide selection of plans that show something just short of or just over the limits given. Then we see whether what we have been told are not hard but soft constraints, or more precisely they are costs or bonuses in our objective.

3. Generally, people are not oriented toward mathematical formulations, but they are oriented toward rules of thumb. We extract rules of thumb from training materials, interviews of subject matter experts, observations of work, and reviews of actual plans. Examples are: “We always allow an extra 15 minutes,” “We generally assign a senior tech to the big jobs,” and “We try to minimize splitting jobs across shifts.” Then, we come up with numerical values to describe what is trying to be accomplished in a format that executives can understand.

4. Take what the business has had trouble quantifying and ask executives how they would deal with something—turn their intuitions into values. We start with some extreme variants and then narrow them down. For example, if a customer called with a last-minute pickup, should an employee drive 50 miles for it? If no, what about driving one mile? We essentially squeeze variants like these to get to the tough calls.

5. Uncover the circumstances that would change the values. For that last-minute pickup, when we reach the area that is iffy, say 5 miles, what would make the business executives say yes or no? They may want to know if it is for a major customer or a new account, or how busy they are at the time. Use different scenarios to extract other variables that impact this cost or bonus.

6. Don’t spend too much time exclusively working with plans. Get out and see what is actually executed. A sample of real data is much more valuable than anecdotes, because it is what they do, not what they plan, that drives the outcomes. What if the plan is not taken very seriously? It could be directional and not necessarily gospel. Often, the plan is a light version of the real set of rules.

7. Present initially scored dummy plans to subject matter experts. Ask them to compare and score the plans in terms of value for the company. Do their answers match those of the new scoring? It is advisable to be in the ballpark with relative weights before proceeding with more scoring.

8. Sometimes it is difficult to get started. How many variables and how much complexity is to much or too little? We can bring in a proven application from another industry, particularly if ours is a first for the company or industry. If we are working on job shop scheduling for a unique complex machine, we can look at an analogous scheduling application for building cars, for example, because the jobs are also very complex, there is a lot of sequencing involved, the duration of each step has a lot of variance, and there are different classes of labor. Another application can show us the variables used and levels of complexity.

9. If there is a problem getting agreement on trade-offs between multiple objectives, a hierarchical approach is worth exploring. For example, we first create a schedule that maximizes profits. Secondly, we say the maximal profit is the lowest cost. With all the jobs together, as a second tier subject to that first function, we reduce some of the shift overlaps. If we started a job on the optimal schedule 15 minutes before the end of the shift, we will let it slide so it starts on a shift boundary. That will bring down the profits slightly, but we will let the profits dip two percent to get L2 done. Once we have gotten L2 done, then and only then, we can spread the work among different work crews so that it balances out—but we don't want that to come at the cost of L1 or L2. That is a hierarchical approach.

10. I am very enthusiastic about SMART options—an alternative to giving a single optimal recommendation. Depending on the circumstances, we will give multiple alternatives that will be accurate. They are in real time, giving a user quick options and responsiveness. They are thematically diverse or attractively different. For example, if we type a phrase into the Google search engine, depending on who it thinks we are based on prior queries, it may give us a completely different set of answers. We would not consider multiple options helpful if the options all basically point to the same website—we want to see different kinds of sites.

To discuss this topic and more with Steve, email us to set up a call.