Skunk Works, Startups and Sponsors: Discussing AI and Optimization Project Risk Management

Wednesday, January 25, 2023

In November, Steve, as a guest speaker at Gurobi Days in New York City, presented a project risk management framework. Following is a lightly edited excerpt from Q&A with practitioners in the audience.

Audience Member: You said a consulting group is brought in when things are not going well. Isn’t there another situation where things are actually going OK? The organization is doing fine but executives are convinced that they can get that two‑percent improvement and millions or billions of dollars in savings and benefits.

Steve: You’re correct. It’s not the case that the only time we’re ever brought in is when an organization’s leaders have said, “Oh, yeah, something’s going wrong here.” A team or group may be advancing an innovation that is secret and competitively valuable. They want to earn the whole win themselves and are very reluctant to bring in outsiders. At Princeton Consultants, we get that, but we like to think we’re not going to steal the glory—we’ll help get the mission done.

For example, we worked with the US Census Bureau, where a group was developing optimization and advanced analytics solutions to schedule and route the enumerators, among other functions, for the 2020 census. Our project was not in the Decennial Census Group; it was a skunk works project outside the group. Some Census Bureau executives charged with oversight had asked the Decennial Group, “Are you using operations research?” They’d answered they were using O.R. but the scheduling and routing were being done in advance, with gameplans and playbooks. A consensus at the Bureau then developed in support of real-time applications, which were ultimately deployed and very successful—they have transformed the way the Census will be done going forward [enjoy this video about the project]. So, that was a case where we were engaged and we helped, but not because the ship was sinking, so to speak.

Audience Member: In considering the various metrics for AI project risk, what if the company is a startup, how would you deal with that case?

Steve: One of the problems at many startups is that everything is single‑thread for the CEO. Particularly with venture capitalists, there can be an almost maniacal view that it’s all about the CEO, who does everything. I am amazed at such CEOs, like Elon Musk, at the “gigantic” scale. But even when there are only eight people, there is still the same problem because everything goes back to the CEO. While 20 initiatives are up for consideration, the CEO might also be engaged in pitching for money, selling, and even recruiting, so a team might find it hard to get speedy answers required for its critical AI project. We use mitigation techniques in this circumstance to identify a product owner and streamline decision making about the project—we may not completely solve the problem, but we can make it work.

Audience Member: Have you been in situations where you realize you’ve got the right sponsor, but then later, once you’re in, realize that the sponsor doesn’t have sufficient standing in the organization and therefore some of the project risks haven’t been fully mitigated or even understood? The sponsor may have the best intentions but is running up against political roadblocks, speed bumps, or a lack of information. That would add complexity or potential risk to a project.

Steve: Definitely. Part of our job is like training a professional athlete: we are expected to make them a winner, not just to get their reps in. In other words, our job isn’t just to build models, it’s to make our sponsor a winner. If needed, we try to identify and shore up areas where the project might be at risk. If we are worried about board members’ buy-in, maybe there are comparable executives we should engage with to see what their issues are. One of the funny phenomena about our work is that the upside can be so spectacular, so we are always mindful of the saying, “Success has many fathers, but failure is an orphan.” We start, say, a scheduling optimization project, “out in the wilderness” and no one believes in our stuff, but then suddenly it starts working, and then other executives want to attach their own initiatives—a new data warehouse, a big SAP installation, etc.—to our successful project. We diplomatically resist; we want to go fast and we don’t want to be dragged down. We use our project management framework to discuss this kind of thing with our sponsor because success itself can cause risk.

Audience Member: When you have the model’s answer generated in a text file or in Excel, what do you consider presenting in the deployed solution?

Steve: To us, the important but subtle design of the system concerns user experience. When there is a human in the loop—a person is reviewing and potentially overriding the software’s recommendations—the human may want to understand the recommendations from a certain angle: “I like this part and this part. What other options do I have?” The data might be refreshing while the human spends 15 minutes walking through a solution. If the system pins everything down, the person might lose degrees of freedom. Figuring out the correct balance for the best user experience is a big part of successful engineering.

To talk with Steve about an AI project, email us to set up a call.