Alumni Profile: Pavel Atanasov ’05 and Sauleh Siddiqui ’07
You don’t need a crystal ball to make rock-solid forecasts. You don’t even need to be an expert in stocks or sports betting. In fact, says Pavel Atanasov ’05, you’re probably better off not being an expert at all.
“Your level of professional expertise and your forecasting accuracy are either unrelated or much less closely related than you might think,” says Atanasov, who specializes in crowd prediction and lives in Brooklyn with his wife, Marina Dimova ’05 and their son.
Atanasov was introduced to the science behind forecasting at F&M, where he did research on economic predictions with former F&M professor Brian A’Hearn. From his experience researching prediction outcomes in medicine and health care, Atanasov suspects that forecasters with critical thinking and analytical skills are often better at predicting the success rate of a clinical drug trial than, say, the dean of a medical school.
Recently, Atanasov teamed up with another alum, Sauleh Siddiqui ’07, to turn the powers of prediction onto the pharmaceutical industry. Atanasov and Siddiqui reconnected at a conference in 2017, only to discover they were both interested in predicting the outcomes of clinical trials for new cancer medications.
Siddiqui, an assistant professor of civil engineering and applied mathematics and statistics at Johns Hopkins University, studies prediction, too—but through the lens of data science and machine learning. His love of math was cultivated by F&M professor Annalisa Crannell, with who supervised his honors thesis on dynamical systems. Thanks to research he led with MIT Collaborative Initiatives, Siddiqui has a machine learning model “that can predict the success of a clinical trial with around 70 to 80 percent accuracy,” he says.
The success of Siddiqui’s algorithm caught the attention of Atanasov and his research partner Regina Joseph, an information systems designer and professional superforecaster. The trio paired up to write—and win—a grant from the National Science Foundation (NSF) for their research, which combines crowd forecasting methods and machine learning.
Most cancer drugs don’t succeed in the clinical testing phases, which makes the clinical trial process a high-cost bet for pharmaceutical companies, explains Atanasov. What if there was a better framework for predicting the outcome of a trial before it started?
“For a prescription drug to become available in the U.S. market, it has to pass through clinical trials,” Atanasov says. “We’re trying to predict if a cancer drug would advance from Phase 2 to Phase 3, then from Phase 3 to the next stage.”
To do this, they’ll collect insights from hundreds of participants with a background in medicine and health care about which criteria might make one clinical trial more successful than another.
“Different experts might see the same trial as related to historical data in different ways,” Atanasov explains. “We thought, well, if we have all these human judgments about what’s relevant and what’s important, we can average [them] and get to an aggregate forecast that way.”
While pharmaceutical companies study outcome prediction for their own clinical trials, explains Siddiqui, they rarely make their data or findings public. Pharma companies aren’t interested in investigating whether there’s a general framework about prediction that could be broadly applied to clinical trials, either. But Atanasov, Joseph and Siddiqui want to see if this framework exists. They also plan to expand the project to predications about COVID-19 treatments and vaccines through further funding from the NSF.
“We wanted to see if there’s a general framework that would be true across all types of trials,” says Siddiqui. “And that’s where our research was different than the prevailing method to try and predict success or failure.”
It’s still early days for Atanasov and Siddiqui’s study. Before they begin, they’ll recruit participants from around the country with backgrounds in biology, healthcare, or forecasting.
There’s even, according to Atanasov, an entire subculture of forecasting enthusiasts who thrive on studies just like his.
It’s easy to see why.
“We make implicit or explicit predictions every day in our professional lives, but we rarely get feedback,” says Atanasov. “Or, when we get feedback, it’s so stressful because it’s real life.
“But this study is a sandbox where you can do some hard thinking,” he adds of forecasting’s appeal. “You can provide your best judgment about the situation and then over time figure out how well you’re doing and improve.”
Interested in joining the study to make predictions with other biology, health care, or forecasting enthusiasts? Email Pavel at email@example.com to learn more about becoming a participant.