o1 pro mode still has a long way to go for Mathematics

Key Points

• The newest Ope­nAI mod­els (o1 and o1 pro) claim greater rea­son­ing skills and mul­ti­modal ca­pa­bil­i­ties, yet prac­ti­cal tests show a lim­it­ed abil­i­ty to ac­cu­rate­ly solve vi­su­al­ly pre­sent­ed math prob­lems.
• In test­ing with pri­ma­ry and sec­ondary-level math ques­tions, the mod­els’ ac­cu­ra­cy im­proved over older ver­sions but still fell short, suc­ceed­ing re­li­ably on only about 67% of the test­ed items.
• For now, stu­dents can’t sim­ply rely on AI for cor­rect an­swers; ed­u­ca­tors can still trust that au­then­tic prob­lem-solv­ing skills re­main nec­es­sary, keep­ing tra­di­tion­al as­sess­ment meth­ods rel­e­vant.

The 12 Days of Ope­nAI began with the full re­lease of o1 and the in­tro­duc­tion of ChatGPT Pro. The release showcase was impressive, with claims of sig­nif­i­cant im­prove­ments in their newer mod­els, a large pro­gres­sion from GPT-4o, to o1 pre­view, to o1, and fi­nal­ly to the newly an­nounced o1 pro mode. Ope­nAI took a lot of care to em­pha­size the en­hance­ments across math­e­mat­ics, cod­ing, and sci­ence do­mains.

Un­like pre­vi­ous mod­els, o1 is po­si­tioned as Ope­nAI’s first model that “thinks” be­fore re­spond­ing—which we can think of as “rea­son­ing” through prob­lems. Ope­nAI has de­scribed o1 as mul­ti­modal, han­dling both text and im­ages, with greater ac­cu­ra­cy and de­tail com­pared to ear­li­er ver­sions. Dur­ing the demon­stra­tion, its ca­pa­bil­i­ties were dis­played through his­to­ry ques­tions about Roman em­per­ors, ther­mo­dy­nam­ics in­volv­ing a hy­po­thet­i­cal space data cen­ter, and chem­istry prob­lems re­quir­ing spe­cif­ic pro­tein con­fig­u­ra­tions. The show­case sug­gest­ed a real im­prove­ment over the mod­els I have been using for the past two years.

How­ev­er, with most of my work day in the Sec­ondary Math­e­mat­ics teach­ing trench, my ques­tion was how good, re­al­ly, has it be­come at Math­e­mat­ics? Ope­nAI is claim­ing that o1 and o1 pro mode are sig­nif­i­cant­ly bet­ter than GPT-4o, and that the model can now solve prob­lems more ac­cu­rate­ly and in­ter­pret ques­tions from im­ages.

Sec­ondary Math­e­mat­ics de­part­ments have been lucky in es­cap­ing the LLM apoc­a­lypse in our class­rooms be­cause, so far, they have been no­to­ri­ous­ly bad at solv­ing math­e­mat­ics prob­lems. After all, their core func­tion is lan­guage pre­dic­tion, not com­pu­ta­tion. For any read­ers not fa­mil­iar with how LLMs work on a tech­ni­cal level, you can think of tools like ChatGPT as sophisticated autocomplete systems. When pre­sent­ed with a query, they pre­dict the most prob­a­ble se­quence of words based on ex­ten­sive tex­tu­al train­ing. Math­e­mat­ics, how­ev­er, de­mands pre­cise an­swers and step-by-step rea­son­ing—skills not in­her­ent­ly aligned with pre­dic­tive text mod­el­ing. For ex­am­ple, when asked, “What is 12 x 8?” a model might re­spond with 96, not be­cause it “un­der­stands” mul­ti­pli­ca­tion, but be­cause it re­calls that “96” is often as­so­ci­at­ed with that ques­tion. The un­der­ly­ing process lacks math­e­mat­i­cal com­pre­hen­sion.

Other ex­am­ples make this point more con­crete­ly, such as ChatGPT’s apparent fondness for the numbers 42 and 7. Colin Fras­er, a data sci­en­tist, rec­og­nized that it seemed to out­put 42 as a ran­dom num­ber near­ly 10% of the time when asked for a ran­dom num­ber be­tween 1 and 100. For the lit­er­ary nerds read­ing this ar­ti­cle, you may be able to rec­og­nize why. After all, 42 is the an­swer to the “ultimate question of life, the universe, and everything.” Fras­er spec­u­lates that there were a lot more 42’s for the AI to see than other num­bers, re­sult­ing in a ran­dom out­put 9% over what should be ex­pect­ed if these tools did un­der­stand and im­ple­ment true math­e­mat­i­cal ran­dom­ness.

My ex­pe­ri­ence with LLMs re­in­forces their lim­i­ta­tions in math­e­mat­ics. No mat­ter how many math prob­lems I have test­ed on ear­li­er mod­els, the ac­cu­ra­cy felt like a coin flip at best. This in­con­sis­ten­cy was, in some ways, a re­lief. Stu­dents who re­lied too heav­i­ly on AI for an­swers with­out en­gag­ing in a crit­i­cal think­ing process risked get­ting things wrong, en­sur­ing that au­then­tic, ex­tend­ed projects re­mained ef­fec­tive for learn­ing and valid for as­sess­ment.

Now we are fac­ing claims from Ope­nAI that o1 can in­ter­pret and solve com­plex prob­lems, even from im­ages. If true, Sec­ondary Math­e­mat­ics class­rooms are going to be in trou­ble. What do you do with as­sess­ment, for­ma­tive or sum­ma­tive, if any prob­lem or project, image or text, can be fed to o1 and be given a valid, rea­soned re­sult? Aside from com­plete­ly closed off and strict­ly se­cured as­sess­ments, any stu­dent mo­ti­vat­ed only by score, and not by process, would just need to mem­o­rize the an­swer and the rea­son­ing that was pro­vid­ed. Re­al­is­ti­cal­ly, I teach with­in a larg­er, as­sem­bly line like sys­tem with lim­it­ed time and re­sourc­ing, so any lofty ide­al­is­tic an­swers on class­room trans­for­ma­tion fly right out the win­dow (and see­ing at how poor­ly schools are han­dling all facets of AI tools ex­is­tence leads me to be­lieve there won’t be any mean­ing­ful change in Sec­ondary Ed­u­ca­tion any­time soon). Again, re­al­is­ti­cal­ly, I also need to put my­self in the shoes of stu­dents who are often in­clined to­ward the path of least re­sis­tance and are like­ly to out­source their prob­lem-solv­ing process en­tire­ly. While I en­cour­age and demon­strate eth­i­cal AI usage to sup­port learn­ing, not all stu­dents are in­trin­si­cal­ly mo­ti­vat­ed enough to re­sist the temp­ta­tion of easy an­swers. This makes it hard­er for stu­dents to ac­tu­al­ly learn the ma­te­r­i­al. Though I would be re­miss to act as if this was new, as cheat­ing is al­ready com­mon. One study of eleven years of college courses found that when students did their homework in 2008, it improved test grades for 86% of them, compared to 45% of students in 2017. This drop was due to half of stu­dents sim­ply look­ing up the an­swers in 2017, so they never got the ben­e­fits of home­work. I con­sid­er LLMs to be just an ex­ten­sion of an al­ready ex­ist­ing trend.

Methodology

So I want­ed to see how true Ope­nAI’s claims are, but for a use case more con­tex­tu­al to my class­room, and less so for the competition level math it was tested on. One of my fa­vorite tasks to layer my class­es with are the Problems of the Week provided by the University of Waterloo’s Centre for Education in Mathematics and Computing (CEMC). I enjoy them so much be­cause the prob­lems are multi-faceted, de­signed to chal­lenge stu­dents across learn­ing strands, and pro­mote crit­i­cal and com­pu­ta­tion­al think­ing across grades three to twelve. Given o1’s show­cased abil­i­ty to han­dle ad­vanced ther­mo­dy­nam­ics and chem­istry prob­lems, I ex­pect­ed it to eas­i­ly nav­i­gate these ques­tion sets.

I up­grad­ed to Chat­G­PT Pro to ac­cess o1 pro mode, the model Ope­nAI claims is their best at rea­son­ing. I want­ed to see the best re­sults I could get, as there is al­ways the pos­si­bil­i­ty of an en­ter­pris­ing stu­dent will­ing to in­vest in these tools. To test its ca­pa­bil­i­ties, I fed o1 Pro mode 12 ques­tions from this aca­d­e­m­ic year set at each of five dif­fi­cul­ty lev­els, for a total of 60 ques­tions.¹¹ You can find an archive of the questions I used at the CEMC website. It is the first 12 questions for each level set from 2024/2025. I also have an archive, which you can contact me to obtain. Each ques­tion was test­ed across four tri­als, my at­tempt at mod­el­ling my test after the “4/4 reliability” framework detailed by OpenAI. For each ques­tion and trial I con­vert­ed the PDF files pro­vid­ed by the CEMC into a PNG, at­tached it, and pro­vid­ed the fol­low­ing prompt.

“This image shows a Math­e­mat­ics prob­lem.

Your job is to pro­vide a so­lu­tion to the prob­lem. While doing so, please do the fol­low­ing.
1. Tell me what prob­lem you are solv­ing.
2. Pro­vide de­tails on how to solve the prob­lem.
3. Pro­vide an an­swer.”

Results

De­spite the hype, I found the re­sults un­der­whelm­ing. o1 pro mode passed the “4/4 re­li­a­bil­i­ty” frame­work on only 40 out of 60 ques­tions — a pass rate of 67%. At the in­di­vid­ual trial level, the model an­swered cor­rect­ly in 177 out of 240 at­tempts, a 74% suc­cess rate.²² You can download a recording of my results as an XLSX file. For now, my math­e­mat­ics class­room, and the ef­fort my stu­dents must put into their work, re­mains rel­a­tive­ly safe, though this is most­ly be­cause it is bad at one par­tic­u­lar style of prob­lem: any­thing that re­quires pars­ing in­for­ma­tion from an image.

o1 pro mode’s poor per­for­mance on image in­ter­pre­ta­tion sur­prised me, es­pe­cial­ly given how heav­i­ly the show­case pro­mot­ed ex­act­ly that ca­pa­bil­i­ty. Many of the Prob­lem B (Grade 5–6) ques­tions in­clud­ed im­ages that need­ed to be parsed, and while hu­mans can eas­i­ly ex­tract the nec­es­sary in­for­ma­tion, o1 pro mode strug­gled sig­nif­i­cant­ly. I first rec­og­nized this prob­lem when be­gin­ning the tests with the high­er-level Prob­lem D (Grade 9–10) and Prob­lem E (Grade 11–12) sets. I was cu­ri­ous if it was the dif­fi­cul­ty of the prob­lems them­selves, or if it was the vi­su­al com­po­nent that re­al­ly tripped up o1 pro and caused er­rors. Even when the prob­lems them­selves were sim­pli­fied, the mere pres­ence of an image ap­peared to ren­der the model in­ef­fec­tive. It failed con­sis­tent­ly. The com­plex­i­ty level it per­formed worst at was the Grade 5–6 set, with a pass rate of 5 out of 12 ques­tions (42%), ap­par­ent­ly be­cause those ques­tions car­ried the heav­i­est vi­su­al com­po­nent.

Analysis

That said, this is a no­tice­able im­prove­ment over ear­li­er mod­els. While I never record­ed pre­vi­ous re­sults, my ear­li­er im­pres­sion of a “coin flip”—where a ques­tion had rough­ly a 50/50 chance of being an­swered cor­rect­ly—no longer holds. The model is more con­sis­tent. o1 pro mode now tends to ei­ther get every trial of a ques­tion cor­rect or fail all tri­als out­right. But it still pro­duces dif­fer­ent wrong an­swers for the same ques­tion, which is strange. For ex­am­ple, in POTWE, Prob­lem 3, when asked which route from Omi­cron to Tau gives the short­est trav­el time the model mis­tak­en­ly omit­ted trav­el path­ways from Pi, Rho, and Sigma dif­fer­ent­ly in three sep­a­rate tests, as well as made sev­er­al var­ied mis­takes on the in­for­ma­tion given for time be­tween cities.³³ Here are images from three (1, 2, 3) of the trials mentioned, where it incorrectly identifies the wrong pathways between cities. This vari­abil­i­ty means users must still close­ly ver­i­fy its re­spons­es, though the con­sis­ten­cy on ques­tions it does solve cor­rect­ly is worth not­ing. Mis­takes often vary from fail­ing to iden­ti­fy the cor­rect in­for­ma­tion the model has to work with to solve, though I wish I could share the chats so peo­ple could take a clos­er look at the over­all out­puts.

Limitations

This is prac­ti­tion­er test­ing with ac­knowl­edged con­straints, not peer-re­viewed method­ol­o­gy. I am one of many, many ed­u­ca­tors (with se­vere time con­straints) try­ing to fig­ure out how to teach ef­fec­tive­ly now that stu­dents have easy ac­cess to LLMs and other AI tools. Un­for­tu­nate­ly, I don’t know a sin­gle sec­ondary school, dis­trict, or board ded­i­cat­ing se­ri­ous time, re­sources, and per­son­nel to mean­ing­ful­ly tack­le these prob­lems. I haven’t seen a sin­gle in­sti­tu­tion across pri­ma­ry and sec­ondary ed­u­ca­tion allocating the necessary personnel and resources to rigorously test emerging AI models against internal, validated, context-specific benchmarks to actually see where the frontiers are for our use cases and implications it has in our classrooms. Though, I do un­for­tu­nate­ly see a lot of Ed­u­ca­tion­al Tech­nol­o­gy Co­or­di­na­tor roles spend­ing a lot of time mak­ing nice look­ing Canva posters (money well spent, I guess). So it is up to me, and oth­ers in the trench like me, even with lim­i­ta­tions on time and money. There are many lim­i­ta­tions to my test. This sam­ple size was small, with only 60 ques­tions test­ed across four tri­als each. I chose the ques­tions be­cause of how “new” they were, how they were like­ly to be a bit more unique, and hope­ful­ly not in the train­ing data set. An­oth­er po­ten­tial lim­i­ta­tion is that tests fo­cused sole­ly on one ques­tion source, which would not fully rep­re­sent the broad­er range of math­e­mat­i­cal prob­lems stu­dents might en­counter in the class­room. Larger, more rigorous benchmarks exist — Frontier Math by Epoch AI, for in­stance. How­ev­er, their con­text is PhD and Field Medal­ist Math­e­mati­cians, not the types of ques­tions that the av­er­age sec­ondary stu­dent would en­counter, nor the ques­tions I would work with in my ed­u­ca­tion con­text.

How accurate does a model need to be be­fore I am im­pressed? Does it need to reach per­fec­tion, or is the more de­tailed rea­son­ing good enough to start stu­dents on a path of crit­i­cal­ly an­a­lyz­ing the out­put of its tools? Sec­ondary Math­e­mat­ics does not vi­tal­ly need 100% ac­cu­ra­cy, re­al­is­ti­cal­ly none of this work is going to re­sult in life or death, so I should prob­a­bly tem­per my ex­pec­ta­tions on what I con­sid­er im­pres­sive. From these tests, the gap is main­ly in image in­ter­pre­ta­tion — a prob­lem with the vi­sion model, not the rea­son­ing ar­chi­tec­ture. If we get true, full, mul­ti­modal, then these test re­sults would have like­ly looked much more im­pres­sive.

I wish I had more time to test a wider va­ri­ety of ques­tions, run more tri­als, and con­trol con­di­tions bet­ter. A break­down of spe­cif­ic error types — prompt mis­in­ter­pre­ta­tion, math rea­son­ing fail­ures, image pars­ing strug­gles — would help clar­i­fy where the model ac­tu­al­ly falls apart.

For now, my stu­dents still need to do the heavy lift­ing when it comes to solv­ing prob­lems. But if the bot­tle­neck is the vi­sion model, and vi­sion ca­pa­bil­i­ties are im­prov­ing with each re­lease cycle, how many it­er­a­tions are we away from a model that pass­es these tests re­li­ably? And when it does, what does sec­ondary math­e­mat­ics as­sess­ment ac­tu­al­ly look like?

Footnotes

1. You can find an archive of the ques­tions I used at the CEMC website. It is the first 12 ques­tions for each level set from 2024/2025. I also have an archive, which you can con­tact me to ob­tain. ↩︎

2. You can download a recording of my results as an XLSX file. ↩︎

3. Here are im­ages from three (1, 2, 3) of the tri­als men­tioned, where it in­cor­rect­ly iden­ti­fies the wrong path­ways be­tween cities. ↩︎