The Challenge: Can AI Check Math Answers?
Our curriculum head, started a discussion in our AI-sharing Slack channel. The idea was to see if we could automate the process of checking answers in our math worksheets. It's a really important QA step, but it takes a lot of time to do manually.
She had already been running a Python script, but the results were mixed. The AI was good, but it was also making simple mistakes, giving a lot of "false positives." The question for the team was: can we make this process more accurate?
My Contribution: Exploring on My Own
For our "AI lab hour," I decided to do some exploring on my own. I took on the task of building my own version of the script to see if I could get better results.
My Python script was built to:
- Parse our complex worksheet files to pull out each question and its answer.
- Clean up and normalize the answers so the AI could understand them easily.
- Use a detailed prompt to tell the AI to act like an expert math teacher.
- Send each problem to different AI models (like OpenAI and Gemini) to test and compare them.
- Collect all the JSON responses from the AI and organize them into a final CSV report for analysis.
Outcome: A Valuable Finding
After running my tests and sharing the results with the team, we confirmed the initial findings. My script worked well, but the AI models were still not consistent enough for this specific task—they were still flagging correct answers as wrong.
Even though it wasn't implemented, this was a successful R&D project. We learned a lot about the current limits of AI for math validation and made a data-driven decision not to spend more time on it for now. It was a great example of our team collaborating to explore new tech.
import openai
from bs4 import BeautifulSoup
import re
# --- PROMPT TEMPLATE ---
CHECK_ANSWER_PROMPT_TEMPLATE = """
You are a meticulous and expert mathematics teacher...
Your task is to analyze a single question... and compare your answer to the provided answer.
...
**INSTRUCTIONS (Follow this order precisely):**
1. **Analyze and Solve:** ...
2. **Normalize Your Answer:** ...
3. **Final Conclusion:** ...
...
**JSON Response Format:**
{{
"is_correct": boolean,
"calculated_answer": "...",
"confidence_score": integer (0, 1, or 2),
"reasoning": "..."
}}
"""
def parse_worksheet(file_path):
# ... (function to parse JSX/XML worksheet files) ...
pass
def check_problem_with_ai(problem, client):
# ... (function to call the LLM with the prompt and problem data) ...
pass
if __name__ == "__main__":
# ... (main script to orchestrate parsing, AI checking, and report generation) ...
pass
Project Overview
R&D, Prototyping, Python Scripting, LLM Evaluation, Collaborative Problem-Solving
Python, OpenAI API, Gemini API, BeautifulSoup