The Challenge: Making Peer QA Consistent
At Mathspace, when a set of new Problem Templates (PTs) is created, it goes through a Peer QA process. My task is to be the quality checker for the PTs my colleague builds. This means I have to check for dozens of things, from math accuracy and correct XML syntax to making sure the language is right for our students in Australia or the US.
Doing all these checks manually is a lot of work and can be inconsistent. Sometimes I might focus more on one area than another. I wanted to build a system to help me standardize my own QA workflow and make sure I never miss a single detail.
My Solution: An AI Agent for My QA Workflow
I built the 'QA Workflow Agent,' a tool I use as a command line to help me review a set of PTs. It acts as my personal QA assistant, managing the whole process from start to finish. I still have to check the output the AI gives to make the final call, of course, but this agent does the heavy lifting and has reduced a lot of time in the process.
The most important part is the "Master Feedback Log," which is a JSON file that keeps track of all the feedback for every PT I'm checking. The agent uses a series of commands to run checks, and it automatically organizes all the findings into the log. In the end, it generates a final, clean report for me that I can share.
How I Built It: A "Megaprompt" Design
The way I made the agent smart was by creating a single, really big "megaprompt." This main prompt contains everything—its personality as a QA specialist, how it remembers things with the JSON log, and all the rules it has to follow.
Inside this megaprompt, I also built a library of smaller, specialized prompts. So when I give the agent a command like `/run check Math_Accuracy`, it knows to use the specific sub-prompt for that QA task. This is how it can do different, complex jobs like checking for localization or analyzing hint structures.
The Commands I Built for My Workflow
The agent is built around a command system that lets me control every step of the QA process, from initial setup to the final report.
Phase 1: Task Initialization
/start - Initializes the QA task and asks for all context files.
/run html - Formats the request document into a clean HTML view.
/run progression_map - Organizes the PTs and creates the Master Feedback Log.
Phase 2: Automated QA Checks
/run alignment - Checks if PTs align with the original request document.
/run comprehensive - Runs a full, holistic review against our entire creation handbook.
/run skill_check - Verifies that each PT is testing the correct curriculum skills.
/run check [check_name] - Runs a specific, targeted check. `[check_name]` can be:
- Hint_Structure
- Forbidden_Words
- Randomization_Logic
- Language_Tone
- Localization
- Grade_Level
- Math_Accuracy
- Solution_Paths
- Realistic_Randomization
Phase 3: Feedback & Reporting
/add_my_feedback [PT_ID] - Lets me add my own manual notes to the log for a specific PT.
/add_general_feedback - Allows me to add high-level notes for the whole project.
/show_feedback_table - Displays the full feedback log as a clean TSV table.
/generate_report - Takes all the feedback from the log and creates the final, synthesized report for my coworker.
/help - Shows the full list of commands.
Project Overview
AI Agent Design, Quality Assurance, Prompt Engineering, System Architecture
CLI, Large Language Models (LLMs), JSON, TSV