PT Curation Agent for Worksheet-PT alignment

The Challenge: Finding the Right Problems

One of our big tasks is to create a set of interactive problems (PTs) that perfectly match a worksheet for a specific topic. It's a really challenging curation job. First, you have to check the worksheet against the PTs in the set to see what's a good match.

For any question that doesn't have a match, you have to search our entire bank to find new ones that can fill the gap. It takes a lot of time, and it's easy to miss things or pick problems that are too similar. So, I thought, maybe I can make a system to make this easier for me.

My Solution: An AI Agent for Curation

I built the 'PT Curator,' which is an AI agent that I use through a command-line interface. It helps me manage the entire curation process from start to finish. I designed it to be very systematic.

It starts with a "gap analysis" to automatically compare the worksheet to our existing content and tells me exactly which skills are missing. Then, I can give it a new set of "candidate" PTs, and the agent evaluates them, finds duplicates, and gives me a clear recommendation on which ones I should select.

Of course, I still QA the AI's output to make the final decision, but the agent does the heavy lifting and has significantly reduced the time it takes to complete this task. Everything is tracked in a "Master Alignment Log" so I can see the whole process.

The Commands I Created

The agent works using a few simple commands that I designed for the workflow:

• /start: This one just starts the process and loads all the files I need.
• /run gap_analysis: This is the main check. It does the comparison and tells me what skills are missing.
• /evaluate: I use this to give the agent a bunch of new PTs, and it gives me a report on which ones are the best fit.
• /organize_and_report: Once I've chosen the PTs, this command organizes them all and gives me the final, clean list.