A.I. Grading Apprentice

Mixed-Methods Case Study


Generative foundational research to understand what problems A.I. could solve in university science courses and what constraints exist on the design space.

The outcome was a roadmap for an A.I. grading apprentice that both supports TA grading efficiency and provides students with individualized feedback.


Student frustrated by the unclear criteria that the A.I. is scoring her work against

"I am an extremely hard-working, motivated student...[BUT] I am still being given dreadful grades...by a COMPUTER."

University student



Dane setting out to tackle the big challenge of A.I. in university science education

This gave me my most important, unique business challenge to date:

Can I translate insights into roadmap priorities by understanding from users:


Project Outline

This is an ongoing research project that started in Summer 2023 and will involve the creation of a proof of concept implementation of A.I. in the Labflow product.

📏 Scope

📦 Deliverables

👥 Roles

Mixed-Methods UX Researcher


How do human conceptions of ML/A.I. impact their expectations & desired behavior of an A.I.-powered grading system?

Research Objectives

Research Methods & Findings

1. Stakeholder Interviews

Participant explains in her own words how how ML/A.I. work, displaying elements of the ELIZA effect

"[Artificial Intelligence] is a way over time to teach a computer how to answer questions for you and learn [what you need]."

 — Carmen, internal stakeholder

Synthesizing across all of the internal an external stakeholders led to insights on what areas of the Labflow product could be improved by A.I.

Admin tasks

Repetitive operations like setting dates and course initialization could be made easier with A.I.

Content authoring

Report grading code, PDFs, graphics with alt text could be generated by an LLM to provide a useful starting point for content team

Student assessment

Student work could be graded by A.I. to speed up the feedback process

Grader coaching

TAs often get little guidance on how to be good graders. A.I. could scale up the ability to provide good professional development to graders

Enhanced analytics

Instructors have access to course analytics in Data Insights, but they have to seek them out. A.I. could make finding insights in data more automated

2. Analysis of User Experience in Rival Product

Faculty member who adopted market rival product explains what issues he thought A.I. could help him solve

"We recruit undergraduate TAs to run first year chem labs. Their content knowledge is not the same as senior graduate students, but I don't have enough time to train them properly.

I still want to be able to ask rigorous questions and know responses will get graded correctly for all students."

 — Chemistry Professor & Coordinator of Undergraduate Laboratories

Students who used the market rival product adopted were also given a questionnaire about their experience. This included fixed choice and open-ended responses.

Dane a bit skeptical that A.I. grading is even a good approach, given that it so strongly detested

To put it bluntly, students had strong feelings about A.I. grading.

Some felt it was a double standard ("why can you use it but I can't?"). Others felt it cheapened the value of their university degree ("I'm paying to be taught by people!")

A.I. grading felt like it might be a risky path.

Q: For graded work within [competitor], do you think you were fairly graded?







Qualitative analysis of student responses for reasons why A.I. did not grade them fairly

Underlying reasons why A.I. feedback was helpful or not
(Positive = reason it helped; negative = reason it didn't help)

One "Aha!" moment I had in this data is A.I. was not universally negative for students. Specific aspects of A.I. drove both negative and positive sentiment.



Research Impact

Example of A.I. concepts ranked against a business prioritization rubric

The prioritization activity yielded some clear winners for an A.I. proof of concept in Labflow. 

The insights I distilled from interviews and questionnaire data really pointed to an A.I. assistant that helps grade rather than replace the TA as having high value.

For now how we plan to implement the specifics are secret, but we'll be excited to share something soon!

A.I. grading assistant

Dane getting excited to build some A.I. goodness

The next step is to roll up our sleeves in someone's garage and build this thing!

Stay tuned...