Time on Task

Mixed-Methods Case Study

 TACTICAL    EVALUATIVE    QUALITATIVE    QUANTITATIVE 

First phase of a mixed-methods research project to design a quantitative time on task model and validate this definition against users' perceptions of time spent grading & effort.

Challenge

Dane thinking about this work within the broader social and political context

As a former graduate student in a union, I was tracking the waves of collective action happening on university campuses.

A number of prominent universities using Labflow had TAs or faculty striking, or threatening to do so.

I wanted to be sure that we at the company level were fully aware of our legal responsibilities under states' and federal law for providing data that potentially be misused in labor relations.

University of California system graduate teaching assistants (GTAs) strike (Nov. 2022). Source: Science

University of Illinois-Chicago faculty threaten to strike over contract negotiation (Jan. 2023). Source: UICUF

Objective

As a Data Scientist & Mixed-Methods UXR, I worked across management, product, and engineering to define the goals and risks of developing time on task measurement techniques. 

I worked closely with some early adopter universities to refine and hone the quantitative approaches to modeling the data and validate its outputs. I also developed a research & development plan for rolling out the functionality and measuring user sentiment.

Project Outline

This is an ongoing project that started in Fall/Winter 2022. The project if currently focused on developing and validating a time on task model for estimating grading time.

📏 Scope

📦 Deliverables

👥 Role

Data Scientist
Mixed-Methods UX Researcher

 RESEARCH QUESTION 

How can we estimate user time time spent grading and to what extent does that align with their own self-reports and perceptions of effort?

Research Objectives

Research Methodologies

"We often get admin [department] pushback. They want to know if we're assigning a balanced workload. I also want to be able to identify my inexperienced TAs."

 — Dr. Jackie Powell, University of Pittsburgh

"I want to be able to see summaries of grading time like averages [for activities], but also have it broken down by individual TAs."

 — Dr. Angela Bischof, Penn State University

Notes from a one-on-one interview with a faculty member interested in grading time data

Conversations with faculty revealed that there are at least two types of uses for this data.

The use of the data at the departmental level presents the clearest case for legal problems.

Most instructors had not even considered the legal implications of this data and were not aware of their university or state regulations.

Grading events are emitted for discrete actions performed by TAs like:

User events also augment this to fill in gaps in time not caught by grading actions.

"Quick Grade" interface in Labflow. Assigning point values, typing feedback, and selecting rubric items all emit individual grading events.

Discrete grading events are captured in BigQuery. I then developed a few computational techniques (Kernel Density Estimation depicted).

Dane considering his model options for estimating time on task

In my R&D, I needed to identify performant computational models for estimating the time on task.

I iterated on a few different possible models and settled on using a KDE model for its handling of distributed clusters.

Clustered bar chart comparing the self-report grading times for three reports against two estimation methods (histogramming, Kernel Density Estimation)

Research Impact

Example of grader data parsed with Kernel Density Estimation to identify divisions between logical clusters in grading events

Prototype of grading time across time separated by individual activities

Prototype of activity level grading data separated by individual grader