Time on Task

Mixed-Methods Case Study

 TACTICAL    EVALUATIVE    QUALITATIVE    QUANTITATIVE 

First phase of a mixed-methods research project to design a quantitative time on task model and validate this definition against users' perceptions of time spent grading & effort.

The outcome of this work was a beta time on task component informed by user interviews as well as updated legal supporting its rollout

Challenge

Dane thinking about this work within the broader social and political context

As a former graduate student in a union, I was tracking the waves of collective action happening on university campuses.

A number of prominent universities using Labflow had TAs or faculty striking, or threatening to do so.

I wanted to be sure that we at the company level were fully aware of our legal responsibilities under states' and federal law for providing data that potentially be misused in labor relations.

University of California system graduate teaching assistants (GTAs) strike (Nov. 2022). Source: Science

University of Illinois-Chicago faculty threaten to strike over contract negotiation (Jan. 2023). Source: UICUF

Objective

Project Outline

This is an ongoing project that started in Fall/Winter 2022. The project if currently focused on developing and validating a time on task model for estimating grading time.

📏 Scope

📦 Deliverables

👥 Role

Data Scientist
Mixed-Methods UX Researcher

 RESEARCH QUESTION 

How can we estimate user time time spent grading and to what extent does that align with their own self-reports and perceptions of effort?

Research Objectives

Research Methods & Findings

1. Faculty Interviews

"We often get admin [department] pushback. They want to know if we're assigning a balanced workload. I also want to be able to identify my inexperienced TAs."

 — Dr. Jackie Powell, University of Pittsburgh

"I want to be able to see summaries of grading time like averages [for activities], but also have it broken down by individual TAs."

 — Dr. Angela Bischof, Penn State University

Notes from a one-on-one interview with a faculty member interested in grading time data

Conversations with 5 faculty revealed that there are at two major for grading time data.

The use of the data at the departmental level presents the clearest case for legal problems.

Most instructors had not even considered the legal implications of this data and were not aware of their university or state regulations.

2. Develop & Validate Quantitative Model

Grading events are emitted for discrete actions performed by TAs like:

User events also augment this to fill in gaps in time not caught by grading actions.

"Quick Grade" interface in Labflow. Assigning point values, typing feedback, and selecting rubric items all emit individual grading events.

Discrete grading events are captured in BigQuery. I then developed a few computational techniques (Kernel Density Estimation depicted)

Dane considering his model options for estimating time on task

I needed to identify performant computational models for estimating the time on task.

I iterated on a few different possible models and settled on using a KDE model for its handling of distributed clusters.

Initial comparisons with self-report indicated the method was accurate for distributed grading actions.

Clustered bar chart comparing the self-report grading times for three reports against two estimation methods (histogramming, Kernel Density Estimation)

3. Legal Compliance Review

The company contacted an employment lawyer on retainer to inquire about laws in the states where we operate. We explored the most restrictive jurisdiction of California to understand what needs to be in place to meet our legal obligations under the California Consumer Privacy Act (CCPA).

4. Visualization Design & Development

Concepts for possible ways to represent time on task for specific activities as well as individual activities

Research Impact

Alpha release candidate of time on task visualization demoing UI functionality

Grading time estimation is helping universities make data-driven decisions when recruiting teaching staff.

The University of North Texas (UNT) decided based on grading time data to rebalance course loads, saving the department $150k!

UNT then reinvested this money in students to lower course costs.

LIMITATION OF LIABILITY

To the extent allowed by Texas law and the U.S. Constitution, in no event shall Catalyst Education LLC, nor its directors, employees, partners, agents, suppliers, or affiliates, be liable for any indirect, incidental, special, consequential, or punitive damages, including without limitation, loss of profits, data, use, goodwill, or other intangible losses, resulting from (i) your access to or use of or inability to access or use the Service; (ii) any conduct or content of any third party on the Service; (iii) any content obtained from the Service; and (iv) unauthorized access, use or alteration of your transmissions or content, whether based on warranty, contract, tort (including negligence) or any other legal theory, whether or not we have been informed of the possibility of such damage, and even if a remedy set forth herein is found to have failed of its essential purpose.

Legal agreements for ToS and Privacy Policy were updated to clearly explain the legal agreement underpinning our data practices.

These changes were included to cover:

We also intend to make time on task an opt-in service.

Dane planning to do gather feedback on the design concept

The beta design is now ready for release to a handful of early adopter universities who have expressed interest in reports on this data.

The next step is to sit down with these early adopters and evaluate areas for improvement.

Stay tuned...