Data Insights: Usability

Mixed-Methods Case Study

 TACTICAL    EVALUATIVE    QUALITATIVE    QUANTITATIVE 

Second part of a design research project to improve the usability of Data Insights based on tasks to be done, usability interviews, and web analytics

Challenge

The Data Insights launch went smoothly, but we started to get questions from faculty instructors on how to do supported actions in the interface as well as actions we had not yet considered. Things like switching between reports should be straightforward, but seeing summative data across the semester was something we had not fully explored. After several meetings with the product team, we concluded that continuous improvement of Data Insights to improve adoption and growth in instructor use needed to be grounded in user research. I set out to (1) identify usability challenges with the current design and to (2) determine whether Data Insights supported the core tasks instructors wanted to accomplish as they checked the health of their course.

Dane focusing on enhancing the usability of Data Insights

This presented me with another business challenge

How could we ensure the continued growth in use of Data Insights by instructors?

Objective

As a Mixed-Methods UX Researcher & Software Engineer, I worked across product and engineering to plan a research program to improve the usability of Data Insights. I tapped relationships I had built with university science departments as well as with high value contacts identified by sales. I designed a qualitative and quantitative framework for evaluating the usability and uptake of Data Insights. As a cornerstone of the sales process, the company wanted to invest in ensuring that Data Insights was addressing the needs of instructors to track their course health. The company mission to "create fans" drives my ongoing effort to ensure a positive reputation to drive a virtuous cycle of adoption in this tight-knit community.

Project Outline

This was a multifaceted project that transpired between the Spring of 2022 and late Winter 2023. We hit an unanticipated snag around our intended release in Dec. 2022 and decided to invest significant resources in redesigning the data model that underpins all course analytics.

📏 Scope

📦 Deliverables

👥 Roles

Mixed-Methods UX Researcher
Software Engineer

 RESEARCH QUESTION 

How can we improve the usability of Data Insights to ensure that it empowers instructors to run their courses in a data-driven way?

Research Objectives

Research Methods & Findings

1. Usability Interviews


Show data over time

Some issues in course administration only become obvious over time. Things like falling behind on grading or disparities in student scores stand out more clearly when shown with time as dimension.


Provide tools to explore data

Instructors often wanted to what data they are looking at to isolate a few key groups like specific graders or sections. These tools were available in the interface, but not placed where users expected.


Enhance visibility of outliers

Outliers are important to know because they map to groups of students at a disadvantage. Making outliers easy to isolate, especially in large courses, is vital for instructor efficiency.

UX Research Data Insights

Complete presentation on the usability findings from user interviews

2. Design & Development

Usability interviews highlighted a number of changes and new views that instructors wanted of their data. I channeled these insights into a number of enhanced visual and functional designs.

Evolution of the Grading Status tab, with initial prototype (left) and production-ready (right) iterations

3. Data Model Redesign

Sample of the process to develop a new data model for Data Insights

Dane recognizing that meeting our users needs required a completely new data model

One of the key insights from usability interviews was that critical data needed to be visible over time.

The data model we had, though, was only transactional—records would be updated and any change history lost.

I worked closely with engineering and QA to completely revamp how our data collection worked to support our users' current needs and serve as the basis for future analytics.

4. Establish & Define Web Analytics 

Collaborative activity to define common instructor goals for using Data Insights to check course health

Aligning user goals to the HEART framework

As a complement to my qualitative research, I worked with the product team to articulate what core goals an instructor would want to accomplish in using Data Insights. I then connected each of these goals to a web analytics metric in the HEART framework.

These measurement targets were sorted into ordinal tiers Level 1, 2, or 3 based on the order I would build up the appropriate measurement & analysis infrastructure. 

Data Insights Quant UX metrics (Summer 2023)

Detailed overview of Level 1 metrics captured with Google Analytics and interpreted within the HEART framework

Research Impact

Annotated Score Distribution component highlighting unified interface controls design

Steady uptick in Data Insights adoption, accelerating noticeable since the release of UX improvements

I also developed training materials to support our instructor users to interpret and effectively leverage their course data.

Example of the training materials ecosystem developed around Data Insights

Dane looking to the future of measuring Level 2 & 3 analytics

Level 2 (data interaction) & Level 3 (data synthesis) move beyond basic engagement metrics to ask whether Data Insights has everyday utility for instructors. 

This means the next phase of quant analytics will need a new strategy. 

Stay tuned...