Data Insights: Usability
Mixed-Methods Case Study
Second part of a design research project to improve the usability of Data Insights based on tasks to be done, usability interviews, and web analytics
Challenge
The Data Insights launch went smoothly, but we started to get questions from faculty instructors on how to do supported actions in the interface as well as actions we had not yet considered. Things like switching between reports should be straightforward, but seeing summative data across the semester was something we had not fully explored. After several meetings with the product team, we concluded that continuous improvement of Data Insights to improve adoption and growth in instructor use needed to be grounded in user research. I set out to (1) identify usability challenges with the current design and to (2) determine whether Data Insights supported the core tasks instructors wanted to accomplish as they checked the health of their course.
Dane focusing on enhancing the usability of Data Insights
This presented me with another business challenge:
How could we ensure the continued growth in use of Data Insights by instructors?
Objective
As a Mixed-Methods UX Researcher & Software Engineer, I worked across product and engineering to plan a research program to improve the usability of Data Insights. I tapped relationships I had built with university science departments as well as with high value contacts identified by sales. I designed a qualitative and quantitative framework for evaluating the usability and uptake of Data Insights. As a cornerstone of the sales process, the company wanted to invest in ensuring that Data Insights was addressing the needs of instructors to track their course health. The company mission to "create fans" drives my ongoing effort to ensure a positive reputation to drive a virtuous cycle of adoption in this tight-knit community.
Project Outline
This was a multifaceted project that transpired between the Spring of 2022 and late Winter 2023. We hit an unanticipated snag around our intended release in Dec. 2022 and decided to invest significant resources in redesigning the data model that underpins all course analytics.
Usability interviews: Apr. 2022
Design & development: May – Dec. 2022
Data model redesign: Jan. – Mar. 2023
Web analytics: Mar. 2023 – present
📏 Scope
Identify the usability challenges that instructors face when interacting & interpreting course data
Determine gaps in the UX between instructor goals and current design
Define and implement web analytics to capture quantitative usability
📦 Deliverables
Presentation on the usability issues identified from faculty interviews
An updated design & release of Data Insights implementing improved recommendations
Reports summarizing the quantitative usability of Data Insights
👥 Roles
Mixed-Methods UX Researcher
Software Engineer
How can we improve the usability of Data Insights to ensure that it empowers instructors to run their courses in a data-driven way?
Research Objectives
Uncover & assess the areas for improvement in Data insights
Identify unmet instructor needs and goals not enabled by the current design
Look for evidence of healthy UX in quantitative streams aligned to the HEART framework
Research Methods & Findings
Methods: Cognitive Think Aloud, Web analytics
Findings:
Users wanted (1) course data presented longitudinally and (2) to be able to interact with and isolate data by different dimensions (e.g., filter it by section or grader, isolate outliers)
Data Insights has has averaged >50% y/y growth each month since release
1. Usability Interviews
❶
Show data over time
Show data over time
Some issues in course administration only become obvious over time. Things like falling behind on grading or disparities in student scores stand out more clearly when shown with time as dimension.
❷
Provide tools to explore data
Provide tools to explore data
Instructors often wanted to what data they are looking at to isolate a few key groups like specific graders or sections. These tools were available in the interface, but not placed where users expected.
❸
Enhance visibility of outliers
Enhance visibility of outliers
Outliers are important to know because they map to groups of students at a disadvantage. Making outliers easy to isolate, especially in large courses, is vital for instructor efficiency.
2. Design & Development
Usability interviews highlighted a number of changes and new views that instructors wanted of their data. I channeled these insights into a number of enhanced visual and functional designs.
Evolution of the Grading Status tab, with initial prototype (left) and production-ready (right) iterations
3. Data Model Redesign
Sample of the process to develop a new data model for Data Insights
Dane recognizing that meeting our users needs required a completely new data model
One of the key insights from usability interviews was that critical data needed to be visible over time.
The data model we had, though, was only transactional—records would be updated and any change history lost.
I worked closely with engineering and QA to completely revamp how our data collection worked to support our users' current needs and serve as the basis for future analytics.
4. Establish & Define Web Analytics
Collaborative activity to define common instructor goals for using Data Insights to check course health
Aligning user goals to the HEART framework
As a complement to my qualitative research, I worked with the product team to articulate what core goals an instructor would want to accomplish in using Data Insights. I then connected each of these goals to a web analytics metric in the HEART framework.
These measurement targets were sorted into ordinal tiers Level 1, 2, or 3 based on the order I would build up the appropriate measurement & analysis infrastructure.
Level 1*: Base usage & adoption
Level 2: Interacting with data (planned)
Level 3: Synthesizing data to improve courses (planned)
Research Impact
Updated UX for Data Insights released for product-wide use
Data Insights has maintained 50–90% y/y growth, adjusting for seasonality
Burgeoning ecosystem of training materials have developed to show how instructors can effectively leverage course data from Data Insights
Annotated Score Distribution component highlighting unified interface controls design
Steady uptick in Data Insights adoption, accelerating noticeable since the release of UX improvements
I also developed training materials to support our instructor users to interpret and effectively leverage their course data.
Example of the training materials ecosystem developed around Data Insights
Dane looking to the future of measuring Level 2 & 3 analytics
Level 2 (data interaction) & Level 3 (data synthesis) move beyond basic engagement metrics to ask whether Data Insights has everyday utility for instructors.
This means the next phase of quant analytics will need a new strategy.
Stay tuned...