In Fall 2020, along with a group of three other students, I conducted a thorough evaluation of the Protect Texas Together mobile app. The app was developed by The University of Texas at Austin for students, faculty, and staff in response to the COVID-19 pandemic. Users can report and review health symptoms, report and review their location on campus, report and review COVID-19 test results, and enable a location tracking feature. Findings from my research were presented as part of an hour-long report. A full copy of my report can be found here.
I began with a heuristic evaluation of an early version of the app, which was based on the 10 Usability Heuristics for User Interface Design published by the Nielsen Norman Group. Each heuristic violation was assigned an urgency level describing the seriousness of the problem and the priority in which it should be addressed. Heuristic violations were also followed by clear, actionable recommendations.
A competitive analysis was conducted on similar COVID-19 apps to identify additional recommendations. The evaluation was conducted on direct and indirect competitors to better capture an array of features. The analysis was followed by clear, actionable recommendations.
Following the heuristic evaluation and competitive analysis, I conducted a usability test of an updated version of the app. Participants were recruited using a screener script that I developed to identify eligibility based on qualifying criteria and a pre-determined demographic quota. Participants were first interviewed to gauge attitudes toward issues such as privacy. Once the interview was completed, participants completed a series of usability tasks using a wireframe that I developed in Figma. Each task was following by a 5-item post-task scale. At the end of the last task, participants completed an exit interview and a post-test system usability scale.
For each participant, interview responses, observations, and comments were recorded in a repository. Data was coded using thematic analysis. Observations related to checking symptom history, for example, were grouped together. Within a group, observations were further divided into more closely-related themes. While checking symptom history, for example, users are assigned a risk level. Notes were, therefore, grouped if they pertained specifically to finding an operational definition of risk. This allowed me to say precisely the number of participants affected for each issue. It also facilitated detection of trends that may have been missed relying on causal observation, or feeling, alone.
A final report was created to highlight the findings of my analysis. Results were reported such that it was clear to developers what issues would arise, when, and where. More urgent issues were emphasized and were followed by recommendations.