1. Usability Test Plan
This evaluation assesses the effectiveness, learnability, and user satisfaction of the ChromaticHarp Tab App
Goals
- Verify users can complete core workflows successfully.
- Identify learnability barriers (especially for novices).
- Measure perceived usability using SUS.
- Capture issues using Nielsen’s heuristics + severity ratings.
Approach
- Heuristic evaluation of the initial prototype.
- Moderated user test with task-based protocol + SUS.
2. Heuristic Evaluation (Initial Prototype)
We used Nielsen’s 10 usability heuristics to identify potential risks prior to user testing. Key areas of concern included error prevention, visibility of system status, and help/documentation for first-time users.
Heuristic Evaluation Table (Initial Prototype)
Severity scale: 0 = No problem · 1 = Cosmetic · 2 = Minor · 3 = Major · 4 = Critical
| S/N | Screen / Feature | Nielsen Heuristic | Agreed Severity | Suggested Improvement |
|---|---|---|---|---|
| 1 | Library | Visibility of system status | 2 | Indicate clearly when search, filters, or sorting are active (visual highlights or status indicators). |
| 2 | Library | Recognition rather than recall | 2 | Make filtering and sorting options more visible so users do not have to remember where options are located. |
| 3 | Library | Consistency and standards | 2 | Align search bar behaviour and icon usage with common Android list/search patterns. |
| 4 | Tab Detail | User control and freedom | 3 | Provide clearer navigation and safe exit paths, especially for destructive actions such as delete. |
| 5 | Tab Detail | Error prevention | 3 | Add safeguards to prevent accidental deletion or unintended actions. |
| 6 | Tab Detail | Match between system and real world | 2 | Improve terminology and iconography to better reflect how users naturally think about tabs and songs. |
| 7 | Editor | Error prevention | 3 | Prevent accidental loss of work by ensuring users are warned before exiting with unsaved changes. |
| 8 | Editor | User control and freedom | 2 | Allow users to easily cancel or undo actions during editing without confusion or fear of losing progress. |
| 9 | Editor | Recognition rather than recall | 3 | Reduce the need to remember notation rules by providing inline hints or visible guidance within the editor. |
| 10 | Practice | Visibility of system status | 3 | Make it clearer which part of the tab is currently being practiced and how progress is structured. |
| 11 | Practice | Flexibility and efficiency of use | 2 | Support efficient navigation for experienced users (e.g., quicker movement between lines). |
| 12 | Practice | Aesthetic and minimalist design | 1 | Minor visual refinements to keep focus on the active practice content without unnecessary distraction. |
| 13 | Global | Consistency and standards | 2 | Ensure consistent visual language, icon behavior, and navigation patterns across all screens. |
| 14 | Global | Help and documentation | 3 | Provide basic in-app guidance or onboarding to explain core concepts and workflows. |
| 15 | Global | Help users recognize & recover from errors | 2 | Use clearer feedback messages when something goes wrong and indicate how the user can recover. |
3. Hypotheses / Testable Questions
- H1: Users can complete all main tasks without failure.
- H2: Users without harmonica experience can still use the app effectively.
- H3: The app achieves a high SUS score (≥ 80).
- H4: Creation/editing tasks require more support than browsing/practice tasks.
4. Planned Data to Collect
Quantitative
- Task outcomes per task: S (Success), P (Partial with hint), F (Fail)
- SUS scores (0–100)
- Participant demographics (age, harmonica experience, tab-app experience)
- Counts of partial completions per task
Qualitative
- Moderator observations during task execution
- Oral feedback and notable quotes
- Usability issues mapped to Nielsen heuristics
- Issue severity ratings (1–4)
5. Methods Used (User Test)
Participants
n = 6, majority novice harmonica users.
Environment
- Android smartphone + Chromatic App prototype
- Think-aloud protocol encouraged
- Sessions recorded for observation notes
Measures
- Task outcome per task (S, P, F)
- System Usability Scale (SUS)
- Heuristic issue mapping + severity (1–4)
Protocol
- Briefing: explain goals, consent, and expectations.
- Warm‑up: short demo of app structure (no task hints).
- Task run: six scenario‑based tasks with think‑aloud.
- Post‑test: SUS questionnaire + open feedback.
Tasks
| Task ID | Scenario | Success Criteria |
|---|---|---|
| T1 | “Find a tab titled Amazing Grace and open it.” | User locates the tab using search or scrolling. |
| T2 | “Show only Medium difficulty tabs and sort them by newest.” | User successfully applies and removes filters and sorting. |
| T3 | “Mark any tab as a favourite and show only your favourite tabs.” | User understands favorite functionality. |
| T4 | “Create a new tab with a title, key, difficulty, tags, and at least two lines of notes. Save it.” | User completes creation without external help. |
| T5 | “Edit the tab you created, then try to leave without saving.” | User understands discard/save dialog. |
| T6 | “Start practice mode and move between different lines.” | User navigates lines and understands practice flow. |
6. Results Overview
Highlights from the evaluation are summarized above; detailed charts are available in the visualizations section.
7. Data Visualizations
7.1 Demographics
Figure 1: Participant demographics (age group, harmonica experience, app experience).
7.2 SUS Results
Individual SUS scores: 95.0, 97.5, 95.0, 97.5, 100.0, 87.5. The mean SUS score (95.42) indicates excellent perceived usability.
7.3 Task Outcomes
| Task | Success | Partial (Hint) | Fail |
|---|---|---|---|
| T1 | 6 | 0 | 0 |
| T2 | 5 | 1 | 0 |
| T3 | 6 | 0 | 0 |
| T4 | 1 | 5 | 0 |
| T5 | 5 | 1 | 0 |
| T6 | 6 | 0 | 0 |
7.4 Issue Severity Distribution
8. Most Relevant Feedback / Comments
“Add a popup prompt before deleting a tab just to confirm.”
“In the editing screen, add an info popup for notation editing to explain how it works.”
“Enter should also enter the tag.”
“The search bar is weird behaviorally.”
“Pressing cancel doesn’t bring up the popup like back.”
“Clear all filters button would help.”
“Show line number out of how many lines in practice mode.”
“Play the note on the editor screen.”
“The library icon looks like a button.”
9. Updates Made Based on Feedback
- Delete confirmation dialog (and optional undo snackbar).
- Info popup / tooltip for notation editing.
- Enter/Done commits tags in the editor.
- Improve search UX (clear state, consistent cancel behavior).
- Add “Clear all filters” action in Library.
- Add “Line X of Y” progress indicator in Practice.
- Add note playback feedback in Editor.
- Refine library icon styling to reduce false affordance.
10. Summary
The usability evaluation indicates excellent user satisfaction (Mean SUS = 95.42) and strong task performance (no failures). Most usability improvements are centered on learnability and feedback, especially for creation/editing workflows (T4). Implementing the proposed changes is expected to improve first-time user confidence and reduce reliance on hints.