HCI - Lecture 7: HCI Evaluation Techniques - Qualitative Methods What is Qualitative Data? · Based on verbal responses · Allows for subjectivity from participants · Allows for flexibility from researchers · Typically more in-depth data · Can accurately answer "why?" · Generate unknowns Think Aloud: " A form of observation research " Ask participants to narrate their thoughts and actions · Pros: o Simple for the participants o Provides useful insight into issues and problems Cons: o Subjective to the participant o Observer effects Cooperative Evaluation: · Modification on Think Aloud . Participants collaborates with evaluation · Both participants and evaluators can interact · Pros: o Less constrained and easier to use o Participant directly encouraged to criticise o Can get clarification from evaluator Cons: o Slower data collection than Think Aloud o Experimenter effects Observations: · Observe participants in a system · Can be quantitative/qualitative " Can be overt/covert . Pros: o Real scenarios and contexts o No chance of interruption from researchers · Cons: o Bias if overt observations o Unclear when data collection finishes " How to measure observations = Protocol Analysis: o Paper and pencil o Audio recordings o Video o Computer logging o User notebooks/diaries Post Task Walkthroughs: · Recordings/transcripts played back to participants . They explain what/how/why they performed a behaviour · Can be immediate/delayed Pros: o Natural environment during initial recording o Helps identify motivations o Helpful when Think Aloud is inappropriate . Cons: o Slower to collect dataff efigfi 2 sessions instead of 1 o Unreliable memory issues if delayed
Questionnaires (Open-Ended): · Set of fixed questions with free-text responses . Pros: o Can be used as a mixed method study o Quick and large data collection Cons: o Less flexible o Drop-out rates o No individual adjustments Interviews: " Direct method of gathering information One-to-one basis or focus groups · Interview questions: o Pre-prepared (interview schedule) o Typically narrows into a specific topic o Maintains consistency o Semi-structured interviews Pros: o Some flexibility o Probing for further information o More in-depth responses o Expects the unexpected o Relatively cheap · Cons: o Very subjective o Time consuming Evaluations Explained: " Test usability and functionality of the system - does it behave as expected? " Can occur in laboratory or in the field · With designers/experts or actual users · Evaluates both design and implementation " Should be considered at all stages in the design life cycle " Focus groups about designff early design changesff up to assessing completed system Goals of Evaluation: . To assess the extent and accessibility of system functionality o Needs to meet the user's requirements and expectations o Measuring the user's performance as a measure of effectiveness in the system . To assess the user's experience of the interaction o How easy it is to learn the systemff the usability and user's satisfaction o Enjoyment and emotional responsesff efigfi leisure or entertainment systems · To identify specific problems with the system o Aspects of the design