Evaluation beyond Usability: SIG

The HCI community has a long and successful history of evaluating its research - however, this evaluation mostly looks at usability. But how do we evaluate non-usability aspects of research?

At CHI 2018 in Montreal, we are organizing a Special Interest Group meeting to discuss those issues with the community. Join us on for our SIG "Evaluating HCI Research beyond Usability"!

WHEN: Tuesday, April 24, 11:00am
WHERE: Room 514B, Palais des congrès, Montreal
CALL: pdf

Evaluation beyond Usability: Paper

Independently, we also tried to provide a first answer to the question specifically for the domain of sustainability. We developed a recipe comprising five ingredients that we believe can provide guidance to researchers in order to identify a suitable evaluation process. Those five ingredients are:

  1. Goals: What is the goal the SHCI research artefact is trying to achieve with regard to sustainability?
  2. Mechanisms: What are the mechanisms that need to be considered to understand and assess the holistic impact of the research artefact in the context of external influencing factors?
  3. Metrics: What are the metrics that can be observed or measured to assess any given mechanism with regard to reaching the desired goal?
  4. Methods: What methods can be employed to assess the metrics for the respective mechanisms to answer the question to what extent the artefact has reached the desired goal?
  5. Scope: What is the scope of mechanisms that can, should, and will be considered for validating the artefact?


Note that this recipe is not meant to be the solution to the evaluation problem - it is just one proposal to jumpstart the discussion that we believe is overdue in our community. Read more about the difficulties of evaluating sustainability and opportunities for furthering the discussion in our paper. If you're interested in examples of how this recipe could potentially be applied to sustainability projects, check out our supplementary file. Also, don't miss our presentation at CHI!

WHEN: Thursday, April 26, 11:00am (2nd paper, starting 11:20)
WHERE: Room 521ABC, Palais des congrès, Montreal
PAPER: pdf / Supplementary file with examples: pdf

Leave a comment:

Name:     Email (will not be published):

Spam prevention: In which city does CHI2018 take place?


Christian wrote on 2018-04-16 00:37:56:
Anyone wants to share their experiences with issues of evaluation? Or do you have an idea for a discussion point during the SIG? Share your thoughts!