Click HERE to join our Google Group/mailing list to stay up-to-date as the discussion about evaluation in HCI evolves!

Evaluation beyond Usability: SIG

The HCI community has a long and successful history of evaluating its research - however, this evaluation mostly looks at usability. But how do we evaluate non-usability aspects of research?

At CHI 2018 in Montreal, we organed a Special Interest Group meeting to discuss those issues with the community. The SIG was called "Evaluating HCI Research beyond Usability" - stay in touch by joining the evaluation-hci mailing list!

chi2018logo
WHEN: Tuesday, April 24, 11:00am
WHERE: Room 514B, Palais des congrès, Montreal
CALL: pdf



Evaluation beyond Usability: Paper

Independently, we also tried to provide a first answer to the question specifically for the domain of sustainability. We developed a recipe comprising five ingredients that we believe can provide guidance to researchers in order to identify a suitable evaluation process. Those five ingredients are:

  1. Goals: What is the goal the SHCI research artefact is trying to achieve with regard to sustainability?
  2. Mechanisms: What are the mechanisms that need to be considered to understand and assess the holistic impact of the research artefact in the context of external influencing factors?
  3. Metrics: What are the metrics that can be observed or measured to assess any given mechanism with regard to reaching the desired goal?
  4. Methods: What methods can be employed to assess the metrics for the respective mechanisms to answer the question to what extent the artefact has reached the desired goal?
  5. Scope: What is the scope of mechanisms that can, should, and will be considered for validating the artefact?

evaluation-figure

Note that this recipe is not meant to be the solution to the evaluation problem - it is just one proposal to jumpstart the discussion that we believe is overdue in our community. Read more about the difficulties of evaluating sustainability and opportunities for furthering the discussion in our paper. If you're interested in examples of how this recipe could potentially be applied to sustainability projects, check out our supplementary file. Also, don't miss our presentation at CHI!

chi2018logo
WHEN: Thursday, April 26, 11:00am (2nd paper, starting 11:20)
WHERE: Room 521ABC, Palais des congrès, Montreal
PAPER: pdf / Supplementary file with examples: pdf