Evaluation beyond Usability: SIG

The HCI community has a long and successful history of evaluating its research - however, this evaluation mostly looks at usability. But how do we evaluate non-usability aspects of research?

• How do we decide if an idea is creative or not?
• How do we evaluate design fiction?
• How do we guage the negative impacts of technology?
• How do we assess if a design is sustainable?
• How do we measure the accuracy of a future scenario?

At CHI 2018 in Montreal, we organed a Special Interest Group meeting to discuss those issues with the community. The SIG was called "Evaluating HCI Research beyond Usability" - stay in touch by joining the evaluation-hci mailing list!

WHEN: Tuesday, April 24, 11:00am
WHERE: Room 514B, Palais des congrès, Montreal
CALL:

Evaluation beyond Usability: Paper

Independently, we also tried to provide a first answer to the question specifically for the domain of sustainability. We developed a recipe comprising five ingredients that we believe can provide guidance to researchers in order to identify a suitable evaluation process. Those five ingredients are:

1. Goals: What is the goal the SHCI research artefact is trying to achieve with regard to sustainability?
2. Mechanisms: What are the mechanisms that need to be considered to understand and assess the holistic impact of the research artefact in the context of external influencing factors?
3. Metrics: What are the metrics that can be observed or measured to assess any given mechanism with regard to reaching the desired goal?
4. Methods: What methods can be employed to assess the metrics for the respective mechanisms to answer the question to what extent the artefact has reached the desired goal?
5. Scope: What is the scope of mechanisms that can, should, and will be considered for validating the artefact?

Note that this recipe is not meant to be the solution to the evaluation problem - it is just one proposal to jumpstart the discussion that we believe is overdue in our community. Read more about the difficulties of evaluating sustainability and opportunities for furthering the discussion in our paper. If you're interested in examples of how this recipe could potentially be applied to sustainability projects, check out our supplementary file. Also, don't miss our presentation at CHI!

WHEN: Thursday, April 26, 11:00am (2nd paper, starting 11:20)
WHERE: Room 521ABC, Palais des congrès, Montreal
PAPER: / Supplementary file with examples:

How do we evaluate HCI research beyond usability? How to decide if a design is sustainable or creative? How to assess if a future scenario is realistic? Join the debate in our SIG at #chi2018 in Montréal! https://t.co/8Kaos82PmV @bates_oliver @jcmankoff @gulliblefish

— Christian Remy (@christianr_hci) April 13, 2018