More than "usability" - feelings of 2017 Chinese CHI forum participation


"Seeing is believing". However, is it the essence of things for what you see?

Think of such a picture, when you're at the mall with your girlfriend, she stands in front of a brand cosmetics counter. When you ask, she laughs and says, "it's fine, just looking around." If the information you get at this time is "Oh, she's just looking around", then is what you see here really the nature of things? Relationships like that, usability testing is also like that. When we are in the implementation of user interviews, users tend to be affected by environment or human factors, who cannot fully express their true feelings, which will affect the accuracy of the information we get. In this case, what should we do?


Capturing nonverbal information is important in life, as well as usability testing. In the usability testing workshop site of 2017 Chinese CHI forum, the insight lab founder and chief research officer, Dr. Li Hongting, introduced the research means to deal with this problem. He led the students to explore a more objective and professional test methods - Performance Measurement.



Dr. Li Hongting, the founder and chief research officer of insight lab, are teaching students.

Different form the concept of "performance evaluation" or "performance assessment" as people usually understand, in usability testing, the "performance measurement" we mentioned is to realize the data acquisition and analysis of the data by taking experiment design method based on certain index system to know clearly user's true feelings. Comparing with the traditional testing methods, this method can further determine the level of availability, understand thoroughly the factors of the impact of operational efficiency, and has higher objectivity and effectiveness, which is suitable for availability research of users’ subjective and objective information collection in mobile terminal, PC terminal and so on.


Nowadays, VR technology is also gradually applied to the usability test. In the workshop of “Usability Test”, Dr. Li Hongting not only introduced the traditional usability test to the participants, but also illustrated the specific application of performance test further combined with the testing research about Human Machine Interface (hereinafter referred to as HMI) conducted by insight lab in 2016. In order to record users` real feelings and feedback about control interfaces in different cars during the process of driving, insight lab develops a test index system based on the characters of testing objectives, and records users` physiological and behavioral performance during their implementation of tasks by the method of user interview and on the combination of eye tracker, driving recorder, multi-conductive physiograph and other third-party measurement tools, so as to help us get more accurate evaluation.

1503903855100367.pnginsight lab cars` HMI evaluation system


In the testing process, the user wears eye tracker, multi-conductive physiograph and other equipment to record their behavior and physiological data

For example, we want to know about the efficiency of air conditioning adjustment under different HMI during user`s driving. And the user would wear head-mounted eye tracker to capture the eye movement data during the operation, then we will make a further analysis about it. 


The real-time pictures of eye tracker in the operation by users

From the eye-tracking chart of Tesla and Mercedes-Benz HMI, we know that Tesla (left) needs 57 fixation points to complete the task, while the Mercedes-Benz (right) only need 24 points, which clearly reflects that the latter is better than the former. At the same time, the distribution of Tesla`s fixation points are relatively mess, while the Benz a line-styled arrangement, indicating that the operation path of Benz is more clear and definite.  Combined with the above two points, we can conclude that Tesla needs more operating energy in air conditioning adjustment, and the efficiency is relatively low. Similarly, we can get a lot of objective data in conjunction with other instruments and equipment, so as to provide more effective information for the final results.


Eye movement data can more accurately reflect the differences between different car HMI

However, what surprised us is that only a few of people with the user experience or college students are aware of or have accessed to performance test, let alone practical operation. Throughout the current enterprises` user research, most of them take user interviews as the method for information collection, although it is intuitive and fast, problems relating to reliability and validity also exist, which may bury certain hidden dangers in the long-term usability test.


User interviews are effective, but there is a certain degree of subjective preference risks

Through the data processing and analysis of the collected objective indicators, we can obtain more objective data support on the basis of user interviews, so as to obtain the user's experience and evaluation of the product in a more comprehensive dimension. Combined subjective and objective information, insight lab can gain an insight into the deeper needs of users, improve the objectivity and utility of the evaluation system, and provide manufacturers with more proper advice and value.

1503904715268518.pngWorkshop site of “Usability Test”

As our slogan--"knowing the cutting-edge, observing the experience" goes, insight lab will continue to promote the research method of performance test in future user experience test and research, and provide a more reliable and accurate evaluation system for user experience industry. In insight lab, we think more.