Original Article

Usability Evaluation of Two National COVID-19 Registration Systems: A Heuristic Evaluation

Please cite this article as follows: Saeedi S, Amirazodi S, Rezayi S, Khodaveisi T. Usability evaluation of two national COVID-19 registration systems: a heuristic evaluation. Avicenna J Care Health Oper Room. 2023; 1(2):38-43. doi:10.34172/ajchor.32

Introduction

China reported cases of severe pneumonia with no known cause in December 2019, and the World Health Organization labeled the illness as a global pandemic on March 11, 2020. (1). This illness, also referred to as severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2), is caused by the same family of viruses as Middle East respiratory syndrome and SARS (2,3). The coronavirus quickly spread to other countries and became a global problem. The clinical symptoms of COVID-19 usually begin less than a week after the coronavirus enters the body and can include shortness of breath, fever, muscle aches, nasal congestion, and other symptoms. The infection can lead to severe illness with shortness of breath and severe chest symptoms related to pneumonia in approximately 75% of patients (4).

Data related to COVID-19 are increasing day by day. Valuable knowledge can be extracted by managing data on COVID-19 registry. Developing a registry can lead to data collection on patients with COVID-19 in a standard, integrated, and unified structure (5). The disease’s progression can be investigated by collecting data in the registry, and the quality of care provided can be examined as well (6,7). On the other hand, drugs’ effectiveness can be evaluated with the help of data recorded in the registry (8). In addition, the data analysis results recorded in the registry can be provided to policymakers for planning. The COVID-19 registry can provide an infrastructure that can be used to investigate the pattern of disease transmission, mortality rate, clinical signs, disease prevalence in different parts of the country, the effectiveness of treatment programs, and thus monitoring and research (9-11).

“Medical Care Monitoring Center” (MCMC) and “Electronic System for Syndromic Surveillance of Infectious Diseases” (ISSS) are two systems that are used to record daily data related to COVID-19 patients in Iran. Both systems developed by the Ministry of Health are utilized as national systems throughout hospitals in Iran and have many users.

One of the problems that leads to the non-acceptance of these information systems is the problem related to user interface design and usability. These problems should be considered at the time of design of the user interface, before the implementation of these systems, and before the everyday use of information systems by users. The existence of usability problems can negatively affect the use of these systems and lead to abandonment and non-use by healthcare providers and, ultimately, the failure of an information system (12). Therefore, these two national systems’ usability should be evaluated to identify problems related to the design of their graphical interfaces in order to prevent further consequences.

One of the most common evaluation methods to identify usability problems is heuristic evaluation (HE). With this method’s help, a large number of system problems can be identified at a low cost, and many resources are not required for evaluation (13). One of the most widely used HE tools is Nielsen’s ten principles. In this method, the evaluators examine the systems independently and consider discrepancies between the items observed in the system design and these principles as a problem (14,15). In a study, Rangraz Jeddi et al evaluated a national system using the heuristic method. The study results showed that 61 unique problems were identified, of which 60.6% were categorized into major and catastrophic groups (16). This study confirmed that even the user interface design of nationally used systems has problems and can lead to non-acceptance by users. To the best of our knowledge, these two systems’ usability has not been studied so far. Therefore, this study investigated the two registries’ usability related to COVID-19 in Iran using the heuristic method.

Materials and Methods

Study Setting and Systems

This evaluation was performed with two national registry systems, namely, MCMC and ISSS. MCMC was created to arrange medical services monitoring through improving the quality of services, improving processes, handling effective communication, accelerating services, and integrating management. The fundamental purpose of establishing the ISSS system was to report community-based epidemics and early detection. In addition, these two surveillance systems are applied by healthcare providers to record COVID-19 patients’ information; however, it is worth noting that they were not designed only to monitor and record COVID-19. Because these two systems are currently used in the COVID-19 period, the primary purpose of this study was to evaluate them. HE is an expert-based inspection, cost-effective, and straightforward method that identifies “Minor” and “Major” violations in a system user interface. The usability evaluators independently appraised the MCMC and ISSS systems’ usability. This study identified usability problems related to the COVID-19 data registry. These results can be used by the designers and custodians of this system, the Ministry of Health, to improve these user interfaces and increase users’ satisfaction.

Characteristics of Evaluators

One health information technology (HIT) specialists and two medical informaticians assessed the user interfaces of two registry systems. The first and second evaluators were Ph.D. students in medical informatics who were familiar with different registry systems and various evaluation methods. They had more than two years of experience

working with “MCMC” and “ISSS”. The third evaluator was an MSc in HIT. She had experience working with two registration systems and was familiar with HE methods.

Heuristics Evaluation

First, three evaluators explored the user interface structures of the MCMC and ISSS systems. Next, they independently evaluated two registration systems’ user interfaces one by one against 10 Nielsen principles (Table 1) and entered the violations of the systems in the data collection checklist; each assessment lasted approximately 5 hours.

Data Analysis and Comparisons

Three checklists that were completed by evaluators for the MCMC system were reviewed in this phase. After duplicate problem removal, a single list of problems was created for this registration system. Therefore, evaluators completed three checklists for the ISSS system, and a single list of violations remained after duplication removal. Furthermore, negotiations were used to settle any disputes that arose between the evaluators. Finally, the severity of the problems was distinguished based on three main criteria, including frequency, impact, and persistence of the problem. Finally, the average severity of each system’s violations (MCMC and ISSS) was computed and stated based on Table 2. The main phases of evaluation are shown in Figure 1.

Close

Figure 1.

Main Steps of the HE Method. Note. HE: Heuristic evaluation

Results

In this study, evaluation was performed based on the HE method for “MCMC” and “ISSS” systems, and a total of 164 usability problems (for two evaluated systems) were identified after duplication removal. In the analysis step, three evaluators recognized 59 usability problems of the ISSS registration system. Further, they identified 105 unique violations of the MCMC registration system. In the ISSS system, the highest frequencies of violations were related to principles 4 (i.e., “Consistency and standard”) and 7 (i.e., “Flexibility and efficiency of use”). In the MCMC system, the highest frequencies of problems belonged to principles 10 (i.e., “Help and documentation”) and 4 (i.e., “Consistency and standards”). Based on ISSS system evaluation results, the highest average severities of violations were determined in the principles “Visibility of system status” and “User control and freedom”, with scores of 3.6 and 2.87, respectively. In the MCMC system evaluation, the highest average severities of usability-oriented problems were observed in principles “Recognition rather than recall” and “User control and freedom”, with scores of 3.28 and 2.88. Table 3 presents the frequencies of identified usability violations based on their severity and the heuristic problems for MCMC and ISSS systems. Of the total determined violations in the ISSS system, 24 and 21 cases were major and minor problems, respectively. Furthermore, in the MCMC system, 50 cases were minor and 36 of the total problems were major. The detailed severity range results for both systems are shown in Figure 2. Figure 3 illustrates several examples of violations, as well as the details of each problem and the number of each problem.

Close

Figure 2.

Heuristic Problem Frequency for ISSS (a) and MCMC (b) Sorted by Severity. Note. ISSS: Electronic system for syndromic surveillance of infectious diseases; MCMC: Medical care monitoring center

Close

Figure 3.

Six Examples of Usability Problems in the ISSS System. Note. ISSS: Electronic system for syndromic surveillance of infectious diseases; MCMC: Medical care monitoring center. P1: The use of colors with high chrome. P2: The title of the window is not distinct due to the small font size. P3: Inconsistency of menu options in language. P4: Numbers must be left-aligned. P5: Fields are not restricted in number of characters. P6: Illegibility and inappropriate arrangement of field labels

Discussion

In this study, MCMC and ISSS systems were evaluated using the heuristic method. The findings demonstrated that the MCMC system had a greater number of usability problems than the ISSS. Overall, 105 and 59 usability problems were identified in the HE of the MCMC and ISSS systems, respectively. In the MCMC system, 49% of the identified problems were major or catastrophic. In the ISSS system, 54% of the identified problems were of major and catastrophic types.

The HE method was employed in this study to determine the usability problems with the two systems pertaining to COVID-19 patient data recording. A study comparing heuristic and user testing evaluation methods concluded that more problems could be identified with the HE method. Moreover, the heuristic method could identify problems more quickly, and therefore it was more efficient and effective than the user testing method (17).

Nearly half of the issues found in the MCMC system were of major and catastrophic types. The assessment of the ISSS system also showed that over half of its usability issues were of the major, catastrophic variety. Three factors are taken into consideration when determining the severity of a problem, including its frequency, impact, and persistence. Solving major problems is somewhat important, and solving catastrophic problems is also highly necessary and has an extremely high priority. The designers of these two systems (MCMC and ISSS) should note that they are used nationally and have many users in hospitals throughout Iran. The existence of these problems can interfere with the correct and accurate registration of information.

With nine problems, the principles of “Consistency and standards” and “Flexibility and efficiency of use” had the highest number of problems found in the ISSS system. The “Consistency and standards” principle was linked to 19.04% of the problems found in the MCMC system as well. This study’s results are consistent with those of studies of Sadeqi Jabali et al and Nabovati et al (18,19). The two studies evaluated “Admission and Medical Record Information Systems” and “Laboratory and Radiology Information Systems”. In the study of Atashi et al, the highest number of problems was related to this principle, and out of 99 unique problems identified, 25% were related to this principle (20). These studies demonstrated that information system designers have not paid enough attention to this principle in user interface design. Therefore, it is recommended to notice user interface design standards when designing information systems.

With 23 problems, the “Help and documentation” principle had the most problems in the MCMC system. The system did not provide any online help to users, and they might perform a function incorrectly or stop performing a function. In the system evaluated by Nabovati et al, this principle was wholly ignored and led to users’ problems performing tasks (19). In another study, problems concerning the “Help and documentation” principle were rated as catastrophic (21). The existence of the before-mentioned problems in the user interface of various information systems, especially those whose users are not very familiar with such systems, can lead to their confusion and, as a result, lead to their dissatisfaction.

The highest average severity of violations was related to the ISSS system and the “Visibility of system status” principle with 3.6 and was graded as a catastrophic problem. In the study of Farzandipour et al, usability problems associated with the principle of “Visibility of system status” were identified as the most severe types (22). To solve these problems, information systems must be developed to inform the user about what is happening through appropriate feedback and at the right time.

This study identified usability problems related to the COVID-19 data registry. These results can be utilized by the designers and custodians of this system, the Ministry of Health, to enhance user interfaces and increase users’ satisfaction.

Strengths and Limitations

This study had some strengths and limitations. The Nielsen questionnaire has some benefits. We can mention several principles that can be used to assess many aspects of a health information system. Compared to other unstructured data collection techniques such as interview and observation, a structured questionnaire, including the Nielsen questionnaire, can facilitate the assessing process, and evaluators can identify more violations during evaluations of information systems. This study was limited by the fact that access to the national system required cooperation with the Ministry of Health, which established it.

Conclusion

The findings of the evaluation of MCMC and ISSS systems revealed that nearly half of the identified issues belonged to the category of major and catastrophic problems. These usability problems can significantly impact the acceptance and continued use of these systems by end-users. Many identified problems can be solved easily. Considering that these two systems are used nationally, it is essential to pay attention to solving the problems identified by these systems’ designers and custodians. On the other hand, evaluating the systems’ user interface in the initial stage and before widespread use is necessary.

Authors’ Contribution

Conceptualization: Soheila Saeedi, Sorayya Rezayi.

Data curation: Soheila Saeedi, Sorayya Rezayi, Taleb Khodaveisi, Shahrzad Amirazodi.

Formal analysis: Sorayya Rezayi.

Funding acquisition: Taleb Khodaveisi.

Investigation: Soheila Saeedi, Sorayya Rezayi, Taleb Khodaveisi, Shahrzad Amirazodi.

Methodology: Sorayya Rezayi, Taleb Khodaveisi.

Project administration: Soheila Saeedi, Sorayya Rezayi.

Supervision: Soheila Saeedi, Sorayya Rezayi.

Writing–original draft: Soheila Saeedi, Sorayya Rezayi, Taleb Khodaveisi, Shahrzad Amirazodi.

Writing–review & editing: Soheila Saeedi, Sorayya Rezayi, Taleb Khodaveisi, Shahrzad Amirazodi.

Competing Interests

None.

Consent for the Publication

Not Applicable.

Ethical Approval

The methodology for this study was approved by the Ethics Committee of Hamadan University of Medical Sciences (Ethics approval number: IR.UMSHA.REC.1402.536).

Funding

In this paper, we received a grant from Hamadan University of Medical Sciences.

References

1. Xiong J, Lipsitz O, Nasri F, Lui LMW, Gill H, Phan L, et al. Impact of COVID-19 pandemic on mental health in the general population: a systematic review. J Affect Disord 2020; 277:55-64. doi: 10.1016/j.jad.2020.08.001 [Crossref] [ Google Scholar]
2. Cucinotta D, Vanelli M. WHO declares COVID-19 a pandemic. Acta Biomed 2020; 91(1):157-60. doi: 10.23750/abm.v91i1.9397 [Crossref] [ Google Scholar]
3. Zhao D, Yao F, Wang L, Zheng L, Gao Y, Ye J, et al. A comparative study on the clinical features of coronavirus 2019 (COVID-19) pneumonia with other pneumonias. Clin Infect Dis 2020; 71(15):756-61. doi: 10.1093/cid/ciaa247 [Crossref] [ Google Scholar]
4. Velavan TP, Meyer CG. The COVID-19 epidemic. Trop Med Int Health 2020; 25(3):278-80. doi: 10.1111/tmi.13383 [Crossref] [ Google Scholar]
5. Freeman EE, McMahon DE, Fitzgerald ME, Fox LP, Rosenbach M, Takeshita J, et al. The American Academy of Dermatology COVID-19 registry: crowdsourcing dermatology in the age of COVID-19. J Am Acad Dermatol 2020; 83(2):509-10. doi: 10.1016/j.jaad.2020.04.045 [Crossref] [ Google Scholar]
6. Peterson ED, Shah BR, Parsons L, Pollack CV Jr, French WJ, Canto JG, et al. Trends in quality of care for patients with acute myocardial infarction in the National Registry of Myocardial Infarction from 1990 to 2006. Am Heart J 2008; 156(6):1045-55. doi: 10.1016/j.ahj.2008.07.028 [Crossref] [ Google Scholar]
7. Mulder DS, Spicer J. Registry-based medical research: data dredging or value building to quality of care?. Ann Thorac Surg 2019; 108(1):274-82. doi: 10.1016/j.athoracsur.2018.12.060 [Crossref] [ Google Scholar]
8. Trifirò G, Gini R, Barone-Adesi F, Beghi E, Cantarutti A, Capuano A, et al. The role of European healthcare databases for post-marketing drug effectiveness, safety and value evaluation: where does Italy stand?. Drug Saf 2019; 42(3):347-63. doi: 10.1007/s40264-018-0732-5 [Crossref] [ Google Scholar]
9. Kazemi-Arpanahi H, Moulaei K, Shanbehzadeh M. Design and development of a web-based registry for coronavirus (COVID-19) disease. Med J Islam Repub Iran 2020; 34:68. doi: 10.34171/mjiri.34.68 [Crossref] [ Google Scholar]
10. Alger HM, Williams JH 4th, Walchok JG, Bolles M, Fonarow GC, Rutan C. Role of data registries in the time of COVID-19. Circ Cardiovasc Qual Outcomes 2020; 13(5):e006766. doi: 10.1161/circoutcomes.120.006766 [Crossref] [ Google Scholar]
11. Mohammadzadeh Z, Keikha L, Maserat E. Toward controlling coronavirus disease 2019 (COVID-19) pandemic by integrated registry systems. Health Scope 2020; 9(3):e104753. doi: 10.5812/jhealthscope.104753 [Crossref] [ Google Scholar]
12. Khajouei R, Hajesmaeel Gohari S, Mirzaee M. Comparison of two heuristic evaluation methods for evaluating the usability of health information systems. J Biomed Inform 2018; 80:37-42. doi: 10.1016/j.jbi.2018.02.016 [Crossref] [ Google Scholar]
13. Ebnehoseini Z, Tara M, Meraji M, Deldar K, Khoshronezhad F, Khoshronezhad S. Usability evaluation of an admission, discharge, and transfer information system: a heuristic evaluation. Open Access Maced J Med Sci 2018; 6(11):1941-5. doi: 10.3889/oamjms.2018.392 [Crossref] [ Google Scholar]
14. Saeedbakhsh S, Saghaeiannejad Isfahani S, Saber M, Yadegarfar G. Hospital information system usability of educational hospitals in Isfahan using heuristic evaluation method. Int J Educ Psychol Res 2017; 3(1):69-73. doi: 10.4103/2395-2296.180308 [Crossref] [ Google Scholar]
15. Mazumder FK, Das UK. Usability guidelines for usable user interface. Int J Res Eng Technol 2014; 3(9):79-82. [ Google Scholar]
16. Rangraz Jeddi F, Nabovati E, Bigham R, Farrahi SR. Usability evaluation of a comprehensive national health information system: a heuristic evaluation. Inform Med Unlocked 2020; 19:100332. doi: 10.1016/j.imu.2020.100332 [Crossref] [ Google Scholar]
17. Maguire M, Isherwood P. A comparison of user testing and heuristic evaluation methods for identifying website usability problems. In: International Conference of Design, User Experience, and Usability. Cham: Springer; 2018.
18. Sadeqi Jabali M, Nabovati E, Farzandipour M, Farrahi SR, Abbasi R. Evaluation of the usability of admission and medical record information system: a heuristic evaluation. J Health Biomed Inform 2019;6(2):80-9. [Persian].
19. Nabovati E, Vakili-Arki H, Eslami S, Khajouei R. Usability evaluation of laboratory and radiology information systems integrated into a hospital information system. J Med Syst 2014; 38(4):35. doi: 10.1007/s10916-014-0035-z [Crossref] [ Google Scholar]
20. Atashi A, Khajouei R, Azizi A, Dadashi A. User interface problems of a nationwide inpatient information system: a heuristic evaluation. Appl Clin Inform 2016; 7(1):89-100. doi: 10.4338/aci-2015-07-ra-0086 [Crossref] [ Google Scholar]
21. Joshi A, Arora M, Dai L, Price K, Vizer L, Sears A. Usability of a patient education and motivation tool using heuristic evaluation. J Med Internet Res 2009; 11(4):e47. doi: 10.2196/jmir.1244 [Crossref] [ Google Scholar]
22. Farzandipour M, Nabovati E, Zaeimi GH, Khajouei R. Usability evaluation of three admission and medical records subsystems integrated into nationwide hospital information systems: heuristic evaluation. Acta Inform Med 2018; 26(2):133-8. doi: 10.5455/aim.2018.26.133-138 [Crossref] [ Google Scholar]