Construction And Evaluation Of A Virtual Learning Environment For Teaching Patient Safety In Drug Therapy

Abstract

Objective: To construct and evaluate a virtual learning environment for teaching patient safety in drug therapy for undergraduate nursing students.

Method: Methodological study developed in 3 steps: integrative literature review (step 1), construction of the virtual environment (step 2) and pedagogical and usability evaluation (step 3). The construction was systematized, according to the Contextualized Instructional Design in the phases: analysis, design, development, implementation and evaluation, and subsidized by the survey of learning needs about patient safety in drug therapy with 45 nursing students from the 7th and 9^th periods of the course.

Results: The pedagogical evaluation obtained a concordance rate of 91.7% among the nurse judges and the usability of 90% among the informatics judges. The constructed virtual environment was called AVA-SPTEME, and is available on the Internet hosted on the Moodle platform, where there are two teaching units: “Patient safety” and “Safety in drug therapy”, both designed with different materials and media: videos, comic books, texts, info graphics, word searches, quizzes, forums and chats.

Conclusion: The virtual environment was validated and positively evaluated by the judges, who considered it a formidable educational technology for teaching patient safety in drug therapy for nursing undergraduate students.

Keywords

Educational technology, Patient safety, Drug therapy, Education, Nursing.

Introduction

Medication errors are reported as one of those responsible for the potential risk to patient safety and are defined as a preventable event that may or may not cause harms to the patient while the medication is under the control of the health professional, patient or consumer [1].

It is estimated that medication errors cause more than 7,000 deaths per year in the USA, entailing important costs to the health system [2]. In Brazil, exclusive statistics on deaths related to medication errors are not yet available. Nevertheless, through the National Notification System for Health Surveillance (NOTIVISA, as per its Portuguese acronym), in theperiod betweenMarch 2014 and July 2017, a total of 3,766 notifications related to incidents involving medications were registered [3], which makes this issue a concern for public health and a challenge for the involved health professionals.

Drug therapy involves a variety of health professionals, including undergraduate students. Although, in general, they are not allowed to prepare and administer medications without teacher supervision before are they graduated, they can start to practice and prepare themselves for many aspects of the safe use of medications [4]. From this perspective, the academic audience of courses in the area of health, especially those in nursing, can receive, from the introduction into the curriculum, a teaching unit directed to the issue of patient safety and, consequently, to the achievement of the goal “risk-free medication”.

Studies related to Nursing Education underline the need for constant teacher improvement in relation to the topic, as well as the use of resources and teaching technologies in teaching practice [5,6]. Regarding this issue, the Brazilian Ministry of Education, through the Law of Guidelines and Bases of National Education (LDB, as per its Portuguese acronym) and Resolution CNE/CES nº 03/2001, which institutes National Curriculum Guidelines (DCNs, as per its Portuguese acronym) for nursing courses, recommend the use of active methodology with the adoption of Information and Communication Technologies (ICTs) as a strategy to develop skills and abilities in nursing education [7].

In this case, the Virtual Learning Environment (VLE or AVA, as per its Portuguese acronym), as it is a computer system available on the internet, which integrates multiple media, languages and resources and is intended to support activities mediated by ICTs, has been increasingly frequent in nursing, being considered a formidable pedagogical strategy for teaching undergraduate and graduate nursing students and continuing education of nurses in clinical practice [8-12].

Accordingly, the current study becomes relevant, since it seeks to contemplate patient safety throughout the medication process, combined with the fact that technologies and their immersion in nursing education are shown in a context of intense transformation, which requires teachers to develop different skills to teach and learn in this digital century. The proposal had the objective of constructing and evaluating a virtual learning environment for teaching patient safety in drug therapy for undergraduate nursing students.

Materials and Methods

This is a methodological study developed in three stages: literature review, construction of the virtual environment and pedagogical/usability evaluation by expert judges, conducted in the Federal Institution of Higher Education of Piauí, Brazil, from March 2019 to November 2020, approved by Research Ethics Committee, Opinion nº 3.739.503, complying with the recommendations of Resolution 466/2012 [13].

The integrative review [step 1] was guided by the research question: what are the educational technologies used in teaching patient safety in drug therapy for undergraduate nursing students?

The following descriptors were considered: Nursing students, Nursing education, Patient safety, Drug therapy, Educational technologies and Distance education, which were selected by consulting the Descriptors em Ciências da Saúde (DeCS) and equivalents, Medical Subject Headings (MeSH) and List of Headings from CINAHL Information Systems, combined with the Boolean connector “OR”, within each component of terms of the PICO strategy, and crossed with the Boolean “AND”. The literature search took place between December 2019 and January 2020, in the following databases: Cumulative Index to Nursing & Allied Health Literature (CINAHL), Latin American & Caribbean Health Sciences Literature (LILACS), via the Virtual Health Library (VHL), and Medical Literature Analysis and Retrieval System Online (MEDLINE), via PubMed, and Web of Science.

A total of 620 publications were identified, of which 19 were included in the final review sample, according to the criteria below: addressing teaching strategies for patient safety in drug therapy (PSDT) for undergraduate nursing students, time cutting between 2009 and 2019, without language limits, excluding educational technologies in aspects other than drug therapy, as well as those directed to audiences other than undergraduate nursing students, review studies and editorials. The selection of publications complied with the recommendations of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).

The construction of the AVA (step 2) was systematized according to the Contextualized Instructional Design [14] in the phases: analysis, design, development, implementation and evaluation. In the analysis phase, the research topic was defined by identifying the learning needs of the target audience regarding the safe use of medications, the characterization of this audience, the use of the Internet, and digital fluency, as well as the survey of institutional strengths and weaknesses, through the application of a questionnaire that covered 45 undergraduate nursing students.

The design consisted of the planning ordered by the creative team: the study supervisor and coordinator of the Research Group on Patient Safety and Nursing Care Systematization; a master’s and a doctoral student, who are members of the group; a graphic designer; a teacher and a student of the Information Systems course from the institution that hosted the study.

In the development phase, the production and adaptation of the digital media, the assembly and configuration of the environment, as well as the scripting of the educational materials for different formats were carried out, namely: texts, hypertexts, links, images, info graphics, cartoons, mascots, videos, comic books, word searches, quizzes, chats and forums, all of which supported by various computer programs and applications (Tagxedo, WikiHow, CorelDRAW X8, Adobe Acrobat Reader DC, Joiner, Video Scribe, Autodesk, Sketch Book, Shotcut, Geniol website, Microsoft Word, version 2013), among others.

Two info graphics were constructed. The first portrayed the milestones of patient safety in a timeline and the second portrayed the complexity of health care, where a mascot called Enfer TEME was used in the presentation of the interfaces of the virtual environment. Four videos are available on the AVA-SPTEME, two of which were produced in light of the Safety Protocol on Prescription, Use and Administration of Medicines by the National Health Surveillance Agency of the Brazilian Ministry of Health and two provided by the World Health Organization and by the Brazilian Institute for Patient Safety (IBSP-Brazil).

Six comic books were produced, whose scripts were created based on the Patient Safety Curriculum Guide: a multidisciplinary edition of the World Health Organization, addressing topics such as the importance of teamwork and effective communication, some types of prescription (urgency, verbal/by phone, if necessary) and the error monitoring.

Word search and quiz were the strategies of choice to train perception skills and evaluate the knowledge acquired at the end of each teaching unit. The level of difficulty of word search was rated as hard. In turn, quiz was prepared with questions in different formats (multiple choice, association and filling out) and levels of difficulty (easy, moderate and hard), with an estimated time to complete it, between 10 to 15 minutes, with the possibility of up to three answer attempts, where the highest score was the considered evaluation method.

Moodle was the platform chosen for the development of the course because it is free software, easy to use, flexible and low cost, which facilitates student interaction for the construction of knowledge, thus enabling collaborative learning. The evaluation of the AVA (step 3) was held by judges, selected through the snowball sampling technique. Obtained by means of convenience, the sample was made up of 12 nurses specializing in patient safety and 9 information technology professionals, who should achieve a minimum score of 5 points in the criteria adapted from the expert classification system of the Fehring validation model [15].

An e-mail was sent to the judges with an invitation letter to participate in the evaluation process and two links, one containing the AVA evaluation electronic forms, the socio demographic questionnaire, the Free and Informed Consent Form, while the other provided access to the AVA-SPTEME platform. A period of up to 20 days for analysis and return was established.

The socio demographic and professional characterization of the judges was carried out through an electronic questionnaire with the variables: gender, age, title, area of expertise, length of experience, research and publication of an article in the areas of patient safety and information technology. The Learning Object Review Instrument (LORI) [16] was the instrument used for the pedagogical evaluation of the AVA, with the following parameters: content quality, learning objectives alignment, feedback and adaptation, motivation, presentation design, interactive use, accessibility and compliance with the standards. This instrument was translated and adapted to the Portuguese language [17] being used nationally by several studies.The judgment variations were scored from 1 to 5, where each item was given a numerical assignment, where 1 corresponds to low suitability and 5 to high suitability.

In order to evaluate usability, an adapted instrument was used [18] containing 10 heuristics: 1- Visibility of system status; 2- Compatibility between the system and the real world; 3- User control and freedom; 4- Consistency and standards; 5- Error prevention; 6- Recognizing rather than remembering; 7- Flexibility and efficiency of use; 8- Minimalist and aesthetic design; 9- Recognition, diagnosis and recovery of errors; and 10- Help and documentation. The degree of severity was interpreted on a scale from 0 to 4 points, where zero = Non-important error, when it does not affect the operation of the interface, not being seen as a usability problem, and 4 = Catastrophic error, when the evaluator does not manage to complete the task because of the problem and must be corrected immediately.

The data obtained were tabulated and organized in the Microsoft Excel 2016 program, and then transferred to the Statistical Package for the Social Sciences (SPSS) software, version 22.0, performing a descriptive analysis of the judges’ characteristics in both areas (nursing and informatics), as well as of pedagogical validation and usability in face of the virtual environment constructed with measures of variance (average) and dispersion (standard deviation).

The concordance rate (CR) was the method used to calculate the percentage among the judges who are in agreement on certain aspects of the instrument; while, regarding the validation of the items, a rate equal to or greater than 0.78 was considered [19].

Results

Concerning the integrative review – step 1 of the study, 19 articles were selected, most (32.0%) identified in the MEDLINE/ PubMed database; written in English (68%); developed predominantly in countries such as Brazil, the United States, Australia and surrounding countries, which showed a considerable increase in the production of educational technologies, especially between the years 2014 and 2019 (83%). In Brazil, no studies were found on the production of educational technologies covering all stages of drug therapy, namely: prescription, dispensing, administration and monitoring.

All studies were conducted with undergraduate nursing students, addressing issues related to safety in drug therapy. Virtual learning environment stood out in 69.0% of the studies as the main developed educational technology, video was as one of the most used media and drug administration was the most discussed topic. There were gaps in knowledge regarding drug therapy in the items related to dispensing and monitoring after drug administration.

Before the construction of the AVA (step 2), the needs for learning about patient safety in drug therapy were surveyed with the target audience, their socioeconomic characteristics, internet use and digital fluency, as well as institutional strengths and weaknesses.

A total of 45 (100%) undergraduate nursing students participated, 33 (73.3%) were females and the average age was 23 (±3.04) years, most single 42 (93.3%), living with the parents 19 (42.2%) or with other people without blood and/or marital ties 14 (31.1%). Moreover, 27 (60%) of the participants called themselves Catholic and 25 (55.6%) reported a family income between 1 and 2 minimum wages.

As for internet use and digital fluency, all confirmed having internet access 45 (100%) with daily use frequency in 42 (93.3%), with smartphones being the most used device for internet access, 42 (77.8%), where home was the predominant access point 36 (80.0%). In terms of institutional strengths and weaknesses, 45 (100%) of the participants affirmed the existence of 2 computing centers, with a standard deviation of 0.77%. A total of 29 (70.7%) informed the availability of 10 to 30 computers, with access confirmed by 23 (51.1%) of the participants and Wi-Fi coverage by 37 (82.2%). With respect to the learning needs about patient safety in drug therapy, both themes (patient safety and medications) were classified as “Essential” among the alternatives, as displayed in Tables 1-3.

Table 1: Characterization of the Learning Needs Identification questionnaire, answered by Undergraduate Nursing Students (N=45) from a Federal Public University in the State of Piauí. Picos (PI), Brazil, 2019-2020.

Table 2: Concordance of Nurse Judges (N=12) for the Pedagogical Validation of the Virtual Learning Environment. Picos (PI), Brazil, 2020.

Table 3: Concordance of Informatics Judges (N=9) for the Usability Validation of the Virtual Learning Environment. Picos (PI), Brazil, 2020.

The score of the judges who evaluated the AVA-SPTEME (step 3 of the study) ranged from 6 to 17 points for the nurse judges and from 7 to 15 points for the informatics professionals.

In this study, 12 nurses specializing in patient safety evaluated the AVA. Most females 9 (75.0%), with an average age of 39 years and standard deviation of 7.1, with a predominance of doctoral degrees, 8 (66.7%), and working in the area of teaching, 10 (83.3%). Professionals have an average of 13.5 years of experience, with a standard deviation of 5.18. Of the total, 5 (41.7%) had their thesis or dissertation themes in the area of patient safety. All said they were researchers in the area of patient safety 12 (100%) and with articles published in this area 9 (75%).

With the exception of the variable related to Accessibility, all the others obtained a concordance rate above 80%, with emphasis on variables 1 “Content quality”, 2 “Learning objectives alignment”, 3 “Feedback and adaptation”, 4 “Motivation” and 8 “Compliance with standards”, which obtained a concordance rate of 100%. The total average of the CR of the variables was 91.7%, with a standard deviation of 14.8.

As for usability, the AVA was evaluated by 9 informatics judges (100%). Most were males, 6 (66.7%), with an average age of 41.3 years. Of these, 7 (77.8%) had a master’s degree, 1 (11.1%) had a doctoral degree and 1 (11.1%) had a specialization. All (100%) developed their graduate works in the area of information technology, more precisely in the segments: computer science, systems and computing. As for the area of expertise, 7 (77.8%) are focused on teaching and 2 (22.2%) on research. The length of experience in the area was 12.4 years, with a standard deviation of 6 years. Of the total, 7 (77.8%) do research in the areas of hypermedia, educational technologies and EaD (Distance education, as per its Portuguese acronym) and 3 (33.3%) published articles in these areas. The table below presents the results of agreement among judges belonging to the area of informatics.

Discussion

COVID-19 Concerning the construction of virtual learning environments, a study points out that, with the advent of computational and technological resources, new challenges and broad demands in all educational spheres have been observed, culminating in the emergence of new theories, methodologies and learning modalities [20] including in the area of patient safety, whose purpose is to improve safety in health care and, consequently, in the medication process.

From this perspective, different strategies and teaching methods were noted, among which, it should be highlighted: Med safe – prototype of a virtual game about preparation and administration of medications (7); Effects of a patient safety course using a flipped classroom approach among undergraduate nursing students [21] use of digital applications in the medicament calculation education for nursing [22] AntibioGame^®: A serious game for teaching medical students about antibiotic use [23] and The use of digital platforms for teaching medication calculation [24].

A review study reveals that, in Brazil, the findings in the literature on technologies for teaching patient safety in drug therapy refer to isolated steps of the process [25]. Therefore, it is important that this theme is addressed in its entirety, given that an error occurred in the prescription (1^st step of the medication) may permeate the entire process until the final destination: the patient. On the other hand, it highlights educational technologies, especially the virtual learning environment. In view of the gap and the contributions that the AVA can offer, the educational technology at stake was developed.

Hypermedia is one of the methods applied in the AVA, understood as the combination of several audio visual tools and resources that seek to provide the user with complete and interactive content, thus making learning easy and pleasurable [26]. Comic books, for example, aggregate, at the same time, visual expression, one of the main means of human communication in all times since prehistory, and verbal writing, being considered a mass communication strategy that attracts millions of readers around the world [27].

A study on the use of digital comic books, as a methodological proposal in the teaching and learning processes of mathematics revealed that it is an important method for teaching and that the experience derived from the constructions of comic books (HQs, as per its Portuguese acronym) by students was well accepted and promoted better learning. Thus, it is believed that the comic books developed in the context of patient safety in drug therapy will produce promising effects for the learning of undergraduate nursing students.

Video is another technology widely used in the educational environment, a tool that facilitates learning, a media with the potential to attract and maintain attention, besides registering student impressions, their power to capture, store, transmit and portray sounds and images in motion, making use of the combination of sound and verbal matrices to enable multiple perceptions [28]. A study used video to check the pedagogical achievement of a health education application for deaf and hearing impaired students in elementary school [29] and concluded that it is an important technology, since it allows the use of several simultaneous and playful resources, in addition to favouring the construction of mental images or visual association, thus enabling learning, memorization and construction of specific skills. Accordingly, it is believed that the videos available on the AVASPTEME can help them to develop the aforementioned skills.

In this study, learning evaluation is supported by the use of quiz and forum. In this process, it is important to consider the student as an integral being, taking into account his/her attitudes, interests and responsibilities, not just being limited to a test, but using several instruments and different elements, always providing the student with the development of autonomy and criticality [30].

As for chat, despite being available on the AVA-SPTEME, as well as on a room for videoconferences, these resources have not yet been explored, given that the implementation phase, where this experience would take place, was not developed in this study. Nevertheless, as it deals with applications for real-time (online) communication, the use of these tools in nursing has been substantially noted nowadays in view of the educational crisis, faced as a result of the pandemic caused by the new coronavirus (COVID-19).

With respect to the evaluation of the virtual environment (step 3), the nurse judges rated the AVA-SPTEME as a pedagogically valid and reliable technology, since their answers focus on the item with the greatest representation (fully suitable), among the instrument’s variables (LORI), where, at the end, a concordance rate equal to 91.7 was obtained, thus reaching values proposed in the pertinent literature. This result indicates the agreement, in most judges, regarding the evaluated items, as well as the quality of the content and the virtual environment. The items “Content quality”, “Feedback and adaptation” and “Interactive usability” reached the maximum concordance rate among the judges. Content quality is related to accuracy, balanced presentation of ideas, appropriate level of details and reusability in varied contexts; Feedback and adaptation, with adaptive content boosted by input or student modelling; in turn, Interactive usability refers to the ease of browsing, the predictability of the user interface and the quality of the interface’s help functions.

Similarly, a study on the pedagogical content evaluation showed agreement, expressed by the categories “strongly agree” by 92.3% of the experts [31]. Although the AVA-SPTEME made available different types of media, such as texts, comic books and videos, the item related to accessibility did not obtain a favourable score among the CR indexes. The non-validation of this item indicated that the design of controls and presentation formats for the material produced could not accommodate students with disabilities on the platform, and this was one of the questions raised by the judges in general.

As for usability, AVA-SPTEME obtained percentages above 80% in the evaluation of heuristics, with emphasis on heuristic 10 – Help and documentation, which obtained a CR of 100%. There was a concentration of judges in the category “G-0”, which corresponds to the “unimportant error”, meaning that the violations found can be adjusted without, however, changing the final essence of the constructed technology. Violations that generate usability problems were identified, which are considered as severe and catastrophic in heuristics.

In this regard, it is worth underlining the discrepancy among the evaluations, justified by the judges’ response time, given that this was not homogeneous, which may have contributed to successive changes in meeting a given recommendation by one judge to the detriment of the other, thus generating comings and goings in some aspects of the AVA.

Low agreement was also noted in the Medsafe study: prototype of a virtual game about preparation and administration of medications, where, of the eight usability evaluation items, three were approved by more than 80% of the participants, while the other items had a low level of agreement, thus reinforcing the need for system improvements to enhance the usability of the constructed game. This elucidates the dual importance of using evaluation methods and the crucial role played out by the evaluator in identifying problems and refining the final product.

Regarding the recommendations made by the judges, heuristic 2 stands out by referring to “compatibility between the system and the real world”, where, on the user registration screen, the acceptance policy was shown in English, and it was suggested to review the plugin for multi-language. Moreover, the option “emblems” appeared in the initial menu, which, despite not being enabled on the website, generated curiosity about its meaning in at least three of the judges. This was related to the modality of the platform chosen for the AVA and was promptly corrected. As limitations, the project was costly and there was no funding for the study, thus limiting the application of more resources to improve the technology, as well as the impossibility of carrying out the implementation phase with undergraduate nursing students, given the sudden changes that came with the covid-19 pandemic and the considerable sample loss of participants in this phase, interfering with the determined time.

Conclusion

The AVA-SPTEME followed a rigorous development process both in relation to the material and the evaluation by expert judges, thus ensuring a product with consistent and relevant information, through simple and accessible language, illustrated with clear, objective and attractive figures for the audience consisted of undergraduate nursing students.

Accordingly, it is believed that, in the short term, the use of this technology will add to teaching practice technological tools capable of improving the practice of care, while strengthening the knowledge and reducing the incidence of errors by nursing students in the medication process; and, in the long term, possible changes in the organization of the curricular matrix of the course by introducing patient safety as a subject to integrate the curriculum, which, consequently, will offer patients undergoing drug therapy a safe and quality care in line with technology and student protagonist.

It should be emphasized the need to implement the AVA-STEME with the target audience for the purpose of evaluating the effectiveness of this technology on the knowledge of undergraduate nursing students.

References

1. World Health Organization (WHO). Medication without Harm: Global Patient Safety Challenge on Medication Safety. Geneva: World Health Organization; 2017.
2. National Coordinating Council for Medication Error Reporting and Prevention (NCCMERP). What is a medication error? Nova York: National Coordinating Council for Medication Error Reporting and Prevention; 2015.
3. ANVISA: Agencia Nacional de Vigilância Sanitária [Internet]. Brasília: ANVISA; 2018. Os relatórios reúnem dados, por estado, de eventos adversos notificados a ANVISA no período de Janeiro de 2014 a julho de 2018.
4. World Health Organization (WHO). World Health Organization Patient Safety Curriculum Guide: Multiprofessional Edition. Rio de Janeiro: Autografia; 2016
5. Pastor AA, Tavares CMM. Literature review of audiovisual practices in nursing education. Rev Bras Enferm.2019; 72:190-191.
6. Ministry of Education and Culture. CNE / CES Resolution No. 3, of November 7, 2001. Brasília: Official Diary of the Union; 2001.
7. Girão ALA, et al. MedSafe: prototype of a virtual game on the preparation and administration of medications. REME rev min enferm. 2019;23:e1239.
8. Chavaglia SRR, et al. Didactic strategies identified by nursing students. Cogitare Enferm, v.23, n.3, p. e53876, 2018.
9. Domenico EBL, Cohrs CR. Moodle platform for the construction of knowledge in intensive care: an experimental study. Acta paul enferm. 2016;29:381-389.
10. Avelino CCV, et al. Teaching-learning evaluation on the ICNP® using virtual learning environment. Rev Bras Enferm. 2017; 70:602-609.
11. Silva AC,et al. Development of a virtual learning environment for cardiorespiratory arrest training. Rev esc enferm USP. 2016; 50:988-995.
12. Tavares APC, et al. Analysis of Brazilian publications on distance education in nursing: integrative review. Rev Bras Enferm. 2018;71:214-222.
13. Ministry of Health. Resolution No. 466, of December 12, 2012. Brasília: Ministry of Health; 2012.
14. Filatro, A. Design instrucional contextualizado: educação e tecnologia. 3 ed. São Paulo: SENAC; 2010.
15. Fehring, R. Methods to validate nursing diagnose. Heart Lung. 1987; 16: . 625-629.
16. Nesbit J, Belfer K, Leacock T. Learning Object Instrument Review (LORI) – user manual.2009.
17. Alvarez AG. Persuasive technology in the learning of acute pain assessment in nursing. Thesis (Doctorate in Nursing) - Postgraduate Program in Nursing, Federal University of Santa Catarina. Florianopolis. 2014.
18. Polit DF, et al. Fundamentos de pesquisa em enfermagem: métodos, avaliação e utilização. 5 ed. Porto Alegre: Artmed; 2004.
19. Martins DO, et al. Virtual Learning Environments (AVAs) as support tools in Complex Learning Environments (ACAs). Brazilian Association of Distance Education.2016; 16:113-131.
20. Kim YM, et al. Effects of a patient safety course using a flipped classroom approach among undergraduate nursing students: A quasi-experimental study. Nurse Education Today.2019; 70:180-187.
21. Pereira FGF, et al. Use of digital applications in the medicament calculation education for nursing. Invest educ enferm. 2016; 34:297-304.
22. Rosy T, et al. AntibioGame®: A serious game for teaching medical students about antibiotic use. Int J Med. Inform. 2020;136: 104074.
23. Renmarker E, Carlson E. Evaluation of Swedish nursing students’ experience of a web-based platform for drug calculation. Nurse Educ Pract. 2019; 38:89-95.
24. Figueiredo IGA, et al. Educational technologies developed for teaching patient safety in drug therapy: Integrative review. Int J Dev Res.2020; 11:50681-50686.
25. Holanda VR, et al. Teaching and learning in a virtual environment: nursing students' attitude. Revista Mineira de Enfermagem.2018; 19:141-147.
26. Leite NM. Digital Comics: a methodological proposal for teaching and learning school mathematics. Monography (Degree in Mathematics), State University of Paraíba. Campina Grande, 2018.
27. Filatro A. Produção de conteúdos educacionais. São Paulo: Saraiva, 2015.
28. Abbasi M, et al. The pedagogical effect of a health Education application for deaf and hard of hearing students in elementary schools. Electr Phys. 2017; 9:9.
29. Menossi LEU, et al. A avaliação da aprendizagem escolar: para além da verificação de resultados. Rev Cient Multidiscip Núcleo Conhec.2019; 1:1629.
30. Gadioli B, et al. Construction and validation of a virtual learning object for the teaching of peripheral venous vascular semiology. Esc Anna Nery. 2018; 22:e20180043.