The Development of an Instrument to Measure Perceived Workload for Certified Registered Nurse Anesthetists (CRNAs)

Since the landmark Institute of Medicine report “To Err Is Human: Building a Safer Health System” [1], Patient safety issues and the reduction of adverse events have been at the forefront of healthcare quality discussions. Providing over 40 million anesthetics each year [2], Certified Registered Nurse Anesthetists (CRNAs) work in various high-risk environments including but not limited to the operating room, remote locations (Endoscopy, Cardiac catheterization lab, Labor and delivery), and office-based settings. While anesthesia patient safety has improved tremendously in the past 30 years as a result of enhanced education and technology, there is still room to reduce morbidity and mortality rates and their associated costs [3]. Negative patient outcomes in the perioperative setting are often attributable to multiple factors, including contextual features of the work environment [4]. Therefore, the CRNA practice environment may be a good intervention point for patient safety because research has shown improved patient safety is associated with higher quality care and fewer adverse events. Perceived workload is considered part of that practice environment.

Workload is conceptualized as a complex and non-linear construct that is affected by external task demands, environmental, organizational, and psychological factors, and perceptive and cognitive abilities [5,6]. The concept of workload varies from profession to profession; it can be defined in a variety of ways and is composed of several domains [7,8,9]. In the field of anesthesia, researchers report that accurate assessment of provider clinical workload may be an important identifier of issues within a complex health system, as psychological, procedural, and physiological workload varies throughout the intraoperative period [6]. This indicates that further research is needed to better understand the complex interaction of anesthesia workload, adverse effects, and patient safety [6].

In nursing literature, workload is defined as “The amount of time and physical and/or cognitive effort…” required to provide both direct and indirect patient care, and activities that support care delivery [10]. Studies have consistently shown that high registered nurse-to-patient staffing ratios and reasonable workloads facilitate positive patient outcomes in hospitals [11,12]. These findings are consistent with research in clinics [13] and hospital settings [14] that revealed significant relationships between high workload scores and poor perceptions of patient safety culture. Research indicates high levels of subjective workload among CRNAs, using instruments developed for professionals other than CRNAs, are significantly associated with their perceptions of:

A. An unsafe work setting,

B. Fair to failing patient safety ratings, and

C. Higher incidence of adverse event reporting [15].

However, the effects of provider workload on the actual outcomes of patients who receive anesthesia have received little or no attention in the medical or nursing literature.

In order to further evaluate how CRNA workload affects patient safety culture and, ultimately, patient safety, it is important to determine exactly what factors comprise CRNA perceived workload because it is unclear whether existing workload tools capture the domains of CRNA workload [15]. Additionally, the workload of a CRNA is vastly different from that of a staff nurse. Staff nurse workload typically is associated with the number and illness severity of patients they care for on a shift; anesthesia practitioners care for one patient at a time, but often function under extreme time pressure [16,17]. To date no workload tool has been validated in CRNAs or other anesthesia providers. The specific aim of this study was to develop an instrument to accurately measure CRNA perceived workload, based on two existing workload instruments that have been validated in inpatient staff nurses.

This exploratory, two-phase sequential mixed method study was designed to collect the necessary data to develop a CRNA perceived workload instrument. Mixed methods research is chosen when it is necessary to collect, analyze, and integrate both qualitative and quantitative data into one study, or for long-term study of a research topic or question [18]. The benefit of using a mixed method design is that the integration of qualitative and quantitative approaches can enrich the data and improve the rigor of the study; in instrument development, the use of qualitative focus groups can add the participants’ view and may help explain differences between qualitative and quantitative findings [19].

Two existing workload scales were used to develop the preliminary CRNA perceived workload instrument: the NASA Task-Load Index (NASA-TLX) and the Revised Individual Workload Perception Scale (IWPS-R). The NASA-TLX is a widely used, multidimensional scale designed to measure subjective workload [20]. The instrument is a self-report visual analogue scale. It consists of 6 subscales that represent independent variable clusters: Mental, Physical, and Temporal Demands, Frustration, Effort, and Performance. An overall workload score is derived based on a weighted average of ratings of the subscales [20]. The assumption is that overall workload is a combination or sum of these scales, and a higher score indicates an increased workload. 7 tested the internal consistency reliability of the NASA-TLX in 757 ICU nurses and reported a Cronbach’s alpha of 0.72 in this population. The Principal Investigator (PI) of the current study reported a Cronbach’s alpha of 0.62 in a study of 336 CRNAs [15].

The 6-item workload subscale of the IWPS-R [21] comprises a 5-point summated Likert-type rating scale for each item, ranging from strongly disagree to strongly agree. Scores can range from 6-30, with a higher score indicating a higher workload. Sample items include “my current workload is reasonable” and “I am able to take at least a 30-minute meal break during my shift.” This workload instrument has also demonstrated good reliability in nursing studies, with a Cronbach’s alpha of > 0.70 [22,23] and a Cronbach’s alpha of 0.76 in a sample of 336 CRNAs [15]. Although this instrument has internal consistency reliability, its validity in measuring CRNA workload is unknown.

Phase 1

Following IRB approval for Phase I of the study, a panel of 14 CRNA clinical practice experts were recruited by the PI through professional networking within the American Association of Nurse Anesthetists (AANA). Participants were asked to evaluate the items on the two existing workload instruments, the NASA-TLX and the IWPS-R (Table 1). These experts were recruited because each had more than 10 years of experience, and functioned in a clinical leadership role with at least 80% dedicated time to providing clinical anesthesia within their respective departments. To reflect the wide variety of practice settings in which CRNAs practice, 2 clinical experts were recruited from each of the following areas: academic medical center>300 beds, private practice urban hospital, free-standing surgical center, and community, VA, military (Department of Defense), and rural hospitals. Data were collected via an online survey. The experts rated each item of the instrument on a 4-point Likert scale as to

A. Relevance to CRNA workload,

B. Clarity and ease in understanding the context of CRNA workload,

C. Contribution to CRNA workload.

Table 1:Phase 1 Expert Review of Instrument.

1. Content Validity Index-Relevance to perceived workload 2. Content Validity Index-Contributes to perceived workload

The experts rated each item of the two instruments using the following ordinal scale: 1=Not at all, 2=Somewhat, 3=Quite, 4=Very. The experts were asked what, if any, domain was missing from the tool.

The CVI is a widely used statistical index in the quantitative evaluation of a sample of items for the construct being measured [24]. If CVI is low, it is likely that the items are not proper operationalization of the construct, that the construct specifications are inadequate, or that experts are biased [25]. With five or fewer experts on the panel, the item content validity index (I-CVI) must be 1.00, which indicates all experts agree that the item is content valid; when there are more than five experts, there can be some disagreement [26]. For a scale item to be considered to have excellent content validity, recommend items have a minimum I-CVI of .78 or higher. The content validity index (I-CVI) was calculated for each survey item by computing the number of experts who rated each item a 3 or a 4. Items which did not have an I-CVI of .78 or greater in either relevance or contribution were evaluated for revision or removal [25].

Phase 2

Following IRB approval for Phase 2 of the study, CRNA focus groups were conducted to confirm the expert work on relevant instrument items from Phase I, and to discover if additional items should be added to examine the influence of CRNA workload. Following approval from AANA, a sample of 32 CRNAs were recruited from the 2016 Annual AANA Congress database. To meet inclusion criteria, the participants had to work full time and have a minimum of 2 years’ experience as a CRNA. Participants were contacted via email 2 weeks in advance of the Annual Congress; the invitation listed the date, times, and location of the focus group sessions, as well as a cell phone number and an email address to RSVP for one of the sessions. The email inviting participation contained information describing the IRB approval, purpose of the study, as well as the voluntary nature of participation; the email also stated that all data collected from the focus group would be deidentified, that audiotapes would be destroyed following transcription, and that no identifying information such as name or workplace should be written or would be recorded during the focus group.

Focus group participants first completed the preliminary workload survey tool developed in Phase 1 of the study (Table 2). Then participants were asked questions about their overall impression of the instrument, ease of use, ease of completion, overall CRNA workload, length of the survey, missing aspects of the workload, and any other workload domains that the tool did not capture. Member checking was completed at the end of each focus group; feedback was solicited from participants until there was no further discussion. The PI verbally summarized the participants’ statements and questions about the instrument, and about the domains of CRNA workload, and asked if they were correct, and if anything had been missed.

Table 2:Phase 1 Revised Instrument

Think about your typical daily workload over the past six months and answer the following questions

Once the focus groups were completed, each of the focus group audiotapes was professionally transcribed and reviewed by the PI for any discrepancies. Qualitative content analysis [27] was performed using NVivo 10 software. The original transcripts were systematically coded to identify the key points and recurring categories and themes that were central to the topics of discussion in the focus groups. The emergent codes and themes were reviewed and analyzed by the research team, which included an experienced qualitative researcher, to assure credibility of the generated findings.

Phase 1

Thirteen clinical expert CRNAs (93% response) provided the initial input into the survey. The participants were 54% Female and 46% Male; 84% were in clinical roles; all had greater than 10 years’ experience, with a range of 11-40 years in clinical practice; 69% were master’s prepared, and 31% were doctorally prepared. The clinical experts rated 2 items of the IWPS-R as not relevant nor contributing to CRNA workload; these items (Questions 4 and 5) had a low I-CVI (Table 1) and so these items were removed as per recommendations discussed in Methods [25,26]. Experts noted that a missing component of CRNA workload was related to the teaching environment in the perioperative period. They also did not like the wording of two questions: they felt the phrase “Meal Break” and the word “unit” should be changed in Questions 1 and 2, respectively. These two domains were added to the discussion during the focus group meetings in Phase 2 of the study. The instrument was revised into a 10-item instrument based on the expert feedback and calculated I-CVI (Table 2).

Phase 2

In Phase 2 of the study, 18 CRNAs (56%) participated in 3 focus groups. The groups ranged in size from 5-7 participants. The characteristics of the focus group participants were: 61% Male, 39% Female; Years of experience ranged from 4 to>30 years. All participants completed the new survey (Table 2) in 2.5 minutes or less. Then each group was asked an identical series of questions about each of the 10 survey items. Survey items 8 and 10 received the most comments; Survey items 4, 6, and 9 received fewer comments (Table 3).

Three themes emerged from the discussion of the survey items:

A. Stress is part of CRNA workload

B. Work environment requires description

C. Workload and work environment are different concepts

Table 3:Phase 2 Summary of Focus Group Feedback.

The survey item summaries are found in Table 3. Overall, the focus group participants felt that stress was a domain of CRNA workload. Stress was not viewed negatively but as a component of a workload. The participants stated there was stress related to teaching others, the politics of the work environment, and patient acuity and time pressure demands. This supported the addition of a survey item about stress. The participants also believed that workload was related to the work setting, or as some described it, the work environment. Further description and differentiation of workload from work environment was deemed necessary by all focus groups. This supports the addition of an item about work environment.

Table 4:Final Revised CRNA Perceived Workload Instrument.

Think about your typical daily workload over the past six months and answer the following questions.

The research team modified the original 10-question instrument to 12 questions; two additional items were added to the new instrument based on CRNA feedback. First, the rating scale for all items was changed to a 5-point Likert scale for ease of use and consistency of the tool. Each response on the Likert scale was directional, ranging from “Strongly Disagree to Strongly Agree,” “Low to High,” or “Not at All to Very.” As discussed above, wording was changed in several of the questions to reflect the focus group input, such as removing “Meal” from question 1 and changing the word “Unit” to “Department” in question 2. In question 3 the word “Reasonable” was changed to “Manageable,” and question 4 was added to reflect the focus group’s desire to include CRNA work setting as part of workload. The word “Monitoring” was added to question 6 about mental demand, and question 8 was changed from “Temporal Demand” to “Time Demand” and the word “Rate” was removed, leaving “Pace.” Question 10 was changed to ask about “Performance at Work” instead of “Job,” and question 11 was added to reflect the focus groups’ belief that stress is a domain of CRNA workload. Finally, question 12 about frustration was modified so all adjectives were removed (Table 4).

Perceived workload for other nursing groups has been measured with a variety of instruments; however, these instruments may not adequately measure the domains of CRNA perceived workload, which are quite different from staff nurses’ workload. For example, because CRNAs take care of one patient at a time, they are not subject to traditional nurse-to-patient staffing ratios, average acuity, and admissions, discharges, and transfers, which are often used to calculate nursing workload [10]. Additionally, CRNAs are under extreme time pressure because of the nature of their work before, during, and immediately following surgical procedures [16,17]. Although bedside staff nurses may be under time pressure for certain activities (e.g., Administering Medications on Time), They generally have 8 to 12 hours with their patients in order to prioritize and accomplish all other expected aspects of care. Although not ideal, bedside staff nurses may defer to the next shift for certain care activities, or simply leave the care unfinished during periods of heavy workload [28,29]. Of course, deferring or not completing routine anesthetic care such as timely reversal of a muscle paralyzing medication, or appropriate airway management such as placement or removal of the breathing tube by a CRNA would result in devastating patient outcomes.

To our knowledge, this is the first effort to develop an instrument to measure CRNA perceived workload. Following a meticulous mixed methods design, a new 12-item CRNA workload instrument was developed using 2 existing nursing workload tools. The mixed method technique provides for enriched data and rigor by combining quantitative and qualitative data; CRNA clinical experts and CRNA focus groups provided input into the tool development, such as the relevance, contribution, and clarity of all items included in the new instrument, and also helped to identify missing domains (Table 4). The limitations of this study may include that the focus group participants were small in number and were obtained from a sample of attendees at a national conference, possibly inducing bias; however, this method was the most efficient way of holding the focus groups to reach a nationally diverse group of CRNAs who practice across the country. The next step in instrument development is to pilot the newly developed tool in a large sample of CRNAs for a comprehensive psychometric analysis which would include exploratory factor analysis and individual item analysis using both item response theory and Mokken scale analyses [30].

It is imperative for CRNAs and the managers who employ them to have a better understanding of the concept of perceived workload in order to establish practice implications, as well as to be able to attach financial attributes and educational efforts to the importance of perceived workload to their anesthesia care [30]. Once perceived workload is measurable, it may then be used in research to correlate CRNA perceived workload with patient outcomes.

The ability to accurately measure CRNA perceived workload can inform further research about the implications of workload on daily practice, patient safety culture, and adverse event reporting. Additionally, a valid and reliable instrument to measure CRNA workload may be used by other researchers to quantify this important concept. It is hoped this instrument may be used to improve patient safety in the perioperative care environment by establishing how CRNA perceived workload may contribute to patient safety culture. For nursing and anesthesia department managers this information may help drive staffing decisions that promote improved patient care in their perioperative departments.

A. Funding Source: University of Alabama at Birmingham School of Nursing Dean’s Scholar Award, University of Alabama at Birmingham IRB Approvals E160623004 & E160713001

B. Thank you to Mrs. Kaitlen Woodfin MSN, CRNA for assistance with focus group management.

1. Kohn L, Corrigan J, Donaldson M (1999) To err is human: Building a safer health system. National Academy Press, Institute of Medicine, Washington DC, USA.
2. American Association of Nurse Anesthetists (AANA) (2016) AANA Membership Statistics.
3. Metzner J, Posner K, Lam M, Domino K (2011) Closed claims’ analysis. Best Practice & Research Clinical Anesthesiology 25(2): 263-276.
4. Jordan L, Quaraishi J (2015) The AANA Foundation Malpractice Closed Claims Study: A descriptive analysis. AANA Journal 83(5): 318-323.
5. Morris R, MacNeela P, Scott A, Treacy P, Hyde A (2007) Reconsidering the conceptualization of nursing workload: Literature Review. Journal of Advanced Nursing 57(5): 463-471.
6. Weinger M, Reddy S, Slagle J (2004) Multiple measures of anesthesia workload during teaching and nonteaching cases. Anesthesia & Analgesia 98(5): 1419-1425.
7. Hoonakker P, Carayon P, Gurses A, Brown R, McGuire K, et al. (2012) Measuring workload of ICU nurses with a questionnaire survey: The NASA Task Load Index. IIE Transactions on Healthcare Systems Engineering 1(2): 131-143.
8. Leedal J, Smith A (2005) Methodological approaches to anesthetists’ workload in the operating theatre. British Journal of Anaesthesia 94(6): 702-709.
9. Young G, Zavelina L, Hooper V (2008) Assessment of workload using NASA-Task Load Index in perianesthesia nursing. Journal of Peri Anesthesia Nursing 23(2): 102-110.
10. Swiger P, Vance D, Patrician P (2016) Nursing workload in the acute-care setting: A concept analysis of nursing workload. Nursing Outlook 64(3): 244-254.
11. Aiken L, Cimiotti J, Sloane D, Smith H, Flynn L, et al. (2011) Effects of nurse staffing and nurse education on patient deaths in hospitals with different nurse work environments. Medical Care 49(12) 1047-1053.
12. Aiken L, Clarke S, Sloane D, Lake E, Cheney T (2008) Effects of hospital care environments on patient mortality and nurse outcomes. Journal of Nursing Administration 38(5): 223-229.
13. Thomas HC, Flynn L (2015) Patient safety culture and nurse-reported adverse patient events in outpatient hemodialysis unit. Research and Theory for Nursing Practice 29(1): 53-65.
14. Aiken L, Sermeus W, Van den Heede K, Sloane DM, Busse R, et al. (2012) Patient safety, satisfaction, and quality of hospital care: Cross sectional surveys of nurses and patients in 12 countries in Europe and the United States. BMJ 344(e1717): 1-14.
15. McMullan SP, Thomas HC, Shirey MR (2017) Certified registered nurse anesthetist perceptions of factors impacting patient safety. Nursing Administration Quarterly 41(1): 56-69.
16. Demaria S, Neustein S (2010) Production pressure, medical errors, and the pre-anesthesia checkout. Middle East Journal of Anesthesiology: 20(5): 631-638.
17. Gaba D, Howard S, Jump B (1994) Production pressure in the work environment: California anesthesiologists’ attitudes and experiences. Anesthesiology 81(2): 488-500.
18. Creswell JW, Plano Clark VL (2007) Designing and conducting mixed methods research (3rd edn), Thousand Oaks, Sage, Canada, pp. 2-16.
19. Wisdom J, Creswell JW (2013) Mixed methods: Integrating quantitative and qualitative data collection and analysis while studying patientcentered medical home models. Agency for Healthcare Research and Quality. AHRQ Publication No. 130028-EF. Rockville, Maryland.
20. Hart SG (2006) NASA-Task Load Index (NASA-TLX): 20 years later. Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting, Santa Monica, Canada, pp. 904-908.
21. Cox KS, Teasley SL, Zeller RA, Lacey SR, Parsons L, et al. (2006) Know staff’s “Intent-to-Stay”. Nursing Management 37(1): 13-15.
22. Flynn L, Thomas HC, Clarke S (2009) Organizational traits, care processes, and burnout among chronic hemodialysis nurses. Western Journal of Nursing Research 31(5): 569-582.
23. Lacey S, Cox K, Lorfing K, Teasley S, Carroll C, et al. (2007) Nursing support, workload, and intent to stay at magnet, magnet-aspiring, and non-magnet hospitals. Journal of Nursing Administration 37(4): 199- 205.
24. Shi J, Mo X, Sun Z (2012) Content validity index in scale development. Journal of Central South University Medical Sciences 37(2): 152-155.
25. Polit D, Beck C, Owen S (2007) Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health 30(4): 459-467.
26. Lynn M (1986) Determination and quantification of content validity. Nursing Research 35(6): 382-386.
27. Krippendorff K (2018) Content analysis: An introduction to its methodology (4th edn), Thousand Oaks, Sage, Canada.
28. Jones TL, Hamilton P, Murray N (2015) Unfinished care, missed care, and implicitly rationed care: State of the science review. International Journal of Nursing Studies 52(6): 1121-1137.
29. Kalisch BJ, Williams RA (2009) Development and psychometric testing of a tool to measure missed nursing care. Journal of Nursing Administration 39(5): 211-219.
30. Clark J, Lang N (1992) Nursing’s next advance: An internal (International) classification for nursing practice. International Nursing Review 39(4): 109-111.