Assessing Spanish-language Patient Education Materials for Comprehension: A Pilot Feasibility Study with Implications for Physical Activity and Wellness Promotion

NoteAuthor Note

Conflict of Interest: We have no conflict of interest to disclose.

Acknowledgments: Emily N. Rodriguez, Kaitlyn I. Kuan, and Giselle Martinez were undergraduate students at the time of the study.

Correspondence: Correspondence concerning this article should be addressed to Jafra D. Thomas, Associate Professor, Department of Kinesiology and Public Health, California Polytechnic State University, 1 Grand Avenue, San Luis Obispo, California 93407. Email: jthoma84@calpoly.edu

Health literacy may influence whether patients can engage with and act on health and wellness information (Samoil et al. 2021), yet nearly half of U.S. adults have limited health literacy (Sudore and Schillinger 2009). When patients are unable to understand information or materials, they may struggle to follow treatment plans, manage chronic conditions, or access services (Berkman et al. 2011; Samoil et al. 2021). Some groups face these challenges more than others. Some of the greatest disparities in health literacy occur among racial and ethnic minority groups from diverse cultural backgrounds (Stormacq et al. 2019). Results from the National Assessment of Adult Literacy found that Hispanic individuals may have the lowest average health literacy scores of all racial/ethnic groups within the United States (Kutner et al. 2006).

Many healthcare settings rely on written patient education materials (PEMs) to recommend and explain health behaviors for preventing or managing chronic health conditions (Thomas et al. 2021). These PEMs and wellness promotion strategies may often target multiple health behaviors simultaneously. For example, a common pairing in wellness promotion is to promote healthy physical activity levels and healthy dietary habits (Greaves et al. 2011). The intention for PEMs is to supplement direct communications with healthcare providers: to back up information that providers give, help people learn important information, and encourage holistic healthy habits. However, these mediated health communication strategies only work if the content makes sense to the readers (Maneze et al. 2019). If the words are complex or if the layout is confusing, the message might not come through as helpful information (May et al. 2022).

Since written materials may influence what patients do (Vallance et al. 2007; Plow et al. 2014), it is important that they are easy to understand for people of all backgrounds. Making sure language is clear and content is relevant is an important step toward better, more inclusive health communication and wellness promotion, which is essential for holistic health care (Thomas and Cardinal 2021). To determine whether PEMs will be understandable for those with limited health literacy skills, researchers (and practitioners) need evaluation tools that go beyond simple reading-grade level formulas (Thomas et al. 2025).

Study Justification

Research investigating comprehension of text-based educational materials written in Spanish and using the cloze procedure is rare. When we initiated our study project in June 2024, to determine whether adapting the cloze procedure to Spanish-language PEMs was feasible and supported in the literature, we only found one published research study. This study was a 1977 dissertation on the construct validity of the cloze procedure when adapted to primary school educational materials written in Spanish (Rodriguez-Trujillo 1977). The dissertation’s preliminary findings suggested good construct validity for the procedure when compared to open-response recall questions (r = .62 to .72, p < .05), which mirrors similar conclusions from initial research comparing cloze scores to multiple-choice quiz questions for materials written in English (Taylor 1957; Bormuth 1968b). However, one patient education study on materials written in English, with older adult participants, reported lower correlation with open-response recall questions, although findings still suggested adequate construct validity (r_s = .43, p < .05) (Friedman and Hoffman-Goetz 2007).

During the write-up of this article’s Discussion section (in August 2025), we eventually located another dissertation published in 2000, which did evaluate comprehension of PEMs written in Spanish in an adult sample using the cloze procedure, but the procedure’s construct validity was not assessed using another direct measure of patient comprehension (Dunlap 2000). In the last several decades, the cloze procedure’s validity has been assessed in PEMs written in English (Holcomb and Ellis 1978; Kicklighter and Stein 1993; Miller et al. 2009), but not in Spanish. Based on the findings from several review studies (Thomas et al. 2021, 2025), it appears that the cloze procedure has been rarely studied in the physical activity and wellness promotion literature, especially in regard to PEMs written in Spanish.

Study Purpose

Given the limited amount of research, the purpose of this study was to evaluate the feasibility of adapting the cloze procedure methods by Rodriguez-Trujillo (1977) to studying PEMs written in Spanish for adult populations. Many of the people who use the medical services upon which our case material was obtained (the MHU) are women who speak predominantly, if not only, Spanish and have low levels of literacy and health literacy. This study contributes to the necessary research on ensuring PEMs disseminated in the primary language of patients and clients are easy to understand (J. D. Thomas et al. 2023). Previous research demonstrates that easy-to-understand PEMs may reduce (or eliminate) health behavior and clinical outcome disparities between low and high literacy patient populations (Clement et al. 2009; Kim and Lee 2016).

Methods

Before participating in this study, participants gave written informed consent. This study was approved by the Institutional Review Board. Participants of this study were 18 years or older.

Study Design

The following study is an exploratory pilot study and a formative evaluation of a protocol to measure and evaluate the comprehension of health promotion PEMs written in Spanish, utilizing the cloze procedure (Taylor 1953). Findings of this evaluation are based upon a systematic replication of procedures developed by Rodriguez-Trujillo (1977) to adapt the cloze procedure to studying PEMs written in Spanish. Rodriguez-Trujillo used a sample of elementary school children within a scholastic educational setting. Our application examines the efficacy of their steps for determining the ability of Spanish-speaking adults to comprehend PEMs written in Spanish and covering topics included in physical activity promotion materials. We conducted two rounds of pilot testing. A counter-balance design was utilized by switching material order in the second pilot test session; this was to account for possible biases due to the order of the materials tested.

Inclusion Criteria

To participate in the study, participants must have expressed the ability to understand text written in Spanish at any level, been 18 years or older, and a current undergraduate or graduate students at the university where the study took place. In our recruitment communication, it was emphasized that people of all Spanish proficiency levels were encouraged to participate, and that fluency was not necessary.

Sampling Procedures

Two rounds of pilot testing were conducted by recruiting participants through word-of-mouth and email announcements. To recruit participants for the first round of pilot testing, an email was sent out to students involved in a Kinesiology and Public Health Department undergraduate research program. Recruitment messages emphasized individuals with any level of Spanish language knowledge were welcome to participate. Additionally, through word-of-mouth, the undergraduate students serving as research assistants on the project spread information about the study to friends and others they knew personally. Via email, we sent out a flyer with a QR code linking to the study’s RSVP form; the flyer also advertised light refreshments would be provided to participants. The first pilot study was on August 8, 2024, using a university department meeting space.

The second round of pilot testing took place on August 13, 2024, and was done with students involved in the university’s Educational Opportunity Program (EOP), a program to help support students from historically underserved backgrounds or those facing economic or educational barriers. Similar recruitment strategies as previously described were used to recruit participants for the second round of pilot testing. The second round of pilot testing took place in an EOP meeting space.

Materials

After each patient visit at the MHU, patients are given a folder full of informational handouts on healthy eating, tips to lose weight, fiber, tips to control diabetes, mammography, cervical cancer, stress, depression, heart health, high blood pressure, and dental care. We thoroughly examined these materials and counted the words on each paper to determine which could be converted into the cloze test format. The only material which seemed to have enough words for conversion into a cloze test was a handout on dietary fiber titled, La Fibra. We confirmed the text of the La Fibra PEM was written at an 8th grade reading level, according to the SMOG reading grade level formula for text written in Spanish (Contreras et al. 1999). The process of converting the material into a cloze test started with retyping the entire paper onto a word document. We began the process by counting and highlighting the first and last 30 words, noting that they should be left as they were. We then began the process of deleting every fifth word. As per the dissertation (Rodriguez-Trujillo 1977) we began deletion with the 31st word and then continued to delete every fifth word until we reached the last 30 words. We included section headings in our word count, so some of the headings had words deleted.

We wanted to find another health material about dietary fiber and to do so, we googled “fibra dieta” to find articles in Spanish that were comparable to the PEM handout on fiber from the MHU. We found a handout from Northwestern University (Northwestern University 2023) titled, Dieta Baja en Fibra, that was about similar content and of a similar length to the MHU PEM. We confirmed the PEM from NWU also was written at an 8th grade reading level using the SMOG conversion formula (Contreras et al. 1999). A limitation to note about our search is that while searching for related handouts in Spanish, the Google search settings were not in Spanish, which might have affected our search results (Antonelli 2025). We used the same process to convert the NWU handout into a cloze format. The main content of the cloze forms, and the open-response recall questionnaires, were all written in Spanish. Participants completed the assessments by responding to prompts in Spanish.

Study Protocol

For materials used to administer the study (e.g., script, assessments), see Supplemental Content #1. Pilot test Group 1 received the MHU material first; Group 2 received the NWU material first.

To conduct pilot testing sessions, a 3-person team of undergraduate students implemented the protocol using a script to ensure sessions were consistent. A script was developed with inclusive language. In each testing session, we provided light refreshments to offer participants as they entered the testing areas. We also asked them to check in by scanning a QR code, which led them to a check-in form. We typically waited about 5 minutes after the hour for students to finish arriving before we began the assessment activities. As they waited, we gave each participant a printed informed consent form to read over and sign before we began.

For the first testing session, students had 15 minutes to complete the MHU cloze test, 10 minutes to complete the MHU open response questions, 13 minutes for the NWU cloze test, 10 minutes for the NWU open response questions, and 5 minutes to fill out the online demographic questionnaire. The demographic questionnaire also contained open-response questions about the quality of participants’ experience. We decided to allocate less time to fill out the NWU cloze test as this article was shorter than that of the MHU PEM.

The order of pilot test one was reversed for pilot test two, meaning participants completed the NWU cloze test and open response questions first, then the assessments for the MHU materials. Each section used the same language as the initial test and had the same time allotted as the initial test; the only difference was the order of the testing materials.

Measures

For the cloze test, scores were calculated by the percentage of blanks participants filled in correctly. We followed the guidelines from Doak et al. (1996), where 40% or higher indicates partial understanding, and 60% or higher indicates full understanding. The open-response questions were scored using a rubric, and interpreted using cut-points by Bormuth (1968a): 75% or higher for partial understanding, and 90% or higher for full understanding.

Analysis Plan

Descriptive statistics were used to summarize participant demographic questionnaire responses and comprehension scores from the two assessments. Responses to the open-ended questions about participants’ experience with participating in the study were summarized using a descriptive-interpretive analysis (Elliot and Timulak 2021). Qualitative results would help gauge the acceptability of the cloze procedure and study protocol.

We also conducted exploratory tests of significance, to determine whether adequate and significant correlation would be observed for cloze and open-response scores. Reported p-values are exact values based on two-sided statistical tests, which were conducted using the Statistical Package for the Social Sciences (SPSS, version 29). The a priori cut-point for statistical significance was set at p \(\leq\) .10, given the study’s small sample size and exploratory nature (Vaske 2019). As Vaske and others have argued, the traditional probability cut-point of p \(\leq\) .05 is arbitrary (Cohen 1994; Vaske 2019; J. R. Thomas et al. 2023); researchers should not use it as a default if not supported by empirical precedence and a large enough sample size (e.g., n = 100; (Vaske 2019, 99)). A nonparametric statistical test (Spearman’s rho) was used to test for statistically significant correlations, since visual inspection of histogram data suggested that the only data normally distributed were the open-response scores (Pallant 2020).

Bonferroni corrections were made to the significance test cut-point, in order to mitigate against an inflated Type 1 error rate due to multiple comparisons (Motulsky 2018). In this investigation, two comparisons were made. Thus, correlation test results must have had a p-value of \(\leq\) .05 to be considered statistically significant. Standardized effect size estimations were used to determine the magnitude of observed differences or associations (DeVon et al. 2007; Vaske et al. 2002; Vaughn and Daniel 2012). Before analyzing the sample as a whole, the two pilot test samples were descriptively compared to one another to confirm whether the counter-balance technique effectively mitigated any potential bias with the material order. Differences between the groups were not tested statistically because of the small sample size of each group.

Results

Descriptive Analysis

As shown in Table 1, the counter-balance technique was largely effective in mitigating any potential bias with material order. Pilot test groups tended to have the same mean comprehension score across both assessment methods, and qualitative interpretation of mean scores were equivalent across both assessment methods. All participants correctly completed the assessment questionnaires, including only using one-word responses for the cloze form.

Table 1: Quality Check: Separated Comprehension by Material Order (Counter-balance Test).

Group & Assessment	n	M% (MHU)	SD (MHU)	Min, Max (MHU)	M% (NWU)	SD (NWU)	Min, Max (NWU)
Order 1	4
Cloze scores		34.55	9.03	25.45, 45.45	35.81	11.75	24.32, 51.35
Open-response		33.33	13.61	16.67, 50.00	40.00	16.33	20.00, 60.00
Order 2	6
Cloze scores		38.79	10.39	25.45, 54.55	32.43	5.92	27.03, 43.24
Open-response		38.89	17.21	16.67, 66.67	46.67	20.66	20.00, 80.00

Note. MHU = mobile health unit; NWU = Northwestern University. M = Mean; SD = Standard deviation; Min, Max = minimum and maximum score range. Order 1 = MHU first, then NWU; Order 2 = NWU first, then MHU. Cloze: ≥ 40% = partial comprehension, ≥ 60% = full comprehension. Open-response: ≥ 75% = partial comprehension, ≥ 90% = full comprehension.

As shown in Table 2, participants reported different levels of Spanish language experience. Most rated their Spanish reading confidence as confident. All had received formal Spanish instruction, most commonly in high school and college. This suggests that participants had enough Spanish exposure to reasonably engage with the study materials.

Table 2: Demographic Characteristics of Participants (N = 10).

Characteristics	Participant Response	Frequency	%
Spanish Reading Confidence	Very Confident	1	10.0
	Confident	6	60.0
	Unsure	3	30.0
Formal Spanish Reading Education	Elementary, Middle, College	1	10.0
	Middle, High, College	2	20.0
	High school, college	3	30.0
	High school	1	10.0
	College	1	10.0
	Not reported	2	20.0
Grew Up Speaking Spanish in Household	Yes	7	70.0
	No	3	30.0
Grew Up in Spanish Speaking Environment	Yes	9	90.0
	No	1	10.0

Note. Percentages are based on N = 10.

Participants were asked to comment on the clarity of instructions after completing the assessment tests. Most felt the instructions were clear, with others stating the process was organized and easy to follow. A few participants mentioned needing more time or experiencing difficulty remembering parts of the task, meaning there were minor areas for improvement with how the assessments were facilitated.

Table 3: Participant Responses on Clarity of Instructions During Cloze Test Activities.

Participant ID	Response
1	The instructions for all activities were very clear
2	Instructions were clear, there was enough time to fill in the blanks.
3	Yes very clear - well organized.
4	The activity instructions were clear but I think I was confused on the first round what exactly I was supposed to be doing since a lot of it is up to your interpretation.
5	Yes the instructions were very clear. I had no confusion.
6	Activity instructions were very clearly explained. I think with the fill in the blank it was difficult sometimes to think of words for when it was naming examples because it was a topic I knew nothing about.
7	The activity instructions were clear, the activity was a little confusing and challenging.
8	Yes they were clear and well explained through the directions.
9	The instructions were very clear and I had no questions, but I haven’t read, or written in Spanish in a long time so it was interesting.
10	The instructions for the activities were clear, but maybe for people without prior knowledge to how fiber works it may be a bit tedious. In terms of the questions, after re-reading the article it should be quite straightforward on how to respond.

Note. Question posed to participants: “Please give us feedback about your experience. Were the activity instructions clear? Please explain your response.” Gathering this feedback helped assess whether the testing process itself was understandable and accessible.

Participants were asked to give feedback on the activity (see Table 4). Some described the experience as enjoyable or interesting, using words like “fun” or “engaging,” while others reported feeling “confused” or said it was “difficult.” This suggests different levels of comfort with the task. Based on Table 2 and Table 3, some participants who expressed difficulty also had limited formal Spanish education, suggesting that background may influence how the activity was experienced. The responses suggest that the task was feasible to complete, even if the materials were challenging.

Table 4: Participant Feedback on Cloze Test Experience.

Participant ID	Response
1	N/A
2	Filling in the blanks was actually kind of fun
3	N/A
4	I think it was sometimes hard to remember the specific numbers or types of food on the follow up questions because I was so focused on trying to find the word that fit. But that may be the purpose of the study to see how well you retain that while trying to comprehend.
5	N/A
6	N/A
7	N/A
8	I enjoyed it and I am glad I did it because I hadn’t been a part of a research project before. It was quite interesting reading the article and sometimes not knowing if maybe the word was simple it’s just the thought of not knowing at the exact moment.
9	It was fun but I was slightly nervous since I never had a formal education in Spanish. I only know what I’ve picked up over the years.
10	I really enjoyed my experience in this study!

Note. Question posed: “Is there anything else you would like to say about your experience? If yes, please state below. If not, put ‘N/A’.” Responses reflect feedback from all 10 participants and are shown exactly as written.

Participants expressed a range of reactions, from confidence to uncertainty, across different parts of the task for completing the cloze form, then attempting the follow-up questions. One participant described completing the cloze form as “actually kind of fun.” This response was not common and shows an aspect of engagement that was not a specific measurement aim.

Of note, the research team introduced several errors when transcribing the MHU material into a cloze format questionnaire. A grammatical error was made by mistake with the title of the MHU patient education material, omitting the acute accent mark over the word, “Que,” when it is used as an interrogative word for asking questions (i.e., Qué) (Temoltzin-Espejel 2025). Several spelling errors caused by the word-processor auto-correct affected two words in the text body. These omissions, however, did not appear to negatively affect participants’ experiences or affect the main findings of this feasibility study, as indicated by the participant feedback and descriptive results shown in Table 1. In this article, the errors were corrected within the MHU material supplemental file document. Post data collection, no other errors were identified—the PEM materials matched their original source in terms of spelling, grammar, and bold font.

Main Analysis

Comprehension scores for both PEMs were below the expected levels, based on the materials’ estimated reading grade level (see Table 5). According to (Doak et al. 1996), cloze scores show partial understanding at 40% or higher and full understanding at 60% or higher. The average cloze score was 37.09% for MHU and 33.78% for NWU, suggesting that neither PEM met the cut-off for comprehension. However, only the NWU mean scores were less than the procedure’s 40% threshold for partial comprehension (p = .042). Descriptively, more participants demonstrated ability to partially comprehend the PEM from the MHU than from NWU (5 versus 2 individuals). Specifically, scoring between 40 and 59% would suggest that some participants could comprehend the educational materials with supplemental instruction. However, open-response scores for the PEM from the MHU and NWU were also low. The average open-response score was 36.67% for MHU and 44% for NWU, far below the 75% threshold suggesting an ability to understand the material with supplemental instruction (Bormuth 1968a).

Cloze scores were more consistent, while open-response scores varied more, showing the potential differences in participant interpretation and knowledge-base. Both methods suggest low material comprehension for the sample, overall. The open-response method, however, occasionally had higher scores, suggesting even when some participants struggled with word-for-word reading they were able to ascertain a certain level of general understanding from the material, supporting accurate recall of key information. This shows that the two methods capture different aspects of information processing and understanding, and together, may provide a better picture of comprehension.

Table 5: Sample-wide PEM Comprehension Scores by Assessment Method.

Assessment Type	N	M% (MHU)	SD (MHU)	Min, Max (MHU)	M% (NWU)	SD (NWU)	Min, Max (NWU)
Cloze Test	10	37.09	9.59	25.45, 54.55	33.78	8.28	24.32, 51.35
Open-response	10	36.67	15.32	16.67, 66.67	44.00	18.38	20.00, 80.00

Note. Scores reflect the combined results across both pilot tests (N = 10). Cloze: ≥ 40% = partial comprehension, ≥ 60% = full comprehension. Open-response: ≥ 75% = partial comprehension, ≥ 90% = full comprehension. MHU = mobile health unit; NWU = Northwestern University. M = Mean; SD = Standard deviation; Min, Max = minimum and maximum score range. For the MHU cloze form, 50% of participants met the threshold for partial comprehension, with all remaining scoring below. Only 20% of participants met the partial comprehension threshold for the NWU cloze form, with all remaining scoring below. Mean cloze scores for the MHU PEM did not statistically differ from the 40% threshold, but mean scores for the NWU material did: MHU (t = .959, df = 9, p = .363, g = .28), NWU (t = 2.374, df = 9, p = .042, g = .69).

While qualitative interpretation of cloze form and open-response scores in this study were equivalent, convergent validity of the two measures were not well-supported in this exploratory study. The strength of correlation between cloze form and open-response scores was adequate for the MHU PEM, and possibly inadequate for the NWU PEM (DeVon et al. 2007; Vaughn and Daniel 2012). Both correlation tests were not statistically significant, after controlling for multiple comparisons using the Bonferroni correction: MHU PEM (rho = .591, p = .072), NWU PEM (rho = .357, p = .311). These results could be due to several factors.

First, while participants felt comfortable speaking Spanish, their health literacy for reading and understanding PEMs in Spanish may not have matched their fluency or confidence level. A modified cloze procedure giving multiple-choice answer options per blank space may have led to stronger and statistically significant correlations with the open-response comprehension scores (Williams et al. 2011). Second, making the cloze form questionnaire “unanticipated” might increase its correlation with the open-response questions (Taylor 1957). This second approach may require a non-cloze version of the PEMs be used first, since the recall questions may not optimally measure comprehension for text with deleted words, as is required for the cloze procedure (Taylor 1957). Previous research seldom measures participant comprehension for the same material using the cloze procedure in conjunction with other methods (Gellert and Elbro 2013; Kicklighter and Stein 1993). Finally, it is fully possible that a study with a larger sample size could result in statistically significant correlations between the two measures (Zhu 2012).

It is worth noting, however, that the primary aim of this study was not to test the convergent validity of the cloze procedure when studying adult comprehension of PEMs in Spanish, nor was it to determine the general ability of patients to comprehend the PEMs tested in this study. Caution should be exercised when making general conclusions about the actual ability of patients utilizing services from the respective healthcare facilities.

Discussion

The purpose of this pilot study was to determine the feasibility of replicating procedures for adapting the cloze procedure in order to study how well Spanish-language PEMs are understood in adult populations. According to counter-balanced results, we produced a reliable protocol that was not difficult to administer and was acceptable to participants. While several participants expressed some uncertainty with how to follow task instructions, all participants correctly completed all assessment questionnaires. None expressed disliking the activity and several found the experience interesting or fun. These qualitative findings mirror findings from an adult education literacy study that assessed the acceptability of the cloze procedure among adults with limited English language skills (Robinson 1973). This feasibility pilot study evidenced that the cloze procedure, and the recall questions for assessing comprehension of PEMs written in Spanish, are replicable, feasible, and may be acceptable to participants with different Spanish reading and writing skills.

Compared to the results of the present study, McWhirter et al. (2011) found somewhat contrasting findings in their study assessing how well Chinese immigrant women in Canada could comprehend a colon cancer screening fact sheet written in English. Nearly all of the participants (89.7%) demonstrated ability to understand the fact sheet as written without the need for supplemental instruction according to cloze score results, and teach-back results mirrored the previous interpretation for several learning objectives assessed in the study (e.g., explain primary screening measures, identify age-point recommended to begin screening by gender). Although statistically significant, the cloze procedure’s construct validity was not strongly supported when correlated against teach-back results (r = .38, p = .004). Stronger correlation, however, was observed when cloze scores were compared to functional health literacy test scores (S-TOFHLA, r = .50, p = .006). While the authors concluded that the cloze test may not be a sufficient measure of health literacy in English language-learner populations, their findings further validate the cloze procedure’s interpretive cut-points for judging patients’ ability to comprehend patient material (Doak et al. 1996).

Still, the secondary findings of the present study, specifically the ability of potential end-users to understand the PEMs, mirrored findings from previous research. In their investigation of how well Spanish-speaking English language learners understood PEMs on diabetes management and vaccine information, Dunlap (2000) found similar findings as our pilot study in their much larger sample (N = 101). Over three-quarters of their sample (88%) could not understand the materials independently and about one-quarter (26%) could not understand the materials as written. In a different, qualitative study, Manzor-Mitrzyk et al. (2024) found that while participants (N = 30) felt the test material (Spanish language health information handouts on depression treatment) contained useful information and they understood key ideas, qualitative interviews revealed 53% of participants were confused by terminology and 10% felt unsure they understood enough to correctly apply the information. Their sample consisted of mixed educational levels with 57% having a high school diploma or higher years of formal education, and who were predominantly low-income and uninsured adults. The authors did not investigate whether education level, income level, or insurance status influenced comprehension results. In research examining comprehension of medical bottle instructions in English and Spanish, Leyva et al. (2005) observed large portions of participants could not correctly demonstrate the correct dosing pattern (50–78%, N = 100). In their study, formal education level and comfort level speaking in English were positively associated with greater odds of following instructions correctly.

Study Implications

While mainly a feasibility study, the preliminary findings about participants’ ability to comprehend the PEMs assessed in this study have important implications. First, when materials are difficult to understand, patients with limited English understanding or health literacy skills may often rely on family or friends to explain or translate the content. However, our study showed that even bilingual adults with formal Spanish education had trouble understanding the materials written at recommended reading grade levels, meaning that informal translators may not fully understand the information either. This is important to understand in the context of physical activity behavior change interventions with objectives to promote health and wellness; many of these interventions rely upon mediated online and print-based PEMs and may address multiple health behaviors (e.g., physical activity plus diet or smoking habits) (Eckman et al. 2012; Espigares-Tribo and Ensenyat 2021). The present study’s findings also challenge the assumption that Spanish as a first language, or classroom exposure, guarantees fluency in both reading and comprehension, especially regarding health information.

Additionally, our results suggest that readability tools are not enough to ensure materials are understandable. Even though the materials were written at an eighth-grade reading level, most participants did not demonstrate ability to fully and easily understand the PEMs assessed in this study. Testing materials with the direct comprehension tests, cloze procedure, and open-response recall questions offered a clearer picture of how well the content may likely be understood in similar demographic groups. These implications highlight the need to use evidence-supported, and theory-grounded health literacy principles to develop and test bilingual PEMs in physical activity promotion interventions, using direct measures of comprehension (Thomas et al. 2024). These additional steps should ensure greater support of individuals with varied levels of health literacy and background knowledge and generally reduce the odds that information is misunderstood or difficult to apply by end-users (Evans et al. 2021).

Study Limitations

There are several limitations to the present study that should be kept in mind when interpreting its results. First, we did not investigate participant health literacy level using an established tool. While the cloze procedure and open-response summary questions do correspond with health literacy constructs, such as prior knowledge and ability to comprehend documents, they are not exact measures of health literacy level skill domains (e.g., interpretation of content numeracy, communication, or critical appraisal; (Institute of Medicine 2004). The results tentatively suggested that the participants had limited health literacy for understanding PEMs on dietary fiber and health, which should be investigated using triangulated methods with established health literacy measurement tools (Yuen et al. 2018). Second, the small, convenience sample of ten participants limits how well results may generalize, and it made it difficult to detect clear differences in comprehension levels. Third, the NWU PEM had fewer than 50 blanks (i.e., 37), which may not meet recommended standards for cloze testing (Doak et al. 1996). In their review of replication studies investigating the cloze procedure, Jongsma (1971) summarized evidence showing that (a) at least 50 blanks were needed to achieve sufficient representation of structural words (e.g., articles, prepositions) and semantic words (nouns and verbs) and (b) cloze forms with less than 50 blanks tend to have inconsistent findings when compared to their counterparts with at least 50. Fourth, participants were college students, not the intended patient group for the materials. Many patients may have lower literacy levels, which could affect how well they understand the materials and may require modifications to how comprehension is measured using the cloze procedure and other methods (Doak and Doak 1980; Keenan et al. 2008; Williams et al. 2011).

Future Directions. Although the results show the protocol is feasible, future studies should include a variety of Spanish speakers and levels from clinical settings. Testing with the material’s intended audience may give a clearer picture of how these materials perform. It would also be useful to examine how comprehension changes when paired with common clinical supports, like visual aids or verbal explanations. Finally, future studies should test the replicability of this study’s findings after addressing the study limitations mentioned in the previous paragraph, including ensuring all cloze tests are based on PEMs with at least 50 blank spaces.

Conclusion

This pilot study, combining the cloze procedure comprehension test paired with open-response recall questions, determined the feasibility of the methods for assessing comprehension for PEMs written in Spanish. Secondary findings also allowed for the preliminary evaluation of how well patients might understand Spanish-language PEMs. Results suggested most participants had inadequate to partial ability to understand the PEMs tested in this study, on the health benefits of dietary fiber for disease prevention and management. While limited by sample size and a focus on assessing the feasibility of the study’s protocol, the preliminary findings of this study support using direct measures to evaluate how well patients and research participants comprehend PEMs used in physical activity and wellness promotion interventions; these direct readability evaluations should occur before wide scale dissemination of the materials or the intervention’s rollout. Larger studies are now needed to confirm the validity of the cloze procedure for assessing PEMs written in Spanish and whether the present study findings replicate in patient populations.

Acknowledgments

The research reported on in this article received funding support from the Bailey College of Science and Mathematics (California Polytechnic State University, San Luis Obispo) William and Linda Frost Fund, through a 2025 Frost Summer Undergraduate Research Award, awarded to Kaitlyn I. Kuan and a 2024 Frost Summer Undergraduate Research Award, awarded to Giselle Martinez, Emily Rodriguez, and Kevin Castaneda.

The research team also benefited from resource and logistical support from the following partners at California Polytechnic State University, San Luis Obispo: the Women’s Mobile Health Unit Coordinator, Cristina Macedo; the previous director for the Center for Health Research, Dr. Suzanne Phelan; the Kinesiology and Public Health Department administrative support staff; and the Educational Opportunity Program.

License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

References

Antonelli, D. 2025. 2 Easy Ways to Change the Language in Google. WikiHow.com. https://www.wikihow.com/Change-Language-on-Google.

Berkman, N. D., S. L. Sheridan, K. E. Donahue, D. J. Halpern, and K. Crotty. 2011. “Low Health Literacy and Health Outcomes: An Updated Systematic Review.” Annals of Internal Medicine 155 (2): 97–107. https://doi.org/10.7326/0003-4819-155-2-201107190-00005.

Bormuth, J. R. 1968a. “Cloze Test Readability: Criterion Reference Scores.” Journal of Educational Measurement 5 (3): 189–96. https://www.jstor.org/stable/1433978.

Bormuth, J. R. 1968b. “The Cloze Readability Procedure.” Elementary English 45 (4): 429–36. https://www.jstor.org/stable/41386340.

Clement, S., S. Ibrahim, N. Crichton, M. Wolf, and G. Rowlands. 2009. “Complex Interventions to Improve the Health of People with Limited Literacy: A Systematic Review.” Patient Education and Counseling 75 (3): 340–51. https://doi.org/10.1016/j.pec.2009.01.008.

Cohen, J. 1994. “The Earth Is Round (\(p\) < .05).” American Psychologist 49 (12): 997–1003. https://doi.org/10.1037/0003-066X.49.12.997.

Contreras, A., R. García-Alonso, M. Echenique, and F. Daye-Contreras. 1999. “The SOL Formulas for Converting SMOG Readability Scores Between Health Education Materials Written in Spanish, English, and French.” Journal of Health Communication 4 (1): 21–29. https://doi.org/10.1080/108107399127066.

DeVon, H. A., M. E. Block, P. Moyle-Wright, et al. 2007. “A Psychometric Toolbox for Testing Validity and Reliability.” Journal of Nursing Scholarship 39 (2): 155–64. https://doi.org/10.1111/j.1547-5069.2007.00161.x.

Doak, C. C., and L. G. Doak. 1980. “Patient Comprehension Profiles: Recent Findings and Strategies.” Patient Counselling and Health Education 2 (3): 101–6. https://doi.org/10.1016/s0738-3991(80)80049-8.

Doak, C. C., L. G. Doak, and J. H. Root. 1996. “Testing Literacy Skills of Patients.” In Teaching Patients with Low Literacy Skills, 2nd ed. J. B. Lippincott. https://www.hsph.harvard.edu/healthliteracy/resources/teaching-patients-with-low-literacy-skills.

Dunlap, C. P. 2000. “The Comprehension of Patient Education Materials Written in Spanish.” PhD thesis, University of Texas at Austin. https://www.proquest.com/openview/40e80fff4e745918f0927f50588376de/1.

Eckman, M. H., R. Wise, A. C. Leonard, et al. 2012. “Impact of Health Literacy on Outcomes and Effectiveness of an Educational Intervention in Patients with Chronic Diseases.” Patient Education and Counseling 87 (2): 143–51. https://doi.org/10.1016/j.pec.2011.07.020.

Elliot, R., and L. Timulak. 2021. “Why a Generic Approach to Descriptive-Interpretive Qualitative Research?” In Essentials of Descriptive-Interpretive Qualitative Research: A Generic Approach, 1st ed. American Psychological Association. https://doi.org/10.1037/0000224-001.

Espigares-Tribo, G., and A. Ensenyat. 2021. “Assessing an Educational Booklet for Promotion of Healthy Lifestyles in Sedentary Adults with Cardiometabolic Risk Factors.” Patient Education and Counseling 104 (1): 201–6. https://doi.org/10.1016/j.pec.2020.06.012.

Evans, H., C. C. Forbes, D. A. Galvão, et al. 2021. “Usability, Acceptability, and Safety Analysis of a Computer-Tailored Web-Based Exercise Intervention (ExerciseGuide) for Individuals with Metastatic Prostate Cancer: Multi-methods Laboratory-Based Study.” JMIR Cancer 7 (3): e28370. https://cancer.jmir.org/2021/3/e28370/.

Friedman, D. B., and L. Hoffman-Goetz. 2007. “An Exploratory Study of Older Adults’ Comprehension of Printed Cancer Information: Is Readability a Key Factor?” Journal of Health Communications 12 (5): 423–37. https://pubmed.ncbi.nlm.nih.gov/17710594/.

Gellert, A. S., and C. Elbro. 2013. “Cloze Tests May Be Quick, but Are They Dirty? Development and Preliminary Validation of a Cloze Test of Reading Comprehension.” Journal of Psychoeducational Assessment 31 (1): 16–28. https://doi.org/10.1177/0734282912451971.

Greaves, C. J., K. E. Sheppard, C. Abraham, et al. 2011. “Systematic Review of Reviews of Intervention Components Associated with Increased Effectiveness in Dietary and Physical Activity Interventions.” BMC Public Health 11: 119. https://doi.org/10.1186/1471-2458-11-119.

Holcomb, C. A., and J. K. Ellis. 1978. “Measuring the Readability of Selected Patient Education Materials: The Cloze Procedure.” Health Education 9 (6): 8–10. https://doi.org/10.1080/00970050.1978.10619379.

Institute of Medicine. 2004. Health Literacy: A Prescription to End Confusion. The National Academies Press. https://doi.org/10.17226/10883.

Jongsma, E. R. 1971. The Cloze Procedure: A Survey of the Research. Indiana University School of Education. https://eric.ed.gov/?id=ED050893.

Keenan, J. M., R. S. Betjemann, and R. K. Olson. 2008. “Reading Comprehension Tests Vary in the Skills They Assess: Differential Dependence on Decoding and Oral Comprehension.” Scientific Studies of Reading 12 (3): 281–300. https://doi.org/10.1080/10888430802132279.

Kicklighter, J. R., and M. A. Stein. 1993. “Factors Influencing Diabetic Clients’ Ability to Read and Comprehend Printed Diabetic Diet Material.” The Diabetes Educator 19 (1): 40–46. https://pubmed.ncbi.nlm.nih.gov/8458298/.

Kim, S. H., and A. Lee. 2016. “Health-Literacy-Sensitive Diabetes Self-Management Interventions: A Systematic Review and Meta-Analysis.” Worldviews on Evidence-Based Nursing 13 (4): 324–33. https://doi.org/10.1111/wvn.12157.

Kutner, M., E. Greenburg, Y. Jin, and C. Paulsen. 2006. The Health Literacy of America’s Adults: Results from the 2003 National Assessment of Adult Literacy. NCES 2006-483. National Center for Education Statistics. https://nces.ed.gov/pubs2006/2006483.pdf.

Leyva, M., I. Sharif, and P. O. Ozuah. 2005. “Health Literacy Among Spanish-Speaking Latino Parents with Limited English Proficiency.” Ambulatory Pediatrics 5 (1): 56–59. https://doi.org/10.1367/A04-093R.1.

Maneze, D., R. Weaver, V. Kovai, et al. 2019. “‘Some Say No, Some Say Yes’: Receiving Inconsistent or Insufficient Information from Healthcare Professionals and Consequences for Diabetes Self-Management.” Diabetes Research and Clinical Practice 156: 107830. https://doi.org/10.1016/j.diabres.2019.107830.

Manzor-Mitrzyk, B., A. I. Lopez-Medina, and K. B. Farris. 2024. “Comprehension and Usefulness of Spanish Language Health Information about Depression Treatment.” Health Education Research 39 (3): 228–44. https://doi.org/10.1093/her/cyae013.

May, P., G. Yeowell, L. Connell, and C. Littlewood. 2022. “An Analysis of Publicly Available National Health Service Information Leaflets for Patients Following an Upper Arm Break.” Musculoskeletal Science and Practice 59: 102531. https://doi.org/10.1016/j.msksp.2022.102531.

McWhirter, J., L. Todd, and L. Hoffman-Goetz. 2011. “Comparing Written and Oral Measures of Comprehension of Cancer Information by English-as-a-Second-Language Chinese Immigrant Women.” Journal of Cancer Education 26: 484–89. https://doi.org/10.1007/s13187-011-0219-x.

Miller, M. J., E. J. Dewitt, M. E. McCleeary, and J. K. O’Keefe. 2009. “Application of the Cloze Procedure to Evaluate Comprehension and Demonstrate Rewriting of Pharmacy Educational Materials.” Annals of Pharmacotherapy 43 (4): 650–57. https://journals.sagepub.com/doi/abs/10.1345/aph.1l642.

Motulsky, H. 2018. “Multiple Comparisons Concepts.” In Intuitive Biostatistics: A Nonmathematical Guide to Statistical Thinking, 4th ed. Oxford University Press.

Northwestern University. 2023. Dieta Baja En Fibra. https://www.nm.org/patients-and-visitors/patient-education.

Pallant, J. 2020. SPSS Survival Manual: A Step by Step Guide to Data Analysis Using IBM SPSS. 7th ed. McGraw Hill.

Plow, M., F. Bethoux, K. Mai, and B. Marcus. 2014. “A Formative Evaluation of Customized Pamphlets to Promote Physical Activity and Symptom Self-Management in Women with Multiple Sclerosis.” Health Education Research 29 (5): 883–96. https://doi.org/10.1093/her/cyu034.

Robinson, R. D. 1973. “The Cloze Procedure: A New Tool for Adult Education.” Adult Education 23 (2): 87–98. https://archive.org/details/sim_adult-education-quarterly_winter-1973_23_2.

Rodriguez-Trujillo, N. 1977. “Adaptation of Cloze Procedure to the Spanish Language.” PhD thesis, University of Chicago. https://www.proquest.com/docview/302857688.

Samoil, D., J. Kim, C. Fox, and J. K. Papadakos. 2021. “The Importance of Health Literacy on Cancer Clinical Outcomes: A Scoping Review.” Annals of Cancer Epidemiology 5 (3). https://doi.org/10.21037/ace-20-30.

Stormacq, C., S. Van den Broucke, and J. Wosinski. 2019. “Does Health Literacy Mediate the Relationship Between Socioeconomic Status and Health Disparities? Integrative Review.” Health Promotion International 34 (5): e1–17. https://doi.org/10.1093/heapro/day062.

Sudore, R. L., and D. Schillinger. 2009. “Interventions to Improve Care for Patients with Limited Health Literacy.” Journal of Clinical Outcome Management 16 (1): 20–29. https://pmc.ncbi.nlm.nih.gov/articles/PMC2799039/.

Taylor, W. L. 1953. “‘Cloze Procedure’: A New Tool for Measuring Readability.” Journalism Quarterly 30 (4): 415–33. https://doi.org/10.1177/107769905303000401.

Taylor, W. L. 1957. “‘Cloze’ Readability Scores as Indices of Individual Differences in Comprehension and Aptitude.” Journal of Applied Psychology 41 (1): 19–26. https://doi.org/10.1037/h0040591.

Temoltzin-Espejel, M. L. 2025. How to Use Interrogatives in Spanish? MangoLanguages.com. https://mangolanguages.com/resources/learn/grammar/spanish/how-to-use-interrogatives-in-spanish.

Thomas, J. D., and B. J. Cardinal. 2021. “Health Science Knowledge Translation: Critical Appraisal of Online Physical Activity Promotion Material.” Nursing and Health Sciences 23 (3): 742–53. https://doi.org/10.1111/nhs.12864.

Thomas, J. D., C. N. Christopher, C. N. Smith, W. Kennedy, and B. J. Cardinal. 2023. “Improving Communication in Kinesiology Through a Practicum Course.” International Journal of Kinesiology in Higher Education 7 (3): 257–68. https://doi.org/10.1080/24711616.2022.2141156.

Thomas, J. D., A. Y. Uwadiale, and N. M. Watson. 2021. “Towards Equitable Communication of Kinesiology: A Critical Interpretive Synthesis of Readability Research.” Quest 73 (2): 151–69. https://doi.org/10.1080/00336297.2021.1897861.

Thomas, J. D., J. C. Wong, R. F. Hockert, Y-S. Wu, and W. Kennedy. 2025. “It Is Not Clear How Well Physical Activity Promotion Materials Are Understood: Findings from a Rapid Literature Review.” Topics in Exercise Science and Kinesiology 6 (1): 6. https://digitalscholarship.unlv.edu/scholarship_kin/vol6/iss1/6/.

Thomas, J. D., J. C. Wong, R. F. Hockert, Y-S. Wu, S. R. Martin, and Z. Zenko. 2024. “Comprehension of Physical Activity Promotion Material: A Retrospective and Comparative Analysis.” Journal of Kinesiology and Wellness 13 (1): 22–35. https://jkw.wskw.org/index.php/jkw/article/view/135.

Thomas, J. R., P. E. Martin, J. L. Etnier, and S. J. Silverman. 2023. “Statistical Issues in Research Planning and Evaluation.” In Research Methods in Physical Activity, 8th ed. Human Kinetics.

Vallance, J. K. H., K. S. Courneya, R. C. Plotnikoff, Y. Yasui, and J. R. Mackey. 2007. “Randomized Controlled Trial of the Effects of Print Materials and Step Pedometers on Physical Activity and Quality of Life in Breast Cancer Survivors.” Journal of Clinical Oncology 25 (17): 2352–59. https://ascopubs.org/doi/10.1200/JCO.2006.07.9988.

Vaske, J. J. 2019. “Hypothesis Testing and Effect Size.” In Survey Research and Analysis, 2nd ed. Sagamore-Venture.

Vaske, J. J., J. A. Gliner, and G. A. Morgan. 2002. “Communicating Judgments about Practical Significance: Effect Size, Confidence Intervals, and Odds Ratios.” Human Dimensions of Wildlife 7 (4): 287–300. https://doi.org/10.1080/10871200214752.

Vaughn, B. K., and S. R. Daniel. 2012. “Conceptualizing Validity.” In Measurement in Sport and Exercise Psychology, edited by G. Tenebaum, R. C. Eklund, and A. Kamata. Human Kinetics.

Williams, S. R., O. Ari, and N. C. Santamaria. 2011. “Measuring College Students’ Reading Comprehension Ability Using Cloze Tests.” Journal of Research in Reading 34 (2): 215–31. https://doi.org/10.1111/j.1467-9817.2009.01422.x.

Yuen, E. Y., M. Thomson, and H. Gardiner. 2018. “Measuring Nutrition and Food Literacy in Adults: A Systematic Review and Appraisal of Existing Measurement Tools.” HLRP: Health Literacy Research and Practice 2 (3). https://doi.org/10.3928/24748307-20180625-01.

Zhu, W. 2012. “Sadly, the Earth Is Still Not Round (\(p\) < .05).” Journal of Sport and Health Science 1 (1): 9–11. https://doi.org/10.1016/j.jshs.2012.02.002.

Supplemental Content

Supplemental Content #1. Study administration materials (script, cloze assessments, open-response questionnaires). Available at: https://jkw.wskw.org/public/176/supplement1.pdf