Applying an Expanded Set of Cognitive Design Principles to Formatting the Kidney Early Evaluation Program (KEEP) Longitudinal Survey

Article Outline

• Abstract
• Methods
  • Assessment of KEEP Follow-up Form Adherence to Cognitive Design Principles and Readability
  • Qualitative Assessment of KEEP Surveys
• Results
  • KEEP Follow-up Form Cognitive Design Assessment
  • KEEP Follow-up Form Readability Assessment
  • Qualitative Assessment
  • Development of the KEEP LS
• Discussion
• Acknowledgements
• Supplementary data
• References
• Copyright

Background

The National Kidney Foundation Kidney Early Evaluation Program (KEEP) is a free community-based health-screening program targeting populations at greatest risk of chronic kidney disease (CKD), those with high rates of diabetes and hypertension, and a high proportion of racial/ethnic minorities. The KEEP Longitudinal Survey will adopt methods similar to those used in KEEP to gather follow-up data to measure CKD-related heath status and gauge program effectiveness for repeated KEEP participants with evidence of CKD stages 3 to 5. KEEP has defined objectives to enhance follow-up survey response rates and target vulnerable populations who bear the greatest CKD risk-factor burdens.

Methods

The KEEP Follow-up Form was assessed for adherence to 6 cognitive design principles (simplicity, consistency, organization, natural order, clarity, and attractiveness) considered to summate the techniques guiding good survey development and for the additional cognitive design principles of readability and variation of readability across survey items.

Results

The KEEP Follow-up Form was found to include violations of each cognitive design principle and readability principle, possibly contributing to item nonresponse and low follow-up rates in KEEP. It was revised according to empirically substantiated formatting techniques guided by these principles and found during qualitative assessment to be more user friendly, simpler, better organized, more attractive, and easier to read. Subsequent development of the KEEP Longitudinal Survey form also was guided by these principles.

Conclusion

To ensure ease of use by populations with limited literacy skills, poor health literacy, and limited survey literacy, survey researchers must apply cognitive design principles to survey development to improve participation and response rates.

Index Words: Chronic kidney disease, cognitive design principles, readability, survey format, survey methods

Health-related surveys vary in formatting and ease of use. Choice of formatting technique usually is based on investigator preference and experience. The formatting goal is to produce surveys that are easy to comprehend, navigate, and respond to regardless of whether self-administered or administered orally. Thus, formatting choices must take into consideration social characteristics of the population to be studied, including educational attainment, literacy skills, disease burden, and cognitive functioning.¹ On the receiving end, respondents read or hear survey questions and process the content in the context of memory and experience to formulate responses. Therefore, responding to surveys is a cognitive exercise, one that is apt to be performed better by respondents who comprehend the purpose of the survey and the items, are healthy, and have higher educational attainment and broader literacy skills.² This is particularly important for patients with chronic kidney disease (CKD) or one or both of its 2 main risk factors, diabetes and hypertension. These patients are at risk of experiencing cognitive decline and decreased literacy skills, making reading health information and responding to surveys challenging.³, ⁴

Formatting techniques used in developing health-related surveys are well documented and were empirically tested by social scientists. Six cognitive design principles were proposed as summating the techniques that guide the development of user-friendly surveys: simplicity, consistency, organization, natural order, clarity, and attractiveness.⁵ The Charles Drew University Biomedical Research Center, Los Angeles, CA, expanded on these by adding 2 components of survey readability: readability of individual survey items and variation in the readability of items across a survey. Assessing readability is an important step in adapting and designing health-related surveys for use with vulnerable populations who tend to have limited literacy skills.⁶ Readability refers to the semantic and syntactic attributes of text. It determines the relative utility of text for persons with varying degrees of reading skill.⁷ Readability of text can be estimated by using one of many readability formulas based on the number of syllables per word and number of words per sentence to estimate the reading skill level needed to decipher and comprehend the text.⁸ Word-reading difficulty and sentence length were found to be the best predictors of text readability. More polysyllabic words and longer sentences are more difficult to read. Desired readability for persons with limited literacy skills is a score of fifth grade level or less, measured using the Flesch-Kincaid Grade Level (F-K) formula, or a score of 80 or higher measured using the Flesch Reading Ease (FRE) formula.

The cognitive design principles of readability of items and variation of readability across items are of particular importance when developing surveys for use in vulnerable populations. Vulnerable populations are more likely to have limited literacy skills and limited health literacy, particularly the elderly and racial/ethnic minorities, who also experience disparities in chronic disease prevalence.⁹, ¹⁰, ¹¹ They bear the largest burden of chronic disease compared with the general population. Therefore, surveys designed to measure behavior and health status over time must take into account limitations in cognition inherent to long-standing chronic disease and aging that influence literacy skills. CKD is especially pertinent in this regard because its 2 main risk factors, diabetes and hypertension, are themselves chronic diseases that are pandemic and contribute to overall cognitive decrease and declining literacy skills.

The Kidney Early Evaluation Program (KEEP), a National Kidney Foundation program, is a free community-based health screening program enrolling individuals 18 years and older with diabetes, hypertension, or a family history of kidney disease, diabetes, or hypertension. All program participants are volunteers. Since the program was launched in August 2000, more than 90,000 participants were screened by 47 National Kidney Foundation affiliates in 49 states and the District of Columbia. The KEEP Follow-up Form is used to assess how participants experienced the screening, what they learned about their health, whether they followed up with a physician visit, and, if so, what health issues were discussed. The KEEP Longitudinal Study will adopt similar survey methods to gather follow-up data to measure CKD-related health status and gauge program effectiveness for repeated KEEP participants with evidence of CKD stages 3 to 5.

Applying cognitive design principles to formatting health-related surveys has 3 objectives: (1) diminish common navigation errors, (2) minimize the administrative burden and cognitive demands on respondents, and (3) increase the ease of negotiating and responding to a survey.¹², ¹³, ¹⁴ Because applying cognitive design principles to developing health-related surveys was shown to diminish item nonresponse, these principles were used to assess the KEEP Follow-up Form and begin development of the KEEP Longitudinal Survey (KEEP-LS). Applying cognitive design principles to developing the KEEP-LS is an iterative and ongoing process. The present version will be field tested as part of the KEEP-LS and likely will be revised as we gain experience in its use and data-collection capability in the context of vulnerable populations.

KEEP targets communities at high risk of CKD and its risk factors. It has detected greater rates of CKD risk factors in targeted communities than in the general population, establishing this approach as justified and productive.¹⁵, ¹⁶, ¹⁷ KEEP Longitudinal Study expands the number of study communities and adds an educational component for providers and program participants.¹⁸ KEEP Longitudinal Study will identify individuals from previous KEEP programs with evidence of CKD stages 3 to 5 and enroll them for long-term participation in a study designed to evaluate the effectiveness of educational programs in improving the process of care and clinical outcomes. Educational programs will be aimed at participants and their health care providers. Survey use is important to this study to accrue baseline and follow-up data. The purpose of this report is to describe the methods used to assess the format design of the KEEP Follow-up Form, measured by using 8 cognitive design principles; develop the KEEP-LS based on assessment of the KEEP Follow-up Form; and qualitatively validate modifications made to the KEEP Follow-up Form in developing the KEEP-LS.

Back to Article Outline

Methods 

We applied these steps to the original KEEP Follow-up Form (version 1 [v1]) to develop the KEEP-LS: (1) assessment of how well v1 adhered to cognitive design principles; (2) assessment of v1 readability at the Charles R. Drew Biomedical Research Center; (3) development of the KEEP Follow-up Form v2 based on steps 1 and 2; (4) conducting cognitive interviews to comparatively assess comprehension, perceived ease of use, and cultural appropriateness of KEEP Follow-up Form v1 and v2; (5) KEEP Follow-up Committee review and revision of v2 to develop KEEP Follow-up Form v3 based on information gained from reports of the Drew Biomedical Research Center; (6) independent assessment of the KEEP Follow-up Form v3 for adherence to the expanded set of cognitive design principles; and (7) development of the KEEP-LS (v1 to v3) at the Charles Drew University Center for Cross-cultural Epidemiologic Studies.

These steps emphasize the iterative nature of methods used to develop the present field version of the KEEP-LS. Moreover, the original version of the KEEP Follow-up Form (v1) underwent 2 independent reviews by the Drew Research Centers in Minority Institutions and the Drew Center for Health Services Research. Reviews focused on the assessment of overall content, language, readability, and format. Based on consensus between reviewers, revisions were made to develop the KEEP Follow-up Form v2 and v3.

The field version of the KEEP-LS also was culturally adapted into Spanish by using rigorous criteria. Discussion of this aspect of the survey development is beyond the scope of this report. However, Fig 1 shows methods for the linguistic and cultural adaptation (including language) of surveys, with a focus on their application to formatting, that were developed at the Drew Research Centers in Minority Institutions.

• 
  • View Large Image
  • Download to PowerPoint

• Figure 1. 

  Charles Drew University protocol for the linguistic and cultural adaptation of surveys. RGL, reading grade level.

Assessment of KEEP Follow-up Form Adherence to Cognitive Design Principles and Readability 

The KEEP Follow-up Form v1 has 14 numbered items consisting of 18 closed-ended and 1 open-ended question. We assessed KEEP Follow-up Form v1 according to the 7 cognitive design principles listed in Table 1. The F-K and FRE readability formulas were used to estimate the readability of items in KEEP Follow-up Form v1 and items developed for the KEEP-LS. The F-K formula rates text on a US grade-school level such that the average eighth grader would be able to read a document that scores 8.0. Scores generated by the F-K formula highly correlated with scores from other commonly used readability formulas.⁸ The FRE formula rates text on a 100-point scale; the higher the score, the easier the document is to read. Both formulas generate scores based on the average number of syllables per word and number of words per sentence. Correspondence between the scores for these 2 methods and the reading difficulty rating for the scores are listed in Table 2.

Table 1. Description of Cognitive Design Principles

Table 2. Reading Difficulty Rating of Flesch Reading Ease Scores and Flesh-Kincaid Grade Level Scores

Because the readability estimate for a passage is equivalent to the average of the readability of its component sentences, we used the F-K and FRE formulas to assess the readability of single items, as well as the survey as a whole. We selected these formulas because they are available in Microsoft Word (Microsoft Corp, Redmond, WA) and therefore are readily available to nearly all investigators interested in assessing text and survey readability. Moreover, use of software decreases the amount of work required to produce readability estimates, eliminates human error inherent in manual calculation, and requires little training.¹⁹

Qualitative Assessment of KEEP Surveys 

Cognitive (intensive) interviews were conducted with 8 participants using KEEP Follow-up Form v3: 5 Hispanics (3 men, 2 women) aged 43 to 67 years and 3 African Americans (1 man, 2 women) aged 50 to 82 years. Four participants had a high school education or equivalency, 2 had some college, and 2 had college degrees (both African American). None of the women reported having diabetes, hypertension, or kidney disease. One Hispanic man had uncontrolled hypertension despite medication, 1 had diabetes treated by diet, and 1 had diabetes and hypertension treated with insulin and antihypertensive medication. All Hispanic men had moderate to severe central obesity and admitted to being overweight. One African-American man had diabetes and kidney disease. All Hispanic participants were fully bilingual.

A cognitive interview script was constructed and used by 2 ethnically matched interviewers (Table 3). Interviews lasted 40 to 60 minutes. The first set of items in the script queried perceptions about the original KEEP Follow-up Form. The last items queried perceptions about the first revision (v2) of the KEEP Follow-up Form. Participants were asked to compare them for comprehension, ease of reading, and preferred format. Comments, opinions, and perception of the KEEP Follow-up Form (v2) were clustered and reported as an item-by-item synopsis to the KEEP Follow-up Committee.

Table 3. KEEP Follow-up Form Cognitive Interview Script

1. What do you think this question is asking?
2. What does this question mean to you?
3. Are the words too technical?	(a) Yes.….How?
	(b) No
4. Should the wording be changed?	(a) Yes.….How? (b) No
5. How would you ask this question?
6. Do you speak Spanish?
7. How would you ask this question in Spanish?
8. Please read the answer choices after this question. Are they understandable?	(a) Yes
	(b) No.….How?
Please look at the form itself. Look at how the questions are written on the pages.
9. Is there too much information on the page?	(a) Yes…. How? (b) No
10. Is the writing easy or hard to read?	(a) Easy (b) Hard…. How?
11. Is there enough space between questions and answers?
12. If you had to change the form, what would you change?
Interviewer: Show the participant the KEEP Follow-up Form v1 and v2.
13. Please compare these 2 surveys. Which of the 2 forms looks easier to read? Why?
14. Please compare these 2 surveys. Which of the 2 forms is easier to read? Why?
Do you have anything else you’d like to say about the KEEP surveys?

Abbreviations: KEEP, Kidney Early Evaluation Program.

Back to Article Outline

Results 

KEEP Follow-up Form Cognitive Design Assessment 

For purposes of brevity, we report only results of our assessment of adherence to cognitive design principles, quantitative assessment of readability, and results of cognitive interviews for the first page of the KEEP Follow-up Form v1 (Fig 2) and KEEP-LS v3 (Fig 3). However, KEEP Follow-up Form v3 and KEEP-LS v3 appear in their entirety as online supplementary materials available at www.ajkd.org to allow readers to better understand how the iterative nature of our methods resulted in the survey’s evolution and as a tool for further study.

• 
  • View Large Image
  • Download to PowerPoint

• Figure 2. 

  Formatting and readability assessment of Kidney Early Evaluation Program (KEEP) Follow-up Form version 1, page 1. The Flesch-Kincaid Grade Level (F-K) formula rates text on a US grade-school level such that the average eighth grader would be able to read a document that scores 8.0. The Flesch Reading Ease (FRE) formula rates text on a 100-point scale; the higher the score, the easier the document is to read.

• 
  • View Large Image
  • Download to PowerPoint

• Figure 3. 

  KEEP Follow-up Survey v2, Part A, page 1.

Overall, KEEP Follow-up Form v1 had deficiencies in each of the 7 cognitive design principle categories (Table 4). As part of the iterative process in the development of the KEEP-LS, adherence to these cognitive design principles was assessed in each step of survey development. For example, the KEEP Follow-up Committee revised v2 and developed v3 based on information gained from the report of the Drew Research Centers in Minority Institutions research group. KEEP Follow-up Form v3 then was independently assessed for adherence to cognitive design principles. This showed that v3 formatting was still difficult to negotiate because it used too many text boxes, making it visually challenging and increasing cognitive demand. It also contained confusing instructions, further adding to cognitive demand when responding (see supplementary materials). In addition, assessment showed that readability of v3 item 13 was reading grade level 12, measured by means of the F-K method; this is considered difficult to read. Thus, this iterative process represents a quality control measure that ensures an end product that will have the greatest utility for gathering valid health-related information from populations with cognitive decrease and limited literacy skills and for diminishing item and survey nonresponse.

Table 4. Assessment of Adherence to Cognitive Design Principles: Kidney Early Evaluation Program Follow-up Form Version 1, Page 1

KEEP Follow-up Form Readability Assessment 

The readability of many KEEP Follow-up Form v1 items was at or less than the desired item readability of 5 or less. However, the readability of many items was considered difficult (Fig 2). Assessing readability across items showed marked variation. Figure 4 shows the variation in readability of items on KEEP Follow-up Form (v1) page 1 compared with KEEP-LS (v3) page 1. Items with difficult readability were simplified by using the for or nor but and yet so method (which reduces long sentences to short simple sentences) developed at the Drew Center for Health Services Research.²⁰

• 
  • View Large Image
  • Download to PowerPoint

• Figure 4. 

  Variation in readability of Kidney Early Evaluation Program (KEEP) Follow-up Form (KEEP-F/U) version 1 (page 1, items 1a, 1b, 1c, 1d, 2, 2b, 3, and 4) and KEEP-Longitudinal Survey (KEEP-LS) version 3 (page 1, items 1 to 6). Abbreviation: F-K, Flesch-Kincaid Grade Level formula.

Qualitative Assessment 

The cognitive interviews validated the initial assessment of KEEP Follow-up Form v1. In summary, v1 did not offer skip patterns and items were not ordered in a way that would help respondents answer questions in logical succession and avoid items that may not be relevant to them. There was consensus agreement across the 2 participant groups that having to negotiate items not relevant to them “is a waste of time,” “can be frustrating,” and may cause participants to stop answering survey questions. Other comments indicated that items and response options were crowded, numbering patterns for questions and response options were confusing, many questions were too long and considered likely to be hard to read for the average person, and the use of technical terms was frustrating. There also was consensus agreement that this version of the survey was not easy to use. Table 5 lists additional commentary and verbatim responses from cognitive interview participants for the first 4 items of KEEP Follow-up Form v1 that are representative of comments made for other items. When asked to compare KEEP Follow-up Form v1 with the KEEP Follow-up Form v2, there was consensus agreement among all participants that the KEEP Follow-up Form v2 was easier to use, better organized, simpler, easier to read, and more attractive (Fig 3).

Table 5. Summary of Cognitive Interview Comments for KEEP Follow-up Form Version 1, Items 1 to 4

Item No.	Comments
1,2	The meaning of “feel” and “glad” in the context of a question about health was unclear to some. One participant suggested merging items 1a and 1b into one by using “satisfied” instead of “glad.” The term “health care provider” was defined in a variety of ways: doctor, HMO, hospital. When asked for an alternative, “doctor” was the preferred term. The alphanumeric designations for items were confusing. Item 2 was considered difficult to comprehend by several participants. “I had to read it more than once.” “Too long and repetitive.” “Most people may not understand this.” Questions within questions were mentioned. “Questions should have separate numbers.” “If I answered no to 2a, why do I have to answer 2b?”
3	Most participants considered the question too long and not clear. “Are you asking about KEEP or a doctor?” “Should be asked differently.” “Make question clear.” “If I don’t have any health problems, why should I answer all these questions?” Two participants astutely suggested that they couldn’t see the point of the question because answering would not tell you if the answer is about KEEP or the provider. The words triglycerides, urinary tract infections, phosphorous, or parathyroid could not be defined by all. “I have no idea.” These terms are used in other items in the KEEP Follow-up Form. Calcium was recognized as a common word: “It’s in milk,” “For the bones.”
4	Participants could not define estimated glomerular filtration rate or creatinine. They were considered too technical. Common questions were “What’s creatinine?” and “Glomerular what?” or “I have no idea what this means.”

Abbreviations: KEEP, Kidney Early Evaluation Program; HMO, health maintenance organization.

Development of the KEEP LS 

Applying expanded cognitive design principles used at the Charles Drew University Biomedical Research Center resulted in the development of KEEP Follow-up Form v2, which was preferred by participants during qualitative assessment. Skip patterns were added with simple graphics to help respondents navigate the survey. However, skip patterns were kept to a minimum and used only twice in the KEEP Follow-up Form v2 (Fig 3) because they have the potential to add to cognitive demand. The KEEP Follow-up Form v2 was reviewed and modified by the KEEP Follow-up Committee. The resulting version (v3) served as the basis for developing the KEEP-LS. The KEEP-LS has 21 closed-ended items and 1 open-ended item that preserved the intent of the KEEP Follow-up Form to gain information about participant perceptions of the program and its impact on their health care–seeking behavior. Importantly, it also preserved the intent of the follow-up form to gain information about the process of care as it relates to CKD and CKD risk factor screening and treatment.

Back to Article Outline

Discussion 

An expanded set of cognitive design principles that includes 2 domains of readability is an important contribution to the survey methods literature. Applying cognitive design principles in formatting the KEEP-LS has the potential to improve item response and diminish survey nonresponse, which has been a challenge to the program. Importantly, the iterative process used in developing the KEEP-LS is a quality-control measure that ensures that investigators adhere to cognitive design principles in formatting health-related surveys. For example, in response to the violations of each of the 7 cognitive design principles in the KEEP Follow-up Form, the KEEP-LS was formatted into 4 main categories (A, KEEP Screening; B, Doctor’s Visit; C, Medicines and Care; and D, Follow-up) that likely will promote user friendliness and diminish cognitive demand (see supplementary materials). In addition, the difficult-to-read item 13 on KEEP Follow-up Form v3 was replaced with 3 short educational sentences that have a combined readability of reading grade level 4 and serve to introduce the last 4 items in the KEEP-LS. This is an innovative approach that not only prepares participants to respond to the last 4 items of the survey, but also may contribute to enhancing their CKD health literacy. (Compare Follow-up Form v3 item 13 with sentences introducing items 19 to 22 of KEEP-LS v3 in the supplementary materials online.)

Surveys designed using these principles may mitigate category fallacy and inaccurate responses by making the survey more user friendly. This is especially important because the KEEP program reaches out to the most vulnerable populations, such as racial/ethnic minorities and the elderly, who are at greatest risk of CKD and may have limited literacy skills and limited survey literacy (limited experience in negotiating and completing surveys).

The purpose of this report was to convey how best to format surveys for vulnerable populations by using cognitive design principles. One limitation of this report is that the psychometric properties of the KEEP-LS were not tested. However, we currently are collecting data to test the instrument’s construct validity and internal consistency reliability. Of interest will be the tally of missing responses from vulnerable populations. To date, most KEEP participants were educated and employed and tended to have health insurance; however, follow-up response rates were less than expected. We hypothesize that it will be more effective at accruing robust data, measured by diminished item and survey nonresponse. Moreover, the Spanish version of the KEEP-LS is undergoing further qualitative evaluation by using focused group discussions with Spanish-only speakers before it is field tested in Latino communities.

By applying cognitive design principles to formatting health-related surveys, researchers may increase the likelihood that participants from all walks of life with differing levels of educational attainment, literacy skills, health literacy, and survey literacy will be able to more easily navigate the surveys. This is crucial for enhancing our understanding of how better to improve preventative care and promote compliance with care in populations at high risk of CKD.

Back to Article Outline

Acknowledgements 

The authors thank Edward Constantini, MA, Shane Nygaard, BA, and Nan Booth, MSW, MPH, of the Chronic Disease Research Group for figure preparation, manuscript preparation, and manuscript editing, respectively.

Support: The Kidney Early Evaluation Program is sponsored by the National Kidney Foundation Inc and supported by Amgen, Abbott, Genzyme, Ortho Biotech Products LP, and Novartis, with additional support provided by Siemens Medical Solutions Diagnostics, Lifescan, Suplena, and OceanSpray Cranberries. Additional support was provided by National Institutes of Health grants RR019234 and MD00148 (JC, KN); RR03026, RR011145, P30AG21684, and RR014616 (JC, EF, KN); Agency for Healthcare Research and Quality grant 1R24-HS014022-01A1 (JC), and CMS grant 1H0CMS300041 (JC).

Financial Disclosure: Dr Vassalotti reports having received grant support from the Centers for Disease Control and Prevention. The authors have no conflicts of interest with its subject matter.

Back to Article Outline

Supplementary data 

Back to Article Outline

References 

1. Sudman S, Bradburn N, Schwarz S. Thinking About Answers: The Application of Cognitive Processes to Survey Methodology. San Francisco, CA: Jossey-Bass; 1995;
   • View In Article
2. Aday L. Designing and Conducting Health Surveys. San Francisco, CA: Jossey-Bass; 1996;
   • View In Article
3. Calderon JL, Zadshir A, Norris K. Structure and content of chronic kidney disease information on the World Wide Web: Barriers to public understanding of a pandemic. Nephrol News Issues. 2004;18:76;78-79, 81-84
   • View In Article
   • MEDLINE
4. Calderon JL, Zadshir A, Norris K. A survey of kidney disease and risk-factor information on the World Wide Web. MedGenMed. 2004;6:3
   • View In Article
   • MEDLINE
5. Mullin PA, Lohr KN, Bresnahan BW, McNulty P. Applying cognitive design principles to formatting HRQOL instruments. Qual Life Res. 2000;9:13–27
   • View In Article
   • MEDLINE
   • CrossRef
6. Calderon JL, Morales LS, Liu H, Hays RD. Variation in the readability of items within surveys. Am J Med Qual. 2006;21:49–56
   • View In Article
   • MEDLINE
   • CrossRef
7. Chall J: The beginning years, in Zakaluk B, Samuels S (eds): Readability: Its Past, Present, and Future. Newark, DE, Sage Publications, 1988, pp 2-13
   • View In Article
8. Meade CD, Smith CF. Readability formulas: Cautions and criteria. Patient Educ Counsel. 1991;17:153–158
   • View In Article
9. Kirsh I, Jungeblut A, Jenkins L, Kolstad A. Adult Literacy in America: A First Look at the Results of the National Literacy Survey. Washington, DC: US Department of Health, Education and Welfare; 2007;
   • View In Article
10. Williams MV, Parker RM, Baker DW, et al. Inadequate functional health literacy among patients at two public hospitals. JAMA. 1995;274:1677–1682
    • View In Article
    • MEDLINE
11. Murphy T, Davis T, Jackson R, Decker B, Long S. Effects of literacy on health care of the aged: Implications for health care professionals. Educ Gerontol. 1993;19:311–316
    • View In Article
12. Wright P, Barnard P. Asking multiple questions about several items: The design of matrix structures on application forms. Appl Ergon. 1978;9:7–14
    • View In Article
    • MEDLINE
    • CrossRef
13. Lohr KN, Aaronson NK, Alonso J, et al. Evaluating quality-of-life and health status instruments: Development of scientific review criteria. Clin Ther. 1996;18:979–992
    • View In Article
    • Abstract
    • Full-Text PDF (869 KB)
    • CrossRef
14. Dillman D, Sinclair M, Clark J. Effects of questionnaire length, respondent-friendly design, and a difficult question on response for occupant-addressed Census mail surveys. Public Opin Q. 1993;57:289–304
    • View In Article
    • CrossRef
15. McGill JB, Brown WW, Chen SC, Collins AJ, Gannon MR. Kidney Early Evaluation Program (KEEP) (Findings from a community screening program). Diabetes Educ. 2004;30:196–206
    • View In Article
    • MEDLINE
    • CrossRef
16. Brown WW, Collins A, Chen SC, et al. Identification of persons at high risk for kidney disease via targeted screening: The NKF Kidney Early Evaluation Program. Kidney Int Suppl. 2003;83:S50–S55
    • View In Article
17. Brown WW, Davis J, Flack JM, Gannon MR. NKF’s KEEP leads to [longitudinal study]. Nephrol News Issues. 2002;16:84–86
    • View In Article
    • MEDLINE
18. Ohmit SE, Flack JM, Peters RM, Brown WW, Grimm R. Longitudinal study of the National Kidney Foundation’s (NKF) Kidney Early Evaluation Program (KEEP). J Am Soc Nephrol. 2003;14(suppl 2):S117–S121
    • View In Article
    • MEDLINE
19. Mailloux SL, Johnson ME, Fisher DG, Pettibone TJ. How reliable is computerized assessment of readability?. Comput Nurs. 1995;13:221–225
    • View In Article
    • MEDLINE
20. Calderon JL, Smith S, Baker R. A simple method for enhancing readability of patient information. Ann Behav Sci Med Educ. 2007;14:20–24
    • View In Article