Adapted Fresno Test

Occupational Therapists: (McCluskey & Bishop, 2009; n = 106; all female OTs; five years post-graduation 80%; did not hold a postgraduate qualification 68%; employed full-time 77%; baseline pretest score of ≤ 78/156 points classified participant as a low scorer (n = 83) and a baseline pretest score  > 78/156 points classified participant as a high scorer (n = 23); posttest AFT occurred after 2-day EBP workshop)

• Based on the authors’ expert opinion, participants scoring ≤ 78/156 points were classified as low scorers and participants scoring > 78/156 points classified as high scorers.

Occupational Therapists: (McCluskey & Bishop, 2009; following EBP training; n = 106; mean score ± SD)

Total (0-156)

57.7 (27.3)

78.3 (18.6)

 

 

Low scorers – participants scoring below 78/156 preworkshop (n=83)

 

 

 

 

Q1

7.0 (2.9)

9.7 (1.6)

 

 

Q2

13.6 (6.4)

15.2 (4.3)

 

 

Q3

3.6 (5.6)

11.9 (4.7)

 

 

Q4

7.9 (6.8)

13.4 (6.1)

 

 

Q5

6.3 (5.3)

4.8 (4.3)

 

 

Q6

7.5 (7.2)

10.6 (7.3)

 

 

Q7

1.2 (2.5)

9.1 (9.1)

 

 

Total

47.4 (19.8)

74.1 (17.7)

 

 

High scorers– participants scoring 78/156 or higher preworkshop (n=23)

 

 

 

 

Q1

9.5 (1.9)

10.2 (1.5)

 

 

Q2

20.9 (2.8)

18.6 (2.9)

 

 

Q3

12.7 (7.2)

13.3 (3.8)

 

 

Q4

15.1 (5.2)

18.8 (3.8)

 

 

Q5

12.3 (4.6)

6.9 (4.8)

 

 

Q6

18.1 (5.3)

14.9 (7.3)

 

 

Q7

6.6 (5.9)

10.7 (5.0)

 

 

Total

95.2 (14.4)

93.3 (13.7)

 

 

Occupational Therapists: (McCluskey & Bishop, 2009; raters were two OT lecturers with expert EBP knowledge)

• Excellent inter-rater reliability; test version 1 (n = 10) (ICC = 0.96)
• Excellent inter-rater reliability; test version 2 (n = 10) (ICC = 0.91)

AFT rater training consisted of a 2-hour training session utilizing scored and unscored copies of the test, after which raters had two weeks to practice rating 20 tests with the scoring matrix (McCluskey & Bishop, 2009).

 

 

Occupational Therapists and Physical Therapists: (Lizarondo et al.,2013; n = 55; OTs and PTs; bachelor’s degree 62%; completed postgraduate degree 38%; less than 50% had prior EBP training or participation in research; participants took the AFT one time; raters consisted of four PTs with varying levels of professional experience; Rater 1 - master’s in sport and musculoskeletal PT, two years outpatient and hospital experience, one year EBP-related research experience, no teaching experience or other professional qualifications; Rater 2 - master’s in manual and sport PT, five years outpatient experience, 2.5 years EBP-related research experience, occasional [clinical demonstration] teaching experience, level 2 sports trainer certification; Rater 3 - PhD candidate and health-related master’s in PT and clinical psychology, internship experience only, seven years EBP-related research experience, 16 year undergraduate and one year postgraduate teaching experience; Rater 4 - PhD candidate and master’s in PT, two years hospital experience, 8.5 years EBP-related research experience, 14 years undergraduate and nine years postgraduate teaching experience, director of research center for five years)

Reliability Estimates for Inexperienced and Experienced raters

 

ICC (95% CI)

Adapted Fresno Test item

Inexperienced*

Experienced†

2. Where might you find answers to these questions? Name as many possible sources of information as you can. List advantages and disadvantages.

0.96 (0.86–0.99)

0.97 (0.86–0.99)

3. What type of study (design) would best answer your clinical question and why?

0

0.94 (0.80–0.98)

4. Describe the search strategy you might use in Medline topics, fields, rationale and limits.

0.66 (−0.04 to 0.90)

0.70 (0.06–0.91)

5. What characteristics of a study determine if it is relevant?

0.24 (−0.34 to 0.70)

0.95 (0.83–0.99)

6. What characteristics of a study determine its validity?

0.70 (−0.02 to 0.91)

0.98 (0.92–0.99)

Total score

0.58 (−0.26 to 0.88)

0.92 (0.72–0.98)

*Inexperienced: Raters 1 and 2.

†Experienced: Raters 3 and 4.

 

Dieticians/Nutritionists, Social Workers, and Speech Pathologists: (Lizarondo et al., 2014; speech pathologists (SP), n = 10; social workers (SW), n = 16; dietician/nutritionists (DN), n = 12; bachelor’s degree 50%; postgraduate degree 50%; previous EBP training or research < 50%; majority in clinical practice more than 10 years; participants took the AFT one time; raters were four individuals experienced in research and teaching EBP to allied health students)

• Excellent SP inter-rater reliability (ICC = 0.93)
• Excellent SW inter-rater reliability (ICC = 0.83)
• Excellent DN inter-rater reliability (ICC = 0.92)

Occupational Therapists:

• Adequate to Excellent internal consistency (Cronbach’s alpha = 0.72 to 0.84) (McCluskey & Lovarini, 2005; pretest with version 1, n = 114; posttest after 2-day EBP workshop with version 2, n = 106; eight month follow-up with version 3, n = 51; majority of participants held an undergraduate degree in OT)
• Adequate internal consistency (Cronbach’s alpha = 0.74) (McCluskey & Bishop, 2009)

 

Speech pathologists (Lizarondo et al., 2014)

• Adequate internal consistency (Cronbach’s alpha = 0.71)

 

Social workers (Lizarondo et al., 2014)

• Poor internal consistency (Cronbach’s alpha = 0.68)

 

Dieticians (Lizarondo et al., 2014)

• Adequate DN internal consistency (Cronbach’s alpha = 0.74)

Occupational Therapists: (McCluskey & Lovarini, 2005)

• Author McCluskey reduced the original FT questions from 12 to seven and included new clinical scenarios relevant to OTs. Questions pertaining to diagnosis or complex statistical calculations were removed.
• Six new clinical scenarios were included, two each for versions 1, 2, and 3, to minimize practice effects.
• The seven questions in the AFT were scored using standardized grading criteria, similar to those reported by Ramos et al. (2003).

Occupational Therapists: (McCluskey & Bishop, 2009)

• Due to low interrater reliability on certain subtest items in the McCluskey and Lovarini (2005) study, original FT questions were further reduced from 12 to seven. Pilot testing with six OTs and two PTs, with a range of experience and knowledge about EBP, assisted in question selection. Questions were removed that pertained to diagnosis or complex statistical calculations.
• Six new clinical scenarios were written, two each for versions 1, 2, and 3, to minimize practice effects.
• Scoring matrix was revised to include examples of “excellent,” “strong,” and “limited” responses, in addition to “not evident” responses that receive no score.

Dieticians/Nutritionists, Social Workers, and Speech Pathologists: (Lizarondo et al., 2014)

• Test modifications for the AFT were established through formal feedback from an expert panel consisting of four practitioners from each discipline. The panel represented practitioners with more than 10 years of clinical experience and previous exposure to EBP training or research. The majority had graduate degrees.
• New clinical scenarios were developed for each discipline. The scoring rubric of the AFT was considered applicable to the new versions except for questions 1, 2, and 4 that were modified slightly to include additional PICO terms or synonyms.

Occupational Therapists: (McCluskey & Bishop, 2009)

• Large effect size = 0.8

Question

 

Mean Change (95% Confidence Interval)

Effect Size*

Q1 (0-12)

2.3

(1.7 to 2.9)

0.8

Q2 (0-24)

0.7

(-0.5 to 1.9)

0.1

Q3 (0-24)

6.8

(5.2 to 8.3)

0.9

Q4 (0-24)

5.1

(3.6 to 6.5)

0.7

Q5 (0-24)

-2.4

(-3.6 to -1.1)

-0.4

Q6 (0-24)

1.7

(-0.2 to 3.6)

0.2

Q7 (0-24)

7.1

(5.3 to 8.8)

1.7

Total (0-156)

20.6

(15.6 to 25.5)

0.8

Q1

2.8

(2.1 to 3.6)

0.9

Q2

1.6

(0.1 to 3.0)

0.3

Q3

8.4

(6.8 to 10.0)

1.5

Q4

5.5

(3.8 to 7.1)

0.8

Q5

-1.5

(-2.9 to -0.1)

-0.3

Q6

3.1

(0.9 to 5.3)

0.4

Q7

7.9

(5.9 to 9.9)

3.2

Total

26.8

(21.6 to 31.9)

1.4

Q1

0.7

(-0.5 to 1.8)

-0.4

Q2

-2.4

(3.9 to -0.8)

0.9

Q3

0.7

(-2.7 to 3.9)

-0.1

Q4

3.7

(0.4 to 7.1)

-0.7

Q5

-5.4

(-7.9 to -2.9)

0.6

Q6

-7.4

(-10.5 to -4.2)

1.4

Q7

4.1

(1.3 to 6.9)

-0.7

Total

1.8

(-6.4 to 10.1)

0.1

*Effect size calculated with the use of mean change scores divided by the standard deviation of preworkshop score.