Health Promotion Graduated Project Evaluation

Based on Healthway Foundation Level 4 Programs (over $60,000)

Basic, Process, Impact and Outcome Evaluations – Level 1 </= $6.000, 2 = $6,001-15,000, 3 = $15,001-$60,000

Jim Grizzell, MBA, MA, CHES

Evaluation Type

Output Measures

Descriptions and Examples

 

Basic

 

Contractual evaluation score

 

Description given by the funding administrators.

6

The outputs of the project exceeded those required by the   health promotion (HP) plan to a very substantial degree. The project delivered outstanding value for money.

150%+

5

The outputs of the project exceeded those required by the HP plan to a very large degree. The project delivered excellent value for money.

120-149%

4

The outputs of the project were consistent with or in the vicinity of those required by the HP plan.  The project delivered good value for money.

95-119%

3

The outputs of the project fell short of those required by the HP plan, but were still within the bounds of acceptability. The project delivered marginally adequate value for money.

75-94%

2

The outputs of the project fell well short or those required by the HP plan.  The project delivered poor value for money.

50-74%

1

The outputs of the project fell short of those required by required by the HP plan to a very substantial degree. The project delivered completely unacceptable value for money.

<50%

 

 

 

 

Basic

 

 

Population measures

 

These measure direct population reach by recording the numbers of organizers, performers/players, participants and spectators participating and attending program activities and events. For a series of events, such as in a season of theater, the population measures are summed over all events in the season.  Population measures are recorded for all health projects.  Indirect population reach is covered under the heading of publicity measures.

 

 

Basic

 

Publication measures

 

Numbers of publications arising from their projects.  These are classified as scientific papers in press or published in refereed scientific journals; other reports and papers; and in-house newsletters. 

 

 

Basic

 

Publicity measures

 

These measures provide an indication of the exposure of the program’s health messages in the print media, radio and television.  For each medium, the number of occasions of coverage is recorded, distinguishing between minor and major media vehicles based on audience or circulation size.  Also recorded is the most prominence of coverage received; i.e., whether coverage was incidental, standard or a feature. 

 

 

Basic

 

 

Structural measures

Measures pertaining to the capacities, technologies, and infrastructure that make up the structure of care (e.g., management information system, number and types of staff, types of facilities

 

Seven areas of structural reform to underpin the program and promote institutionalization are sought:

1.  Mental Health - provision of resources for counseling, education and support to care for stress, sleep, relationship difficulties and attempted suicide

2.  Physical Activity & Fitness - provision of resources for exercise programming, education and exercise.

3.  Safe, legal and moderate use of alcohol - safe alcohol serving practices; and availability of low or non-alcohol beverages, food, designated drivers, friends.

4.  Responsible Sexual Behavior – resources for social norms promotion, sex educations.

5.  Nutrition - provision of low fat, fruit and vegetables, or differential pricing in favor of healthy food choices.

6.  Relationship Violence and Sexual Assault - provision of staff counselors, peer educators, plus facilities and staff for enhancing healing process of survivors.

7.  Access to Health Care - adoption of policies that encourage 1) participation of disadvantaged groups health enhancing actions, and 2) elimination of health disparities by promoting total health insurance coverage and blood pressure checks to such as student on very low incomes and people with cultural barriers.
 

Basic

 

 

Development measures

 

1. Campus Community Involvement - students or their representatives were consulted or surveyed about their needs and concerns, and ideas for action by the program staff.

2. Campus Community participation - students or their representatives actually took part in the carrying out of program staff action.

3. Community Responsibility - students or their representatives took prime responsibility for the project and its implementation.
 
Process

 

Promotional measures

 

Promotional measures:  This is the first set of output measures required only at GPE level 2 or higher.  These apply only to sponsorship projects.  Six categories of promotional benefits are recorded:

1. General profile - naming rights, presentation rights, media coverage, media advertising and event promotion.

2. Signage - perimeter signs, large signs, banners, A-frame signs, posters, bunting and flags.

3. Personal endorsement - official or participant clothing, role modeling of healthy behavior (e.g., non-smoking; exercise; moderate eating; safe, legal, moderate drinking), acknowledgement by announcer or celebrities, personal endorsement by performers.

4. Bulk materials - promotional leaflets, entry forms and admission tickets, program advertising, program editorial, distribution of campaign materials.

5. Hospitality - complimentary seating for opinion leaders, introduction to opinion leaders; association with catering.

6. Interactive activities - campaign demonstrations or displays, campaign related competitions, interactive promotional activities.

Under each of the six headings the particular types of benefit applicable to the specific project are identified and their promotional visibility as a group is scored on a three-point scale (i.e., 'low', 'medium' and 'high').

 
 

Process

 

 

Educational measures

 

These apply to health projects (at GPE levels 2 or higher) that provide classroom activities, workshops, demonstrations, health check-ups, telephone or direct counseling services.  The measures consist of the total number of participants exposed to educational activities; the average frequency of exposure; and the average duration of exposure per session.  In addition, a separate record is made of the number of teachers/leaders/coaches/trainers inserviced as part of the project and whether customized resource materials were produced.

 
 
Impact

 

Target measures

 

The requirement for a survey of those targeted (college students, possible by a demographic for a specific objective [i.e., to eliminate health disparities]) by the health interventions is introduced at GPE level 3.  According to the dollar value and type of project, the minimum permissible survey size varies between 50 and 300 subjects.  In some instances both pre-project and post-project surveys are required, as well as end of season surveys for sponsorships that extend over more than one year.

The interviewer-administered questionnaire takes an average of five minutes to complete.  The self-administered questionnaire is somewhat shorter, having several sections omitted, including the section on prompted recall of health messages.  Health projects are too diverse for a standard survey instrument, and therefore customized questionnaires are used instead.

The same target measures are obtained from all forms of the survey.  They are the numbers of participants and spectators who fell within the pre-defined target group for the health message or intervention, expressed as a proportion of the number of people surveyed.  A distinction is also made, where appropriate, between primary and secondary target groups.

 
 

Impact

 

 

Cognitive / attitudinal / knowledge measures

 

For health message projects, survey instruments are designed to collect the following six impact measures.  They represent a hierarchy of cognitive/attitudinal effects within the primary or secondary target groups (Donovan & Robinson, 1992).

1. Unprompted recognition - the number who were able to recall the project's message without any prompting.

2. Prompted recognition - the number (in 1) who were aware of the project's health message and could either recall it unprompted or recognize it from a prompt list.

3. Comprehension - the number (in 2) who understood the health message according to predefined criteria.

4. Acceptance - the number (in 3) who agreed with the health message.

5. Intention - the number (in 4) who intended to act on the health message.

6. Action - the number (in 5) who actually took some form of action as a result of the health message, including actions such as seeking further information.

The survey instruments used in the evaluation of health message projects include data on demographics, health behaviors (i.e., mental health, alcohol intake, fruit and vegetables consumption, exercise and relationship violence, access to health care, responsible sexual behavior) and cognitive questions, including salience of health messages.

The cognitive/attitudinal/knowledge measures for health education projects are as follows.

1. Knowledge - the number who correctly answered the key knowledge question (i.e., the single most important knowledge question in the survey instrument).

2. Attitude - the number who reported agreement with the key attitude question.

3. Skill - the number who possessed the key skill at the level targeted by the project.

4. Intention - the number who expressed the key behavioral intention sought by the intervention.

5. Action - the number who took some form of immediate action to improve their health or well-being as a result of the intervention.

6. Behavior - the number who reported the key behavior sought by the intervention.

Separate records are made for pre-project and post-project surveys and are based only the primary target group.

 
 
Outcome

 

Outcome report

 

The final component of GPE applies only to $60,000+ health promotion programs.  It requires the funding recipients to furnish administration with a customized quantitative report on the outputs of the intervention in terms of long-term attitudinal, perception, behavioral or biological changes in the target group.

 

Sources of Cal Poly Pomona student and university concerns and key priority health outcome areas (prevalent diseases, including chronic conditions, behavioral health, etc. are:

  • Mental health (sleep difficulties, relationship difficulties, stress, attempted suicide)
  • Physical activity and fitness
  • Nutrition
  • Responsible sexual behavior
  • Responsible, legal and safe use of alcohol
  • Sexual assault and relationship violence
  • Access to health care (insurance, blood pressure checks)

 

Important problems and concerns of the students and University are:

·         enhancement of learning and reduction of health impediments to learning,

·         CSU Chancellor and other administrative mandates (alcohol, cultural competence),

·         obligations to those pay for programs and services (VAWA grant, students, University),

·         accreditation standards to have quality improvement activities with evidence of improving quality or addressing the sources of problems and concerns (prevalent disease, chronic conditions, behavior health, etc.),

·         health risk areas with the largest gaps between our campus and national, and Healthy Campus 2010 baseline data and targets.

 

The student learning outcomes resulting from health education programs and to be measured are to help students:

  • integrate the information that they observe and analyze in academic courses by application in daily life (i.e., decision making, relationships, etc.)
  • develop a healthy sense of identity, self-awareness, and self-confidence
  • identify individual accomplishments that they believe contribute to their self-esteem.
  • Incorporate healthy lifestyle choices in their daily living.

 

 
 

Outcome

 

 

Outcome report

 

Additional requirements for the outcome report.

Along with the data supplied in the statistical evaluation forms, grant recipients are required to provide a qualitative report on their project.  This describes the project performance in terms of the originally proposed objectives, strategies, target group, time frame, budget and inter-agency relations.  Detailed financial statements, samples of any resource materials produced, copies of media coverage and completed standard survey questionnaires also are submitted with the final report.

With the exception of the outcome reports, the nine sets of GPE output measures are entered into a project electronic data file, together with supplementary data obtained mainly from the original grant application.  The supplementary variables include project ID and type (if given); amount of grant; grant recipient type; health message(s); geographic location; targeted age group and special needs groups; and completion date. A separate survey electronic data file has been created to accumulate the standard survey data from the GPE levels 3-4 sponsorship projects. The two data files can  be the object of comparison to other funding recipients and detailed analysis by the funding organization. 

NOTES: Accreditation standards from the Association for Accreditation for Ambulatory Health Centers (AAAHC) require quality improvement activities with evidence of improving quality or addressing problems. In this process, important problems or concerns are identified. Sources of identifiable problems include, . . . .: prevalent disease, including chronic conditions, behavior health, etc. Important problems and concerns of the students and university are enhancement of learning, obligations to grant funders, CSU Chancellor and other administrative mandates, and areas with the largest gaps in Healthy Campus 2010 baseline data and targets.

 

 

REFERENCES

Donovan, R.J.,  and Robinson, L. (1992)  Using the mass media in health promotion: The Western Australia immunization campaign.  Australian Journal of Public Health, in press.

Dupe, C. (1992)  sponsorship in health promotion.  Substance, March 1992: 9-11.

Green, L.W., Kreuter, M.W., Deeds, S.G. and Partridge, K.B. (1980)  Health Education Planning. A Diagnostic Approach.  Mayfield Publishing Company, Palo Alto.

Hastings, G.B. (1984)  Sponsorship works differently from advertising.  International Journal of Advertising, 3: 171-176.

Hastings, G.B., MacAskill, S., McNeill, R.E.S. and Leathar, D.S. (1988)  Sports sponsorship in health education.  Health Promotion, 3: 161-169.

Holman, R. Donovan and B. Corti. (1993)  Evaluating projects funded by the Western Australian Health Promotion Foundation: a systematic approach. Health Promotion International 1993; 8(3).

MacFarlane, J. (1991)  A Review of Sponsorship Strategies Used by the Smoking and Health Program, Health Promotion Services Branch, Health Department of W.A., from 1987 to February 1991.  Health Promotion Monograph, Health Department of Western Australia, Perth.

Naccarella, L., Borland, R. and Hill, D. (1991)  Quit sponsorship of a Victorian Football League Club.  In Quit Evaluation Studies No. 5 1989.  Victorian Smoking & Health Program, Melbourne.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Gpelevel4hp.doc July 2, 2001

Jim Grizzell, MBA, MA, CHES; jvgrizzell@csupomona.edu