CSU Information Competence Assessment

Phase Two Summary Report

 

This is a summary of the full report of findings from the Phase 2 qualitative study issued by the Social and Behavioral Research Institute for the CSU Information Competence Assessment Task Force.

 

Introduction

 

Background

Two very different studies have now been completed as part of the California State University's commitment to information competence assessment.  Phase I conducted in Spring 2000 used scenarios as part of a questionnaire-based quantitative study to establish a baseline of student information competence.    Phase II conducted in Spring 2001 studied information competence through qualitative methods designed to uncover how students seek, evaluate, analyze, and use information. The Social and Behavioral Research Institute (SBRI) at CSU San Marcos conducted both studies with input and guidance from the California State University Information Competence Assessment Task Force.  The report discusses the variation found in student work processes and products, explicates the reasons students achieve certain results, presents findings about areas where students lack proficiency, and offers suggestions for imparting such skills.  

While this summary highlights important findings from the study, it cannot do justice to the richness of this data.  For example, included in the report are many examples of student work and verbatim comments from students.  Reading the report itself is the best way to understand the findings and their implications.

 

Research Methods

 

·        A sample of 100 students was randomly selected from the Offices of Institutional Research at each of the 21 participating campuses. 

·        Seventh-six students, 20 librarians, and 10 faculty members participated in this study.  The number of student and librarian participants approximated the desired sample. There was an inadequate representation of teaching faculty.

·        Study conducted on four CSU campuses in different parts of the state: Fullerton, Northridge, San Jose, Sacramento.  Task Force members participated in developing the assessment tasks, took part in the four assessment days, and assisted with site preparations at each assessment site. 

·        Study engaged students, faculty, and librarians in a full day of assessment activities centered around four information tasks designed specifically for this study. 

·        Data gathering methods included:

Ø     Video and screen capture of students’ computer activity;

Ø     Video and audio capture of librarian/faculty/student focus groups and discussions;

Ø     Ethnographic observations of and conversations with students working on the information tasks;

Ø     Task sheets requiring students to fill out answers to essay questions regarding their research and to provide information about their background and experience with the subject of the task and with the library in general; and

Ø     Demographic data on all student participants.

 

 FINDINGS BY DATA COLLECTION METHOD

 

Studying student information seeking behavior with different, but complimentary research methods may lead to both similar and dissimilar, even conflicting or contradictory, explanations of observed behavior.  The findings from the screen capture data, the ethnographic data, and the videotaped data exhibit both the consistency and the contradictions inherit in a multi-method research approach. 

 

 SCREEN CAPTURE DATA 

The work of 63 students was recorded through screen capture using Camtasia Recorder software.   This data provide specific instances of students' information searching procedures, but do not display any student's complete knowledge of how to seek information.  The data suggests that each student displays some of his or her routine search procedures, and indicates tendencies toward information seeking routines where each one applies their specific knowledge about what constitutes information adequate for completing common tasks, and ways to find such information. 

 

Findings of the analysis of the screen capture data:

 

• Students tend to use the WWW rather than library catalogs and databases
• Reliance on the WWW as a source of information may discourage students from developing the skills and evaluative criteria supported by the Association of College and Research Libraries (ACRL) Information Literacy Competence Standards.[1]
• The online environment may influence the ways students choose and refine search terms, leading to similar search strategies.  Often they do not select databases best suited to their task.
• Students selection and revisions of search terms does not meet the ACRL standards
• Students do not use online databases and catalogs effectively
• Students often give up easily when trying to use library databases and catalogs

 

Generally, most students search in the following ways:

 

• Begin with more specific search terms and then try broader terms when the more specific search terms yield 0 results;
• Begin with broad search terms, narrow them,  and then return to the broad search terms when the narrower terms fail to produce results; and
• Begin with a very broad search term and scroll through pages and pages of results, linking to various citations rather than narrowing the search.

 

ETHNOGRAPHY DATA 

The purpose of the ethnographic study is to provide insight into the ways students think about, seek, and apply information. The guiding questions for this study of student work processes include: Where do students seek information?  How do they find it? and, s How do student select and apply the information that they find?  Analysis of the findings indicates general patterns and tendencies.  They do not characterize every student's information seeking routines and procedures.  The full report explains these trends and the exceptions to them.

Where Do Students Seek Information?

 

• Students embrace the virtual library for its convenience, flexibility, timeliness,

and access to large amounts of up-to-date information.

• Students use the Internet as a “gateway resource” for introductory information

and as a stepping-stone to other search media.

• Students use favorite search engines and databases for their efficiency; ease of

use, and for the amount of quality information.

• Students rely less on traditional “bricks and mortar” library resources.
• Students achieve well-rounded searches using non-technological sources.
• Technologically based information sources telescope (and possibly limit) the

information that students retrieve.

 

How Do Students Find Information?

 

• “Plans of action” guide student searches.
• Students approach searches holistically, assessing the distinctive contributions of

diverse information resources.

• Students struggle to come up with key words.
• Unfamiliar search environments pose significant barriers to the use of information

resources.

• Students engage in general searches to generate a robust pool of information from

which to discover “gateway” resources.

 

How Do Students Select and Apply Information?

 

• Search methods (including the characteristics of the on-line environment), rather

than critical evaluation of resources, guide selection of materials.

• Students with little knowledge of the topic selected sources based on background

knowledge and personal experience.

• Students selected information based on expectations of the assignment.
• Students selected information based on perceived needs and preferences of their

audiences.

• Students’ concerns for credibility and balance led to seeking primarily fact-based,

rather than idea-based, information.

• Students’ concerns for credibility and balance led to selecting information based

on form (e.g., representing both sides) rather than critical evaluation of content.

• Successful searches require strategies for expanding students’ knowledge bases.
• Selection and application of information required orienting and reorienting to the

task.

 

 

VIDEOTAPED DATA 

This sections reports on information gleaned from analysis of the videotaped data from focus group sessions.

Importance of Context

·        Most students recognized that they needed a general overview in order to understand the task.

·        Students who believe that they needed to only find the facts to answer a question at the end of a task, erroneously searched for bits and pieces of information rather than striving for a contextual understanding of the topic.

·        Understanding a question in context is central to accomplishing their research

o       The struggle to find effective keywords may be related to lack of knowledge about and context for a research problem.

In addition to what the students show us about their use of  the strategies listed in the Standards, these findings have implications for libraries and librarians.

·        Students do not often make distinctions between academic and popular literature whether this material in on the WWW or in the library.

·        Many library resources such as microfiche or periodical indexes go unused because students are not familiar with them and can manage to get by without them.

·        Students rely on professors and other students more than on librarians for help in completing their research.

 

Student Suggestions For Best Practices from the Focus Groups

·        Students prefer targeted learning sessions directed towards one task rather than a broad overview or tour of library facilities and resources.

·        Students appreciate specific information about databases.

·        Students appreciate faculty and librarians who work closely together.

·        Students would like help in negotiating the physical library, for example locating materials.

·        Some students would respond better to a friendlier library environment, for example overstuffed chairs, cafes.

 

Comparison of Two Students Followed through the Test Day

This section of the report compares two students who were tracked across the test day through videotaped, screen capture, and ethnographic data.    Their activities are compared against Standards One, Two, and Three of the ACRL Information Literacy Competency Standards.  The comparison raises a key issue in regard to information competence standards that calls for further study, and that is, what critical threshold do students need to meet in order to be considered information competence.

 

SUGGESTIONS FOR IMPROVING STUDENT INFORMATION COMPETENCE SKILLS

 

Suggestions from the Analysis of Screen Capture Data

 

·        The data show that students find very little appropriate information using the web. Students could have gathered more useful information with more efficient use of time had they used effective database searching skills as promoted in the ACRL standards.

• Libraries can enhance students' information competence skills by expanding their function as places for teaching and learning information competency. Expertise in systematically refining searches is key to efficiently finding a range of valid and reliable information about a topic.  This type of information competence is central to the ACRL standards, and valuable in the workplace.
• The accessibility of a multitude of resources by itself does not service the function of teaching students and helping them to learn information competence

 

Suggestions from Analysis of the Videotape and Ethnographic Data

Information competence needs to be seen as the province of both faculty and librarians:

• Set up faculty-librarians dialogues about the findings of this report;
• Encourage coordination between faculty and librarians to develop specific library services and holdings applicable to a particular class or topic;
• Teach students to read critically and evaluate materials using criteria other than whether the source is scholarly or popular, especially important now that the lines between academic literature and popular press literature are increasingly blurred;

• Encourage critical thinking and analytical and reasoning skills by increasing student awareness of the range of information available from "facts" to discussion of abstract ideas; and

• Encourage students to use both technological sources and non-technological sources and explain how various sources complement each other.

 

 

CONCLUDING REMARKS

 

The report includes discussion and analysis of student information competence skills as evidenced in a one-day exercise.   While the findings cannot be claimed to pertain to all CSU students, they are suggestive of the range of skills that may be found in the CSU student population and of the types of challenges students experience in conducting research. 

 

 

 

 

 

 Summary prepared by

Kathleen Dunn, Chair

CSU Information Competence

Assessment Task Force

kkdunn@csupomona.edu

January 2002

 

 

 

 

 

---