|
Technical Advisory Committee MeetingOctober 9, 2008
OverviewThe meeting of the Technical Advisory Committee (TAC) of the Partnership for Accessible Reading Assessment (PARA) was held on October 9, 2008, in Washington, D.C. The purpose of the meeting was to update TAC members on PARA's progress since its previous meeting on December 7, 2007, in Washington, D.C. TAC members were updated on dissemination of PARA findings, overviews of two PARA studies (the Motivation Study and the Reading Pen Study), and descriptions of the PARA field trial. TAC members also learned about and discussed the upcoming research for each project and the next steps for PARA. Welcome and GreetingsMartha Thurlow Martha Thurlow welcomed and thanked all the TAC members for their attendance. She highlighted the day's agenda and brought special attention to the project publications and presentations completed to date and those in progress. Martha also discussed the current activities of the PARA project. Newly completed project papers and reports can be found in the meeting binder. Reports and past presentations can be found on the PARA Web site. Overview of PARA Projects - Motivation StudyDeborah Dillon Deborah Dillon shared an overview of the PARA Motivation Study, which is designed to determine whether improving the motivational characteristics of a large-scale reading assessment increases its accessibility for students with disabilities, and, in so doing, provides a more valid assessment of these students' reading proficiency due to their increased engagement. The specific questions of the Motivation Study are: Does the option of exerting choice in the selection of passages on a large-scale reading assessment improve the accessibility of the assessment and lead to increased comprehension due to improved motivation and engagement for students with disabilities and general education students? When students with disabilities and general education students are given the option to choose literary and informational passages that are included on large-scale reading assessments, do they perform better on one text type or the other? Are students who are identified as being more motivated to read on general motivation to read measures (e.g., scales and questionnaires) more proficient in reading when given a choice of reading passages on large-scale assessments? Are students who are identified as being more situationally motivated to read, as reported on questions following each passage, more proficient in reading those passages? Deborah Dillon discussed recent changes in the study design, particularly increasing the number of subjects to 512 (256 in each in each grade) to increase validity of study findings. The study instruments, the Motivations for Reading Survey and the passages for rating and selection, were originally designed as computer-based means of data collection, but it was later decided to administer the motivation assessment as a paper and pencil test. Deborah Dillon also discussed the completed Calibration Study, the purpose of which was to calibrate the measurement tools that will be used in a large-scale reading assessment for students with disabilities. Results from the test will determine passages and items that will be used for the Motivation Study. Further discussion entailed the Motivation Study's instrument design, content of assessments, and other methodological logistics. Discussion with TAC members focused on the claims that can be made in the Motivation Study analysis based on the researched motivation constructs and how those are related to perseverance and self-efficacious reading. Discussion also included approaches to reporting the findings in case they indicate that being able to choose the passages students read on an assessment allows regular education students to have the same boost as students with disabilities. Finally, discussion addressed concerns of presenting data findings in case they indicate no significant differences between Choice and No Choice groups. Overview of the PARA Projects - Reading Pen StudyRoss Moen Ross Moen provided an overview of the Reading Pen Study. The objective of the study is to evaluate how use of the Reading Pen (a small hand-held device that can provide pronunciations, synonyms, and definitions of words scanned by a user) might improve the accessibility of reading assessments for students with disabilities in grades 5-8. Ross Moen began the overview by describing the Reading Pen Study, which employs a split plot design with one between-subjects factor (students with and without disabilities) and one within subjects factor (taking a test with and without the Reading Pen, assigned randomly). The study is designed to include up to 120 students from intact classrooms between 5th and 8th grades, including students with and without IEPs. Ross Moen also described the study hypothesis that the Reading Pen can be used as an effective accommodation by decreasing barriers of poor decoding or fluency, which can confound measures of a student's ability to comprehend text. Unlike the read-aloud accommodation, which is typically administered for the entire text, the Reading Pen allows students to decode words selectively depending on their individual needs. The study is also designed to investigate whether the Reading Pen is capable of providing students with disabilities with a "differential boost", meaning that use of the pen would result in more improved scores for students with disabilities than for students without disabilities. By providing a means of temporarily overcoming limited decoding or fluency, some students might have increased "access" to reading tests in that they would be able to more fully demonstrate their skills in reading comprehension. TAC members discussed feasibility and marketability of the general strategy of providing on-demand reading assistance as the way to measure components of reading separately. Viability of using the digital pen as a tool for facilitating the reading process was another focus of the meeting. Finally, the TAC team generated recommendations for particular computer-based, reading pen, or other solutions for providing on-demand reading assistance. PARA Field TrialMartha Thurlow Martha Thurlow and Jamal Abedi shared an overview of the PARA field trial, which entails administering a prototype assessment for 150 students from each of the four disability groups: specific learning disabilities; speech/language impairments; intellectual disabilities; deafness/hard of hearing. Another 150 students from each of the four disability groups will receive the non-accessible assessments. The non-accessible assessment consists of the same content as in the accessible assessment with the same question type and format but with no accessibility features included. The presenters focused on key issues of accessibility, validity, and comparability of assessment components. Discussion of accessibility was based on the premise that PARA field trial features should make the assessment more accessible without changing the test construct. Validity of assessment will be evaluated by analyzing the effect sizes. Assessment comparability will be estimated based the performance differences between students without disabilities who take the accessible assessments with students without disabilities who take the non-accessible assessments. A group of 300 students without disabilities will take the accessible assessments and another group of 300 of students without disabilities will take the non-accessible assessment. The following questions were addressed by the TAC during open discussion: How might the researchers disentangle the effects of accessibility features given the small sample size for students with different categories of disabilities? If the validity study discovers issues concerning validity of the accessible assessment, then how could the researchers identify the feature (or features) that contributed to the validity problem? What external criteria should be used for examining the validity of accessible reading assessment? Is the state assessment outcome a valid criterion for validation of the accessible reading assessment that was developed by PARA? Should the developmental aspect of reading be considered in the validation process (e.g., latent-class approach)? Principles and Guidelines EvidenceMartha Thurlow Martha Thurlow shared the progress on developing five evidence-based principles for making large scale assessments of reading proficiency more accessible for students with disabilities. Each of these principles is supported by specific guidelines that address the implementation of the principles. The five principles are:
TAC members commented on the generated principles and guidelines and discussed types of evidence that support the accessibility principles. Part of the discussion was devoted to considerations of additional evidence which could be added to the collected evidence about the reading assessment principles. PARA Next StepsDeborah Dillon Deborah Dillon concluded the meeting by thanking the TAC for their thoughtful participation in the discussions. She stated that the presentations from the day's meeting would be added to the Web site so that TAC members are able to share and talk about them with their colleagues. PARA will send out regular updates on the recent activities and future research for the TAC members to distribute among their constituents. |
|||||||||||||||||||||||||||||||
|
Contact readingassess@umn.edu © 2009 Regents of the University of Minnesota |
||||||||||||||||||||||||||||||||
