Always Learning

Research Reports

Find Research:
TITLE
DESCRIPTION
AUTHOR
DATE
A Universal Design for Learning-based Framework for Designing Accessible Technology-Enhanced Assessments

The increased capabilities offered by digital technologies offer new opportunities to evaluate students’ deeper knowledge and skills and on constructs that are difficult to measure using traditional methods. Such assessments can also incorporate tools and interfaces that improve accessibility for diverse students, as well as inadvertently introduce new accessibility barriers. Designing these technology-enhanced tasks according to universal design principles is one way to address these accessibility concerns, but requires a grounded understanding of students’ diverse abilities and the ways they interact with the tasks. A thorough consideration of the factors that impact construct validity, with an emphasis on identifying and eliminating sources of construct-irrelevant variance, is essential

Dolan, Robert P.
Burling, Kelly
Harms, Michael
Strain-Seymour, Ellen
Way, Walter (Denny)
Rose, David H.
04-28-2013
An Adaptive-within-Testlet Item Selection Method with Both Testlet Level and Test Level Content Balancing in CAT

The purpose of this study is to propose a heuristic item selection procedure, which selects a testlet and a subset within the selected testlet to be administered with consideration of content balancing and being adaptive within the testlet.

Chien, Yuehmei Shin, Chingwei, David 01-01-2013
Halo Effects and Analytic Scoring: A Summary of Two Empirical Studies

We address the issue of whether unique information is provided by analytic scores assigned to student writing, beyond what is depicted by holistic scores, and to what degree multiple analytic scores assigned by a single rater display evidence of a halo effect.

Lai, Emily R., Wolf, Edward W., Vickers, Daisy H. 11-01-2012
Online Scoring vs. Materials Scoring for Portfolio Assessments: An Exploration of Score Stability

The study is designed to investigate whether scores assigned to portfolio submissions are comparable between materials-based scoring and online scoring conditions, and to evaluate how scorers perceive the ease of using the online scoring platform and in facilitating the scoring process.

Hau Wei 10-02-2012
A Literature Review of Gaming in Education

This research report provides an overview of the theoretical and empirical evidence behind five key claims about the use of digital games in education.

McClarty, Katie Larsen, Orr, Aline, Frey, Peter M., Dolan, Robert P., Vassilev, Victoria, McVay, Aaron 06-30-2012
Value-Added Models in the Evaluation of Teacher Effectiveness: A Comparison of Models and Outcomes

This study compared five value-added models and illustrated the impact of model choice on the estimates of teacher effectiveness.

Wei, Hua, Hembry, Tracey, Murphy, Daniel L., McBride, Yuanyuan 06-01-2012
Collaboration: A Literature Review

Collaboration is the “mutual engagement of participants in a coordinated effort to solve a problem together.”

Lai, Emily R. 06-01-2011
Critical Thinking: A Literature Review

Critical thinking includes the component skills of analyzing arguments, making inferences using inductive or deductive reasoning, judging or evaluating, and making decisions or solving problems.

Lai, Emily R. 06-01-2011
Motivation: A Literature Review

Motivation refers to reasons that underlie behavior that is characterized by willingness and volition.

Lai, Emily R. 04-12-2011
Metacognition: A Literature Review

Metacognition is defined most simply as "thinking about thinking."

Lai, Emily R. 04-01-2011
Cognitive Lab Evaluation of Innovative Items in Mathematics and English Language Arts Assessment of Elementary, Middle, and High School Students

This research report examines a study in which a set of prototype items were developed to align with specific Common Core State Standards and administered to students in a series of cognitive labs. The report details results and offers implications and recommendations for future use.

Dolan, Robert P., Goodman, Joshua, Strain-Seymour, Ellen, Adams, Jeremy, Sethuraman, Sheela 03-01-2011
Universally Designed Computer Based Testing: UD-CBT Guidelines

This report's table of contents is hyperlinked for your convenience. These Universally Designed Computer Based Testing Guidelines aim to help item and test developers understand the cognitive processes involved in interacting with different item stimuli and response methods and, thereby, help identify sources of construct irrelevant variance.

Burling, Kelly, Dolan, Bob, Hanna, Elizabeth, Harms, Michael, Nichols, Amy, Strain-Seymour, Ellen, Way, Denny, In collaboration with CAST 10-01-2010
A Cognitive Lab Report for the American Diploma Project Algebra II End-of-Course Exam

This cognitive lab study was an exploratory study that allowed for an in-depth investigation of students’ familiarity with ADP Algebra II exam items as well as the strategies that students are engaging in when attempting to solve the problems.

Test, Measurement & Research Services 10-01-2010
A Cognitive Lab Report for the American Diploma Project Algebra I End-of-Course Exam

This cognitive lab study was an exploratory study that allowed for an in-depth investigation of students’ familiarity with ADP Algebra I Exam items as well as the strategies that students are engaging in when attempting to solve the problems.

Test, Measurement & Research Services 10-01-2010
Rater Effects as a Function of Rater Training Context

This study examined the influence of rater training and scoring context on the manifestation of rater effects in a group of trained raters.

Wolfe, Edward W., McVay, Aaron 10-01-2010
An Exploratory Teacher Survey Related to the American Diploma Project Algebra I End-of-Course Exam

A survey-based exploratory study was conducted to better understand the gaps between curriculum and instruction, and what content knowledge is expected of students on the American Diploma Project (ADP) Algebra I End-of-Course Exam.

Test, Measurement & Research Services 09-01-2010
An Exploratory Teacher Survey Related to the American Diploma Project Algebra II End-of-Course Exam

A survey-based exploratory study was conducted to better understand the gaps between curriculum and instruction, and what content knowledge is expected of students on the American Diploma Project (ADP) Algebra II End-of-Course Exam.

Test, Measurement & Research Services 09-01-2010
American Diploma Project Algebra II End-of-Course Exam: Standard Setting Briefing Book

The Briefing Book includes an overview of 1) the American Diploma Project, 2) the ADP  Algebra II End-of-Course Exam, 3) the standard setting process, and 4) the validity studies conducted to  inform standard setting. Concurrent, cross sectional, and judgment studies are also included.

Pearson 07-01-2010
Narrative Structures in the Development of Scenario-Based Science Assessments (Large-Scale Assessment Technical Report 3)

A study was conducted to determine if the explication of Narrative Structures in storyboard development improves the quality and efficiency of storyboard writing. Research and evaluative findings suggest that Narrative Structure recognition and use may aid in the storyboard writing process. Note: This PDF is downloaded directly from ECDLarge.

Fulkerson, Dennis, Nichols, Paul, In collaboration with industry peers 03-01-2010
Leveraging Evidence-Centered Design in Large-Scale Test Development (Large-Scale Assessment Technical Report 4)

This report depicts ECD as a series of integrated layers describing an assessment design process that includes analyzing and modeling domains, specifying arguments in terms of student, task and evidence models, and implementing the assessment and executing operational processes. Note: This PDF is downloaded directly from ECDLarge.

Fulkerson, Dennis, Nichols, Paul, in collaboration with industry peers 03-01-2010
Informing Design Patterns Using Research on Item Writing Expertise (Large-Scale Assessment Technical Report 9)

This technical report presents a study in which verbal reports from expert item writers were collected and analyzed. Findings from the study are used to suggest modifications to the development of design patterns. Note: This PDF is downloaded directly from ECDLarge.

Fulkerson, Dennis, Nichols, Paul 03-01-2010
A Comparison of Training & Scoring in Distributed & Regional Contexts—Reading

This study examined the influence of rater training and scoring context on the following outcomes: (a) training time, (b) scoring time, (c) qualifying rate, (d) quality of ratings, and (e) rater perceptions.

Wolfe, Edward W. 07-01-2009
A Comparison of Training & Scoring in Distributed & Regional Contexts—Writing

This study examined the influence of rater training and scoring context on the following outcomes: (a) training time, (b) scoring time, (c) qualifying rate, (d) quality of ratings, and (e) rater perceptions.

Wolfe, Edward W. 07-01-2009
Strategies and Processes for Developing Innovative Items in Large-Scale Assessments

In this paper we describe processes for developing high-quality innovative items in the context of large-scale assessments.

Strain-Seymour, Ellen, Way, Walter "Denny", Dolan, Robert P. 06-01-2009
PADI Online Assessment Design System and Minnesota Science Assessment Glossary of Terms (Large-Scale Assessment Technical Report 1)

This paper presents a Glossary of terms about features of the Application of Evidence-Centered Design to State Large-Scale Science Assessment project, which is partnering with the state of Minnesota. Note: This PDF is downloaded directly from ECDLarge.

Fulkerson, Dennis, In collaboration with industry peers 05-01-2009
A Design Pattern for Observational Investigation Assessment Tasks (Large-Scale Assessment Technical Report 2)

Drawing on research development in assessment design, this paper provides a design pattern to help assessment designers create tasks assessing students’ complex scientific reasoning skills in observational investigation.Note: This PDF is downloaded directly from ECDLarge.

Fulkerson, Dennis, Nichols, Paul, In collaboration with industry peers 05-01-2009
Weighted Penalty Model for Content Balancing in CAT

This research report proposes a new model called the Weighted Penalty Model (WPM) for content balancing in computer adaptive testing.

Chien, Yuehmei, Shin, Chingwei David, Swanson, Len, Way, Walter Denny 04-01-2009
Perspective™-Integrated Assessment and Instructional Resources System

The Learning Locator is the mechanism that connects students with appropriate learning materials based on their assessment performance.

Meyers, Jason L., Nichols, Paul, Shin, David 03-01-2008
The Validity Case for Assessing Direct Writing by Computer

Technology continues to provide opportunities for changing how teachers give instruction and how students learn.

Davis, Laurie L., Ph.D., Strain-Seymour, Ellen, Ph.D., Way, Walter D., Ph.D. 01-01-2008
Incremental Validity of Numerical Reasoning over Critical Thinking

This study was conducted to evaluate the incremental validity of numerical reasoning over critical thinking in predicting job performance and overall potential as measured by supervisors’ ratings.

Ejiogu, Kingsley C., Rose, Mark, Trent, John, Yang, Zhiming 08-01-2007
Evidence of Test Score Use in Validity: Roles and Responsibilities

This paper has three goals.

Nichols, Paul D. 08-01-2007
A Comparison of Methods of Estimating Subscale Scores for Mixed-Format Test

Because the world is complex and resources are often limited, test scores often serve to both rank individuals and provide diagnostic feedback (Wainer, Vevea, Camacho, Reeve, Rosa, Nelson, Swygert, and Thissen, 2000).

Shin, David 05-01-2007
An Empirical Investigation of Growth Models

With the recent legislation of NCLB, there has been an increasing interest to measure students’ growth over the course of their schooling.

O'Malley, Kimberly, Tong, Ye 08-01-2006
Adolescent/Adult Sensory Profile

The Adolescent/Adult Sensory Profile enables clients from 11 through 65+ years to use a Self-Questionnaire for evaluating their behavioral responses to everyday sensory experiences.

06-01-2006
Understanding the Relationship Between Critical Thinking and Job Performance

This study was conducted to evaluate the relationship between a measure of critical thinking ability and job performance as measured by supervisors’ ratings.

Ejiogu, Kingsley C., Rose, Mark , Trent, John, Yang, Zhiming 05-01-2006
Practical Questions in Introducing Computerized Adaptive Testing for K-12 Assessments

In this paper, a number of practical questions related to introducing CAT for K-12 assessments are discussed.

Way, Walter D. 04-01-2006
Miller Analogies Research Report

Once a mainstay of college admission tests, many educators have come to see the analogy format as obsolete, something like a manual typewriter in the age of word processing and text messaging.

Meagher, Don 03-01-2006
WISC-IV Technical Report #7: Extended Norms Cayton, Tom, PhD, Gabel, Amy, PhD, Weiss, Larry, PhD, Zhu, Jianjun, PhD
WISC-IV Technical Report #5: WISC-IV and Children's Memory Scale Drozdick, Lisa W., Holdnack, James, Rolfhus, Eric, Weiss, Larry
WISC-IV Technical Report #6: Using the Cognitive Proficiency Index in Psychoeducational Assessment

The Cognitive Proficiency Index (CPI) summarizes performance on the WISC-IV working memory and processing speed indices in a single score.

Gabel, Amy Dilworth, Ph.D, Weiss, Lawrence G., PhD
ABAS-Second Edition

The Adaptive Behavior Assessment System-Second Edition (ABAS-II) provides a comprehensive norm-referenced assessment of the adaptive skills of individuals ages birth to 89 years.

Harrison, Patti L., Oakland, Thomas
DEAP Technical Report

The Diagnostic Evaluation of Articulation and Phonology, U.S. Edition (DEAP) is a comprehensive, individually administered, norm-referenced battery designed to provide differential diagnoses of speech disorders in children ages 3.0-8.11 years.

Crosbie, Sharon, Dodd, Barbara, Holm, Alison, Ozanne, Anne
The VIP (Validity Indicator Profile) Test in Court

VIP users may be called upon to present their clinical findings and conclusions, either in person or by report, to a judge, jury, disability board, or agency that makes decisions about remunerative awards.