Item Development
Tasks delivered by the Ordinate® testing system are designed to be simple and intuitive both for native speakers and for proficient non-native speakers of English. Items are designed to cover a broad range of skill levels and skill profiles, and to elicit responses that can be analyzed automatically to produce measures of fluency, listening, vocabulary, pronunciation and oral reading ability.

To ensure conversational content, test developers specified that the test materials should conform to the vocabulary that is actually used in conversational English. Spontaneous conversations from 540 North Americans guided the design of test items. Conversation samples were geographically and gender balanced, encompassed a variety of topics, and represented every major dialect of American English. All words used in test items appeared at least four times in the spontaneous conversations. To accommodate test-takers who have been trained to a British English standard, items have been reviewed by two British linguists to ensure conformity to colloquial usage in the United Kingdom and Australia. Test items were also reviewed for fairness and bias-free usage by an independent committee of language experts. The audio item prompts are spoken by a diverse sample of educated native speakers of North American English.

Validation/Norming

Norming
Prototype versions of Versant for English were administered in a series of validation studies to over 4,000 native and non-native speakers. The native norming group comprised 376 literate adults, geographically representative of the U.S. population aged 18 to 50. It had a female/male ratio of 60/40 and was 18% African-American. The non-native norming group was a stratified random sample of 514 callers sampled from a larger group of more than 3,500 non-native callers.  Stratification was aimed at obtaining an even representation for gender and for native language.

Over 40 different languages were represented in the non-native norming group, including Arabic, Chinese, Spanish, Japanese, French, Korean, Italian, and Thai.  Ages ranged from 17 to 79, and the female/male ratio was 50/50.

Because of the introduction of several modifications to Versant for English in the current version, a number of additional validation studies were performed. These studies used a native norming group of 775 native speakers of English from the U.S. and the U.K. and a non-native norming group of 603 speakers from a number of countries in Asia, Europe and South America.  The native norming group consisted of approximately 33% speakers from the U.K. and 66% speakers from the USA and had a female/male ratio of 55/45. Ages ranged from 18 to 75. The non-native norming group had a female/male ratio of 62/38. Ages ranged from 12 to 56.

Validity
Versant for English machine-generated scores generally correspond as they should with human ratings.  At the Overall score level, Versant for English machine-generated scores are virtually indistinguishable from a scoring that is done by careful human transcriptions and repeated independent human judgments.  The correlation between the two is 0.97.

Over the years, Ordinate and third parties have collected data on parallel administrations of Versant for English and other well-established language examinations, enabling a measure of concurrent validity. Correlations with other instruments for assessing oral skills, which focus mainly or entirely on speaking, range from .75 to .94. The data suggest that the Versant for English measure overlaps substantially with that of other instruments designed to assess spoken language skills.

The following academic and commercial organizations across North America, Europe and Asia participated in the development and validation testing of the system.

• Bologna University, Italy
• Cañada College, California
• CITO, National Institute for Educational Measurement, Netherlands
• CUNY, New York
• Defense Language Institute English Language Center, Lackland Air Force Base,
• Texas Deloitte and Touche
• Eastern Michigan University
• Economics Institute, Boulder, Colorado
• EF International Language School, Washington
• Foothill College, Los Altos Hills, California
• Hong Kong Baptist University, Hong Kong
• IIBC, Institute for International Business Communication, Japan
• Indiana University, Indiana
• Iowa State University, Iowa
• Lancaster University, UK
• Monroe Community College, New York
• Monterey Institute of International Studies, California
• New York University American Language Institute
• Oklahoma State University, Oklahoma
• Patel Institute of English, India
• Point Loma Nazarene College English Institute, San Diego, California
• Rainbow Idiomas, Buenos Aires, Argentina
• San Francisco State University American Language Institute
• Sapporo International University, Japan
• Sierra Academy of Aeronautics, Oakland, California
• Stanford University Linguistics Department
• Teacher Training Institute, Seoul, Korea
• University of Luton, UK
• University of North Carolina at Charlotte, Office of International Programs
• University of Findlay, Ohio
• University of Pennsylvania, English Language Programs
• University of Southern Mississippi
• University of Szeged, Hungary
• University of York, UK
• Waseda University, Japan