TEAC and ICAO Doc 10197: Aligning Evidence-Based Testing With Regulatory Standards

With the release of ICAO Doc 10197 (Test Design Guidelines: Handbook on the Design of Tests for the ICAO Language Proficiency Requirements), the aviation English testing community faces an important moment of change. This new handbook incorporates the ICAEA Test Design Guidelines (TDGs) into ICAO’s regulatory framework and aims to support harmonization of test design across providers.

It is important to clarify that ICAO Doc 10197 does not replace or modify ICAO LPR provisions but instead serves as a guidance document to help ensure consistency in how tests are evaluated. The ICAO Language Proficiency Requirements (LPRs), as outlined in ICAO Doc 9835, remain the governing regulatory framework.

At Lenguax, we fully comply with ICAO Doc 9835 and EASA Part FCL.055 while aligning with the core principles of ICAO Doc 10197. Our Test of English for Aeronautical Communication (TEAC) is designed to meet ICAO’s key objectives of validity, reliability, fairness, and operational relevance. However, some aspects of the TDGs require further empirical validation, particularly regarding:

• The requirement for scripted role-play tasks (RPTs).
• The requirement for separate listening sections.
• The elevation of authenticity as a central criterion in test evaluation.

In this post, we examine how TEAC aligns with ICAO’s regulatory goals, where our approach differs, and why an evidence-based perspective is critical to ensuring fair and effective testing.

Where TEAC Aligns With ICAO Doc 10197

TEAC is designed to assess plain English proficiency in aviation contexts, ensuring a construct-valid, fair, and reliable assessment. Here’s how TEAC aligns with key ICAO Doc 10197 criteria:

1. Radiotelephony Contexts (Criterion 1)

ICAO Doc 10197 emphasizes that test tasks should assess language use in radiotelephony (RT) contexts, particularly in non-routine situations requiring plain English. TEAC achieves this through:

• Explaining an RT Communication (Part 2) – Candidates analyze and summarize RT messages, demonstrating comprehension and the ability to convey the meaning of unexpected communications.
• Reporting Pilot/Controller Messages (Part 3) – Candidates listen to non-routine RT communications and provide structured responses, simulating how aviation personnel process and relay important information.

While these tasks assess comprehension and response formulation within RT settings, they do not and are not intended to simulate real-time interaction with an interlocutor. TEAC focuses on ensuring clear and accurate information relay, which is an important component of aviation safety.

Unlike some tests that conflate phraseology and plain English, TEAC separates these constructs to prevent construct interference, ensuring the test measures what it is intended to assess: plain English in aviation contexts.

Acknowledging the limitations of single-play listening

While TEAC’s listening comprehension tasks effectively assess a candidate’s ability to understand and process operational messages, we recognize that single-play listening presents challenges.

In real-world aviation communication, when a message is not understood, a pilot or controller would typically request repetition or receive a paraphrased clarification to ensure accurate information transfer.

However, in a testing environment, a candidate who mishears an RT message may have no opportunity to recover meaning, even if they would have done so in an operational setting. If the reason for mishearing is anything other than distraction, a single play might not provide a fair reflection of the candidate’s true comprehension ability.

To address this and improve fairness, TEAC is exploring enhancements to its listening comprehension tasks, including:

• Allowing candidates to request a second version of a message, which is slightly modified (paraphrased) and possibly clearer, simulating how RT messages are often rephrased in operational settings.

2. Separate Tests for Pilots and Controllers (Criterion 2)

TEAC meets the ICAO requirement for role-specific assessments by offering tailored test versions for:

• Pilots
• Air Traffic Controllers (ATCs)
• Other aviation professionals

Each version ensures test content is operationally relevant to the candidate’s role, fully aligning with ICAO’s guidance on role differentiation.

3. Distinct Sections and Task Variety (Criterion 4)

TEAC features four distinct sections, ensuring a balanced and comprehensive assessment:

1. Image-Based Interview – Candidates discuss routine and non-routine aviation topics.
2. RT Communication Tasks – Candidates analyze and summarize operational RT exchanges.
3. Message Reporting – Candidates demonstrate comprehension of non-routine RT messages.
4. Picture Comparison & Discussion – Candidates engage in extended discussions, expressing opinions and speculating about aviation-related topics.

This structured format aligns with ICAO’s emphasis on varied task types while maintaining a clear assessment construct.

4. Interactive Communication (Criterion 5)

TEAC is designed to be highly interactive, ensuring candidates engage in spontaneous communication:

• Image-Based Interviews (Part 1) and Discussion Tasks (Part 4) encourage real-time interaction, requiring candidates to respond dynamically, clarify ideas, and negotiate meaning.
• This satisfies ICAO’s goals for interactive tasks while avoiding overly scripted or predictable exchanges.

5. Aviation-Relevant Language Use (Criterion 7 – Authenticity)

ICAO Doc 10197 emphasizes authenticity, though it does not mandate how authenticity should be implemented (nor, indeed, does it define this term). TEAC aligns with this principle by ensuring:

• Situational relevance: All test tasks are embedded in aviation-specific contexts, rather than artificial or generic conversations.
• Functional relevance: Candidates are required to use and respond to language functions they encounter in the course of their duties.

However, we argue that authenticity should remain an aspect of validity, not a standalone test criterion. A strict focus on authenticity risks making test tasks overly rigid, scripted, or impractical. TEAC maintains a balanced approach, ensuring that real-world relevance does not come at the cost of test reliability or fairness.

Where TEAC Takes an Evidence-Based Approach

While TEAC aligns with most of ICAO Doc 10197, we take a research-driven approach in two key areas:

1. Role-Play Tasks (RPTs)

ICAO Doc 10197 recommends Role-Play Tasks (RPTs) for assessing plain English in aviation. However, we believe that mandatory scripted role-plays raise practical and construct validity concerns:

• Limited Plain English Elicitation: Many RPTs produce formulaic, scripted responses rather than natural speech.
• Construct Overlap with RT Phraseology: Poor performance could stem from lack of phraseology knowledge, not plain English ability.

TEAC’s Alternative Approach:

• Open-Ended Discussion Tasks (Part 4) achieve the same interactive goals without relying on rigidly scripted interactions.
• Scenario-Based RT Analysis (Parts 2 & 3) assess comprehension and reasoning without confusing phraseology skills with plain English.

Proposal: We invite empirical research comparing RPTs vs. open-ended discussions to determine which best elicits spontaneous, role-relevant language.

2. Separate Listening Sections

ICAO Doc 10197 mandates standalone listening tasks, assuming this improves assessment validity. However, we believe this requires further validation:

• Listening is rarely an isolated skill in aviation – it is typically integrated with response and reasoning.
• Assessing comprehension via spoken responses better reflects real-world communication demands.

TEAC’s Approach:

• Integrated Listening in Parts 2 & 3: Candidates summarize and explain RT messages, demonstrating both comprehension and communication ability.
• Oral responses serve as a more operationally relevant measure of listening proficiency.

Proposal: We support collaborative studies comparing separate vs. integrated listening assessments to establish best practices.

Balancing Authenticity, Validity, and Practicality

The TDGs elevate authenticity as a central testing goal, but ICAO Doc 10197 does not mandate any single approach to authenticity. TEAC ensures aviation relevance while maintaining:

• Construct Validity: Testing only what needs to be tested.
• Fairness & Practicality: Avoiding overly scripted, rigid tasks.
• Reliability: Ensuring scores are consistent and meaningful across versions.

Collaboration for a Stronger Future

TEAC complies fully with ICAO Doc 9835 and EASA Part FCL.055 while meeting the core principles of ICAO Doc 10197.

Our approach:

✔ Ensures validity and reliability through evidence-based design.

✔ Maintains fairness and practicality without unnecessary complexity.

✔ Supports further research to refine best practices in aviation English testing.

We welcome collaboration with regulators, test providers, and industry stakeholders to strengthen aviation English testing standards. If you share our commitment to evidence-based testing, let’s work together to create fair, reliable, and practical solutions.