Weighing QA Evidence for Better Judgements

In further education and skills, we are surrounded by data. Learner feedback, observation notes, achievement rates, and meeting minutes all contribute to our picture of quality. But not all evidence is created equal. Before we use this information to form judgements and drive improvement, it's crucial to pause and weigh its quality. Relying on weak, biased, or indirect evidence can lead to flawed conclusions and wasted effort. A discerning approach to your evidence base is the foundation of a reliable quality system.

Direct vs. Indirect Evidence

The first step is to distinguish between evidence you gather at the source and evidence that is reported to you.

• Direct evidence is gathered first-hand. It includes your own observation of a teaching session, a review of a learner's portfolio, a recording of a professional discussion with an apprentice, or the output of a joint activity with staff. It provides rich, contextualised insight but often reflects a single point in time.
• Indirect evidence is second-hand or aggregated information. This includes survey results, qualification achievement rates (QAR), a manager's summary report on their team's performance, or destination data. It is excellent for spotting trends and patterns across large groups but can obscure important context and individual experiences.
• Practical tip: Use one to validate the other. If high-level achievement data looks strong, triangulate this with a direct scrutiny of learners' work to confirm the quality and depth of learning match the grades.

Recency and Relevance

Not all data stays fresh or relevant. The context in which evidence was gathered is critical to its value.

• Recency: Old evidence has a short shelf-life. A thematic review from two years ago may be irrelevant if the curriculum, staff, or learner cohort have since changed. Quality assurance is a live process, and your evidence should reflect current, not historical, practice.
• Relevance: Does the evidence truly answer the question you are asking? For example, high learner attendance is positive, but it is not direct evidence of high-quality curriculum, teaching and training. It tells you learners are present, not what they are learning or how well.
• Practical tip: Date-stamp all key evidence sources. When writing evaluative summaries, be explicit about what a piece of evidence can and cannot tell you. This disciplined thinking prevents you from making unsupported logical leaps.

Sample Size and Potential Bias

Whom you ask, and how many, has a significant bearing on the weight you can give to the evidence.

• Sample size: The weight of evidence often depends on its scale. Feedback from a focus group of five engaged learners provides a valuable anecdote and deep insight. An end-of-course survey completed by 500 learners provides a dataset. Both are useful, but they must be interpreted differently.
• Potential bias: Always consider where the evidence came from. Self-selecting surveys tend to attract participants with very strong positive or negative views. A focus group of volunteers may not represent the typical learner experience. Evidence collected by a manager about their own team may be unconsciously optimistic.
• Practical tip: Be intentional with sampling. For learner voice, use stratified sampling to ensure you hear from different provision types, levels, and groups (e.g., those with high needs). Acknowledge any potential bias in your analysis to maintain credibility.

Typicality Over Showcase Performance

Is the evidence you are reviewing a reflection of everyday reality or a carefully prepared performance?

• The 'showcase' risk: Evidence gathered during a formal, high-stakes quality week can often represent a 'performance' rather than typical practice. A portfolio curated specifically for review might only contain a learner's best work, hiding their true developmental journey.
• Seeking typicality: The most valuable evidence often comes from your provider's everyday, business-as-usual activities. Scrutiny of randomly selected work, notes from informal 'corridor conversations', and low-stakes professional dialogue often provide a more authentic picture of quality.
• Practical tip: Design your quality cycle to collect evidence throughout the academic year. Mix formal, scheduled activities with more frequent, informal checks to build a picture of what is typical, not just what is possible on a good day.

Where this fits in QualityHero: Building a defensible evidence base is a core function of quality assurance. The SAR and QIP modules in QualityHero allow you to link multiple evidence documents directly to your judgements and actions. By tagging sources from across the platform - such as observations from Toolkit Areas or feedback from the Learner Voice module - you can easily triangulate direct and indirect evidence, ensuring your conclusions are balanced, credible, and focused on typical practice.