Robust QA Sampling: A Practical Guide

A provider’s understanding of its own quality is only as reliable as the evidence it is built on. While anecdotal evidence has its place, robust quality assurance requires a systematic approach to gathering information. A well-designed sampling strategy is the engine of this process, ensuring the insights that inform your self-assessment and improvement planning are valid, reliable, and representative of typical practice.

This guide outlines how to move from convenience to conviction in your quality assurance sampling.

Define Your Purpose and Scope

Before you sample anything, you must be clear on the question you are trying to answer. A focused inquiry leads to meaningful evidence. Without a clear purpose, sampling becomes a tick-box exercise. Start by defining your 'why'.

• Start with the question: Are you evaluating the consistency of assessment feedback? The impact of a new teaching strategy? The effectiveness of support for learners with high needs? Your question dictates what you sample.
• Define the population: Are you looking at all Level 3 learners, apprentices on a specific standard, sessional staff, or a particular curriculum area? Be specific.
• Consider the evaluation area: Frame your inquiry around the inspection toolkit. For example, for 'Curriculum, teaching and training', you might sample lesson plans. For 'Achievement', you might sample learner portfolios to review progress from starting points.
• Establish your criteria: What does 'good' look like? Agree on the criteria for your evaluation before you begin sampling. This ensures everyone involved makes consistent judgements.

Choose Your Sampling Method

The right method depends entirely on your purpose. A combination of methods often provides the richest picture. Avoid defaulting to the same approach for every task.

• Stratified Sampling: This is often the most robust method. Divide your population into meaningful sub-groups (e.g., by provision type, site, mode of study, year of entry) and draw a sample from each group. This ensures that no significant part of your provision is overlooked.
• Purposeful (or Risk-Based) Sampling: This involves intentionally selecting specific cases because they are likely to yield important information. You might sample a new programme, a course with declining achievement rates, or from a team with many new staff. This is vital for targeted support and intervention.
• Random Sampling: In this method, every item in the population has an equal chance of being selected. While it can be useful for getting a broad, unbiased overview of a large provision, it can also miss isolated pockets of weak or exceptional practice.
• Thematic Sampling: This involves following a specific theme or 'golden thread' across the organisation. For instance, you could sample how careers education is integrated across different curriculum areas, or how digital skills are developed for learners in practical and academic subjects.

Determine a Sensible Sample Size

There is no single magic number for sample size. The goal is not to hit a specific percentage, but to gather enough evidence to be confident in your findings. Bigger is not always better.

• Focus on saturation: A good sample is large enough to reflect the diversity of your provision and identify recurring themes. You have reached 'data saturation' when new samples stop telling you anything new.
• Consider the stakes: If you are making a high-stakes judgement that will inform your SAR, a larger, more robust sample is required than for a quick, informal check-in.
• Balance rigour with reality: Your sampling plan must be manageable. A small, well-chosen and deeply analysed sample is far more valuable than a large, superficial one. For small cohorts, you may decide to review all evidence (a census) rather than sampling.

Mitigate Bias in Your Process

Bias can creep into any sampling process, undermining the validity of your findings. Being aware of potential biases is the first step to mitigating them.

• Challenge 'convenience sampling': Be wary of only reviewing work that is easy to access, or observing teachers who volunteer. This rarely provides a typical picture.
• Ensure objectivity: Where possible, the person selecting the sample should be different from the person who created the evidence (e.g., the teacher). Consider using an admin team member or MIS to generate a random or stratified list.
• Involve multiple perspectives: Have more than one person review the sampled evidence. This helps to calibrate judgements and reduces the impact of individual bias.
• Document your rationale: Record why you chose a particular sampling method and size. This demonstrates a thoughtful and deliberate approach to quality assurance.

Where this fits in QualityHero

A robust sampling strategy is fundamental to generating the evidence that feeds your quality cycle. In QualityHero, the insights from your purposeful sampling would populate evidence logs within specific Toolkit Areas, directly inform the evaluative judgements in your SAR, and lead to focused, evidence-based actions in your QIP. This ensures the data surfaced in your Leadership Reports is built on a valid and reliable view of whole-provider and provision-type practice.