Design, Deliver, Verify: A Quality Learning Series -- Would You Pass Your Own Secret Shopper Test?
Suppose the numbers all look right — steady enrollment, solid completion rates, strong evaluation scores. It's tempting to read that as confirmation the program is working. But numbers like these are all lagging indicators of satisfaction — and satisfaction is not the same as impact. A learner can rate a session five stars while still walking away without the capability the session was designed to build. The gap between "well-received" and "actually working" is easy to miss, precisely because nothing about it looks obviously wrong.
What a "secret shopper" mindset actually means
Borrowed from a practice more commonly associated with customer experience, the idea is straightforward: rather than only asking people what they thought afterward, someone with real distance from the program experiences it directly — as an actual learner would — and observes what a survey can't gather. Did the pacing hold attention or lose it partway through? Did the stated objectives match what was actually delivered? Were accessibility needs genuinely met, or just checked off? Did engagement look different in practice than the design intended?
This isn't about catching anyone doing something wrong. It's not a gotcha exercise, and it's not a critique of the people delivering the program. It's a structured, neutral way of seeing the system as it actually functions — because the people closest to a program are often too close to see it clearly, and that's not a failure of effort, it's simply how proximity works.
Why this matters even when things seem fine
The instinct is to reserve this kind of scrutiny for programs that are clearly struggling. In practice, the opposite is often more valuable. Stable evaluation scores can mask real erosion — declining engagement in a specific segment, content that no longer matches what learners need, or a format that's coasting on habit rather than continuing to earn its place. The absence of complaints is not evidence that everything is working. It's evidence that no one complained.
The most useful moment to look closely at a program is often when it appears to be succeeding, because that's precisely when structural issues are easiest to overlook and hardest to raise internally.
Turning fresh eyes into real evidence
A single observation is a data point, not a conclusion. It becomes useful when layered against other evidence: historical performance over multiple offerings, what learners report through surveys, and candid conversations with the people involved in designing and delivering the program. When those layers point to the same conclusion, it stops being a hunch and becomes something an organization can act on with confidence.
The real question isn't whether your program is "good"
It's whether it's still intentionally designed for the learners you have today, not the assumptions you made when it was first built. Quality learning experiences, strong curriculum, and skilled facilitation — the three things we've covered in this series — all matter enormously. But none of them tell you, on their own, whether the combination is actually working. Sometimes the only way to know is to look, with fresh eyes, at what's really happening inside the experience you've built.
Start looking
That's the same spirit behind CEU accreditation itself — a standard exists because self-assessment alone isn't enough. Verification from outside your own perspective is what makes "quality" more than a claim.
You don't need to hire an outside expert to start. Pick a program your organization already runs, and sit in on the next offering the way an actual learner would — no laptop open to "just check on things," no side conversations with staff, no slipping out before it wraps. Notice where your attention drifts, where the pacing lags, and whether what's delivered actually matches what was promised. That alone will tell you more than another round of survey data.
Want a deeper framework for applying this kind of analysis? Read our white paper, Your Event Looks Fine On Paper, for the full methodology.
About the Author
Aaron Wolowiec, FASAE, CAE, CMP Fellow, is founder and president of Event Garde, where he and his team have partnered with more than 150 associations and mission-driven organizations on facilitation, strategic planning, learning design, and event strategy — from curriculum development to multi-track conference education. A certified Technology of Participation (ToP) facilitator and mentor trainer, Aaron co-created FACILIT8me, a certificate course and community of practice dedicated to elevating the craft of facilitation.