I've been sitting with this one for a while because it touches something I've watched play out enough times, including from the inside, that I wanted to make sure I was saying it right.
That distinction is costing us more than we acknowledge.
I remember sitting in a conference room after a skills-first pilot review a few years ago. The mood was optimistic. The data looked promising. The pilot had produced stronger candidate pools, managers were saying the right things, and someone had already started talking about how to scale it across the organization.
And yet, I had this uncomfortable feeling that almost none of it was going to stick. Because the pilot had succeeded under conditions that didn't exist anywhere else in the business: the managers involved had extra support; the assessments had been carefully designed; the right leaders were paying attention; everyone participating knew they were part of something important.
But underneath the pilot, the actual operating system of the organization remained mostly untouched. The job architecture hadn't changed. Performance management hadn't changed. Incentives hadn't changed. And the managers who would eventually inherit this work were still being asked to navigate the same pressures, risks, and ambiguities they always had.
Over the past several years, organizations across industries have launched thoughtful, well-intentioned skills-first pilots. Some focused on hiring, while others focused on internal mobility, apprenticeships, reskilling, or new assessment approaches. Many of them produced genuinely encouraging results.
But then something predictable happens. The pilot ends. Executive attention shifts. The support structures begin to disappear. And slowly, often quietly, the organization starts reverting back to the same proxies and decision-making patterns it was trying to move beyond. Degrees reappear in job descriptions. Managers fall back on pedigree and familiarity. Hiring teams default to speed over signal. Learning investments drift away from workforce demand. And what once felt like transformation becomes another promising initiative that "never fully scaled."
What's striking is that organizations often interpret this as an adoption problem. I've heard executives say that perhaps managers weren't ready, or the culture is resistant to change, or the business moved too fast. But in many cases, the real issue is something deeper and more structural: the pilot succeeded without requiring the system itself to change.
The conversation that stayed with me
A few weeks after one pilot officially wrapped, I caught up with a Talent Acquisition leader who had been one of its strongest advocates. During the pilot, she'd talked openly about candidate quality, how much more structured and equitable the interviews felt, and how surprised she was by the talent they'd surfaced through skills-based assessments.
I asked whether her team was planning to continue using the approach more broadly. She paused, then said:
I asked why not.
What followed had almost nothing to do with whether she believed in skills-first. She talked about hiring timelines. Managers already feeling overloaded. Not enough people trained to assess skills consistently. Concerns from senior leaders if a nontraditional hire struggled in a high-visibility role. Recruiters quietly defaulting back to degrees because it was faster and easier to defend.
Then:
That sentence has stayed with me because she was describing organizational reality. The pilot had not redesigned the system. It had temporarily made the existing system safer to work around. When the pilot ended, the system resumed.
This is the pattern worth naming clearly. Most managers caught in this dynamic are not rejecting skills-first philosophically. They are trying to operate inside systems that still reward legacy behavior. They're still being evaluated on speed, short-term delivery, and execution under pressure. They're still working with role definitions built around degrees, tenure, and pedigree. They're still making difficult talent decisions without consistent assessment infrastructure or clear organizational standards.
So when pressure increases, they revert to the signals the system already understands because the system around them never fully changed.
Why the design flaw hides
This is where the design flaw usually hides. Pilots are often built to prove that something can work under supported conditions. Systems, however, must operate under normal conditions, with competing priorities, inconsistent managers, time pressure, imperfect data, organizational politics, and changing business demands. Those are very different environments.
And many skills-first pilots quietly depend on conditions that are difficult to sustain at enterprise scale: extra coaching, specialized implementation teams, hand-picked managers, carefully curated candidate pools, temporary flexibility around policy or process. None of those things are inherently bad. In fact, many are necessary early on. But organizations rarely stop to ask the harder question: what happens when the pilot becomes the responsibility of the broader system? That's usually where the real friction begins.
A pilot should not simply demonstrate that an idea can work. It should reveal what the organization would actually need to redesign in order for that idea to persist under normal operating conditions. That's a very different goal, and it requires a very different kind of design.
In practice, most pilots remain too isolated from the systems they are eventually expected to influence. They sit adjacent to the business rather than inside it. They rarely force organizations to confront the harder questions around governance, incentives, accountability, job architecture, or managerial risk. And because those underlying systems remain intact, organizations mistake early enthusiasm for readiness to scale.
What a pilot should actually test
So what would it look like to design a pilot that actually sets up scaling? A few things I've come to believe matter:
Stress-test the system, not just the concept. Before launching, ask which existing structures this pilot will eventually have to survive. Job architecture? Performance management? Manager incentives? Then deliberately test at least one of those pressure points during the pilot. Don't leave them for later.
Name the scaffolding explicitly. Every pilot runs on supports that won't exist at scale. Name them out loud at the start. Not to eliminate them, but to be honest about what the organization will need to build or redesign before those supports come down.
Treat "what broke" as the most important data. Most pilot reviews focus on what worked. The more valuable question is where did the system push back? Where did managers hedge? Where did the old logic creep back in? Those friction points are the redesign roadmap.
Define scaling conditions early. Define, in advance, what would need to be true before this pilot could expand. Not metrics alone, but structural conditions. Who owns the governance? What changes to job architecture are required? What manager support infrastructure needs to exist? If you can't answer those questions before the pilot ends, you're not ready to scale.
Designing better pilots matters. But it only gets you so far. Because what pilots ultimately expose, whether organizations intend it or not, is whether leadership is actually willing to redesign the systems around them.
The uncomfortable questions
And eventually, every organization trying to scale these efforts runs into the same uncomfortable questions:
- → Who actually owns skills standards across the enterprise?
- → How are managers being trained and protected when making skills-based decisions?
- → What incentives reinforce or undermine the behavior we say we want?
- → What happens when the pilot collides with performance management, hiring timelines, or compensation structures that were never designed for skills-based decisions?
And perhaps most importantly: Organizations eventually have to decide whether they are willing to build a new talent system or simply layer new language onto the old one. Because eventually every skills-first effort collides with the same reality.
That TA leader knew what was missing. She could name it precisely. What she couldn't do was redesign the system alone. That was a leadership decision that had not been made yet. And until it is, the system will keep winning.