Advice & News

February 15, 2021

Pilot Testing: A Low-Risk, High-Return Approach to Innovation


Iurii Motov/Shutterstock

"Do things to discover what to do," Peter Sims said. In his book, "Little Bets: How Breakthrough Ideas Emerge from Small Discoveries," Sims wrote about using small experiments to undercover large results -- a practice Google, Amazon, and other tech titans have used for decades. Little bets, he said, are "concrete actions taken to discover, test, and develop ideas that are achievable and affordable."

Last month I wrote about some techniques to get faculty talking and generating ideas for their institutions. An astute reader commented, "These are great ideas, [but] this will only work if there is leadership that allows free thought and is committed to trying the new ideas that result." True enough, once new ideas are born, leaders need to either move them along into reality or shelve them. Either way, they are making a choice.

Sometimes leaders may feel uncomfortable committing to a new idea or intimidated by the prospect of starting a large initiative. This is often how we think of leadership action -- taking some grand step to implement a major project. In reality, however, the most effective leadership actions can be the smallest. This is where pilot testing comes in.

Pilot testing is a key part of the design thinking process. Pilot testing allows you to test, measure, improve, and retest until you get the concept right. Some of the largest innovations in higher education have started as small, obscure experiments. For example, the idea that small "nudges" via text message can influence student enrollment, financial aid, and degree completion started as a pilot testing project. The history of online education is replete with small innovations building upon each other to create a strong foundation of support.

Why is pilot testing so powerful, and how can leaders use it effectively?

Proving Concepts

The first obvious benefit of pilot testing is proving if a concept works or not. Faculty may produce the best, most detailed proposals to back up their ideas, but testing the concept yields insight not possible any other way. One dean of education at a large public university shared her philosophy for responding to a new faculty proposal. She stated, "I might say, 'I can't afford to do that,' or, 'the budget doesn't allow it, that's a lot of money.' But then again, I might say that since they took the initiative, and this is a great idea, then why not do a pilot program for two years, collect data, and see if it's worth the college's investment in that?"

Often, it's not possible to run a full version of an idea, like an entire degree program, to see if it is feasible. The average new program takes 9-10 months to achieve final approval and 24 months until the first students matriculate. Leaders need shorter, lower-stakes ways to prove concepts. A great way to do this is to offer individual courses, continuing education classes, or certificate programs prior to full-scale degree programs. This allows leaders to understand the demand signal from the market without committing to the long-term.

Preserving Faculty Investment

One of the worst demotivators for faculty is when ideas are solicited but no meaningful action results. 'I spent all this time away from my teaching and research, and nothing ever came of it!' they may think. One business dean at a small private institution cautioned, "I'm always careful about not asking people to put together proposals that you are not going to consider…I don't do that because it exhausts people, and it undermines peoples' trust in you."

On the other hand, if a faculty member feels their ideas are considered, they can become more invested in the institution. Another education dean at a mid-sized private university shared how a faculty member from another college approached her with the idea for a cross-disciplinary online global development program. His idea had been ignored in his home college due to the amount of work required. Instead of punting the idea back to the other college, the dean tasked the faculty member to pull together a committee, develop a proposal, and test the concept in various ways. With this encouragement, the faculty member was empowered to pursue his idea and completed the necessary work to prove it successful.

Other people's ideas are like delicate flowers. With nourishment, they can bloom and brighten. With neglect or rough handling, the petals can wither away or fall off altogether.

Inspiring More Innovation

Lastly, innovation can beget more innovation in a virtuous cycle. Sims said, "Once a small win has been accomplished, forces are set in motion that favor another small win." For example, a faculty member at one private college in the Northeast, with very few graduate degrees, had the idea to start a clinical mental health master's program. It was not popular with colleagues in her department, but her dean decided to back her. They brought the results of their pilot test and research to the provost, who later recounted, "When we looked at the research she pulled together, it was really good, and it was clear that she was right. …This new graduate program now seven years in has generated over $4 million in net revenue, 160 students, and it's a great success story." Since that time, the school has opened numerous other graduate programs for working adults and has been recognized as among the most innovative colleges in the nation.

In conclusion, simply gathering the best ideas is not enough to spur innovation and change. Spending a small amount of energy to test concepts now may actually help conserve a larger amount of energy in the future and uncover breakthroughs that could not be seen before. This is as applicable to university faculty, departments, and colleges as it is to entrepreneurs and startups.

How have you seen pilot testing work in your institutions? What pitfalls or snags should we be aware of, and how can leaders successfully use this practice?