NBC Relies on Canvs Surveys to Get the Full Story During Pilot Season


AI-powered Open-Ended Responses Processing Empowers TV Researchers to
Uncover Critical Insights Faster, More Consistently, and at Scale.

The Challenge

With over 500 new TV series airing each year, competition for viewers is fierce. The shows that win have one thing in common – they create intense emotional connections with viewers. But how can networks measure Emotional Reactions with any sort of accuracy, especially at the breakneck speed and scale of TV’s pilot test season? NBC found its magic bullet in Canvs Surveys.

In the following study, NBC compared the performance of three promising drama pilots using Canvs Surveys by analyzing open-ended responses. The dramas had scored nearly identically in close ended questions, but researchers suspected deeper differences. Uploading the open-ended responses to Canvs Surveys, they immediately saw how the drama pilots stacked up emotionally, findings that were validated by real viewership numbers in the following season.

The Methodology

Processing open-ended responses is as simple as uploading a spreadsheet. Canvs Surveys immediately jumps into action, using its one-of-a-kind unnatural language processing engine to decipher trillions of words and expressions, including slang, acronyms, emojis, misspellings, and subtle nuances such as sarcasm and “shade.”

Within minutes, NBC researchers were able to view and analyze a detailed breakdown of the Emotional Reactions captured in their open-ended responses, as well as the Emotional Drivers behind them, such as character, acting, or music. For context, Canvs Surveys also compared the results to benchmarks for NBC dramas over the past two years.  

NBC then compared the results of all three pilots, quantifying the differences in their emotional impact and the reasons for it.


Canvs quantified the intensity of Emotional Reactions in NBC’s dramas, and indicated what drove the emotion.


The Findings

When the Emotional Reactions for all three drama pilots were compared, four stood out as the most intense and most significant: Love, Enjoy, Interesting, and Dislike. While all three dramas performed well against historical benchmarks, there were significant differences between them:

  • More Love and Enjoyment Reactions signaling more intense emotional connections.

  • Less Interesting and Dislike Reactions indicating fewer challenges and barriers.

Drama #3’s Canvs Emotional Score for Love, Enjoyment, Interesting & Dislike performed well above the emotional norm for dramas. Therefore, NBC was able to successfully predict that Drama #3 would receive the highest average S1 (Live + Same Day) ratings, as reported by Nielsen. In fact, the data was so compelling that NBC used Canvs’ Emotion Graph to guide its advertising allocations for the three dramas.

Validated by Viewership
In the season that followed, the insights revealed by Canvs Surveys were proven true by actual viewership data. Drama #3 outperformed the others in terms of viewers and repeat viewers, no doubt driven by the fact that these three emotions significantly over-performed against the drama relative norm:

Screen Shot 2018-09-05 at 4.05.31 PM.png
  • Drama #1 had 3.6 million live + same day for the premiere, and over the course of the season averaged about 2.2 million.

  • Drama #2 had 7.5 million live + same day, and a season average of 3 million

  • Drama #3 had 10.1 million live + same day, and a season average of 3.7 million

The best part? While finding critical insights like these, NBC has on average saved 15 hours in analysis time per survey.

Key Takeaways

To get the full story about the potential performance of a pilot drama, NBC researchers have to use open ended survey questions. Close-ended questions miss too many important nuances, not least of which is Emotional Reactions.

With Canvs Surveys, NBC was able to automate the processing of open ends and make a giant leap forward in its pilot testing research. Canvs Surveys delivered quick, accurate, consistent results at scale, so NBC can keep the drama of pilot testing in the acting and out of the research department.

Uncover Critical Insights Faster, More Consistently, and at Scale.


Contact Us to Understand How
People Truly Think & Feel in Your Open-Ended Responses