Closed captioning

Hundreds of millions around the world rely on closed captioning to be able to understand what they’re watching on TV. While the idea seems simple—just add words to relay the dialogue and describe any sounds—it took decades to mandate processes for making entertainment accessible to the deaf and hard of hearing community, as well as the elderly.

Inspired by text in silent films, closed captioning’s progress in the US started with captioning pre-recorded newscasts and weekly primetime movie nights in the 1970s and 1980s. Back then, people had to shell out money for extra devices to enable the CC option on their TVs. Now, video viewership has exploded on social media, and new forms of technology are trying to improve often-imperfect live transcription (but still failing to reach 100% accuracy). Just one day before this email was published, Instagram announced the immediate rollout of captioning on IGTV.

It’s all happening. Flip on the CC setting, and let’s get to reading.

5%: Share of the global population that is deaf or hard of hearing

15%: Proportion of American adults who report hearing issues

80%: Proportion of UK closed captioning users who aren’t deaf or hard of hearing

$520: Price of a commercial TV decoder that would translate coded captions into text on TV screens in 1980. The device would cost $1,600 today.

250+: Words per minute a real-time captioner has to be able to handle

>1,600: Complaints the US Federal Communications Commission received over accuracy of TV captioning in 2014

$100,000: Annual salary a US stenographer could earn in 2015

Silent films of the early 1900s inserted cards with lines of text for dialogue or narration, but the cards were cast aside when sound was introduced to film in 1927. Those intertitles were the first use of open captioning (OC), the term for dialogue text that is permanently fused into the film for everyone to see regardless of choice. This was also used in the first captioned TV production, Julia Child’s The French Chef, in 1972.

Closed captioning (CC) allows the viewer to choose whether to have the text transcription on screen. The first closed captioning programs in the US aired on ABC, NBC, and PBS in March 1980. Old style captioning is often white text on black background lines, although new technology may let TV owners redesign the appearance of the captioning. Then, of course, there are subtitles. There’s a difference between the two—captioning includes text descriptions of music, background noise, or even audience applause. Subtitles are straight translations, often from one language to another, of only spoken audio. In other words, subtitles assume you can hear, but don’t know the original language.

“What I saw was great irony. Our country’s scientists could send a spaceship to the moon and back, but we couldn’t put captions on television for millions of deaf people who were watching it!”

—Harry Lang, professor emeritus at Rochester Institute of Technology, told TIME about the lack of captioning during the Apollo moon landing

1947: Emerson Romero, a deaf former silent film actor, develops the first technique for captioning movies with sound by splicing images with text between picture frames, similar to the text cards of the silent film era.

1949: Captioned Films for the Deaf, now the Described and Captioned Media Program, is established. Its activities are taken over by the US government in 1958.

1972: The Caption Center, now the Media Access Group, starts work at WGBH in Boston, Massachusetts, first designing the CC logo.

1979: The federal National Captioning Institute nonprofit organization is established in the United States, with the first closed captioned TV programs airing a year later.

1985: The first Super Bowl with closed captioned live commentary airs.

1990: US Congress requires all television set makers to include built-in captioning decoders, nixing the need for deaf and hard of hearing people to purchase expensive decoders.

1996: The US passes the Telecommunications Captioning Act to require it on all TV programming.

2009: Google rolls out automated captioning for any video upload.

2010: The US signs the 21st Century Communications and Video Accessibility Act, which requires captioning for programming on smaller screens or the internet.

There are a slew of ways to add closed captioning, but no process is foolproof. For pre-recorded shows, like primetime TV episodes, captions are done in post production before going to air. Before you think, “Oh I could type a provided script in my sleep,” there’s a lot more to it: The editing task entails the screen placement of captions and making sure any ad libbed or non-speech sounds are added.

For live sports events or newscasts, it gets harder—a lot harder. Some local news operations or small broadcast programs are limited to “text crawl” captioning that displays what’s in the teleprompter as it’s audibly said. But this doesn’t account for speed or any ad libbed dialogue or reporting. (There’s actually a petition to ban teleprompter use in captioning.) The live captioning process is typically done by captioners, or stenographers, who quickly type on a special keyboard. There can be a lag of a few seconds, and there can be errors since the stenographers are processing hundreds of words a minute.

New technologies are now trying to potentially improve viewer experiences with closed captioning, using artificial intelligence and machine learning to provide faster captions with higher accuracy. For example, IBM Watson offers video speech to text technology that can be used for captioning. There’s also automated speech recognition programs, similar to Siri and Alexa, but they still meet the same challenges around accuracy and viewership satisfaction. Perhaps human captioning is best, at least for now.

Maybe if commercials were captioned, viewers would be able to focus on them for longer. As it stands, there are a lot more demands on our time than there used to be.

The last 20 years have also seen the rise of entirely new vehicles for advertising—web search and social media—that attract billions of eyeballs. Consumers are everywhere and advertisers are struggling to figure out how best to reach them.

There has been a noticeable rise in closed captioning usage by… Gen Z. Younger, hearing people have opted to turn on CC for various reasons, from multitasking to education to ADHD to comfort.

Captioners at National Captioning Canada share their process for describing and transcribing live productions.

Netflix wasn’t really game to caption content. The National Association for the Deaf brought a class-action lawsuit against the streaming platform in 2010, arguing that Netflix was violating the Americans with Disabilities Act (ADA). The platform spent more than two years trying to get the lawsuit thrown out, saying it didn’t need to meet ADA requirements. A judge ultimately disagreed, and Netflix agreed to caption all of its content by 2014.

The company processed captioning in the order of most-watched to least-watched films or TV shows. Then, it agreed to caption new content within seven days of release by 2016. Now Netflix has set the bar high and says, “Ultimately, we need to change the way we think about subtitles and closed captions. They are no longer secondary assets in a world where content knows no physical borders.”

We hear you. Our Hitting the Open Road sweepstakes runs through Sep. 17, so don’t forget to enter for a chance to win a (socially distanced) vacation, including a National Parks pass, REI gift card, and a lot more. If you’re in the US, check it out.