At last October’s introduction of the new iPad Air, the creators of a clever iOS app named Replay were invited on stage. To get there, they went through a selection process that illustrates Apple’s perfectionism—and hidden application sophistication.
In September 2014, while at the Stupeflix Paris office, Nicolas Steegmann got a call from Apple in Cupertino. Once the caller identified herself, Nicolas knew something up. The contact came after Stupeflix’s presentation to Apple’s team in Paris. In rather elliptic terms, Steegman’s interlocutor said it would be great if two members of the company—a developer and a designer—could be in Cupertino the next day. “They will have to stay at least two weeks,” she said. Forty-eight hours later, the team was on Apple’s campus. They quickly found themselves in a windowless room and given a straightforward brief: Devise the coolest possible demo for your app. No more details, no promises whatsoever.
Replay is a clever iOS application that focuses on a “simple” issue: Automating the process of making videos, without going through the convoluted steps of a dedicated movie editing app. With Replay, you shoot with your iPhone (or your iPad), select the clips you want, pick one of the proposed themes, and you’re done. The app will assemble the clips in the smartest possible way, making visual corrections, and adjusting the soundtrack selected from your iTunes library (or drawn from a proposed catalog) to the pace of the movie. If you have the time and inclination, additional settings let you fine-tune your production. But even if you just stay with basic preset themes, the result is stunning. In literally a few seconds, you end up with a clip perfectly suited to quick sharing on YouTube, Instagram, or Facebook.
Behind Replay’s simplicity are years of work and a great deal of sophisticated programming. The company’s roots are in an automated video generation system originally designed for completely different goals.
As is often the case, this company’s final product has little to do with the original intent.
Stupeflix is a pure engineers’ startup. It was created in 2008 by Nicolas Steegmann, an engineering and mathematics graduate from École Centrale de Paris, and François Lagunas, who holds a PhD in computer science and linguistics from Polytechnique and École des Mines. Their first product was an automated video generator that scrapped images and text from Wikipedia and other sources, and inserted text-to-speech voice-overs, to create 45 second glances at various cities and places around the world. The result was more a demonstrator than a commercial product (you can still access hundreds of automatically generated videos here on YouTube).
The concept paved the way for a much more marketable product: a system to create videos entirely online, with preset themes—the ancestor to the Replay app. Its business model was (and still is) based on the proven freemium mechanism: Casual use is free, scaling to a professional/intensive use requires a subscription.
The same went for the next iteration: a home-brewed API allowing third parties to use all the tools Stupeflix developed to create videos. As a result, digital advertising agencies Publicis, Saatchi, and TBWA jumped on it. Hundreds of thousands of videos were created for Coca-Cola, Red Bull, or Sprint, to be used in countless promotional operations. Stupeflix still derives significant revenue from its API business.
Technically speaking, editing and rendering a video is CPU intensive—GPU intensive to be more precise—and Stupeflix’s APIs suck a lot of graphic processing power. At the time that this was developed, explains Nicolas Steegmann, graphics rendering was outsourced to a specialized server farm in Texas (where the oil industry consumes loads of computational power for geophysics modeling). Now, Stupeflix relies on Amazon Web Services, which has since cornered the CPU/GPU for-hire market.
It took 18 months to port the video rendering engine to iOS. Many invisible features had to be pared down to match the capabilities of the iPad/iPhone processor, because unbeknownst to the user, Replay performs many complex graphics tasks. For instance, it analyses each piece of raw media material picked by the user. Color palette and saturation, exposure, motion, and pace are decomposed and translated into mathematically useable components. These chunks of data are then fed into a “cinematographic grammar” hard-coded by Replay’s programmers (all movie enthusiasts). Each theme or skin selected by the user reflects a Quentin Tarantino or Alfred Hitchcock inspiration that will direct transitions, colorimetry, beat, as well as soundtrack sync. And an embedded machine-learning engine also devises new rules by itself.
The fluidity of Replay’s performance caught Apple’s attention during the summer of 2014, a couple of months before Tim Cook unveiled the new iPad Air.
Now secluded in their room on the Apple campus—always escorted when entering or exiting—Stupeflix’s team is hard at work devising the most mind-blowing demonstration of their app. Early on, they had a hunch that the whole process was in fact a competition among applications (12 contenders, as they would later discover). For two weeks, a quiet selection process took place, with a stream of people visiting the Stupeflix team, testing its app on the last version of a new iPad—camouflaged in a thick neoprene enclosure to conceal its size and shape. Each successive visit was made by someone ranking higher and higher in the chain of command—as the team realized by Googling the reviewers. They knew they were on the short list when their demo was shown to Phil Schiller, Apple SVP for Worldwide Marketing. The next day, the pair was taken to a conference room where their work was reviewed by Tim Cook in person. They knew it was a go. It was time for a series of full rehearsals.
On D-day, the two-minute presentation was to be made by Jeff Boudier, the Stupeflix man in San Francisco (and co-founder of the company), assisted by François Lagunas controlling the iPad. It went well, except when a slip of a finger (due to an excess of makeup applied to the demonstrator’s hand) caused the auto-correct to transform the title “Utah Road Trip” into a weirder “It’s a road trip.” After the show, Apple staff asked to re-record the demo for spotless posterity (the re-edited version is here on Apple’s site, while the original is here; the demonstration starts at about 00:55:10 on the two keynotes). This says much about Apple’s attention to details.
An epilogue: Replay became a hit, generating substantial revenue thanks to the in-app purchase system. Stupeflix now employs 23 people, all of the same caliber as the founders. To stimulate the team’s creativity, management keeps holding internal hackathons, and they continue to build on the uniqueness of their video algorithms and rendering engines. The company recently came up with Steady, a spectacular app that gives the impression your iPhone is mounted on a Steadicam (a complex system crops each frame in real-time to compensate for unwanted motion) and Legend, that animates texts on the fly. They are now working on another movie capture app that will further transfer the burden of filmmaking from the user to the software. Call it talent by proxy.
This post originally appeared at Monday Note.