Zoox Gives First Look at Startup’s Mysterious Robo-Taxi

Of the almost 50 companies that have permits to test autonomous vehicles in California, the most radical and mysterious member of this self-driving car club is Zoox. At long last, however, the startup with the funny name is ready to show some of what it has been up to on San Francisco’s streets.

Founded in 2014, Zoox Inc. has raised more than $250 million. It has used this money not only to develop self-driving technology but also to rethink the car altogether. Unlike most, if not all, of its rivals, Zoox has tried to build a new type of vehicle—one that assumes self-driving abilities from the get-go. This means no steering wheel, seats that face each other like those in a limo and a host of novel safety features that a true robotic car would need in order to communicate with other drivers and pedestrians to make its intentions clear. Add to this an Uber-like service for requesting rides and managing a fleet of cars, and, well, Zoox has taken on an awful lot. The company is poised to either revolutionize the transportation industry or flop with rarefied skill, and, my word, do many of us want to know which it will be.

In mid-November, Tim Kentley-Klay, Zoox‘s co-founder and chief executive officer, took me for a test drive in one of the company’s vehicles to try and make the case for revolution over ruin. Since its inception, Zoox has tried to operate in near-total secrecy, and so this marked a rare peek at its technology.

Kentley-Klay and I stood on a sidewalk near San Francisco’s Ferry Building and summoned a car with Zoox’s prototype app. A couple minutes later, a matte black Toyota Highlander arrived with a roof rack full of sensors, a trunk full of computers and a passenger cabin full of screens showing what the car can see. After a tap on the app to begin the journey, the SUV merged into traffic, and our 10-minute or so drive commenced. (Zoox’s more radical, futuristic vehicle without the steering wheel has yet to hit the San Francisco roads but is being tested at its research facility in Menlo Park, California.)

Autonomous cars already do pretty well on the highway, where conditions are more controlled and predictable. The great puzzle in need of solving has been city driving, particularly in a place like San Francisco with its funky roads, streetcars, thick traffic and oblivious pedestrians who value checking their Instagram feeds over maintaining long, productive lives.

The company that has received the most press to date for driving in a dense, urban environment has been General Motors Co.’s Cruise Automation—which gave a number of reporters test drives of its Chevy Volt in San Francisco this week.Waymo, which is Alphabet Inc.’s self-driving tech arm, has tested its vehicles for years on the sleepier streets of Mountain View, California, and more recently in suburban Phoenix, and Uber has done some city driving, too. Since its start, Zoox has been all about making a car suitable for dense, urban streets and seems well on its way based on what I experienced.

During my drive, there were two people sitting up front making sure nothing went really wrong. The driver had her hand covering the wheel at all times, and the passenger had a laptop that revealed in gory detail what the car’s computers could see and what the car was expected to do. Kentley-Klay says that human intervention remains necessary on some of the drives, but the humans were never summoned during my test. This was a 100 percent robot affair.

The screens inside the car depict a world rich with information. The computer assigns different colors to different objects: Buildings are gray; cars are purple; stoplights are pink; and pedestrians are orange. The computer gives a unique identifier to every object and provides details on that object’s location and speed. Even while classifying objects, the Zoox software can do pretty nuanced things like telling the difference between a person on a motorcycle and another on a bicycle going at similar speeds or when it should give a parked car some extra space because a person is about to open the door.

During the straightforward parts of the drive, the Zoox vehicle behaves much as one guided by a human would. It follows the car in front at a safe distance and observes the actions going on around it. The major thing I noticed during the stop-and-go traffic was that the car seemed, now and again, to stop more abruptly than a decent human driver would.

One of the big tests on city streets is a left turn where the car has to pause at an intersection, and judge the speed of oncoming traffic and the flow of pedestrians before cutting across lanes of traffic. Kentley-Klay, sitting to my left in the middle row of seats, called out this challenge to me early on during our drive, and the car handled the task as well as a person. It held steady at the intersection and let about 10 cars pass to its left and then made sure all the people crossing the road had finished before proceeding with a turn on a green light.

Another big challenge came at a four-way stop, where cars were hitting the intersection at different times and with different levels of aggression. The Zoox vehicle knew precisely when its turn to go had arrived and went quickly to the stop sign, paused briefly and then zoomed across the street. It was a subtle move that helped it get ahead of a feisty driver to the left who was trying to jump the line and seemed exactly like something a human would do to get a fair shake at the intersection and to avoid confusion.

The car was doing an “intersection precedence” maneuver where it tries to balance the right-of-way rules with how people actually behave. “You want to be assertive and still safe,” Jesse Levinson, Zoox’s co-founder and chief technology officer. “If you wait until the intersection is totally clear, you will annoy a lot of people.”

Kentley-Klay, wearing a floppy driving cap, readily admitted that the drive had been set up to succeed. Zoox has practiced a lot on the circular route we took around the Embarcadero. This means that the car has an intricate picture of its surroundings and that its artificial intelligence software has been trained to deal with the obstacles it’s likely to encounter. That said, the test drive did come with some of the hardest challenges. “Sometimes these drives are boring,” says Kentley-Klay. “Yours was nice because it was action packed.”

To train its cars, Zoox pre-drives routes with a human at the wheel. It then builds out the rich, 3-D map of the surroundings and, as needed, does some manual annotations of the streets with an engineer noting a crosswalk or making sure other road markings are correct. “Once we have done that, the car will drive autonomously quite well,” says Levinson. “We have a pretty high degree of proficiency driving in downtown San Francisco now.” Zoox has yet to have a traffic incident while in autonomous mode, although a San Francisco driver did crash into one of its vehicles that was being driven by a human.

Overall, the test felt incredibly impressive. I’ve been in a host of autonomous vehicles, and, without question, Zoox trumped them all in terms of the complexity of situations it could handle. We’re not privy to the inner-workings of all the self-driving car companies, but, from what has been displayed publicly, it seems that Zoox and Cruise are the furthest along with urban driving.

The bummer, of course, is that there’s little more to say about how close Zoox is to its ultimate goals. My ride took place in a modified Toyota—not one of the company’s homemade, futuristic cars. Despite being one of the top-funded automotive startups, Zoox still needs to raise more money, manufacture its own vehicles, hire more engineers and perfect the self-driving technology while locked in a battle for the ages with well-moneyed Silicon Valley competitors like Waymo, Tesla Inc. and Uber Technologies Inc. and, you know, companies that already know how to make cars like GM, Ford and BMW. There’s a chance that Zoox gets acquired along the way, although Kentley-Klay has long been adamant that all of his competitors have been too conservative with their car designs and that he wants to build a massive, lasting company.

Zoox has some theoretical echoes to a company like Apple Inc. Kentley-Klay is an Australian designer by background and has been playing the role of Steve Jobs—a brash salesman with good taste and hyperbolic levels of ambition. Levinson is the quiet, genius engineer playing the role of Steve Wozniak. Levinson has worked on self-driving technology for about a decade, going back to his time as one of the heads of a Stanford University team that performed well in early autonomous vehicle competitions. His expertise has allowed Zoox to attract other top engineers and gives the company the technical credibility to take such a big swing.

Zoox, of course, will need to live up to its world-changing hopes and dreams if it’s to one day be a Silicon Valley giant. For the moment, the founders are taking solace in the belief that they no longer see any technological hurdles that will prevent self-driving cars from coming to market. Zoox is sticking by its plan to begin picking up customers in autonomous vehicles by 2020. “At this point, we don’t consider self-driving or Zoox to be a moonshot,” says Kentley-Klay. “Now it comes down to execution.”