CUPERTINO, California -- The Rainbow Mansion sits high on a hill overlooking Silicon Valley. With a roof of Spanish tile and a chandelier in the foyer, this 5,000-square-foot West Coast mini-palace was once home to a man who made his millions selling chips for PC graphics cards and CD-ROM drives. But today, it's a kind of Silicon Valley commune, a place where the Young Turks of the tech world can live and share their work.
Googlers live here. And NASA engineers. One resident builds electric cars at Tesla, and yes, Apple engineers have lived here too. But the whole idea of the place is that residents aspire to something more than a day job. Just off the foyer, there's a communal library that hosts regular salons -- free-form discussions where residents have called for everything from the death of nationalism to a world where the free flow of information across the net outweighs the need for personal privacy. In the garage, there's a makeshift hardware lab where Mansionites build miniature submarines, satellites, and unmanned aerial vehicles you can fly with your Android phone. And over the years, more than a few startups have been launched from these same rooms.
If you want to live at the Rainbow Mansion, says one former resident, you have to want to change the world. That's an easy thing to say. But the Mansionites have already changed the world, bringing their communal ethos to the most important open source project of the decade: OpenStack, the Linux of cloud computing.
OpenStack is software anyone can use to build their own version of Amazon's Elastic Compute Cloud, the massively popular web service that gives developers and businesses instant access to virtual servers. The roots of OpenStack stretch back only about four years to a skunk works project inside of NASA, but it has already overturned the status quo in both the private sector and the public. After catching the eye of Vivek Kundra -- the country's first CIO -- it's used not only by NASA but by other operations across the federal government. After it was launched to the rest of the world through an unlikely partnership between the space agency and Rackspace -- the Texas outfit that trails only Amazon in the cloud computing game -- it's now backed by over 150 companies worldwide. And it's shaping the future of such names as HP, Cisco, Dell, and -- if the rumors are true -- IBM.
"There are a lot of folks out there taking this very seriously," says Zorawar “Biri” Singh, the man who oversees HP's effort to build a cloud service atop OpenStack. "You can look at the list of members, but we're talking to others, including some big-time architects who say they have big plans for this. There's a lot of work to be done, but we're positive this is something we can really scale."
>"This could have fallen apart in a million different ways, from the beginning. In fact, it all seemed impossible"
OpenStack has many founders across NASA, Rackspace, and beyond. But several of the most important players were a regular part of the tech commune that thrives at the Rainbow Mansion, including Chris C. Kemp, 34, one of the freethinkers who founded the Mansion when they joined NASA's Ames Research Center in 2006. "We were just looking for a place to live," says Kemp. "But it turned into a place where the idea was to recruit interesting people -- interesting people to have dinner with, to run into in the common areas, to be around a lot -- people who could expand our understanding of the world."
Kemp went on to become the chief information officer at Ames and later the chief technology officer of NASA as a whole. While there, working alongside several others with close ties to the Rainbow Mansion, he spearheaded the creation of NASA Nebula, an effort to bring Google's web genius to the rest of the world. And after two years of struggle, a key part of this project -- an open source platform called Nova -- would merge with a complementary platform from Rackspace and give birth to OpenStack.
Like Linux, OpenStack is a bit of a miracle. The odds were against Kemp even getting Nebula off the ground at NASA -- not only because it's somewhat tangential to the agency's mission, but because the NASA bureaucracy was so unsuited to the creation of something openly shared with the rest of the world. And NASA is only half the story. It's even more remarkable that a project created at NASA would so quickly find a home among the giants of the tech world.
"This could have fallen apart in a million different ways, from the beginning. In fact, it all seemed impossible," says Rick Clark, who worked at Rackspace when OpenStack was in its infancy and now helps drive the project at Cisco. "You have to please NASA and the NASA legal team and the Rackspace legal team and the Rackspace board of directors, and you have to do it in a way that still have something that's palatable to developers everywhere else. It's amazing that it actually happened."
NASA's partnership with Rackspace was rooted in a blog post that appeared on the web in the early summer of 2010 and was read by almost no one. "Launched Nova," wrote Joshua McKenty, a NASA contractor and a regular guest at the Rainbow Mansion. "Apache-Licensed Cloud Computing, in Python. It’s live, it’s buggy, it’s beta. Check it out."
Built with the Python programming language, Nova was basically a mimic of Amazon's Elastic Compute Cloud. Amazon EC2 lets you run software applications without setting up your own physical machines, spinning up more virtual servers as you need them. The difference was that Nova was open source, and since it carried an Apache license, there were no restrictions on anyone who might want to use the platform as a way of building his own version of EC2.
A week after Nova hit the web, someone told Rick Clark it was there. Clark had joined Rackspace the previous September from Canonical -- the outfit behind the Ubuntu Linux distro -- and he says his mission was to create an open source platform for building "infrastructure clouds." Several Rackspace executives were pushing for the same thing, including vice president of corporate development Jim Curry and chief strategy officer Lew Moorman.
As Clark looked over NASA's Nova code, he couldn't get over how similar it was to the platform Rackspace was planning to open source on its own, and he quickly sent an e-mail to the big guns at Rackspace. "[NASA is] about three months ahead of us," he wrote. "We need to talk to them and find a way of working with them or fork the code."
Clark didn't for a second think NASA would actually work with Rackspace. But the company had karma on its side. Clark's boss -- vice president of corporate development Jim Curry -- is the son of a NASA engineer who for 40 years worked on the thermal protection systems for the Space Shuttle and other spacecraft. He grew up in the shadow of the agency's space center in Houston, Texas. "I'd take my Boy Scout troop on tours of the engineering lab," he says.
Curry tracked down Kemp at NASA, and a week later, several Rackspace employees flew to Silicon Valley for a sit-down with the NASA team. They met at Ames and later at a Thai restaurant nearby, and the good karma continued.
>"We got the two development teams together -- none of whom had ever met each other -- and we both said, 'Wow, you just wrote the code that we were going to write.'"
"It was like finding a long-lost twin," says McKenty. "It was the weirdest experience I've ever had. We were using the same tools. We had made the same language decisions. We got the two development teams together -- none of whom had ever met each other -- and we both said: 'Wow, you just wrote the code that we were going to write.'"
More than that, their code bases were complementary. While NASA was building Nova, Rackspace had built a platform called Swift. Nova provided virtual servers -- i.e., processing power -- and Swift provided storage. Whereas Nova mimicked Amazon EC2, Swift was analogous to Amazon's Simple Storage Service (S3). At dinner, the two parties resolved to combine the two projects and open source them as one under the banner "OpenStack."
But it was still a long shot. When the check came at the end of the meal, the engineers realized there was no easy way to pay for it. At NASA, both the civil servants and the contractors must pay for their own meals. "We had like twenty people at dinner and we'd shared all this Thai food and there we were trying to divide this check," says Rackspace's Jonathan Bryce. "I thought: 'If this is what working with the government is like, the project will never work.'"
That is what working with the government is like. It was surprising that Nova even saw the light day.
Chris Kemp was hired to change NASA. In 2006, a new director took over at NASA Ames, intent on bringing the agency's Silicon Valley outpost into the internet age, and he immediately brought in some young blood, including Kemp and four others who moved into the Rainbow Mansion -- about a 15-minute drive from Ames.
Kemp had worked as a systems engineer at a Silicon Graphics field office in Huntsville, Alabama, where he supported rocket scientists at NASA's Marshall Space Flight Center. But more recently, he'd served as the chief architect of Classmates.com, that icon of the late '90s dot-com boom.
Seven months after setting up shop in the basement of Building 200 at NASA Ames, Kemp was wheeling and dealing. He inked a partnership with Google that would see the web giant serve NASA's moon and Mars images across the public web. Then, with a second venture, he let Google's top executives park their private jets at the Ames air field.
>"With Google, infrastructure was treated as if it was infinite. They had this laissez-faire attitude that said: 'Sure, we can find a few petabytes of storage whenever we need it.'"
The jet deal brought NASA an extra $1.3 to $2.3 million a year. But Google Moon and Google Mars had a larger impact. Kemp had worked with Microsoft on a similar service, and he was struck by how differently Google and Microsoft handled their backend infrastructure.
"With Google, infrastructure was treated as if it was infinite," Kemp says. "They had this laissez-faire attitude that said: 'Sure, we can find a few petabytes of storage whenever we need it.' With Microsoft, it was just the opposite. The storage was owned by some business unit, and you had to beg, borrow, and steal for the compute resources. It was a much more difficult relationship, because NASA didn't even have the infrastructure to host this stuff."
Over the years, Google had built a sweeping infrastructure -- a software platform spread evenly across a vast array of servers that its developers scale their applications up and down as needed. It was only used by Google itself -- not shared with the rest of the world as Amazon's EC2 was -- but the underlying technical concepts were similar.
And that, Kemp decided, was what NASA needed. He wanted the agency to be able host its own high-res images -- without help from Google or Microsoft or Amazon.
"I felt it was NASA's responsibility to host the golden copy of this data in formats that could be consumed by platforms such as Google Earth and Microsoft WorldWide Telescope," Kemp says. "We couldn't just say that it was Google's problem or Microsoft's problem. But we didn't have the infrastructure. And that was the spark for the project."
He dubbed the project "NASA Nebula," and this would eventually give rise to Nova. Over the next few years, says Joshua McKenty, most of the strategy sessions and a bulk of the after-hours coding happened at the dining room table inside the Rainbow Mansion.
NASA Nebula might have ended up as just another EC2 mimic -- or just another failed government project. But in 2009, it was spotted by Vivek Kundra, the country's first federal CIO. When Barack Obama took office that year, he created the CIO post in an effort to overhaul the government's technical infrastructure, and NASA Nebula was an opportunity to do so. Kundra saw it as a platform that could run applications not only at NASA, but across the federal government.
"The technology was still early on," Kundra says. "But I could see the potential to disrupt how we deployed the old model of technology -- what we called 'digital oil.' The old model was very inefficient, required a lot of resources. Each agency would build up enough capacity for that one day when demand would peak, but for the rest of the year, most of it wouldn't get used."
>"Whenever I would talk in Washington about this cloud technology enabling data centers to run without people, this was interpreted as jobs going away"
According to Kundra, the government's data centers were using only about 40 percent of their total storage -- and about 27 percent of their processing power. Nebula could change that, serving up processing power and storage to each agency only as they were needed.
So Kundra summoned Chris Kemp to the White House, and he eventually used NASA Nebula to launch USAspending.gov -- a site that shared the government's spending with the world at large -- while drawing up plans to expand the platform to other agencies as well.
The problem was that certain U.S. lawmakers and NASA bureaucrats were intent on killing the project. Chief among them was Senator Richard Shelby, the chairman of the Senate Appropriations Committee, according to Kemp. Shelby's office didn't respond to an inquiry from Wired, but Kemp says that the senator saw Nebula as a jobs-killer. "Whenever I would talk in Washington about this cloud technology enabling data centers to run without people, this was interpreted as jobs going away," Kemp says. "There was a serious political challenge to the project...and I was called before the NASA administrator -- of the whole agency -- to explain it."
Kundra puts it a bit more diplomatically -- "there was resistance to a new idea," he says -- and Ray O'Brien, the current CTO for IT at NASA Ames, says much the same. But according to Kemp, the project was very much in danger of losing its funding -- on multiple occasions.
The pressure from Washington nearly reached a breaking point in the fall of 2009, when Kundra flew into Silicon Valley to formally launch the federal government's cloud computing strategy at Ames, lending his support to Nebula both publicly and privately. "We would not have gotten this done without Vivek at the White House," Kemp says. "My team was spending one hundred percent of its time putting together documentation for a [potential] Congressional inquiry. What stopped it was Vivek."
The other problem is that NASA Nebula didn't work quite the way it was supposed to work.
The first version of Nebula was built atop an existing open source software platform called Eucalyptus created by a team of academics at the University of California, Santa Barbara. But according to Chris Kemp and Joshua McKenty, Eucalyptus didn't scale as readily as his team wanted it to. "There were fundamental problems with the [Eucalyptus] architecture that didn't match what we were trying to do," McKenty says. "And then there were some code quality issues. There were some really serious bugs."
The added problem is that the founders of Eucalyptus had recently created a commercial company around the project, and it seemed that they wouldn't let certain code into the open source platform because they wanted to a save it for a version they could sell to businesses. This "open-core" business model -- where the core of a project is open source but code around it is not -- didn't jibe with what Kemp and McKenty and the rest of the NASA team were trying to do: create an open source that platform that anyone could not only use, but contribute to.
>"Nebula was meant for collaboration. But NASA wasn’t an environment designed to make that sort of thing happen"
Kemp, McKenty, and Jesse Andrews -- the software engineering lead on the project who would crash at the Rainbow Mansion on weekends -- had long discussed whether NASA could just start a new open source project from scratch. But they knew that would be difficult. The way the NASA bureaucracy worked, it could easily use code from existing open source project, and after jumping through a few legal hoops, it could take code it had built and contribute to such a project. But NASA didn't really know how to start a project of its own where it would, on a daily basis, collaborate with the outside world.
"This is a federal agency that typically develops mutli-billion-dollar systems. The software in these systems is often the product of decades of investments and typically contains all sorts of intellectual property and probably information involving sensitive national security," Kemp says. "That’s the environment where we were trying to open source our code. Nebula was meant for collaboration. But NASA wasn’t an environment designed to make that sort of thing happen. We had to completely reinvent the software release and open source process.”
It was McKenty and Andrews who found a loophole: What if the team spent a weekend of its own time building its own version of Eucalyptus? It wouldn't be a NASA project -- and then, when they went back to work on Monday as NASA contractors, they could contribute to it. "The problem of NASA releasing code would go away because NASA would technically be contributing to an outside project," McKenty says.
And that's what they did. They hacked through the weekend at HouseKu, another collaborative group house in San Francisco where a NASA contractor named Andy Smith lived. But they needed more than a weekend. The prototype was finished about two weeks later, and this was Nova.
Five weeks after NASA and Rackspace dreamed up the idea for OpenStack at a Thai restaurant near Ames, they were set to release their combined code at the OSCON open source trade show in Portland, Oregon. Several days earlier, the two outfits had met with several others companies in Texas, and some had already signed on to the project. But the evening before the planned announcement, Kemp still hadn't gotten the final approval from NASA to open source the Nova code under an Apache license.
Yes, Andrews and McKenty had already opened sourced Nova. But the work they'd done since -- as NASA contractors -- was still closed. And the project needed NASA's implicit approval.
It would have been easier to release the Nova code under the NOSA license, a license NASA had created for open sourcing software several years earlier. But Kemp and Rackspace's Jim Curry agreed that if OpenStack couldn't be open sourced under an Apache license -- and with the support of both outfits -- it wouldn't be widely adopted. Today, looking back on the decision, HP's Zorawar Singh agrees. The platform is used by so many vastly different operations, he says, because the Apache license lets anyone use it without having to contribute code back to the open source project.
But the NASA bureaucracy just wasn't prepared to give away code in this way. "It was nearly impossible to get NASA to open source it under Apache," Kemp says. "I had technologists on my team who had graduated from law school but were not NASA lawyers to support me in my crusade to collaborate with NASA's lawyers to make this possible." And eventually, he won out. Kemp received the final NASA signature in the early morning of July 19, just hours before OpenStack was officially revealed to the rest of the world.
>"I had technologists on my team who had graduated from law school but were not NASA lawyers to support me in my crusade to collaborate with NASA's lawyers to make this possible"
The announcement was everything Kemp and the Rackspace team had hoped for -- and then some. "It just exploded," McKenty says. "It was probably about thousand times bigger than any of us expected." The project was promptly embraced by NTT Data, the IT consulting arm of Japanese telecom giant NTT. Just four months after its debut, it won the support of Microsoft -- a company that wasn't exactly known for backing open source software. And by the following year, Dell, HP, and Cisco had joined the project.
When Rick Clark was at Canonical, the company bundled Eucalyptus with its Ubuntu Linux distro, hoping to spawn a world of open source clouds. But in May of 2011 -- less than a year after OpenStack was launched -- Canonical booted Eucalyptus from Ubuntu in favor of the new platform. Like Kemp, McKenty, Andrews, and other developers at NASA, Canonical wanted an open source platform that anyone could use, but also one that anyone could contribute to. This was the only way you could compete with Amazon.
That's what OpenStack is: a way for the rest of the world to compete with Amazon. "Amazon [is] at war with every IT vendor out there," says Sebastian Stadil, the CEO of an open source cloud management outfit Scalr, the founder of the Silicon Valley Cloud Computing group, and a former resident of the Rainbow Mansion. "I think one of the reasons OpenStack is getting so much traction -- despite, to be frank, iffy stability -- is that it represents the industry's only hope to survive."
It's hard to tell how widely Amazon is used. But last year, the company's Amazon Web Services business grew 70 percent, and it's on the verge of a $1 billion annual run rate. If you use Amazon EC2, you're not buying servers from the likes of HP and Dell. You're not buying a lot of the software you would have used to manage those servers. And if you're using the services Amazon offers atop EC2 and S3 -- including databases and other tools -- you're not buying a lot of other stuff as well.
OpenStack is a way of building public web services that compete directly with Amazon. But it's also an alternative to such public services. With OpenStack, everyday businesses can build build private services dedicated to their own particular operations. Many companies are reluctant to put their corporate data onto someone else's servers, and if they use OpenStack in their own data centers, they get the benefits of an Amazon EC2 without giving their data to Amazon.
>"Amazon is at war with every IT vendor out there. I think one of the reasons OpenStack is getting so much traction -- despite, to be frank, iffy stability -- is that it represents the industry's only hope to survive"
Amazon overturned the status quo in the IT world. And now OpenStack wants to overturn Amazon. But this is a collective effort to change things. It's not one company trying to disrupt the market. It's many. Chris Kemp is now running Nebula Inc., a company named after his project at NASA that sells a hardware and software cloud computing system based on OpenStack. Joshua McKenty is CEO of Piston Cloud Computing, an outfit that offers an OpenStack distro for traditional enterprise outfits. And Jesse Andrews and several other NASA contractors who worked on Nebula are now at Rackspace, helping build the company's public cloud service but also supporting an services operation that helps other outfits run their own OpenStack clouds.
According to Josh McKenty, this was the idea driving Nebula as far back as 2008. And it very much grew out of the group environment that developed at Rainbow Mansion. Kemp founded the house together with Jessy Schingler, who was part of the original Nebula development team. McKenty lived there part-time while commuting back and forth to Canada. Jesse Andrews and another big contributor, Devin Carlen, were also regulars at the house, and so were other team members such as Rahan Khozein, Creon Levit, and Jeff Lindsay.
These are just some of the founders of OpenStack. There were many more at Rackspace, including Jim Curry, Lew Moorman, Jonathan Bryce, Mark Collier, and Rick Clark, and still others working across the industry, most notably Thierry Carrez, an Ubuntu developer who discussed the idea of an truly open source Amazon EC2 with Rick Clark at Canonical before Clark moved to Rackspace. Rainbow Mansion only played a part. But that's the point.
Now, just two years after its debut, OpenStack is running inside Dell and HP and IBM and Cisco. The world didn't use to work that way. But it does now.
