Two talks: 'Cosmos: Big Data and Big Challenges' and 'Condos and Clouds'

Cosmos: Big Data and Big Challenges
The underlying plumbing for Bing includes a massively parallel storage and computation layer running over tens of thousands of servers in many data centers. Cosmos stores data triply-replicated for high availability in simple byte streams. The store layer of Cosmos holds hundreds of petabytes of data. Applications are written in a high-level set-oriented language called SCOPE which is similar to SQL. SCOPE applications are executed in a distributed fashion across the servers in a massively parallel way. Optimizations try to ship the computation close to the data. Cosmos is an ongoing service with hundreds of departments using it for the back-end calculations supporting Bing. Recent innovations have included the development of structured stream. With structured streams, Cosmos records information about the meta-data, indexing, and data affinitization within the stream as it is being generated. This meta-data is then used by SCOPE jobs which read the data to dramatically improve the overall performance of the system.

This talk will describe Cosmos, the mechanism for handling huge amounts of data and computation, the challenges faced in supporting this real live service.

Condos and Clouds
Over the last 100+ years, the way people design, build, and use buildings has evolved. It is now normal to construct a building without knowing in advance who will occupy the building. In addition, we increasingly have shared occupancy of our homes (apartments and condos), retail, and office space. To accomplish this change, the way we use the buildings has evolved. There is a new trust relationship, customs, and laws that establish the relationship between the occupants and the building managers.

Recently, our industry has been moving to implement Cloud Computing. This has been very successful in some applications and very challenging in others. This talk posits that many of the challenges we've seen in cloud computing can be understood by looking at what has happened in buildings and their occupancy. Standardization, usage patterns, legal establishment of rights and responsibilities are all nascent in the area of cloud computing. We examine a very common pattern in the implementation of "software and a service" and propose ways in which this pattern may be better supported in a multi-tenant fashion.