Innovation and the Data Platform Continuum

All of us who work in technology-related fields have experienced disruptions that seem to change everything, and figuring out how to manage through the transition can mean the survival or demise of an enterprise. When faced with innovation, it is critical to determine when, where, and how to adopt the change.

At Microsoft, we talk a lot about the shift to cloud computing. As we continue to invest in this shift, we are developing and delivering outstanding on-premises data platform solutions, capabilities to bridge between the cloud and on-premises, and advancing cloud technologies for next generation applications.

As examples, consider:

• The breakthrough performance of In-Memory OLTP and In-Memory DW in SQL Server 2014.
  
• Hybrid scenarios like enrolling an Azure-hosted virtual machine running SQL Server with an on-premises Availability Group or high-performing appliances running queries or moving data across both on-premises and cloud environments.
  
• Completely new ways for developers to write mission critical applications using services like Azure SQL Database, with built-in high availability, performance, and business continuity.
  

At Microsoft, I lead the program management team responsible for data platform technologies such as SQL Server, Parallel Data Warehouse, Azure SQL Virtual Machines, and Azure SQL Database. I’ve spent nearly my entire professional career building enterprise—and now cloud—software.

While on this journey, I have had the privilege of talking to customers around the world, to be part of major technology changes, to make some pretty big mistakes, and recently to go through one of the largest engineering culture changes in my career. The shift from building box-cycle software to a cloud computing cadence has been exhilarating, to say the least.

In this post, I hope to share some insight into how we think about our data platform and what’s to come.

PLATFORM CONTINUUM

The North Star that guides our work on the modern data platform is to deliver products and integrated services that empower developers to build high performance applications that are flexible and scale free. We talk about the data platform as a continuum of capabilities, spanning on-premises databases, hybrid or cloud-attached solutions, and Azure assets that each provide a unique set of capabilities.

The concept of a continuum of capabilities enables developers to continue to use SQL Server on-premises, to easily virtualize and move database workloads into Azure, and to attach Azure services and build new cloud applications all from one data platform.

WHY IT’S IMPORTANT & WHAT’S TO COME

The Microsoft data platform addresses a broader set of scenarios than anyone else because it encompasses solutions for on-premises, hybrid (cloud-attached), and full cloud data services. In fact, you could think of this data platform continuum as the New Tier1 for modern mission critical applications.

On-premises—While we anticipate massive growth in the use of cloud computing, it is also true that there will always be opportunities for on-premises, highly regulated database workloads that might never move to the cloud. To support these workloads, we continue to embrace hardware trends that cater to very high-scale applications and deliver strategic capabilities which maximize performance and availability via on-premises SQL Server databases.

I’ve met many developers and IT professionals who can explain in detail every aspect of their mission critical OLTP applications. It’s both impressive and demanding. They require full environmental control, with good reason. At the same time, these customers and others are exploring new and more efficient ways of doing things.

Hybrid—I have worked with many customers who have spent months designing a disaster recovery strategy and then additional months and large expense just trying to realize it. Today, many on-premises SQL Server customers have discovered new ways of planning for disaster recovery by way of the cloud. One example is by enrolling an Azure-hosted virtual machine running SQL Server with an on-premises SQL Server Availability Group. Another example is configuring on-premises SQL Server to encrypt and continuously backup to Azure data management services. Both scenarios tap into a use of Windows Azure, which provides an infinite level of scale and geo redundancy for a fraction of the cost.

A function like SQL Server Windows Azure backup is going to reap the rewards of continuous reduced storage costs in the cloud and maintain the benefits of mission critical business continuity for pennies on the dollar. I expect an extraordinary amount of innovation to happen in the hybrid space. Frankly, what’s mentioned here is just the tip of what’s possible.

Mission Critical—Lately, I’ve been thinking about how customers talk about their applications as “mission critical.” In the past, truly mission critical applications were a small but very important set of applications, usually constrained by cost in terms of development expense and talent. Going forward, I expect a much larger number of mission critical applications will emerge because the economics and technology advances can allow the highest service-level-agreements at price points that were unimaginable in the past.

Virtualization—An extraordinary number of companies have virtualized their environments, lifted and shifted applications to the cloud, and built development and test environments that spin up and down in Azure Virtual Machines. We’re talking millions of compute hours and platform telemetry providing insight unlike ever before. Imagine petabytes of telemetry from a global footprint of virtual machines powering millions of applications that humans and machines interact with to understand usage patterns or predict failures, etc.

Big Data—If you believe that almost every vital object on earth will one day emit data to be captured, stored, and analyzed—and many already are—then, you know how many breakthroughs are really just waiting to be discovered in the world. Think about what a medical researcher with a $5,000 grant could actually do several years ago compared to what they can do today, and then imagine tomorrow. Wow! I imagine this researcher driving into work with a new hypotheses, after she was up all night pondering about how to cure a disease, and being able to ask a question that thousands of computers then process and answer (at fractional cost). The world of medicine could be forever changed by enabling researchers to do things that were just not feasible in the past. Or, imagine what an automobile manufacturer could learn from every single car it built emitting telemetry. This could result in smarter, safer, and more efficient cars.

Appliances—Another important data platform asset is Microsoft Parallel Data Warehouse (PDW). It’s a mission critical, pre-built data warehouse appliance that enables analytics across structured and unstructured data at massive scale (up to 10 PB) with 10-100X performance gains relative to SMP DW w/SQL Server. PDW includes a built-in HDI Insight Region and Polybase, a remarkable innovation because it enables new classes of hybrid insights. With a focus on scenarios that start on-premises, Polybase enables queries to run across Azure SQL Database and HDInsight data stores in the cloud and on-premises using TSQL. In both on-premises and hybrid scenarios, existing BI tools work transparently out of the box.

Cloud Computing—I love what our customers are doing with Azure SQL Database. They are redefining what it means to build a data-centric application in the cloud. To me, the developer promise of Platform as a Service solutions like SQL Database means that all applications can be Tier1-enabled and achieve the highest service-level-agreements (SLAs) across the Azure platform without IT effort. This is major step forward as most data applications require both IT professionals and developers to design, build, and operate the entire system. With SQL Database, a developer no longer needs to think about designing availability, performance, and business continuity into the system because it’s built-in and driven by SLAs for the platform. And, going forward, developers will no longer spend time elaborating over scale concerns because the platform includes a simple contract to elastically scale applications without rewrites and extraordinary burden.

SUMMARY

Looking forward, I’m more excited about the future of data platform technology than I’ve ever been because we are seeing breakthroughs in every industry segment: manufacturing, finance, healthcare, entertainment, agriculture, etc., and maybe these breakthroughs result in a better world for generations to come. I’ve never seen anything like this in my career, and it’s humbling to know that we all have an opportunity to be part of it.

We believe the continuum of on-premises, hybrid or cloud-attach, and Windows Azure assets provides a unique connection of elements that customers won’t find anywhere but Microsoft and enables the incredible ideas and aspirations of generations to come.

For more information, see SQL Server, Windows Azure Data Services, Parallel Data Warehouse (PDW).

Shawn Bice

Director of Program Management

Data Platform Group

Microsoft Corp.

Continue reading...