Databricks has made a name for itself as one of the most popular commercial services around the Apache Spark data analytics platform (which, not coincidentally, was started by the founders of Databricks). Now it’s coming to Microsoft’s Azure platform in the form of a preview of the imaginatively named “Azure Databricks.”
This isn’t just your run-of-the-mill “partnership,” where Databricks makes it service available in the Azure Marketplace and where the actual cooperation doesn’t go much further than a joint press release. Microsoft and Databricks have actually worked on this integration since 2016, and this is making Databricks a first-party service on Azure. That’s a first for both Microsoft and Databricks, and it means that users will be able to spin up Databricks in the Azure Portal just like they would any other native Azure service.
“This was a no-brainer,” Databricks CEO Ali Ghodsi told me. “There was such overwhelming demand for having Databricks on Azure.” With this cooperation, Databricks users don’t just get easy access to their analytics platform on Azure, but also deep integrations with all of the other Azure services that allow for building a modern data analytics pipeline. Think Azure SQL Data Warehouse, Azure Blob Storage and Azure Data Lake, but also tools like Azure Active Directory.
Indeed, as both Ghodsi and Microsoft’s general manager for its database systems, Rohan Kumar, told me, this integration with Active Directory and other security tools was key for many of the potential customers they both talked to.
Azure users will be able to spin up Databricks with a single click and the service can scale automatically as the user’s needs change. They also will get access to Databricks’ interactive and collaborative notebooks for building their models and pipelines, as well as all of Databricks’ built-in monitoring and security tools. These notebooks are part of the core value of using a service like Databricks (besides its fast runtime and other optimizations). As Ghodsi noted, building a data pipeline and analytics on top of it is usually an iterative service, so the faster you can go from experiment to results to the next experiment, the better.
Because this is essentially a first-party Azure service, users will also get all of the usual Azure SLAs and be able to deliver support to their joint customers — a group of Databricks engineers will actually be on call and ready to support Azure users.
Both Kumar and Ghodsi stressed that this was a very unusual partnership. Kumar noted that Microsoft doesn’t have any similar cooperation in the plans right now, but the combination of Databricks’ analytics smarts and Microsoft’s cloud computing platform clearly makes sense, and it allows both parties to leverage their own strengths. Microsoft could have built its own Spark-based service, of course. Instead, though, it decided to partner, and that’s probably a good thing as it will give the company access to Databricks’ existing customers, who Ghodsi noted are looking for a dependable enterprise cloud.
The service is now in preview. Microsoft hasn’t announced any pricing yet, though Kumar stressed that all of the billing will go through Microsoft, so chances are that Databricks will get a share of that revenue.
Featured Image: Erik Dreyer/Getty Images