Databricks made several announcements at its annual conference, signaling the company’s intent to capture the data intelligence market brick by brick — from Artificial and Business to Catalog and, its roots, Data intelligence. Last year, Databricks acquired MosaicML to bolster its AI capabilities. This year’s announcements around AI, business intelligence (BI), and open-sourcing its Unity Catalog help Databricks capture the BI and conversation intellgence flags, as well! The acquisition of Tabular strengthens its core data technology roots and helps it expand capabilities and customer base. While these advancements are impressive, we missed seeing anything about real-time and automated data quality, critical for AI applications and a comprehensive semantic layer to achieve holistic data intelligence from a business perspective. That notwithstanding, the generative AI revolution, 16,000 onsite attendees at the conference, and a star-power keynote appearance by NVIDIA’s Jensen Huang signal that Databricks has the necessary tailwinds to fly higher and faster and has places to go.
Here are some key takeaways from the summit:
Unity Catalog goes open source on the heels of Snowflake’s announcement. Databricks announced open-sourcing Unity Catalog, making it the summit’s biggest news. Launched in 2021 and being used by over 10,000 enterprises, Unity Catalog integrates data and AI governance across various clouds and data formats. The decision to open-source is expected to enhance interoperability and broaden Databricks’ use cases and ecosystem. This comes shortly after Snowflake’s unveiling of Polaris, its new open-source data catalog for Apache Iceberg. Databricks’ decision underscores its strong commitment to open-source standards.
Databrick removes format barriers with acquisition of Tabular. Rather than competing directly with the Apache Iceberg format, which is supported by multiple providers including Snowflake, Cloudera, and Dremio, Databricks has chosen to embrace it. Databricks has announced its acquisition of Tabular, enabling the platform to seamlessly read and write to Apache Iceberg and Hudi formats, thereby eliminating barriers associated with table formats.
The company aims for analytics democratization with AI/BI integration. Embedded within the Data Intelligence Platform, AI/BI features AI-powered low-code dashboards and Genie, a conversational interface. It enhances reasoning capabilities using a compound AI system that continuously adapts and learns from human feedback, employing specialized AI agents for BI tasks. Integration with Unity Catalog and the Photon engine, along with lakehouse federation (described below), allows customers to eliminate data extraction and leverage governed high-volume data without performance issues.
Databricks goes serverless for the entire stack. Databricks has announced that its entire platform will now be available as serverless, aligning with offerings from other leading data platform providers. This shift allows customers to load and run queries without the need to manage the underlying data infrastructure, thereby accelerating AI initiatives and reducing costs. Additionally, this move enables Databricks to streamline its solution, enhancing version control, technical support, and rapid delivery of innovative products.
Partnership with NVIDIA shows promising potential. At the summit, the CEOs of NVIDIA and Databricks discussed leveraging NVIDIA’s full stack with the Databricks platform to accelerate generative AI use cases. Databricks unveiled plans to integrate NVIDIA’s Compute Unified Device Architecture (CUDA) technology into its data intelligence platform, aiming to enhance the performance and scalability of AI processing and queries. By natively supporting NVIDIA’s GPUs and leveraging NVIDIA’s accelerated computing in its vectorized query engine Photon, Databricks aims to significantly boost computational efficiency.
Lakehouse improvements focus on distributed data. Databricks, known for pioneering the lakehouse concept, further announced enhancing its solution with the introduction of lakehouse federation in Unity Catalog. This new feature provides unified data management, discovery, categorization, and governance across data platforms. This will enable customers to seamlessly discover, query, and govern any data regardless of its location.
Share your thoughts and discuss any of these announcements or data capabilities in detail with us by scheduling an inquiry/guidance session.
