The data cloud wars are starting anew. For those who might not know, a data cloud is a unified data management ecosystem that enables easy reuse, portability, and consumability of all data. “Non-builder” organizations – or those that do not have the in-house access to IT to build a custom solution - are clear that they now want and need a data cloud to simplify their data management, data pipelines, data sharing, and data analytics to obtain high-quality data-driven insights faster.
These organizations are clear that they need three things from a data cloud:
1) All Data – fundamentally, they want a data platform that can ingest, manage and provide automation and orchestration for all data types – structured semi-structured, and unstructured in one location
2) All Analytics – they want self-service access, automation and orchestration of data for any analytic workload
3) Usability – they want self-service data cloud simplicity for both their IT and data teams
Figure 1: 3 Data Cloud Options
To achieve these ends, Non-Builder organizations have really three options: 1) Do Nothing and pay the potential price of being left behind in the market; 2) choose a common solution such as Snowflake, or 3) explore the broader benefits of the Amorphic Data Cloud for AWS.
THE DATA CLOUD 2.0 – THE MODERN DATA ARCHITECTURE
The Data Cloud 1.0 market so far has been led by Snowflake as well as Databricks. Both have been successful to date; however both offerings were built to support singular data and analytic solutions: Snowflake was built to support SQL analytics in the cloud and Databricks was built to support Spark in the cloud. Neither solution is comprehensive enough to address more than a few components of the Data Cloud 2.0 modern data architecture and Unified Data Infrastructure as defined by illustrated in the graphic by Matt Bornstein, Jennifer Li, Martin Casado of A16z, see below:
Figure 2: The Data Cloud Unified Data Infrastructure 2.0 by Matt Bornstein, Jennifer Li, Martin Casado of A16z
From: https://future.com/emerging-architectures-modern-data-infrastructure/
This Unified Data Infrastructure 2.0 expands the modern data architecture beyond just the data warehouse to include the data warehouse, data lake + lakehouse architectures, a multiplicity of sources including batch and streaming of all data to break down all data silos, data transformation, ETL and Analysis plus BI Dashboarding and Visualization, and Data Discovery/Search, RBAC Governance, Data Observability & Data Sharing, and Security. In this architecture, Snowflake has notable solution elements in just the Data Warehouse and Event Collector categories, while Databricks is recognized in a few more categories. Only the Amorphic Data Cloud for AWS - which natively integrates over 65 AWS services into a seamless user-interface with built-in data discovery and data governance - was designed to meet these modern data architecture and Unified Data Infrastructure 2.0 platform requirements.
HOW THE AMORPHIC DATA CLOUD FOR AWS BEATS SNOWFLAKE
Built to address the requirements of both the data warehouse and the data lake + lakehouse architecture by natively integrating more than 65 AWS data and analytics and other services into a single user interface, the Amorphic Data Cloud for AWS seamlessly unifies the entire modern data architecture stack into a single data cloud platform for all data types. No other data cloud does this. Amorphic ingests all data types and natively enables the full stack of AWS Data & Analytics services without third-party assistance – in stark contrast to Snowflake which requires multiple additional third-party licenses, at greater cost and complexity - to achieve the same functionality. Amorphic natively enables on AWS comprehensive data pipeline creation, data sharing, data visualization and overall data management with full data discovery and data governance - checking off 26 of the 31 the categories outlined in A16z’s Unified Data Architecture 2.0 shown in Figure 2.
With Amorphic for AWS, organizations can seamlessly eliminate data silos, empower secure and governed access to data and remove infrastructure complexity - natively in their AWS account – gaining the freedom to drive holistic insights across their business and address new market opportunities at lower cost and faster time- to-decision. (see Figure 3 below).
Figure 3: The Amorphic Data Cloud for AWS natively integrates 65+AWS Services – comparison to Snowflake Native Data Cloud
While Snowflake capably handles production-level Business Intelligence workloads, Snowflake’s single-purpose design as a data warehouse creates a significant amount of technical debt. As a data cloud platform 2.0 for all data and analytic workloads, Snowflake falls short for Non-Builder organizations in these 5 data cloud categories:
1. Data Ingestion and Streaming. Bulk data processing is limited in Snowflake, and most data ingestion use cases requires Non-Builder organizations to buy and integrate expensive third-party subscriptions from FiveTran, Matilion, Talend, Informatica or Confluent to ingest their data into Snowflake, which increases cost and complexity.
2. All Data. Snowflake was designed to support structured and semi-structured data, not unstructured data. This gap in Snowflake’s ‘Data Cloud’ architecture has major implications, and while Snowflake has been working to fix this, there are still gaps in this capability.
3. Data Catalog and Search. Snowflake requires users to buy and integrate expensive third-party enterprise data catalog tools from third-party vendors such as Alation or Collibra, which increases cost and complexity for Non-Builders seeking a seamless, lower-cost solution.
4. Advanced Analytics. Snowflake was designed as a data warehouse, not an advanced analytics platform. As such, artificial intelligence (AI) and machine learning (ML) analytic use cases are not supported by Snowflake natively and require the additional purchase and integration of third-party advanced analytic providers such as Alteryx, AWS SageMaker, Big Squid, Dataiku, Databricks, DataRobot, or others, causing further data and analytics limitations and usability for Non-Builder organizations.
5. Usability. Snowflake is great if all the organization needs and wants is a cloud data warehouse and has simple structured and semi-structured data. Beyond the classic SQL analytic use cases, Snowflake, becomes very difficult to use as a ‘Data Cloud’ platform for all data and analytic workloads. The cost of third-party software and IT integrations required makes Snowflake a dead-end data cloud platform for most.
SGN, one of the largest energy companies in the UK, selected Amorphic Data Cloud for AWS in 2019 to power its next generation cloud analytics for faster time to insight across multiple business units. Using Amorphic, SGN effectively transformed their Business Intelligence (BI), Management Information (MI), Artificial Intelligence (AI), and Machine Learning (ML) capabilities, while saving over $4.2 Million USD.
SGN was looking for an AWS-native, self-service, scalable, and secure cloud data analytics platform which could allow SGN to decentralize access to data and allow business users to self-serve. The solution would further allow the current BI and MI reporting to run effectively, modernizing the data warehouse traditionally powered by Oracle databases on Amazon Elastic Compute Cloud (Amazon EC2) instances. The SGN team also wanted to reduce the overall technical debt that had built up within the BI database. The team needed a platform to support multiple ingestion mechanisms, with data coming from different sources including CSV files, databases, and APIs.
SGN is one of the top innovators in its industry and had a clear cloud strategy. It wanted to take its systems and applications to the public cloud to increase the agility and pace of innovation that the cloud provides. Although SGN had a large cloud transformation project underway, the idea of making data available to the business and letting the business ‘own’ the insights was missing.
According to an AWS APN Customer Innovation Spotlight Video with Abigail Burgess, SGN Analytics Program Manager, and an AWS APN Customer Innovation Spotlight Webinar with Suchi Nagar, SGN Head of Architecture and Data, SGN chose Amorphic over other solutions because it allowed SGN to:
1. Break data silos easily and migrate all data to AWS to be managed by the Amorphic Data Cloud
2. Consolidate data into a single, analytics-ready source of truth
3. Innovate faster and make better data-driven decisions
4. Create new monetization streams and data-driven applications
5. Do more faster with less skilled resources and at a lower cost
Using Amorphic, SGN IT is now able to provide the line of business with new high-value data and analytics-as-a-service capabilities enabling rapid business innovations across all lines of business. Additionally, the Amorphic Data Cloud has allowed SGN to expand and accelerate its capabilities on AWS to drive faster innovation at a lower cost.
Most organizations today want a self-service data cloud. Selecting a data cloud that supports natively all data and analytic workloads is the most important information technology decision for any organization. The Data Cloud 1.0 platforms solved only some of the data and analytic requirements. Data Cloud 2.0 platforms are designed to unlock the power of the data cloud for all organizations – including the “Non-Builders” who might not have the time, skilled resources, or budgets to build data cloud platforms. Selecting the Amorphic Data Cloud for AWS provides your organization with:
1. The ability to break data silos and infrastructure barriers for analytics
2. One data cloud for all data and analytic workloads
3. IT and data team self-service usability for EDW, ETL, BI, Search, AI and ML
4. The ability to create new insights, products, services, and revenues with data and analytics
5. Faster data and analytic results with less skilled resources and at lower cost
If any of these goals are important for your organization, then reach out to me at mark@cloudwick.com and we can discuss, qualify, and align an Amorphic Data Cloud consultation with you and your AWS account manager.
References
1. https://www.trustradius.com/products/snowflake/reviews?qs=pros-and-cons
2. https://www.oracle.com/hk/a/ocom/docs/database/adw-melts-snowflake-report.pdf
3. https://pages.awscloud.com/apn-tv-aws-partner-success-stories-ep-029.html
4. https://aws.amazon.com/partners/success/sgn-cloudwick/
5. https://future.com/author/matt-bornstein/
6. https://pages.awscloud.com/GLOBAL-partner-OE-pmc-aiml-cloudwick-oct-2021-confirmation.html