Understanding Google Cloud Platform’s Cloud Architecture
Cloud analytics is surging, and the trend is showing no signs of slowing down. Making the move to modern analytics by transitioning data to cloud platforms is becoming a necessity for organizations of all sizes and industries. But for those just beginning their modern analytics journey, navigating the cloud platform marketplace can be difficult. That’s why it’s essential to gain a better understanding of what options are available, what they offer, and how they can best suit your company’s needs.
In this demo, we dive into Google Cloud Platform (GCP) to discover the powerful tools that can be used to address your data analytics needs. We’ll walk you through an example cloud data pipeline in which we’ll migrate the data from a representational state transfer (REST) application programming interface (API) into GCP’s serverless data warehouse, BigQuery, where it can be leveraged by data visualization tools.
By applying lessons learned from this demo, you’ll be able to:
- Solidify a reference architecture that can be used across all cloud engagements
- Translate traditional data and analytics skillsets to the cloud
- Raise awareness and engagement in the cloud analytics space
- Train and equip your team as you move to the cloud
Exploring GCP Cloud Architecture
The graphic above represents a sample GCP cloud data pipeline through which data is processed and moved between different storage and analytics services at specified intervals. Let’s take a closer look at the steps through which the data flows through this pipeline:
- Source Data – The data begins its journey from the REST API.
- Data Orchestration – Leveraged Google Cloud Functions (serverless compute solution) to move the data from the API into a Data Lake. If a low to no code integration or ETL tool is preferred, GCP also offers other options, such as Cloud Data Fusion and Cloud Dataflow (not used in this pipeline).
- Data Lake 3. Data Staging (in a Data Lake) – The data is then staged in a data lake (or landing zone), where Google Cloud Storage leverages cloud-based object storage to preserve historical data and enable data science activities.
- Data Orchestration – Once loaded into the data lake, Google Cloud Functions is used to move the data into a Cloud Data Warehouses (CDW).
- Cloud Data Warehouse – The data is stored in Google Big Query Service, which is Google’s native cloud service for all reporting activities.
- Data Visualization – Now that your data is transformed, it can then be consumed by any number of visualization tools, including Looker, Power BI, QuickSight, and Tableau.
Advantages of GCP
GCP offers several advantages over other cloud platforms that can help you on your modern analytics journey:
- Google Cloud Functions allows you to run your code with zero server management, and you only have to pay for what you use, based on the functions’ execution time and metered to the nearest millisecond.
- The user’s table that stores raw data in Google Cloud Storage uses an external data configuration, which allows you to directly query a CSV file, as well as many other file formats.
- The ability to directly query files containing raw data avoids having to redundantly store both the raw file in Google Cloud Storage and the raw data in BigQuery.
- You can avoid the use of data manipulation languages such as SQL insert or update statements, as the table’s external data configuration can automatically parse new files.
- This data can be further transformed in BigQuery to create a native table, which is more suitable for reporting and ad hoc querying than external tables.
- This SQL transformation can be set to trigger automatically using a cloud function or with other tools such as BigQuery scheduled queries.
How to Select a Cloud Platform
Having your data in the cloud allows your organization to move toward more modern, self-service analytics and make data-driven decisions. However, the modern analytics landscape can be confusing, and many people are still trying to figure out where to start. For those in the beginning stages, it’s essential to understand the differences between legacy and modern analytics, the advantages of cloud architectures and other available platforms, and the next steps you should pursue. To learn more about preparing for a cloud analytics transformation, download our eBook, Modern Analytics: Reaching Beyond the Clouds.
When exploring different cloud platforms, it’s important to perform a thorough side-by-side comparison of other providers that allows you to measure against organizational criteria to better understand what options are available, what they offer, and how they can best suit your company’s needs.
Even with a thorough comparison and review of your organization’s guidelines, it can be hard to determine which cloud analytics platform will serve you best. If you need guidance or expert recommendations, our team of experts can help you develop and deploy a modern cloud analytics plan that will work best for your company. When you’re ready, contact us, and we’ll get you on your way.
Sense Corp is a leading professional services firm transforming organizations for the digital era. We help clients solve their toughest challenges by bridging the gap between what is and what’s possible.