Understanding Cloud Platforms: Google Cloud Platform (GCP)
Making the move to modern analytics by transitioning data to cloud platforms is becoming a necessity for organizations of all sizes and industries. But for those just beginning their modern analytics journey, navigating the cloud platform marketplace can be difficult.
In this Demo, we dive into Google Cloud Platform (GCP) to discover the powerful tools that can be used to address your data analytics needs. We’ll walk you through an example cloud data pipeline, beginning by extracting data from a rest API and loading the data into GCP’s serverless data warehouse, BigQuery, where Data Visualization tools can use it. We will visit each of the steps along the way with goals to:
- Solidify a reference architecture that can be used across all cloud engagements
- Take traditional data and analytics skills and translating it to the cloud
- Raise awareness and engagement in the cloud analytics space
- Train and equip your team as you move to the cloud
Exploring GCP Cloud Architecture
- Source Data – Rest API
- Data Orchestration – Leveraged Google Cloud Functions (cloud-based ETL tool) to move the data from our API into our Data. Lake. If a low to no code integration or ETL tool is preferred, GCP also offers other options, such as Cloud Data Fusion and Cloud Dataflow (not used in this pipeline).
- Data Lake (Landing Zone) – Google Cloud Storage leverages cloud-based object storage to preserve historical data and enable data science activities.
- Data Orchestration – Once loaded into the data lake, Google Cloud Functions is used to move the data into a Cloud Data Warehouses (CDW).
- Cloud Data Warehouse – The data, now in Google Big Query Service, is now used as the source for all reporting activities
- Data Visualization – Now that your data is transformed, it can then be consumed by several tools; among these are visualization tools such as Looker, Power BI, QuickSight, and Tableau.
Advantages of Google Cloud Platform
- Google Cloud Functions allows you to run your code with zero server management. Additionally, you only pay for what you use, based on the functions execution time, metered to the nearest .001 millisecond.
- The users table containing raw data uses an external data configuration, which allows you to directly query a CSV file as well as many other file formats stored in Google Cloud Storage.
The ability to directly query files containing raw data avoids having to redundantly store the data in Google Cloud Storage and BigQuery itself.
- Data manipulation languages such as SQL insert or update statements are no longer necessary, as the table’s external data configuration can automatically parse new files.
- The data can be further transformed in BigQuery to create a native table, which is more suitable for reporting and ad hoc querying than external tables.
- The SQL transformation can be set to trigger automatically using a cloud function or with other tools such as BigQuery scheduled queries.
How to Select a Cloud Platform
The modern analytics landscape can be confusing, and many people are still trying to figure out where to start. For those in the beginning stages, it’s essential to understand the differences between legacy and modern analytics, cloud architecture and available platforms, as well as the next steps you should pursue. To learn more about preparing for a cloud analytics transformation, download our eBook, Modern Analytics: Reaching Beyond the Clouds. For those exploring different cloud platforms, it’s important to perform a thorough side-by-side comparison of other providers that you can measure against organizational criteria to better understand what options are available, what they offer, and how they can best suit your company’s needs.
Even with a thorough comparison and review of your organization’s guidelines, it can be hard to determine which cloud analytics platform will serve you best. If you need guidance or expert recommendations, our team of experts can help you develop and deploy a modern cloud analytics plan that will work best for your company. When you’re ready, contact us, and we’ll get you on your way.
Sense Corp is a leading professional services firm transforming organizations for the digital era. We help clients solve their toughest challenges by bridging the gap between what is and what’s possible.