Google Cloud Data Training Overview
This instructor-led, hands-on Data Engineering on Google Cloud training course teaches attendees how to design and build data processing systems on Google Cloud. Attendees learn how to design data processing systems, build end-to-end data pipelines, and analyze data. This class also covers structured, unstructured, and streaming data.
Location and Pricing
This course is taught as a private, live online class for teams of 3 or more. All our courses are hands-on, instructor-led, and tailored to fit your group’s goals and needs. Most Accelebrate classes can be flexibly scheduled for your group, including delivery in half-day segments across a week or set of weeks. To receive a customized proposal and price quote for online corporate training, please contact us.
In addition, some courses are available as live, instructor-led training from one of our partners.
Objectives
- Extract, load, transform, clean, and validate data
- Design pipelines and architectures for data processing
- Create and maintain machine learning and statistical models
- Query datasets, visualize query results, and create reports
Prerequisites
- Completed Google Cloud Big Data and Machine Learning Fundamentals course or have equivalent experience
- Basic proficiency with common query language such as SQL Experience with data modeling, extracting, transforming, and load activities
- Experience developing applications using a common programming language such as Python
- Familiarity with basic statistics
Outline
Expand All | Collapse All
Introduction to Data Engineering
- Explore the role of a data engineer
- Analyze data engineering challenges
- Intro to BigQuery
- Data Lakes and Data Warehouses
- Federated Queries with BigQuery
- Transactional Databases vs Data Warehouses
- Website Demo: Finding PII in your dataset with DLP API
- Partner effectively with other data teams
- Manage data access and governance
- Build production-ready pipelines.
- Review GCP customer case study
- Analyzing Data with BigQuery
Building a Data Lake
- Introduction to Data Lakes
- Data Storage and ETL options on GCP
- Building a Data Lake using Cloud Storage
- Optional Demo: Optimizing cost with Google Cloud Storage classes and Cloud Functions
- Securing Cloud Storage
- Storing All Sorts of Data Types
- Video Demo: Running federated queries on Parquet and ORC files in BigQuery
- Cloud SQL as a relational Data Lake
- Loading Taxi Data into Cloud SQL
Building a Data Warehouse
- The modern data warehouse
- Intro to BigQuery
- Demo: Query TB+ of data in seconds
- Getting Started
- Loading Data
- Querying Cloud SQL from BigQuery
- Loading Data into BigQuery
- Exploring Schemas
- Exploring BigQuery Public Datasets with SQL using INFORMATION_SCHEMA
- Schema Design
- Nested and Repeated Fields
- Nested and repeated fields in BigQuery
- Working with JSON and Array data in BigQuery
- Optimizing with Partitioning and Clustering
- Partitioned and Clustered Tables in BigQuery
- Transforming Batch and Streaming Data
Introduction to Building Batch Data Pipelines
- EL, ELT, ETL
- Quality considerations
- How to carry out operations in BigQuery
- ELT to improve data quality in BigQuery
- Shortcomings
- ETL to solve data quality issues
Executing Spark on Cloud Dataproc
- The Hadoop ecosystem
- Running Hadoop on Cloud Dataproc
- GCS instead of HDFS
- Optimizing Dataproc
- Running Apache Spark jobs on Cloud Dataproc
Serverless Data Processing with Cloud Dataflow
- Cloud Dataflow
- Why customers value Dataflow
- Dataflow Pipelines
- A Simple Dataflow Pipeline (Python/Java)
- MapReduce in Dataflow (Python/Java)
- Side Inputs (Python/Java)
- Dataflow Templates
- Dataflow SQL
Manage Data Pipelines with Cloud Data Fusion and Cloud Composer
- Building Batch Data Pipelines visually with Cloud Data Fusion
- Components
- UI Overview
- Exploring Data using Wrangler
- Building and executing a pipeline graph in Cloud Data Fusion
- Orchestrating work between GCP services with Cloud Composer
- Apache Airflow Environment
- DAGs and Operators
- DAGs and Operators
- Workflow Scheduling
- Event-triggered Loading of data with Cloud Composer, Cloud Functions, Cloud Storage, and BigQuery
- Monitoring and Logging
- An Introduction to Cloud Composer
Introduction to Processing Streaming Data
- Processing Streaming Data
Serverless Messaging with Cloud Pub/Sub
- Cloud Pub/Sub
- Publish Streaming Data into Pub/Sub
Cloud Dataflow Streaming Features
- Cloud Dataflow Streaming Features
- Streaming Data Pipelines
High-Throughput BigQuery and Bigtable Streaming Features
- BigQuery Streaming Features
- Streaming Analytics and Dashboards
- Cloud Bigtable
- Streaming Data Pipelines into Bigtable
Advanced BigQuery Functionality and Performance
- Analytic Window Functions
- Using With Clauses
- GIS Functions
- Mapping Fastest Growing Zip Codes with BigQuery GeoViz
- Performance Considerations
- Optimizing your BigQuery Queries for Performance
- Creating Date-Partitioned Tables in BigQuery
Introduction to Analytics and AI
- What is AI?
- From Ad-hoc Data Analysis to Data-Driven Decisions
- Options for ML models on GCP
Prebuilt ML model APIs for Unstructured Data
- Unstructured Data is Hard
- ML APIs for Enriching Data
- Using the Natural Language API to Classify Unstructured Text.
Custom Model building with SQL in BigQuery ML
- BigQuery ML for Quick Model Building
- Train a model with BigQuery ML to predict NYC taxi fares
- Supported Models
- Option 1: Predict Bike Trip Duration with a Regression Model in BQML
- Option 2: Movie Recommendations in BigQuery ML
Conclusion
Training Materials
All GCP training attendees receive comprehensive courseware.
Software Requirements
Students must have the Google Chrome web browser and Internet access.
