All rights reserved. Do not distribute. Cloud Endpoints helps you create and maintain APIs. â Distributed API manageme
1 2 3
4 5 6
Getting Started
7 8
With Google Cloud
9 10 11 12 13 14 15 16 17 18
1 2
Cloud OnBoard
3 4
{
5
(’Module 1’) Introducing Google Cloud Platform
Page
2 - 21
6
(’Module 2’) Getting Started with Google Cloud Platform
Page
22 - 52
7
(’Module 3’) Google Compute Engine and Networking
Page
53 - 78
8
(’Module 4’) Google Kubernetes Engine
Page
79 - 99
9
(’Module 5’) Serverless on Google Cloud
Page
100 - 123
10
(’Module 6’) Google Cloud Platform Storage Options
Page
124 - 150
11
(’Module 7’) Big Data and Machine Learning
Page
151 - 186
Page
187 - 193
12
Igniting your Career with Google Cloud | Training & Certification
13 14 15 16 17 18
}
Cloud OnBoard
Welcome to Cloud OnBoard #GoogleCloudOnBoard
©Google Inc. or its affiliates. All rights reserved. Do not distribute. ©Google Inc. or its affiliates. All rights reserved. Do not distribute. May only be taught by Google Cloud Platform Authorized Trainers.
Cloud OnBoard
Introducing Google Cloud Platform Google Cloud Platform Fundamentals: Core Infrastructure V3.0
©Google Inc. or its affiliates. All rights reserved. Do not distribute. ©Google Inc. or its affiliates. All rights reserved. Do not distribute. May only be taught by Google Cloud Platform Authorized Trainers.
Agenda 1
Introduction to Google Cloud Platform
2
Quiz
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
3
Computing trends toward pay-as-you-go, fully automated services Next
Now
Storage
Processing
Memory
Network
Storage
Physical/Colo User-configured, managed, and maintained
Processing
Memory
Virtualized
Network
Serverless/No-Ops Fully automated
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
4
Every company is a data company
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
5
GCP offers a range of computing architectures
Compute Engine
Kubernetes Engine
App Engine
Cloud Functions
IaaS
Hybrid
PaaS
Serverless logic
Toward managed infrastructure
Managed services Automated elastic resources
Toward dynamic infrastructure ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
6
Google network: 100,000s of miles of fiber cable, 8 subsea cables
FASTER (US, JP, TW) 2016
Unity (US, JP) 2010
SJC (JP, HK, SG) 2013
PLCN (HK, LA) 2019
Monet (US, BR) 2017
Network Network sea cable investments
Junior (Rio, Santos) 2017
Edge points of presence >100
Tannat (BR, UY, AR) 2017
Edge node locations >1000 Indigo (SG, ID, AU) 2019
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
7
Google Cloud Platform is organized into regions and zones
Netherlands London Oregon 3 3 3
Mumbai
2
Tokyo
Los Angeles
3 3
Iowa
Montreal
4 3
3
3
Finland
2 3
3 3
Frankfurt
Belgium
N Virginia
S Carolina
Taiwan
Singapore
Future region and number of zones 3
Sydney
Current region and number of zones
3
São Paulo
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
8
Google is committed to environmental responsibility
100% carbon neutral
One of the world’s
First data centers to
since 2007
largest corporate
achieve ISO 14001
purchasers of renewable energy
certification
Google offers customer-friendly pricing innovations Billing in sub-hour
Discounts for
Custom VM instance
increments
sustained use
types
For virtual machines
Automatically applied
Pay only for the
and containers in the
to virtual machine use
resources you need for
cloud; data processing
over 25% of a month
your application
and other services too
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
10
Open APIs and open source mean customers can leave Open APIs;
Open source for a rich
Multi-vendor-friendly
compatibility with open-source services
ecosystem
technologies
Google Stackdriver
Kubernetes
Cloud Bigtable
Forseti Security Cloud Dataproc
Kubernetes Engine
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
11
Security is designed into Google’s technical infrastructure Layer
Notable security measures (among others)
Operational security
Intrusion detection systems; techniques to reduce insider risk; employee U2F use; software development practices
Internet communication
Google Front End; designed-in Denial of Service protection
Storage services
Encryption at rest
User identity
Central identity service with support for U2F
Service deployment
Encryption of inter-service communication
Hardware infrastructure
Hardware design and provenance; secure boot stack; premises security
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
12
Why choose Google Cloud Platform? Google Cloud Platform enables developers to build, test, and deploy applications on Google’s highly secure, reliable, and scalable infrastructure.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
13
Review: Google Cloud Platform offers a range of compute services Compute
Compute Engine
Kubernetes Engine
App Engine
Cloud Functions
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
14
Google Cloud Platform offers a range of storage services Compute
Compute Engine
Kubernetes Engine
Storage
App Engine
Cloud Functions
Bigtable
Cloud Storage
Cloud SQL
Cloud Spanner
Cloud Datastore
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
15
Google Cloud Platform offers services for getting value from data Compute
Compute Engine
Kubernetes Engine
Storage
App Engine
Cloud Functions
Bigtable
Cloud Storage
Big Data
BigQuery
Pub/Sub
Dataflow
Cloud SQL
Cloud Spanner
Cloud Datastore
Machine Learning
Dataproc
Datalab
Natural Vision API Language API
Machine Learning
Speech API
Translate API
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
16
Agenda 1
Introduction to Google Cloud Platform
2
Quiz
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
17
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
18
Quiz Name some of Google Cloud Platform’s pricing innovations.
Name some benefits of using Google Cloud Platform other than its pricing.
Quiz Name some of Google Cloud Platform’s pricing innovations.
● Sub-hour billing ● Sustained-use discounts ● Compute Engine custom machine types
Name some benefits of using Google Cloud Platform other than its pricing.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
19
Quiz Name some of Google Cloud Platform’s pricing innovations.
● Sub-hour billing ● Sustained-use discounts ● Compute Engine custom machine types
Name some benefits of using Google Cloud Platform other than its pricing.
● Commitment to environmental responsibility ● Commitment to open-source technologies ● Robust infrastructure
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
20
Resources Why Google Cloud Platform? https://cloud.google.com/why-google/ Pricing philosophy https://cloud.google.com/pricing/philosophy/ Data centers https://www.google.com/about/datacenters/ Google Cloud Platform product overview http://cloud.google.com/products/ Google Cloud Platform solutions http://cloud.google.com/solutions/ ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
21
Cloud OnBoard
Getting Started with Google Cloud Platform Google Cloud Platform Fundamentals: Core Infrastructure V3.0
©Google Inc. or its affiliates. All rights reserved. Do not distribute. ©Google Inc. or its affiliates. All rights reserved. Do not distribute. May only be taught by Google Cloud Platform Authorized Trainers.
Agenda 1
Google Cloud Platform Projects
2
Identity and Access Management (IAM)
3
Interacting with Google Cloud Platform
4
Quiz
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
23
Projects organize resources ● All Google Cloud Platform services you use are associated with a project. ● Use the project to: ○ Track resource and quota usage. ○ Enable billing. ○ Manage permissions and credentials. ○ Enable services and APIs.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
24
Projects are uniquely named ● Projects have three identifying attributes: ○ Project Name ○ Project Number
○ Project ID (Application ID for Google App Engine)
● Interact with projects using the Cloud Platform Console or the Cloud Resource Manager API.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
25
●
Resources inherit policies from parent. ○
●
Each policy contains a set of roles and role members.
example.com
bookshelf
Resource policies are a union of parent and resource.
A less restrictive parent policy overrides a more restrictive resource policy.
static-assets
stream-ingest
Compute Engine
App Engine
Cloud Storage
Cloud Storage
Cloud Pub/Sub
BigQuery
instance_a&
queue_a&
bucket_a&
bucket_b&
topic_a&
dataset_a&
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
Policy Inheritance
○
Project
A policy is set on a resource.
Resources
●
Organization
The organization node organizes projects
26
The organization node organizes projects ● Organization node is root node for Google Cloud resources ● Notable organization roles: ○ Organization Policy Administrator: Broad control over all cloud resources ○ Project Creator: Fine-grained control of project creation
[email protected]
example.com
OrganizationAdmin
Create
[email protected]
project_1
project_2
Project Creator
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
27
Agenda 1
Google Cloud Platform Projects
2
Identity and Access Management (IAM)
3
Interacting with Google Cloud Platform
4
Quiz & Lab
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
28
Google Cloud Identity and Access Management defines...
Who
can do what
on which resource
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
29
There are three types of IAM roles Primitive
Predefined
CustomBeta
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
30
IAM primitive roles: for fixed, coarse-grained levels of access
Owner
Editor
Invite members Deploy applications Remove members Modify code Can delete project Configure services Includes Editor rights Includes Viewer rights
Viewer
Billing administrator
Read-only access
Manage billing Add administrators Remove administrators
A project can have multiple owners, editors, viewers and billing administrators. ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
31
IAM predefined roles: for fine-grained permissions on particular services Google Group InstanceAdmin Role
project_a
List of Permissions ✔ ✔ ✔ ✔ ✔ ✔
compute.instances.delete compute.instances.get compute.instances.list compute.instances.setMachineType compute.instances.start compute.instances.stop ...
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
32
IAM custom rolesBeta : let you define a precise set of permissions Google Group InstanceOperator Role
List of Permissions ✔ ✔ ✔ ✔
compute.instances.get compute.instances.list compute.instances.start compute.instances.stop ...
project_a
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
33
Service Accounts control server-to-server interactions ● Provide an identity for carrying out server-to-server interactions in a project ● Used to authenticate from one service to another ● Used to control privileges used by resources ○ So that applications can perform actions on behalf of authenticated end users
● Identified with an email address:
[email protected]&
[email protected] ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
34
Service Accounts and IAM ● Service accounts authenticate using keys.
Identity
IAM Role
Service Account
InstanceAdmin Role
Resource
○ Google manages keys for Compute Engine and App Engine.
● You can assign a curated or custom IAM role to the service account.
Compute Instances
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
35
Example: Service Accounts and IAM ● VMs running component_1 are granted Editor access to project_b using Service Account 1. ● VMs running component_2 are granted objectViewer access to bucket_1 using Service Account 2. ● Service account permissions can be changed without recreating VMs.
project_a
component_1
Service Account 1 Editor
component_2
Service Account 2 Storage. objectViewer
project_b
bucket_1 ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
36
Agenda 1
Google Cloud Platform Projects
2
Identity and Access Management (IAM)
3
Interacting with Google Cloud Platform
4
Quiz
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
37
There are four ways to interact with GCP Cloud Platform
Cloud Shell and
Cloud Console
Console
Cloud SDK
Mobile App
Web user
Command-line
For iOS and
interface
interface
Android
REST-based API For custom applications
>_
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
38
Google Cloud Platform Console ● Centralized console for all project data ● Developer tools ○ Cloud Source Repositories ○ Cloud Shell ○ Test Lab (mobile app testing)
● Access to product APIs ● Manage and create projects
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
39
Google Cloud SDK ● SDK includes CLI tools for Cloud Platform products and services ○ gcloud, gsutil (Cloud Storage), bq (BigQuery)
● Available as Docker image ● Available via Cloud Shell ○ Containerized version of Cloud SDK running on Compute Engine instance
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
40
RESTful APIs ● Programmatic access to products and services ○ Typically use JSON as an interchange format ○ Use OAuth 2.0 for authentication and authorization
● Enabled through the Google Cloud Platform Console ● Most APIs include daily quotas and rates (limits) that can be raised by request ○ Important to plan ahead to manage your required capacity
● Experiment with APIs Explorer ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
41
Cloud Console Mobile App ● Manage virtual machines and database instances ● Manage apps in Google App Engine ● Manage your billing ● Visualize your projects with a customizable dashboard
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
42
APIs Explorer ● The APIs Explorer is an interactive tool that lets you easily try Google APIs using a browser. ● With the APIs Explorer, you can: ○ Browse quickly through available APIs and versions. ○ See methods available for each API and what parameters they support along with inline documentation. ○ Execute requests for any method and see responses in real time. ○ Easily make authenticated and authorized API calls.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
43
Client Libraries ● Cloud Client Libraries ○ Community-owned, handcrafted client libraries ● Google API Client Libraries ○ Open source, generated ○ Support various languages ■ Java, Python, JavaScript, PHP, .NET, Go, Node.js, Ruby, Objective-C, Dart
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
44
Cloud Launcher gives quick access to solutions ● A solution marketplace containing pre-packaged, ready-to-deploy solutions ○ Some offered by Google ○ Others by third-party vendors
● You pay for the underlying GCP resource usage. ○ Some solutions also assess third-party license fees.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
45
Agenda 1
Google Cloud Platform Projects
2
Identity and Access Management (IAM)
3
Interacting with Google Cloud Platform
4
Quiz
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
46
Quiz True or False: If a Google Cloud IAM policy gives you Owner permissions at the project level, your access to a resource in the project may be restricted by a more restrictive policy on that resource. True or False: All Google Cloud Platform resources are associated with a project.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
47
Quiz True or False: If a Google Cloud IAM policy gives you Owner permissions at the project level, your access to a resource in the project may be restricted by a more restrictive policy on that resource.
False: Policies are a union of the parent and the resource. If a parent policy is less restrictive, it overrides a more restrictive resource policy.
True or False: All Google Cloud Platform resources are associated with a project.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
48
Quiz True or False: If a Google Cloud IAM policy gives you Owner permissions at the project level, your access to a resource in the project may be restricted by a more restrictive policy on that resource.
False: Policies are a union of the parent and the resource. If a parent policy is less restrictive, it overrides a more restrictive resource policy.
True or False: All Google Cloud Platform resources are associated with a project.
True: All Google Cloud Platform resources are associated with a project.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
49
Quiz: Service Accounts Service accounts are used to provide which of the following? ❏ Authentication between Google Cloud Platform services ❏ Key generation and rotation when used with App Engine and Compute Engine ❏ A way to restrict the actions a resource (such as a VM) can perform ❏ A way to allow users to act with service account permissions ❏ All of the above
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
50
Quiz: Service Accounts Service accounts are used to provide which of the following? ❏ Authentication between Google Cloud Platform services ❏ Key generation and rotation when used with App Engine and Compute Engine ❏ A way to restrict the actions a resource (such as a VM) can perform ❏ A way to allow users to act with service account permissions ✓ All of the above
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
51
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
52
More Resources Google Cloud Platform security https://cloud.google.com/security/ Configuring permissions https://cloud.google.com/docs/permissions-overview Identity and Access Management (IAM) https://cloud.google.com/iam/ Cloud SDK installation and quick start https://cloud.google.com/sdk/#Quick_Start gcloud tool guide https://cloud.google.com/sdk/gcloud/
Cloud OnBoard
Google Compute Engine and Networking Google Cloud Platform Fundamentals: Core Infrastructure V3.0
©Google Inc. or its affiliates. All rights reserved. Do not distribute. ©Google Inc. or its affiliates. All rights reserved. Do not distribute. May only be taught by Google Cloud Platform Authorized Trainers.
Agenda 1
Google Compute Engine Overview
2
Google Cloud Networking
3
Operations and Tools
4
Comparing Compute Options
5
Quiz ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
54
Compute Engine offers managed virtual machines ● High CPU, high memory, standard and sharedcore machine types ● Persistent disks ○ Standard, SSD, local SSD ○ Snapshots
● Resize disks with no downtime ● Instance metadata and startup scripts
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
55
Compute Engine offers innovative pricing ● Per-second billing, sustained use discounts ● Preemptible instances ● High throughput to storage at no extra cost ● Custom machine types: Only pay for the hardware you need
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
56
Agenda 1
Google Compute Engine Overview
2
Google Cloud Networking
3
Operations and Tools
4
Comparing Compute Options
5
Quiz ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
57
Virtual Private Cloud Network ● Managed networking functionality for Google Cloud Platform resources ● Provision Cloud Platform resources, connect them to each other, and isolate them from one another in a Virtual Private Cloud (VPC).
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
58
Google Cloud VPCs are global; subnets are regional My VPC us-east1 my-subnet1
us-east1-b
us-east1-c
10.0.0.0/24 10.0.0.2
10.0.0.3 ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
59
VPC Network offers many internetworking features ● Fine-grained networking policies ● Fine-grained IP address range selection ● Routes ● Firewalls ● Virtual Private Network (VPN) ● Cloud Router
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
60
Google Cloud Platform offers many interconnect options
Carrier Interconnect
Direct Peering
Enterprise-grade connections provided by carrier service providers
Connect your business directly to Google
CDN Interconnect Allows select CDN providers to establish direct interconnect links with Google’s edge network at various locations
Google Cloud Interconnect ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
61
Cloud DNS is highly available and scalable ● Create managed zones, then add, edit, delete DNS records ○ Programmatically manage zones and records using RESTful API or command-line interface
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
62
Cloud Load Balancing: HTTP(S) ● Balance HTTP-based traffic across multiple Compute Engine regions ● Global, external IP address routes traffic ● Traffic is directed only to instances that pass health checks ● Scalable, requires no pre-warming and provides resilience, fault tolerance
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
63
Cloud Load Balancing: TCP/SSL, UDP ● Spread TCP/SSL and UDP traffic over pool of instances within a Compute Engine region ● Traffic is directed only to instances that pass health checks ● Scalable, requires no pre-warming
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
64
Cloud CDN (Content Delivery Network) ● Use Google's globally distributed edge caches to cache HTTP(S) load-balanced content far closer to your users than your instances ○ Faster delivery of content to users while reducing costs
● Cloud CDN uses caches at network locations to store responses generated by instances
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
65
Agenda 1
Google Compute Engine Overview
2
Google Cloud Networking
3
Operations and Tools
4
Comparing Compute Options
5
Quiz ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
66
Google Stackdriver
● Integrated monitoring, logging, diagnostics ● Powerful data, analytics tools ● Collaborations with PagerDuty, BMC, Splunk, others Works across Google Cloud Platform, Amazon Web Services ● Open source agents, integration
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
67
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
68
Google Stackdriver’s areas of focus Monitoring
Logging
Platform, system, and application metrics Uptime/health checks Dashboards and alerts
Platform, system, and application logs Log search/view/filter Log-based metrics
Trace
Error Reporting
Latency reporting and sampling Per-URL latency and statistics
Error notifications Error dashboard
Debugger Debug applications
Google Cloud Deployment Manager
● Infrastructure management service ● Create a .yaml template describing your environment and use Deployment Manager to create resources ● Provides repeatable deployments
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
69
Google Cloud Source Repositories
● Fully-featured Git repositories hosted on Google Cloud Platform ● Supports collaborative development of cloud apps ● Includes integration with Stackdriver debugger
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
70
Agenda 1
Google Compute Engine Overview
2
Google Cloud Networking
3
Operations and Tools
4
Comparing Compute Options
5
Quiz ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
71
Comparing Compute solutions Service
Compute Engine
Language support
Any
Any
Java, Python, Go and PHP
Java, Python, Node.js, Ruby, PHP, .NET core, and Go; or supply your own runtime
JavaScript
Service model
IaaS
Hybrid
PaaS
PaaS
Serverless
Use cases
General computing workloads
Container-based workloads
Web and mobile applications
Web and mobile applications, container-based workloads
Ephemeral functions responding to events
Kubernetes Engine
App Engine Standard
App Engine Flex
Cloud FunctionsBeta
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
72
Agenda 1
Google Compute Engine Overview
2
Google Cloud Networking
3
Operations and Tools
4
Comparing Compute Options
5
Quiz ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
73
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
74
Quiz Name 3 robust networking services available to your applications on Google Cloud Platform. Name 3 Compute Engine pricing innovations.
True or False: Google Cloud Load Balancing lets you balance HTTP traffic across multiple Compute Engine regions.
Quiz Name 3 robust networking services available to your applications on Google Cloud Platform.
Cloud Virtual Network, Cloud Interconnect, Cloud DNS, Cloud Load Balancing, and Cloud CDN.
Name 3 Compute Engine pricing innovations.
True or False: Google Cloud Load Balancing lets you balance HTTP traffic across multiple Compute Engine regions.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
75
Quiz Name 3 robust networking services available to your applications on Google Cloud Platform.
Cloud Virtual Network, Cloud Interconnect, Cloud DNS, Cloud Load Balancing, and Cloud CDN.
Name 3 Compute Engine pricing innovations.
Per-second billing, custom machine types, preemptible instances.
True or False: Google Cloud Load Balancing lets you balance HTTP traffic across multiple Compute Engine regions.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
76
Quiz Name 3 robust networking services available to your applications on Google Cloud Platform.
Cloud Virtual Network, Cloud Interconnect, Cloud DNS, Cloud Load Balancing, and Cloud CDN.
Name 3 Compute Engine pricing innovations.
Per-second billing, custom machine types, preemptible instances.
True or False: Google Cloud Load Balancing lets you balance HTTP traffic across multiple Compute Engine regions.
True.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
77
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
78
Resources Google Compute Engine https://cloud.google.com/compute/docs/ Google Cloud Platform VPC https://cloud.google.com/compute/docs/vpc/ Google Cloud Stackdriver https://cloud.google.com/stackdriver/docs/ Google Cloud Source Repositories gcloud tool guide https://cloud.google.com/source-repositories/docs/
Cloud OnBoard
Google Kubernetes Engine Google Cloud Platform Fundamentals: Core Infrastructure V3.0
©Google Inc. or its affiliates. All rights reserved. Do not distribute. ©Google Inc. or its affiliates. All rights reserved. Do not distribute. May only be taught by Google Cloud Platform Authorized Trainers.
Agenda 1
Introduction to Containers
2
Kubernetes
3
Google Kubernetes Engine
4
Quiz
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
80
Review: IaaS and PaaS
Toward managed infrastructure
Compute Engine
Kubernetes Engine
App Engine
IaaS
PaaS
Raw compute, storage, and network More granular control
Preset run-times Java, Go, PHP, Python... Focus is application logic
Pay for what you allocate More management overhead
Pay for what you use Less management overhead
Toward managed services
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
81
Containers mean virtualization inside the operating-system layer
● Separates operating system from application code and dependencies ● Isolates individual processes ● Popular implementations include Docker
Container Code and libraries
Container Code and libraries
Container Code and libraries
Shared libraries Operating system Virtual Machine
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
82
Why use containers? Consistency
Loose coupling
Across
Between
development, testing, and
application and operating system
Simplified
production
layers
premises and
environments
Workload migration
Agility Agile development and operations
between oncloud environments
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
83
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
84
Agenda 1
Introduction to Containers
2
Kubernetes
3
Google Kubernetes Engine
4
Quiz
Kubernetes is a container cluster orchestration system ● Also known as k8s ● Automates deployment, scaling, and operations for container clusters ● Open source, based on Google’s experience over 10+ years ● Built for a multi-cloud world ○ Public, private, hybrid
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
85
Kubernetes eases application management ● Workload portability ○ You can run in many environments, across cloud providers. ○ Implementation is open and modular.
● Rolling updates ○ You can upgrade applications without downtime.
● Persistent storage ○ Details of how storage is provided are abstracted from how it is consumed. ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
86
Kubernetes makes applications more elastic ● Multi-zone clusters ○ Run a single cluster in multiple zones
● Load balancing ○ External IP address routes traffic to correct port
● Autoscaling ○ Automatically adapt to changes in workload
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
87
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
88
Agenda 1
Introduction to Containers
2
Kubernetes
3
Google Kubernetes Engine
4
Quiz
Kubernetes Engine manages and runs containers ● Fully managed cluster management and orchestration system for running containers ○ Based on Kubernetes ○ Uses Compute Engine instances and resources
● Uses a declarative syntax to manage applications ○ Declare desired application configuration, Kubernetes Engine implements, manage
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
89
Why use Kubernetes Engine? ● Decouples operational, development concerns ● Manages and maintains ○ Logging, health management, monitoring ● Easily update Kubernetes versions as they are released
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
90
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
91
Comparing available solutions Service
Compute Engine
Language support
Any
Any
Java, Python, Go and PHP
Java, Python, Node.js, Ruby, PHP, .NET core, and Go; or supply your own runtime
JavaScript
Service model
IaaS
Hybrid
PaaS
PaaS
Serverless
Use cases
General computing workloads
Container-based workloads
Web and mobile applications
Web and mobile applications, container-based workloads
Ephemeral functions responding to events
Kubernetes Engine
App Engine Standard
App Engine Flex
Cloud FunctionsBeta
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
92
Deploying Apps: Kubernetes Engine vs App Engine Kubernetes Engine
App Engine Standard
App Engine Flexible
Language support
Any
Java, Python, Go & PHP
Any
Service model
Hybrid
PaaS
PaaS
Web and mobile applications
Web and mobile applications, container-based workloads
Primary use case Container-based workloads
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
93
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
94
Agenda 1
Introduction to Containers
2
Kubernetes
3
Google Kubernetes Engine
4
Quiz
Quiz Name two reasons for deploying applications using containers.
True or False: Kubernetes lets you manage container clusters in multiple cloud providers. True or False: GCP provides a private, high-speed container image storage service for use with Kubernetes Engine.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
95
Quiz Name two reasons for deploying applications using containers.
Consistency across development, testing, and production environments; Simpler to migrate workloads; Loose coupling; Agility
True or False: Kubernetes lets you manage container clusters in multiple cloud providers. True or False: GCP provides a private, high-speed container image storage service for use with Kubernetes Engine.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
96
Quiz Name two reasons for deploying applications using containers.
Consistency across development, testing, and production environments; Simpler to migrate workloads; Loose coupling; Agility
True or False: Kubernetes lets you manage container clusters in multiple cloud providers.
True.
True or False: GCP provides a private, high-speed container image storage service for use with Kubernetes Engine.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
97
Quiz Name two reasons for deploying applications using containers.
Consistency across development, testing, and production environments; Simpler to migrate workloads; Loose coupling; Agility
True or False: Kubernetes lets you manage container clusters in multiple cloud providers.
True.
True or False: GCP provides a private, high-speed container image storage service for use with Kubernetes Engine.
True.
©Google Inc. or its affiliates. All rights reserved. Do not distribute.
98
Resources Kubernetes Engine https://cloud.google.com/container-engine/docs/ Kubernetes Engine tutorials https://cloud.google.com/container-engine/docs/tutorials Kubernetes http://kubernetes.io/ Google Cloud Container Builder https://cloud.google.com/container-builder/docs/ Google Container Registry https://cloud.google.com/container-registry/docs/ ©Google Inc. or its affiliates. All rights reserved. Do not distribute.
Cloud OnBoard
Serverless on Google Cloud Platform Google Cloud Platform Fundamentals: Core Infrastructure V3.0
©Google Inc. or its affiliates. All rights reserved. Do not distribute. ©Google Inc. or its affiliates. All rights reserved. Do not distribute. May only be taught by Google Cloud Platform Authorized Trainers.
99
Agenda 1
Overview and Customer Stories
2
Google App Engine Standard Environment
3
Google App Engine Flexible Environment
4
Google Cloud Endpoints and Apigee Edge
5
Quiz
6 ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 101
IAAS and PAAS
Towards managed infrastructure
Compute Engine
App Engine
IaaS
PaaS
Raw compute, storage and network More granular control
Preset run-times Java, Go, PHP, Python... Focus is application logic
Pay for what you allocate More management overhead
Towards managed services
Pay for what you use Less management overhead
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 102
App Engine is a PaaS for building scalable applications ● App Engine makes deployment, maintenance, and scalability easy so you can focus on innovation ● Especially suited for building scalable web applications and mobile backends
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 103
Snapchat
“App Engine enabled us to focus on developing the application. We wouldn’t have gotten here without the ease of development that App Engine gave us.” Bobby Murphy, CTO
Snapchat sends
700 million
photos and videos each day
Google App Engine scaled seamlessly during growth to
Small team is able to innovate quickly and expand
millions of users
globally
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 104
Google Cloud Functions Beta ● Create single-purpose functions that respond to events without a server or runtime ○ Event examples: New instance created, file added to Cloud Storage ● Written in Javascript, execute in managed Node.js environment on Google Cloud Platform ● Perfect for IoT, microservices. ● Mobile first with with Firebase integration
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 105
Agenda 1
Overview and Customer Stories
2
Google App Engine Standard Environment
3
Google App Engine Flexible Environment
4
Google Cloud Endpoints and Apigee Edge
5
Quiz
6 ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 106
App Engine standard environment ● Easily deploy your applications ● Autoscale workloads to meet demand ● Economical ○ Free daily quota ○ Usage based pricing ● SDKs for development, testing and deployment
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 107
App Engine standard environment: Requirements ● Specific versions of Java, Python, PHP, and Go are supported ● Your application must conform to sandbox constraints: ○ No writing to local file system ○ All requests time out at 60 seconds ○ Third-party software installations are limited
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 108
Example App Engine standard workflow: Web applications 3
Develop & test the web application locally
1
App Engine automatically scales & reliably serves your web application
Project
Memcache
App Engine
Task queues
App Servers
Use the SDK to deploy to App Engine
2
App Engine can access a variety of services using dedicated APIs
Application instances
Scheduled tasks
Application instances
Search
Application instances
Logs ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 109
Agenda 1
Overview and Customer Stories
2
Google App Engine Standard Environment
3
Google App Engine Flexible Environment
4
Google Cloud Endpoints and Apigee Edge
5
Quiz
6
Quiz ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 110
App Engine flexible environment ● ● ● ● ●
Build and deploy containerized apps with a click No sandbox constraints Can access App Engine resources Standard runtimes: Python, Java, Go, Node.js Custom runtime support: Any language that supports HTTP requests ○ Package your runtime as a Dockerfile
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 111
112
Comparing the App Engine environments Standard Environment
Flexible Environment
Instance startup
Milliseconds
Minutes
SSH access
No
Yes (although not by default)
Scaling
Manual, basic, automatic
Manual, automatic
Write to local disk
No
Yes (but writes are ephemeral)
Support for 3rd-party binaries
No
Yes
Network access
Via App Engine services
Yes
Pricing model
After free daily use, pay per instance
Pay for resource allocation per hour; no
class, with automatic shutdown
automatic shutdown
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 112
Deploying Apps: App Engine vs Kubernetes Kubernetes Engine
App Engine Standard
App Engine Flexible
Language support
Any
Java, Python, Go & PHP
Any
Service model
Hybrid
PaaS
PaaS
Web and mobile applications
Web and mobile applications, container-based workloads
Primary use case Container-based workloads
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 113
Comparing available solutions Service
Compute Engine
Language support
Any
Any
Java, Python, Go and PHP
Java, Python, Node.js, Ruby, PHP, .NET core, and Go; or supply your own runtime
JavaScript
Service model
IaaS
Hybrid
PaaS
PaaS
Serverless
Use cases
General computing workloads
Container-based workloads
Web and mobile applications
Web and mobile applications, container-based workloads
Ephemeral functions responding to events
Kubernetes Engine
App Engine Standard
App Engine Flex
Cloud FunctionsBeta
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 114
Agenda 1
Overview and Customer Stories
2
Google App Engine Standard Environment
3
Google App Engine Flexible Environment
4
Google Cloud Endpoints and Apigee Edge
5
Quiz
6
Quiz ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 115
Cloud Endpoints helps you create and maintain APIs ● Distributed API management through an API console ● Expose your API using a RESTful interface ● Control access and validate calls with JSON Web Tokens and Google API keys ○ Identify web, mobile users with Auth0 and Firebase Authentication
● Generate client libraries
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 116
Cloud Endpoints: Supported platforms ● Supports App Engine standard or flexible environment, Compute Engine, Kubernetes Engine ● Use Java or Python open-source Frameworks or any other framework and language ● Supports iOS, Android, and JavaScript clients
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 117
Apigee Edge helps you secure and monetize APIs ● A platform for making APIs available to your customers and partners ● Contains analytics, monetization, and a developer portal
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 118
Agenda 1
Overview and Customer Stories
2
Google App Engine Standard Environment
3
Google App Engine Flexible Environment
4
Google Cloud Endpoints and Apigee Edge
5
Quiz
6 ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 119
Quiz Name 3 advantages of using the App Engine flexible environment over App Engine standard.
What is the difference between Cloud Endpoints and Apigee Edge?
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 120
Quiz Name 3 advantages of using the App Engine flexible environment over App Engine standard.
The flexible environment allows SSH access, allows disk writes, and supports third-party binaries (also allows stack customization and background processes).
What is the difference between Cloud Endpoints and Apigee Edge?
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 121
Quiz Name 3 advantages of using the App Engine flexible environment over App Engine standard.
The flexible environment allows SSH access, allows disk writes, and supports third-party binaries (also allows stack customization and background processes).
What is the difference between Cloud Endpoints and Apigee Edge?
Cloud Endpoints helps you create and maintain APIs; Apigee Edge helps you secure and monetize APIs.
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 122
Resources Google App Engine https://cloud.google.com/appengine/docs/ Google App Engine Flexible Environment https://cloud.google.com/appengine/docs/flexible/ Google App Engine Standard Environment https://cloud.google.com/appengine/docs/standard/ Google Cloud Endpoints https://cloud.google.com/endpoints/docs/ Apigee Edge http://docs.apigee.com/api-services/content/what-apigee-edge
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 123
Cloud OnBoard
Google Cloud Platform Storage Options Google Cloud Platform Fundamentals: Core Infrastructure V3.0
©Google Inc. or its affiliates. All rights reserved. Do not distribute. ©Google Inc. or its affiliates. All rights reserved. Do not distribute. May only be taught by Google Cloud Platform Authorized Trainers.
Agenda 1
Google Cloud Storage
2
Google Cloud Datastore and Google Cloud Bigtable
3
Google Cloud SQL and Google Cloud Spanner
4
Comparing Storage Options
5
Quiz ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 125
Cloud Storage is binary large-object storage ● High performance, internet-scale ● Simple administration ○ Does not require capacity management
● Data encryption at rest ● Data encryption in transit by default from Google to endpoint ● Online and offline import services are available ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 126
Your Cloud Storage files are organized into buckets
Bucket attributes:
Bucket contents
● Globally unique name
● Files (in a flat namespace)
● Storage class
● Access Control Lists
● Location ○ Region or multi-region ● IAM policies or Access Control Lists ● Object versioning setting ● Object lifecycle management rules
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 127
Choosing among Cloud Storage classes Multi-regional
Regional
Nearline
Coldline
Intended for data that is...
Most frequently accessed
Accessed frequently within a region
Accessed less than once a month
Accessed less than once a year
Availability SLA
99.95%
99.90%
99.00%
99.00%
Access APIs
Consistent APIs
Access time
Millisecond access
Storage price
Price per GB stored per month
Retrieval price Total price per GB transferred
Use cases
Content storage and delivery
In-region analytics, transcoding
Long-tail content, backups
Archiving, disaster recovery
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 128
TERABYTES
of data from more than
200 PER GENOME RAW
GIGABYTES
1,300 WHOLE
MSSNG project could easily surpass a
GENOMES
PETABYTE
to Google Cloud Storage
10,000
Making MSSNG world’s largest single repository of autism-related DNA sequencing data
PEOPL E
WHOLE GENOMES FROM
OF DATA
UPLOADED
100
UP TO
“Thanks to Google Cloud Platform and the Google Genomics team, the greatest minds in science from around the world will be able to study trillions of data points in one single database.”
Agenda 1
Google Cloud Storage
2
Google Cloud Datastore and Google Cloud Bigtable
3
Google Cloud SQL and Google Cloud Spanner
4
Comparing Storage Options
5
Quiz ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 130
Cloud Datastore is a horizontally scalable NoSQL DB ● NoSQL designed for application backends ● Fully managed ○ Uses a distributed architecture to automatically manage scaling
● Built-in redundancy ● Supports ACID transactions
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 131
Google Cloud Datastore: benefits ● Schemaless access ○ No need to think about underlying data structure
● Local development tools ● Includes a free daily quota ● Access from anywhere through a RESTful interface
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 132
Mobile Client Geolocation Map cache
Niantic Server Logic (Kubernetes deployment, 10K+ CPU cores) Player Data Spatial Service
Niantic Dataflow
NoSQL database (Google Cloud Datastore)
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 133
Cloud Datastore Transactions Per Second
1X
Target Traffic
5X Worst Case Estimate
50X Actual Traffic Original Launch Target
Estimated Worst Case
Actual Traffic
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 134
Cloud Bigtable is managed NoSQL ● Fully managed NoSQL, wide-column database service for terabyte applications ● Integrated ○ Accessed using HBase API ○ Native compatibility with big data, Hadoop ecosystems
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 135
Why choose Cloud Bigtable? ● Replicated storage ● Data encryption in-flight and at rest ● Role-based ACLs ● Drives major applications such as Google Analytics and Gmail
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 136
Bigtable Access Patterns Application API Data can be read from and written to Cloud Bigtable through a data service layer like Managed VMs, the HBase REST Server, or a Java Server using the HBase client. Typically this will be to serve data to applications, dashboards, and data services.
Streaming Data can be streamed in (written event by event) through a variety of popular stream processing frameworks like Cloud Dataflow Streaming, Spark Streaming, and Storm.
Batch Processing
Cloud Bigtable
Data can be read from and written to Cloud Bigtable through batch processes like Hadoop MapReduce, Dataflow, or Spark. Often, summarized or newly calculated data is written back to Cloud Bigtable or to a downstream database.
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 137
Overview:
Challenges:
Data to process: Data in the Consolidated Audit Trail (CAT). A data repository of all equities and options orders, quotes, and events
How to process the CAT and organize 100 billion market events into an “order lifecycle” in a 4 hour window Store 6 years (~30PB) of data
Cloud Bigtable to process and run queries and tolerate volume increases
6 BILLION MARKET EVENTS WRITTEN PER HOUR
1.7 GIGs PER SECOND
6 TBs PER HOUR
10 BN
1.7 GIGABYTES
WRITTEN PER HOUR BURSTS
10 TERABYTES
PER SECOND PER HOUR
Google confidential │ Do not distribute
Datastore vs Bigtable: use cases Cloud
Bigtable
Datastore Type
Best for
Use cases
NoSQL
NoSQL
document
wide column
Getting started, App Engine
“Flat” data, Heavy read/
applications
Getting started, App Engine applications
Cloud
Cloud
Cloud
Storage
SQL
Spanner
Blobstore
BigQuery
Relational
Relational
Relational
SQL for OLTP
SQL for OLTP
SQL for OLAP
Structured and unstructured
Web frameworks,
Large-scale database
Interactive querying, offline
write, events, analytical data
binary or object data
existing applications
applications (> ~2 TB)
analytics
AdTech, Financial and IoT data
Images, large media files, backups
User credentials, customer orders
Whenever high I/O, global consistency is needed
Data warehousing
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 139
Agenda 1
Google Cloud Storage
2
Google Cloud Datastore and Google Cloud Bigtable
3
Google Cloud SQL and Google Cloud Spanner
4
Comparing Storage Options
5
Quiz ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 140
Cloud SQL is a managed RDBMS ● Offers MySQL and PostgreSQLBeta databases as a service ● Automatic replication ● Managed backups ● Vertical scaling (read and write) ● Horizontal scaling (read) ● Google security
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 141
Cloud Spanner is a horizontally scalable RDBMS Cloud Spanner supports: ● ● ● ●
Automatic replication Strong global consistency Managed instances with high availability SQL (ANSI 2011 with extensions)
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 142
Agenda 1
Google Cloud Storage
2
Google Cloud Datastore and Google Cloud Bigtable
3
Google Cloud SQL and Google Cloud Spanner
4
Comparing Storage Options
5
Quiz ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 143
Comparing storage options: technical details Cloud
Bigtable
Datastore Type
NoSQL
NoSQL
document
wide column
Transactions
Yes
Single-row
Complex queries
No
Capacity Unit size
Cloud
Cloud
Cloud
Storage
SQL
Spanner
Relational
Relational
Relational
SQL for OLTP
SQL for OLTP
SQL for OLAP
No
Yes
Yes
No
No
No
Yes
Yes
Yes
Terabytes+
Petabytes+
Petabytes+
500 GB
Petabytes
Petabytes+
1 MB/entity
~10 MB/cell
5 TB/object
Determined
10,240 MiB/
10 MB/row
by DB engine
row
~100 MB/row
Blobstore
BigQuery
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 144
Comparing storage options: use cases Cloud
Bigtable
Datastore Type
Best for
Use cases
NoSQL
NoSQL
document
wide column
Getting started, App Engine
“Flat” data, Heavy read/
applications
Getting started, App Engine applications
Cloud
Cloud
Cloud
Storage
SQL
Spanner
Blobstore
BigQuery
Relational
Relational
Relational
SQL for OLTP
SQL for OLTP
SQL for OLAP
Structured and unstructured
Web frameworks,
Large-scale database
Interactive querying, offline
write, events, analytical data
binary or object data
existing applications
applications (> ~2 TB)
analytics
AdTech, Financial and IoT data
Images, large media files, backups
User credentials, customer orders
Whenever high I/O, global consistency is needed
Data warehousing
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 145
Agenda 1
Google Cloud Storage
2
Google Cloud Datastore and Google Cloud Bigtable
3
Google Cloud SQL and Google Cloud Spanner
4
Comparing Storage Options
5
Quiz ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 146
Quiz Your application transcodes large video files. Which storage service should you consider first?
You stream huge amounts of data from devices with sensors. Which storage service should you consider first?
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 147
Quiz Your application transcodes large video files. Which storage service should you consider first?
Google Cloud Storage
You stream huge amounts of data from devices with sensors. Which storage service should you consider first?
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 148
Quiz Your application transcodes large video files. Which storage service should you consider first?
Google Cloud Storage
You stream huge amounts of data from devices with sensors. Which storage service should you consider first?
Google Cloud Bigtable
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 149
Resources Overview of Cloud Storage https://cloud.google.com/storage/ Getting started with Google Cloud SQL https://cloud.google.com/sql/docs/quickstart Cloud Bigtable https://cloud.google.com/stackdriver/docs/ Cloud Spanner https://cloud.google.com/spanner/docs/ Cloud Datastore https://cloud.google.com/datastore/docs/
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 150
Cloud OnBoard
Big Data and Machine Learning Google Cloud Platform Fundamentals: Core Infrastructure V3.0
©Google Inc. or its affiliates. All rights reserved. Do not distribute. ©Google Inc. or its affiliates. All rights reserved. Do not distribute. May only be taught by Google Cloud Platform Authorized Trainers.
Agenda
1
Google Cloud Big Data Platform
2
Google Cloud Machine Learning Platform
3
Quiz
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 152
Google Cloud’s big data services are fully managed and scalable
Cloud BigQuery
Cloud Dataflow Cloud Dataproc Managed Hadoop MapReduce, Spark, Pig, and Hive service
Stream & batch processing; unified and simplified pipelines.
Cloud DataPrep
Cloud Pub/Sub
Analytics database; data streaming at 100,000 rows/second
Interactive data exploration with Jupyter Notebooks
Scalable and flexible enterprise messaging
Prepare, clean your data and build your Dataflow pipeline
Cloud Datalab
Cloud Data Studio A collaborative BigQuery visualization experience ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 153
Cloud Dataproc is managed Hadoop ● Fast, easy, managed way to run Hadoop and Spark/Hive/Pig on Google Cloud Platform ● Create clusters in 90 seconds or less on average. ● Scale clusters up and down even when jobs are running.
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 154
Typical Spark/Hadoop clusters... Scale
Obtain servers
Configure servers
Install OSS
Reconfigure
Configure OSS
… vs Cloud Dataproc clusters
Create cluster (t+0)
Optimize OSS
Debug OSS
Process data
Scale anytime
Configure cluster (t+20 sec)
Process data (t+90 sec) ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 155
Why use Cloud Dataproc? ● Easily migrate on-premises Hadoop jobs to the cloud. ● Quickly analyze data (like log data) stored in Cloud Storage; create a cluster in 90 seconds or less on average, and then delete it immediately. ● Use Spark/Spark SQL to quickly perform data mining and analysis. ● Use Spark Machine Learning Libraries (MLlib) to run classification algorithms.
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 156
Cloud Dataflow offers managed data pipelines ● Processes data using Compute Engine instances. ○ Clusters are sized for you ○ Automated scaling, no instance provisioning required ● Write code once and get batch and streaming. ○ Transform-based programming model
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 157
Dataflow pipelines flow data from a source through transforms Source BigQuery
Transforms
Sink
Cloud Storage
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 158
Not great at writing data pipelines? Use Cloud Dataprep! ● Data Preparation and Data cleansing using an easy to use visual interface ● An intelligent cloud data service to visually explore, clean, and prepare data for analysis
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 159
Why use Cloud Dataflow? ● ETL (extract/transform/load) pipelines to move, filter, enrich, shape data ● Data analysis: batch computation or continuous computation using streaming ● Orchestration: create pipelines that coordinate services, including external services ● Integrates with GCP services like Cloud Storage, Cloud Pub/Sub, BigQuery, and Bigtable ○ Open source Java and Python SDKs
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 160
BigQuery is a fully managed data warehouse ● Provides near real-time interactive analysis of massive datasets (hundreds of TBs) ● Query using SQL syntax (SQL 2011) ● No cluster maintenance is required.
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 161
BigQuery runs on Google’s high-performance infrastructure ● Compute and storage are separated with a terabit network in between ● You only pay for storage and processing used ● Automatic discount for long-term data storage
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 162
Cloud Pub/Sub is scalable, reliable messaging ● Supports many-to-many asynchronous messaging ○ Application components make push/pull subscriptions to topics ● Includes support for offline consumers ● Based on proven Google technologies ● Integrates with Cloud Dataflow for data processing pipelines
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 163
Why use Cloud Pub/Sub? ● Building block for data ingestion in Dataflow, Internet of Things (IoT), Marketing Analytics ● Foundation for Dataflow streaming ● Push notifications for cloud-based applications ● Connect applications across Google Cloud Platform (push/pull between Compute Engine and App Engine)
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 164
Cloud Datalab offers interactive data exploration ● Interactive tool for large-scale data exploration, transformation, analysis, and visualization ● Integrated, open source ○ Runs on Compute Engine ○ Built on Jupyter (formerly IPython)
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 165
Why use Cloud Datalab? ● Create and manage code, documentation, results, and visualizations in intuitive notebook format. ○ Use Google Charts or matplotlib for easy visualizations.
● Analyze data in BigQuery, Compute Engine, and Cloud Storage using Python, SQL, and JavaScript. ● Easily deploy models to BigQuery. ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 166
Cloud Data Studio offers BigQuery visualization ● Web-based data visualization ● Direct access to BigQuery data (and many other sources) ● Collaborative editing experience (using Drive model) ● Zero-install ● Free
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 168
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 169
Agenda
1
Google Cloud Big Data Platform
2
Google Cloud Machine Learning Platform
3
Quiz
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 170
Machine Learning APIs enable apps that see, hear, and understand
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 171
Cloud Machine Learning Platform Open source tool to build and run neural network models ●
Wide platform support: CPU or GPU; mobile, server, or cloud
Fully managed machine learning service
Cloud ML
●
Familiar notebook-based developer experience
●
Optimized for Google infrastructure; integrates with BigQuery and Cloud Storage
Pre-trained machine learning models built by Google ●
Speech: Stream results in real time, detects 80 languages
●
Vision: Identify objects, landmarks, text, and content
●
Translate: Language translation including detection
●
Natural language: Structure, meaning of text
Machine Learning APIs ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 172
Inject AI into your business with the Google Cloud Machine Learning platform For structured data
For unstructured data
Classification and regression
Image and video analytics
Recommendation Text analytics Anomaly detection ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 173
Cloud Vision API ● Analyze images with a simple REST API ○ Logo detection, label detection, etc ● With the Cloud Vision API, you can: ○ ○ ○ ○
Gain insight from images Detect inappropriate content Analyze sentiment Extract text
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 174
Cloud Speech API ● Recognizes over 80 languages and variants ● Can return text in real time ● Highly accurate, even in noisy environments ● Access from any device ● Powered by Google’s machine learning ● Used by Google Assistant
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 175
Cloud Natural Language API ● Uses machine learning models to reveal structure and meaning of text. ● Extract information about items mentioned in text documents, news articles, and blog posts. ● Analyze text uploaded in request or integrate with Cloud Storage.
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 176
Cloud Translation API ● Translate arbitrary strings between thousands of language pairs ● Programmatically detect a document’s language ● Support for dozens of languages
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 177
Cloud Video Intelligence API ● Annotate the contents of videos ● Detect scene changes ● Flag inappropriate content ● Support for a variety of video formats
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 178
Dialogflow API (previously called API.AI) BETA ● Build chatbots capable of natural and rich interactions between users and your business. ● Uses ML to recognize the intent & context of a conversation, allowing your interface to provide an efficient and accurate response. ● Use cases:
○ Customer Service ○ Commerce ○ IOT Devices ○ Enterprise Productivity ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 179
Cloud Job Discovery API PRIVATE BETA ● Power your job site with Machine Learning ● Improve job engagement and candidate conversion.
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 180
Agenda
1
Google Cloud Big Data Platform
2
Google Cloud Machine Learning Platform
3
Quiz
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 181
Quiz When would you use Cloud Dataproc?
Name two use cases for Cloud Dataflow.
Name three use cases for the Google machine learning platform.
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 182
Quiz When would you use Cloud Dataproc?
You can use it to migrate on-premises Hadoop jobs to the cloud. You can also use it for data mining and analysis of cloud-based data.
Name two use cases for Cloud Dataflow.
Name three use cases for the Google machine learning platform.
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 183
Quiz When would you use Cloud Dataproc?
You can use it to migrate on-premises Hadoop jobs to the cloud. You can also use it for data mining and analysis of cloud-based data.
Name two use cases for Cloud Dataflow.
ETL, orchestration
Name three use cases for the Google machine learning platform.
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 184
Quiz When would you use Cloud Dataproc?
You can use it to migrate on-premises Hadoop jobs to the cloud. You can also use it for data mining and analysis of cloud-based data.
Name two use cases for Cloud Dataflow.
ETL, orchestration
Name three use cases for the Google machine learning platform.
Fraud detection, sentiment analysis, content personalization
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 185
Resources
Google Big Data Platform https://cloud.google.com/products/big-data/ Google Machine Learning Platform https://cloud.google.com/products/machinelearning/
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 186
Cloud OnBoard
Igniting Your Career with Google Cloud Training & Certification
©Google Inc. or its affiliates. All rights reserved. Do not distribute. ©Google Inc. or its affiliates. All rights reserved. Do not distribute. May only be taught by Google Cloud Platform Authorized Trainers.
Google Cloud training - A roadmap for learners Introduction to GCP
Next Level Training
GCP Fundamentals - Core Infrastructure -
IaaS Architecting with GCP: Core Infrastructure Architecting: Design & Process Develop & Deploy Windows Applications on GCP
GCP Fundamentals - Data and ML 7 Qwiklabs - GCP Essentials GCP for AWS Professionals G Suite Admin Fundamentals
Advanced Advanced Training
8 Qwiklabs - Cloud Architecture
Getting Started with Kubernetes and Google Container Engine Coming Soon Advanced Data Engineering Advanced Machine Learning
Data and Machine Learning Data Engineering on GCP From Data to Insights with GCP 8 Qwiklabs - Data Engineering 8 Qwiklabs - Scientific Data Processing PaaS Developing Applications with GCP 7 Qwiklabs - Developing Applications ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 188
Find more training courses
cloud.google.com/training ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 189
Google Cloud Self Paced Qwiklabs
google.qwiklabs.com ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 190
After Cloud OnBoard
Continue Learning with Qwiklabs Receive a follow up email 1 day after the event
Create Qwiklabs account
●
Google Chrome is recommended
●
Use the email address with which you registered for the class
●
You must confirm your account before logging in
Open your email and confirm account
Open incognito browser tab
Return to Qwiklabs and log in
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 191
Make Certification Your Goal
Getting your Google cloud certification shows the world that you can design, develop, manage and administer application infrastructure and data solutions on Google Cloud technology. Gain industry recognition Validate your technical expertise Take your career to a level beyond your wildest dreams
cloud.google.com/certification/ ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 192
Join the GDG Cloud Bangkok community!
or bit.ly/gdgcloudbkk
©Google Inc. or its affiliates. All rights reserved. Do not distribute. 193
1 2 3 4 5 6
7
https://cloud.google.com/training/
8
9 10 11 12 13 14 15 16 17 18