Cloud OnBoard

1 2 3

4 5 6

Getting Started

7 8

With Google Cloud

9 10 11 12 13 14 15 16 17 18

1 2

Cloud OnBoard

3 4

{

5

(’Module 1’) Introducing Google Cloud Platform

Page

2 - 21

6

(’Module 2’) Getting Started with Google Cloud Platform

Page

22 - 52

7

(’Module 3’) Google Compute Engine and Networking

Page

53 - 78

8

(’Module 4’) Google Kubernetes Engine

Page

79 - 99

9

(’Module 5’) Serverless on Google Cloud

Page

100 - 123

10

(’Module 6’) Google Cloud Platform Storage Options

Page

124 - 150

11

(’Module 7’) Big Data and Machine Learning

Page

151 - 186

Page

187 - 193

12

Igniting your Career with Google Cloud | Training & Certification

13 14 15 16 17 18

}

Cloud OnBoard

Welcome to Cloud OnBoard #GoogleCloudOnBoard

©Google Inc. or its affiliates. All rights reserved. Do not distribute. ©Google Inc. or its affiliates. All rights reserved. Do not distribute. May only be taught by Google Cloud Platform Authorized Trainers.

Cloud OnBoard

Introducing Google Cloud Platform Google Cloud Platform Fundamentals: Core Infrastructure V3.0


Agenda 1

Introduction to Google Cloud Platform

2

Quiz

©Google Inc. or its affiliates. All rights reserved. Do not distribute.

3

Computing trends toward pay-as-you-go, fully automated services Next

Now

Storage

Processing

Memory

Network

Storage

Physical/Colo User-configured, managed, and maintained

Processing

Memory

Virtualized

Network

Serverless/No-Ops Fully automated


4

Every company is a data company


5

GCP offers a range of computing architectures

Compute Engine

Kubernetes Engine

App Engine

Cloud Functions

IaaS

Hybrid

PaaS

Serverless logic

Toward managed infrastructure

Managed services Automated elastic resources

Toward dynamic infrastructure ©Google Inc. or its affiliates. All rights reserved. Do not distribute.

6

Google network: 100,000s of miles of fiber cable, 8 subsea cables

FASTER (US, JP, TW) 2016

Unity (US, JP) 2010

SJC (JP, HK, SG) 2013

PLCN (HK, LA) 2019

Monet (US, BR) 2017

Network Network sea cable investments

Junior (Rio, Santos) 2017

Edge points of presence >100

Tannat (BR, UY, AR) 2017

Edge node locations >1000 Indigo (SG, ID, AU) 2019


7

Google Cloud Platform is organized into regions and zones

Netherlands London Oregon 3 3 3

Mumbai

2

Tokyo

Los Angeles

3 3

Iowa

Montreal

4 3

3

3

Finland

2 3

3 3

Frankfurt

Belgium

N Virginia

S Carolina

Taiwan

Singapore

Future region and number of zones 3

Sydney

Current region and number of zones

3

São Paulo


8

Google is committed to environmental responsibility

100% carbon neutral

One of the world’s

First data centers to

since 2007

largest corporate

achieve ISO 14001

purchasers of renewable energy

certification

Google offers customer-friendly pricing innovations Billing in sub-hour

Discounts for

Custom VM instance

increments

sustained use

types

For virtual machines

Automatically applied

Pay only for the

and containers in the

to virtual machine use

resources you need for

cloud; data processing

over 25% of a month

your application

and other services too


10

Open APIs and open source mean customers can leave Open APIs;

Open source for a rich

Multi-vendor-friendly

compatibility with open-source services

ecosystem

technologies

Google Stackdriver

Kubernetes

Cloud Bigtable

Forseti Security Cloud Dataproc

Kubernetes Engine


11

Security is designed into Google’s technical infrastructure Layer

Notable security measures (among others)

Operational security

Intrusion detection systems; techniques to reduce insider risk; employee U2F use; software development practices

Internet communication

Google Front End; designed-in Denial of Service protection

Storage services

Encryption at rest

User identity

Central identity service with support for U2F

Service deployment

Encryption of inter-service communication

Hardware infrastructure

Hardware design and provenance; secure boot stack; premises security


12

Why choose Google Cloud Platform? Google Cloud Platform enables developers to build, test, and deploy applications on Google’s highly secure, reliable, and scalable infrastructure.


13

Review: Google Cloud Platform offers a range of compute services Compute

Compute Engine

Kubernetes Engine

App Engine

Cloud Functions


14

Google Cloud Platform offers a range of storage services Compute

Compute Engine

Kubernetes Engine

Storage

App Engine

Cloud Functions

Bigtable

Cloud Storage

Cloud SQL

Cloud Spanner

Cloud Datastore


15

Google Cloud Platform offers services for getting value from data Compute

Compute Engine

Kubernetes Engine

Storage

App Engine

Cloud Functions

Bigtable

Cloud Storage

Big Data

BigQuery

Pub/Sub

Dataflow

Cloud SQL

Cloud Spanner

Cloud Datastore

Machine Learning

Dataproc

Datalab

Natural Vision API Language API

Machine Learning

Speech API

Translate API


16

Agenda 1

Introduction to Google Cloud Platform

2

Quiz


17


18

Quiz Name some of Google Cloud Platform’s pricing innovations.

Name some benefits of using Google Cloud Platform other than its pricing.


●  Sub-hour billing ●  Sustained-use discounts ●  Compute Engine custom machine types



19


●  Sub-hour billing ●  Sustained-use discounts ●  Compute Engine custom machine types


●  Commitment to environmental responsibility ●  Commitment to open-source technologies ●  Robust infrastructure


20

Resources Why Google Cloud Platform? https://cloud.google.com/why-google/ Pricing philosophy https://cloud.google.com/pricing/philosophy/ Data centers https://www.google.com/about/datacenters/ Google Cloud Platform product overview http://cloud.google.com/products/ Google Cloud Platform solutions http://cloud.google.com/solutions/ ©Google Inc. or its affiliates. All rights reserved. Do not distribute.

21

Cloud OnBoard

Getting Started with Google Cloud Platform Google Cloud Platform Fundamentals: Core Infrastructure V3.0


Agenda 1

Google Cloud Platform Projects

2

Identity and Access Management (IAM)

3

Interacting with Google Cloud Platform

4

Quiz


23

Projects organize resources ●  All Google Cloud Platform services you use are associated with a project. ●  Use the project to: ○  Track resource and quota usage. ○  Enable billing. ○  Manage permissions and credentials. ○  Enable services and APIs.


24

Projects are uniquely named ●  Projects have three identifying attributes: ○  Project Name ○  Project Number

○  Project ID (Application ID for Google App Engine)

●  Interact with projects using the Cloud Platform Console or the Cloud Resource Manager API.


25

● 

Resources inherit policies from parent. ○ 

● 

Each policy contains a set of roles and role members.

example.com

bookshelf

Resource policies are a union of parent and resource.

A less restrictive parent policy overrides a more restrictive resource policy.

static-assets

stream-ingest

Compute Engine

App Engine

Cloud Storage

Cloud Storage

Cloud Pub/Sub

BigQuery

instance_a&

queue_a&

bucket_a&

bucket_b&

topic_a&

dataset_a&


Policy Inheritance

○ 

Project

A policy is set on a resource.

Resources

● 

Organization

The organization node organizes projects

26

The organization node organizes projects ●  Organization node is root node for Google Cloud resources ●  Notable organization roles: ○  Organization Policy Administrator: Broad control over all cloud resources ○  Project Creator: Fine-grained control of project creation

[email protected]

example.com

OrganizationAdmin

Create

[email protected]

project_1

project_2

Project Creator


27

Agenda 1


2


3


4

Quiz & Lab


28

Google Cloud Identity and Access Management defines...

Who

can do what

on which resource


29

There are three types of IAM roles Primitive

Predefined

CustomBeta


30

IAM primitive roles: for fixed, coarse-grained levels of access

Owner

Editor

Invite members Deploy applications Remove members Modify code Can delete project Configure services Includes Editor rights Includes Viewer rights

Viewer

Billing administrator

Read-only access

Manage billing Add administrators Remove administrators

A project can have multiple owners, editors, viewers and billing administrators. ©Google Inc. or its affiliates. All rights reserved. Do not distribute.

31

IAM predefined roles: for fine-grained permissions on particular services Google Group InstanceAdmin Role

project_a

List of Permissions ✔  ✔  ✔  ✔  ✔  ✔ 

compute.instances.delete compute.instances.get compute.instances.list compute.instances.setMachineType compute.instances.start compute.instances.stop ...


32

IAM custom rolesBeta : let you define a precise set of permissions Google Group InstanceOperator Role

List of Permissions ✔  ✔  ✔  ✔ 

compute.instances.get compute.instances.list compute.instances.start compute.instances.stop ...

project_a


33

Service Accounts control server-to-server interactions ●  Provide an identity for carrying out server-to-server interactions in a project ●  Used to authenticate from one service to another ●  Used to control privileges used by resources ○  So that applications can perform actions on behalf of authenticated end users

●  Identified with an email address: [email protected]& [email protected] ©Google Inc. or its affiliates. All rights reserved. Do not distribute.

34

Service Accounts and IAM ●  Service accounts authenticate using keys.

Identity

IAM Role

Service Account

InstanceAdmin Role

Resource

○  Google manages keys for Compute Engine and App Engine.

●  You can assign a curated or custom IAM role to the service account.

Compute Instances


35

Example: Service Accounts and IAM ●  VMs running component_1 are granted Editor access to project_b using Service Account 1. ●  VMs running component_2 are granted objectViewer access to bucket_1 using Service Account 2. ●  Service account permissions can be changed without recreating VMs.

project_a

component_1

Service Account 1 Editor

component_2

Service Account 2 Storage. objectViewer

project_b

bucket_1 ©Google Inc. or its affiliates. All rights reserved. Do not distribute.

36

Agenda 1


2


3


4

Quiz


37

There are four ways to interact with GCP Cloud Platform

Cloud Shell and

Cloud Console

Console

Cloud SDK

Mobile App

Web user

Command-line

For iOS and

interface

interface

Android

REST-based API For custom applications

>_


38

Google Cloud Platform Console ●  Centralized console for all project data ●  Developer tools ○  Cloud Source Repositories ○  Cloud Shell ○  Test Lab (mobile app testing)

●  Access to product APIs ●  Manage and create projects


39

Google Cloud SDK ●  SDK includes CLI tools for Cloud Platform products and services ○  gcloud, gsutil (Cloud Storage), bq (BigQuery)

●  Available as Docker image ●  Available via Cloud Shell ○  Containerized version of Cloud SDK running on Compute Engine instance


40

RESTful APIs ●  Programmatic access to products and services ○  Typically use JSON as an interchange format ○  Use OAuth 2.0 for authentication and authorization

●  Enabled through the Google Cloud Platform Console ●  Most APIs include daily quotas and rates (limits) that can be raised by request ○  Important to plan ahead to manage your required capacity

●  Experiment with APIs Explorer ©Google Inc. or its affiliates. All rights reserved. Do not distribute.

41

Cloud Console Mobile App ●  Manage virtual machines and database instances ●  Manage apps in Google App Engine ●  Manage your billing ●  Visualize your projects with a customizable dashboard


42

APIs Explorer ●  The APIs Explorer is an interactive tool that lets you easily try Google APIs using a browser. ●  With the APIs Explorer, you can: ○  Browse quickly through available APIs and versions. ○  See methods available for each API and what parameters they support along with inline documentation. ○  Execute requests for any method and see responses in real time. ○  Easily make authenticated and authorized API calls.


43

Client Libraries ●  Cloud Client Libraries ○  Community-owned, handcrafted client libraries ●  Google API Client Libraries ○  Open source, generated ○  Support various languages ■  Java, Python, JavaScript, PHP, .NET, Go, Node.js, Ruby, Objective-C, Dart


44

Cloud Launcher gives quick access to solutions ●  A solution marketplace containing pre-packaged, ready-to-deploy solutions ○  Some offered by Google ○  Others by third-party vendors

●  You pay for the underlying GCP resource usage. ○  Some solutions also assess third-party license fees.


45

Agenda 1


2


3


4

Quiz


46

Quiz True or False: If a Google Cloud IAM policy gives you Owner permissions at the project level, your access to a resource in the project may be restricted by a more restrictive policy on that resource. True or False: All Google Cloud Platform resources are associated with a project.


47

Quiz True or False: If a Google Cloud IAM policy gives you Owner permissions at the project level, your access to a resource in the project may be restricted by a more restrictive policy on that resource.

False: Policies are a union of the parent and the resource. If a parent policy is less restrictive, it overrides a more restrictive resource policy.

True or False: All Google Cloud Platform resources are associated with a project.


48

Quiz True or False: If a Google Cloud IAM policy gives you Owner permissions at the project level, your access to a resource in the project may be restricted by a more restrictive policy on that resource.

False: Policies are a union of the parent and the resource. If a parent policy is less restrictive, it overrides a more restrictive resource policy.

True or False: All Google Cloud Platform resources are associated with a project.

True: All Google Cloud Platform resources are associated with a project.


49

Quiz: Service Accounts Service accounts are used to provide which of the following? ❏  Authentication between Google Cloud Platform services ❏  Key generation and rotation when used with App Engine and Compute Engine ❏  A way to restrict the actions a resource (such as a VM) can perform ❏  A way to allow users to act with service account permissions ❏  All of the above


50

Quiz: Service Accounts Service accounts are used to provide which of the following? ❏  Authentication between Google Cloud Platform services ❏  Key generation and rotation when used with App Engine and Compute Engine ❏  A way to restrict the actions a resource (such as a VM) can perform ❏  A way to allow users to act with service account permissions ✓  All of the above


51


52

More Resources Google Cloud Platform security https://cloud.google.com/security/ Configuring permissions https://cloud.google.com/docs/permissions-overview Identity and Access Management (IAM) https://cloud.google.com/iam/ Cloud SDK installation and quick start https://cloud.google.com/sdk/#Quick_Start gcloud tool guide https://cloud.google.com/sdk/gcloud/

Cloud OnBoard

Google Compute Engine and Networking Google Cloud Platform Fundamentals: Core Infrastructure V3.0


Agenda 1

Google Compute Engine Overview

2

Google Cloud Networking

3

Operations and Tools

4

Comparing Compute Options

5

Quiz ©Google Inc. or its affiliates. All rights reserved. Do not distribute.

54

Compute Engine offers managed virtual machines ●  High CPU, high memory, standard and sharedcore machine types ●  Persistent disks ○  Standard, SSD, local SSD ○  Snapshots

●  Resize disks with no downtime ●  Instance metadata and startup scripts


55

Compute Engine offers innovative pricing ●  Per-second billing, sustained use discounts ●  Preemptible instances ●  High throughput to storage at no extra cost ●  Custom machine types: Only pay for the hardware you need


56

Agenda 1


2


3


4


5


57

Virtual Private Cloud Network ●  Managed networking functionality for Google Cloud Platform resources ●  Provision Cloud Platform resources, connect them to each other, and isolate them from one another in a Virtual Private Cloud (VPC).


58

Google Cloud VPCs are global; subnets are regional My VPC us-east1 my-subnet1

us-east1-b

us-east1-c

10.0.0.0/24 10.0.0.2

10.0.0.3 ©Google Inc. or its affiliates. All rights reserved. Do not distribute.

59

VPC Network offers many internetworking features ●  Fine-grained networking policies ●  Fine-grained IP address range selection ●  Routes ●  Firewalls ●  Virtual Private Network (VPN) ●  Cloud Router


60

Google Cloud Platform offers many interconnect options

Carrier Interconnect

Direct Peering

Enterprise-grade connections provided by carrier service providers

Connect your business directly to Google

CDN Interconnect Allows select CDN providers to establish direct interconnect links with Google’s edge network at various locations

Google Cloud Interconnect ©Google Inc. or its affiliates. All rights reserved. Do not distribute.

61

Cloud DNS is highly available and scalable ●  Create managed zones, then add, edit, delete DNS records ○  Programmatically manage zones and records using RESTful API or command-line interface


62

Cloud Load Balancing: HTTP(S) ●  Balance HTTP-based traffic across multiple Compute Engine regions ●  Global, external IP address routes traffic ●  Traffic is directed only to instances that pass health checks ●  Scalable, requires no pre-warming and provides resilience, fault tolerance


63

Cloud Load Balancing: TCP/SSL, UDP ●  Spread TCP/SSL and UDP traffic over pool of instances within a Compute Engine region ●  Traffic is directed only to instances that pass health checks ●  Scalable, requires no pre-warming


64

Cloud CDN (Content Delivery Network) ●  Use Google's globally distributed edge caches to cache HTTP(S) load-balanced content far closer to your users than your instances ○  Faster delivery of content to users while reducing costs

●  Cloud CDN uses caches at network locations to store responses generated by instances


65

Agenda 1


2


3


4


5


66

Google Stackdriver

●  Integrated monitoring, logging, diagnostics ●  Powerful data, analytics tools ●  Collaborations with PagerDuty, BMC, Splunk, others Works across Google Cloud Platform, Amazon Web Services ●  Open source agents, integration


67


68

Google Stackdriver’s areas of focus Monitoring

Logging

Platform, system, and application metrics Uptime/health checks Dashboards and alerts

Platform, system, and application logs Log search/view/filter Log-based metrics

Trace

Error Reporting

Latency reporting and sampling Per-URL latency and statistics

Error notifications Error dashboard

Debugger Debug applications

Google Cloud Deployment Manager

●  Infrastructure management service ●  Create a .yaml template describing your environment and use Deployment Manager to create resources ●  Provides repeatable deployments


69

Google Cloud Source Repositories

●  Fully-featured Git repositories hosted on Google Cloud Platform ●  Supports collaborative development of cloud apps ●  Includes integration with Stackdriver debugger


70

Agenda 1


2


3


4


5


71

Comparing Compute solutions Service

Compute Engine

Language support

Any

Any

Java, Python, Go and PHP

Java, Python, Node.js, Ruby, PHP, .NET core, and Go; or supply your own runtime

JavaScript

Service model

IaaS

Hybrid

PaaS

PaaS

Serverless

Use cases

General computing workloads

Container-based workloads

Web and mobile applications

Web and mobile applications, container-based workloads

Ephemeral functions responding to events

Kubernetes Engine

App Engine Standard

App Engine Flex

Cloud FunctionsBeta


72

Agenda 1


2


3


4


5


73


74

Quiz Name 3 robust networking services available to your applications on Google Cloud Platform. Name 3 Compute Engine pricing innovations.

True or False: Google Cloud Load Balancing lets you balance HTTP traffic across multiple Compute Engine regions.

Quiz Name 3 robust networking services available to your applications on Google Cloud Platform.

Cloud Virtual Network, Cloud Interconnect, Cloud DNS, Cloud Load Balancing, and Cloud CDN.

Name 3 Compute Engine pricing innovations.



75




Per-second billing, custom machine types, preemptible instances.



76




Per-second billing, custom machine types, preemptible instances.


True.


77


78

Resources Google Compute Engine https://cloud.google.com/compute/docs/ Google Cloud Platform VPC https://cloud.google.com/compute/docs/vpc/ Google Cloud Stackdriver https://cloud.google.com/stackdriver/docs/ Google Cloud Source Repositories gcloud tool guide https://cloud.google.com/source-repositories/docs/

Cloud OnBoard

Google Kubernetes Engine Google Cloud Platform Fundamentals: Core Infrastructure V3.0


Agenda 1

Introduction to Containers

2

Kubernetes

3

Google Kubernetes Engine

4

Quiz


80

Review: IaaS and PaaS

Toward managed infrastructure

Compute Engine

Kubernetes Engine

App Engine

IaaS

PaaS

Raw compute, storage, and network More granular control

Preset run-times Java, Go, PHP, Python... Focus is application logic

Pay for what you allocate More management overhead

Pay for what you use Less management overhead

Toward managed services


81

Containers mean virtualization inside the operating-system layer

●  Separates operating system from application code and dependencies ●  Isolates individual processes ●  Popular implementations include Docker

Container Code and libraries



Shared libraries Operating system Virtual Machine


82

Why use containers? Consistency

Loose coupling

Across

Between

development, testing, and

application and operating system

Simplified

production

layers

premises and

environments

Workload migration

Agility Agile development and operations

between oncloud environments


83


84

Agenda 1


2

Kubernetes

3


4

Quiz

Kubernetes is a container cluster orchestration system ●  Also known as k8s ●  Automates deployment, scaling, and operations for container clusters ●  Open source, based on Google’s experience over 10+ years ●  Built for a multi-cloud world ○  Public, private, hybrid


85

Kubernetes eases application management ●  Workload portability ○  You can run in many environments, across cloud providers. ○  Implementation is open and modular.

●  Rolling updates ○  You can upgrade applications without downtime.

●  Persistent storage ○  Details of how storage is provided are abstracted from how it is consumed. ©Google Inc. or its affiliates. All rights reserved. Do not distribute.

86

Kubernetes makes applications more elastic ●  Multi-zone clusters ○  Run a single cluster in multiple zones

●  Load balancing ○  External IP address routes traffic to correct port

●  Autoscaling ○  Automatically adapt to changes in workload


87


88

Agenda 1


2

Kubernetes

3


4

Quiz

Kubernetes Engine manages and runs containers ●  Fully managed cluster management and orchestration system for running containers ○  Based on Kubernetes ○  Uses Compute Engine instances and resources

●  Uses a declarative syntax to manage applications ○  Declare desired application configuration, Kubernetes Engine implements, manage


89

Why use Kubernetes Engine? ●  Decouples operational, development concerns ●  Manages and maintains ○  Logging, health management, monitoring ●  Easily update Kubernetes versions as they are released


90


91

Comparing available solutions Service

Compute Engine

Language support

Any

Any



JavaScript

Service model

IaaS

Hybrid

PaaS

PaaS

Serverless

Use cases






Kubernetes Engine

App Engine Standard

App Engine Flex

Cloud FunctionsBeta


92

Deploying Apps: Kubernetes Engine vs App Engine Kubernetes Engine

App Engine Standard

App Engine Flexible

Language support

Any

Java, Python, Go & PHP

Any

Service model

Hybrid

PaaS

PaaS



Primary use case Container-based workloads


93


94

Agenda 1


2

Kubernetes

3


4

Quiz

Quiz Name two reasons for deploying applications using containers.

True or False: Kubernetes lets you manage container clusters in multiple cloud providers. True or False: GCP provides a private, high-speed container image storage service for use with Kubernetes Engine.


95


Consistency across development, testing, and production environments; Simpler to migrate workloads; Loose coupling; Agility

True or False: Kubernetes lets you manage container clusters in multiple cloud providers. True or False: GCP provides a private, high-speed container image storage service for use with Kubernetes Engine.


96



True or False: Kubernetes lets you manage container clusters in multiple cloud providers.

True.

True or False: GCP provides a private, high-speed container image storage service for use with Kubernetes Engine.


97



True or False: Kubernetes lets you manage container clusters in multiple cloud providers.

True.

True or False: GCP provides a private, high-speed container image storage service for use with Kubernetes Engine.

True.


98

Resources Kubernetes Engine https://cloud.google.com/container-engine/docs/ Kubernetes Engine tutorials https://cloud.google.com/container-engine/docs/tutorials Kubernetes http://kubernetes.io/ Google Cloud Container Builder https://cloud.google.com/container-builder/docs/ Google Container Registry https://cloud.google.com/container-registry/docs/ ©Google Inc. or its affiliates. All rights reserved. Do not distribute.

Cloud OnBoard

Serverless on Google Cloud Platform Google Cloud Platform Fundamentals: Core Infrastructure V3.0


99

Agenda 1

Overview and Customer Stories

2

Google App Engine Standard Environment

3

Google App Engine Flexible Environment

4

Google Cloud Endpoints and Apigee Edge

5

Quiz

6 ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 101

IAAS and PAAS

Towards managed infrastructure

Compute Engine

App Engine

IaaS

PaaS

Raw compute, storage and network More granular control

Preset run-times Java, Go, PHP, Python... Focus is application logic

Pay for what you allocate More management overhead

Towards managed services

Pay for what you use Less management overhead

©Google Inc. or its affiliates. All rights reserved. Do not distribute. 102

App Engine is a PaaS for building scalable applications ●  App Engine makes deployment, maintenance, and scalability easy so you can focus on innovation ●  Especially suited for building scalable web applications and mobile backends


Snapchat

“App Engine enabled us to focus on developing the application. We wouldn’t have gotten here without the ease of development that App Engine gave us.” Bobby Murphy, CTO

Snapchat sends

700 million

photos and videos each day

Google App Engine scaled seamlessly during growth to

Small team is able to innovate quickly and expand

millions of users

globally


Google Cloud Functions Beta ●  Create single-purpose functions that respond to events without a server or runtime ○  Event examples: New instance created, file added to Cloud Storage ●  Written in Javascript, execute in managed Node.js environment on Google Cloud Platform ●  Perfect for IoT, microservices. ●  Mobile first with with Firebase integration


Agenda 1


2


3


4


5

Quiz


App Engine standard environment ●  Easily deploy your applications ●  Autoscale workloads to meet demand ●  Economical ○  Free daily quota ○  Usage based pricing ●  SDKs for development, testing and deployment


App Engine standard environment: Requirements ●  Specific versions of Java, Python, PHP, and Go are supported ●  Your application must conform to sandbox constraints: ○  No writing to local file system ○  All requests time out at 60 seconds ○  Third-party software installations are limited


Example App Engine standard workflow: Web applications 3

Develop & test the web application locally

1

App Engine automatically scales & reliably serves your web application

Project

Memcache

App Engine

Task queues

App Servers

Use the SDK to deploy to App Engine

2

App Engine can access a variety of services using dedicated APIs

Application instances

Scheduled tasks


Search


Logs ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 109

Agenda 1


2


3


4


5

Quiz

6

Quiz ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 110

App Engine flexible environment ●  ●  ●  ●  ● 

Build and deploy containerized apps with a click No sandbox constraints Can access App Engine resources Standard runtimes: Python, Java, Go, Node.js Custom runtime support: Any language that supports HTTP requests ○  Package your runtime as a Dockerfile


112

Comparing the App Engine environments Standard Environment

Flexible Environment

Instance startup

Milliseconds

Minutes

SSH access

No

Yes (although not by default)

Scaling

Manual, basic, automatic

Manual, automatic

Write to local disk

No

Yes (but writes are ephemeral)

Support for 3rd-party binaries

No

Yes

Network access

Via App Engine services

Yes

Pricing model

After free daily use, pay per instance

Pay for resource allocation per hour; no

class, with automatic shutdown

automatic shutdown


Deploying Apps: App Engine vs Kubernetes Kubernetes Engine

App Engine Standard

App Engine Flexible

Language support

Any

Java, Python, Go & PHP

Any

Service model

Hybrid

PaaS

PaaS



Primary use case Container-based workloads


Comparing available solutions Service

Compute Engine

Language support

Any

Any



JavaScript

Service model

IaaS

Hybrid

PaaS

PaaS

Serverless

Use cases






Kubernetes Engine

App Engine Standard

App Engine Flex

Cloud FunctionsBeta


Agenda 1


2


3


4


5

Quiz

6


Cloud Endpoints helps you create and maintain APIs ●  Distributed API management through an API console ●  Expose your API using a RESTful interface ●  Control access and validate calls with JSON Web Tokens and Google API keys ○  Identify web, mobile users with Auth0 and Firebase Authentication

●  Generate client libraries


Cloud Endpoints: Supported platforms ●  Supports App Engine standard or flexible environment, Compute Engine, Kubernetes Engine ●  Use Java or Python open-source Frameworks or any other framework and language ●  Supports iOS, Android, and JavaScript clients


Apigee Edge helps you secure and monetize APIs ●  A platform for making APIs available to your customers and partners ●  Contains analytics, monetization, and a developer portal


Agenda 1


2


3


4


5

Quiz


Quiz Name 3 advantages of using the App Engine flexible environment over App Engine standard.

What is the difference between Cloud Endpoints and Apigee Edge?



The flexible environment allows SSH access, allows disk writes, and supports third-party binaries (also allows stack customization and background processes).




The flexible environment allows SSH access, allows disk writes, and supports third-party binaries (also allows stack customization and background processes).


Cloud Endpoints helps you create and maintain APIs; Apigee Edge helps you secure and monetize APIs.


Resources Google App Engine https://cloud.google.com/appengine/docs/ Google App Engine Flexible Environment https://cloud.google.com/appengine/docs/flexible/ Google App Engine Standard Environment https://cloud.google.com/appengine/docs/standard/ Google Cloud Endpoints https://cloud.google.com/endpoints/docs/ Apigee Edge http://docs.apigee.com/api-services/content/what-apigee-edge


Cloud OnBoard

Google Cloud Platform Storage Options Google Cloud Platform Fundamentals: Core Infrastructure V3.0


Agenda 1

Google Cloud Storage

2

Google Cloud Datastore and Google Cloud Bigtable

3

Google Cloud SQL and Google Cloud Spanner

4

Comparing Storage Options

5


Cloud Storage is binary large-object storage ●  High performance, internet-scale ●  Simple administration ○  Does not require capacity management

●  Data encryption at rest ●  Data encryption in transit by default from Google to endpoint ●  Online and offline import services are available ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 126

Your Cloud Storage files are organized into buckets

Bucket attributes:

Bucket contents

●  Globally unique name

●  Files (in a flat namespace)

●  Storage class

●  Access Control Lists

●  Location ○  Region or multi-region ●  IAM policies or Access Control Lists ●  Object versioning setting ●  Object lifecycle management rules


Choosing among Cloud Storage classes Multi-regional

Regional

Nearline

Coldline

Intended for data that is...

Most frequently accessed

Accessed frequently within a region

Accessed less than once a month

Accessed less than once a year

Availability SLA

99.95%

99.90%

99.00%

99.00%

Access APIs

Consistent APIs

Access time

Millisecond access

Storage price

Price per GB stored per month

Retrieval price Total price per GB transferred

Use cases

Content storage and delivery

In-region analytics, transcoding

Long-tail content, backups

Archiving, disaster recovery


TERABYTES

of data from more than

200 PER GENOME RAW

GIGABYTES

1,300 WHOLE

MSSNG project could easily surpass a

GENOMES

PETABYTE

to Google Cloud Storage

10,000

Making MSSNG world’s largest single repository of autism-related DNA sequencing data

PEOPL E

WHOLE GENOMES FROM

OF DATA

UPLOADED

100

UP TO

“Thanks to Google Cloud Platform and the Google Genomics team, the greatest minds in science from around the world will be able to study trillions of data points in one single database.”

Agenda 1


2


3


4


5


Cloud Datastore is a horizontally scalable NoSQL DB ●  NoSQL designed for application backends ●  Fully managed ○  Uses a distributed architecture to automatically manage scaling

●  Built-in redundancy ●  Supports ACID transactions


Google Cloud Datastore: benefits ●  Schemaless access ○  No need to think about underlying data structure

●  Local development tools ●  Includes a free daily quota ●  Access from anywhere through a RESTful interface


Mobile Client Geolocation Map cache

Niantic Server Logic (Kubernetes deployment, 10K+ CPU cores) Player Data Spatial Service

Niantic Dataflow

NoSQL database (Google Cloud Datastore)


Cloud Datastore Transactions Per Second

1X

Target Traffic

5X Worst Case Estimate

50X Actual Traffic Original Launch Target

Estimated Worst Case

Actual Traffic


Cloud Bigtable is managed NoSQL ●  Fully managed NoSQL, wide-column database service for terabyte applications ●  Integrated ○  Accessed using HBase API ○  Native compatibility with big data, Hadoop ecosystems


Why choose Cloud Bigtable? ●  Replicated storage ●  Data encryption in-flight and at rest ●  Role-based ACLs ●  Drives major applications such as Google Analytics and Gmail


Bigtable Access Patterns Application API Data can be read from and written to Cloud Bigtable through a data service layer like Managed VMs, the HBase REST Server, or a Java Server using the HBase client. Typically this will be to serve data to applications, dashboards, and data services.

Streaming Data can be streamed in (written event by event) through a variety of popular stream processing frameworks like Cloud Dataflow Streaming, Spark Streaming, and Storm.

Batch Processing

Cloud Bigtable

Data can be read from and written to Cloud Bigtable through batch processes like Hadoop MapReduce, Dataflow, or Spark. Often, summarized or newly calculated data is written back to Cloud Bigtable or to a downstream database.


Overview:

Challenges:

Data to process: Data in the Consolidated Audit Trail (CAT). A data repository of all equities and options orders, quotes, and events

How to process the CAT and organize 100 billion market events into an “order lifecycle” in a 4 hour window Store 6 years (~30PB) of data

Cloud Bigtable to process and run queries and tolerate volume increases

6 BILLION MARKET EVENTS WRITTEN PER HOUR

1.7 GIGs PER SECOND

6 TBs PER HOUR

10 BN

1.7 GIGABYTES

WRITTEN PER HOUR BURSTS

10 TERABYTES

PER SECOND PER HOUR

Google confidential │ Do not distribute

Datastore vs Bigtable: use cases Cloud

Bigtable

Datastore Type

Best for

Use cases

NoSQL

NoSQL

document

wide column

Getting started, App Engine

“Flat” data, Heavy read/

applications

Getting started, App Engine applications

Cloud

Cloud

Cloud

Storage

SQL

Spanner

Blobstore

BigQuery

Relational

Relational

Relational

SQL for OLTP

SQL for OLTP

SQL for OLAP

Structured and unstructured

Web frameworks,

Large-scale database

Interactive querying, offline

write, events, analytical data

binary or object data

existing applications

applications (> ~2 TB)

analytics

AdTech, Financial and IoT data

Images, large media files, backups

User credentials, customer orders

Whenever high I/O, global consistency is needed

Data warehousing


Agenda 1


2


3


4


5


Cloud SQL is a managed RDBMS ●  Offers MySQL and PostgreSQLBeta databases as a service ●  Automatic replication ●  Managed backups ●  Vertical scaling (read and write) ●  Horizontal scaling (read) ●  Google security


Cloud Spanner is a horizontally scalable RDBMS Cloud Spanner supports: ●  ●  ●  ● 

Automatic replication Strong global consistency Managed instances with high availability SQL (ANSI 2011 with extensions)


Agenda 1


2


3


4


5


Comparing storage options: technical details Cloud

Bigtable

Datastore Type

NoSQL

NoSQL

document

wide column

Transactions

Yes

Single-row

Complex queries

No

Capacity Unit size

Cloud

Cloud

Cloud

Storage

SQL

Spanner

Relational

Relational

Relational

SQL for OLTP

SQL for OLTP

SQL for OLAP

No

Yes

Yes

No

No

No

Yes

Yes

Yes

Terabytes+

Petabytes+

Petabytes+

500 GB

Petabytes

Petabytes+

1 MB/entity

~10 MB/cell

5 TB/object

Determined

10,240 MiB/

10 MB/row

by DB engine

row

~100 MB/row

Blobstore

BigQuery


Comparing storage options: use cases Cloud

Bigtable

Datastore Type

Best for

Use cases

NoSQL

NoSQL

document

wide column

Getting started, App Engine

“Flat” data, Heavy read/

applications

Getting started, App Engine applications

Cloud

Cloud

Cloud

Storage

SQL

Spanner

Blobstore

BigQuery

Relational

Relational

Relational

SQL for OLTP

SQL for OLTP

SQL for OLAP

Structured and unstructured

Web frameworks,

Large-scale database

Interactive querying, offline

write, events, analytical data

binary or object data

existing applications

applications (> ~2 TB)

analytics

AdTech, Financial and IoT data

Images, large media files, backups

User credentials, customer orders

Whenever high I/O, global consistency is needed

Data warehousing


Agenda 1


2


3


4


5


Quiz Your application transcodes large video files. Which storage service should you consider first?

You stream huge amounts of data from devices with sensors. Which storage service should you consider first?









Google Cloud Bigtable


Resources Overview of Cloud Storage https://cloud.google.com/storage/ Getting started with Google Cloud SQL https://cloud.google.com/sql/docs/quickstart Cloud Bigtable https://cloud.google.com/stackdriver/docs/ Cloud Spanner https://cloud.google.com/spanner/docs/ Cloud Datastore https://cloud.google.com/datastore/docs/


Cloud OnBoard

Big Data and Machine Learning Google Cloud Platform Fundamentals: Core Infrastructure V3.0


Agenda

1

Google Cloud Big Data Platform

2

Google Cloud Machine Learning Platform

3

Quiz


Google Cloud’s big data services are fully managed and scalable

Cloud BigQuery

Cloud Dataflow Cloud Dataproc Managed Hadoop MapReduce, Spark, Pig, and Hive service

Stream & batch processing; unified and simplified pipelines.

Cloud DataPrep

Cloud Pub/Sub

Analytics database; data streaming at 100,000 rows/second

Interactive data exploration with Jupyter Notebooks

Scalable and flexible enterprise messaging

Prepare, clean your data and build your Dataflow pipeline

Cloud Datalab

Cloud Data Studio A collaborative BigQuery visualization experience ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 153

Cloud Dataproc is managed Hadoop ●  Fast, easy, managed way to run Hadoop and Spark/Hive/Pig on Google Cloud Platform ●  Create clusters in 90 seconds or less on average. ●  Scale clusters up and down even when jobs are running.


Typical Spark/Hadoop clusters... Scale

Obtain servers

Configure servers

Install OSS

Reconfigure

Configure OSS

… vs Cloud Dataproc clusters

Create cluster (t+0)

Optimize OSS

Debug OSS

Process data

Scale anytime

Configure cluster (t+20 sec)

Process data (t+90 sec) ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 155

Why use Cloud Dataproc? ●  Easily migrate on-premises Hadoop jobs to the cloud. ●  Quickly analyze data (like log data) stored in Cloud Storage; create a cluster in 90 seconds or less on average, and then delete it immediately. ●  Use Spark/Spark SQL to quickly perform data mining and analysis. ●  Use Spark Machine Learning Libraries (MLlib) to run classification algorithms.


Cloud Dataflow offers managed data pipelines ●  Processes data using Compute Engine instances. ○  Clusters are sized for you ○  Automated scaling, no instance provisioning required ●  Write code once and get batch and streaming. ○  Transform-based programming model


Dataflow pipelines flow data from a source through transforms Source BigQuery

Transforms

Sink

Cloud Storage


Not great at writing data pipelines? Use Cloud Dataprep! ●  Data Preparation and Data cleansing using an easy to use visual interface ●  An intelligent cloud data service to visually explore, clean, and prepare data for analysis


Why use Cloud Dataflow? ●  ETL (extract/transform/load) pipelines to move, filter, enrich, shape data ●  Data analysis: batch computation or continuous computation using streaming ●  Orchestration: create pipelines that coordinate services, including external services ●  Integrates with GCP services like Cloud Storage, Cloud Pub/Sub, BigQuery, and Bigtable ○  Open source Java and Python SDKs


BigQuery is a fully managed data warehouse ●  Provides near real-time interactive analysis of massive datasets (hundreds of TBs) ●  Query using SQL syntax (SQL 2011) ●  No cluster maintenance is required.


BigQuery runs on Google’s high-performance infrastructure ●  Compute and storage are separated with a terabit network in between ●  You only pay for storage and processing used ●  Automatic discount for long-term data storage


Cloud Pub/Sub is scalable, reliable messaging ●  Supports many-to-many asynchronous messaging ○  Application components make push/pull subscriptions to topics ●  Includes support for offline consumers ●  Based on proven Google technologies ●  Integrates with Cloud Dataflow for data processing pipelines


Why use Cloud Pub/Sub? ●  Building block for data ingestion in Dataflow, Internet of Things (IoT), Marketing Analytics ●  Foundation for Dataflow streaming ●  Push notifications for cloud-based applications ●  Connect applications across Google Cloud Platform (push/pull between Compute Engine and App Engine)


Cloud Datalab offers interactive data exploration ●  Interactive tool for large-scale data exploration, transformation, analysis, and visualization ●  Integrated, open source ○  Runs on Compute Engine ○  Built on Jupyter (formerly IPython)


Why use Cloud Datalab? ●  Create and manage code, documentation, results, and visualizations in intuitive notebook format. ○  Use Google Charts or matplotlib for easy visualizations.

●  Analyze data in BigQuery, Compute Engine, and Cloud Storage using Python, SQL, and JavaScript. ●  Easily deploy models to BigQuery. ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 166

Cloud Data Studio offers BigQuery visualization ●  Web-based data visualization ●  Direct access to BigQuery data (and many other sources) ●  Collaborative editing experience (using Drive model) ●  Zero-install ●  Free



Agenda

1


2


3

Quiz


Machine Learning APIs enable apps that see, hear, and understand


Cloud Machine Learning Platform Open source tool to build and run neural network models ● 

Wide platform support: CPU or GPU; mobile, server, or cloud

Fully managed machine learning service

Cloud ML

● 

Familiar notebook-based developer experience

● 

Optimized for Google infrastructure; integrates with BigQuery and Cloud Storage

Pre-trained machine learning models built by Google ● 

Speech: Stream results in real time, detects 80 languages

● 

Vision: Identify objects, landmarks, text, and content

● 

Translate: Language translation including detection

● 

Natural language: Structure, meaning of text

Machine Learning APIs ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 172

Inject AI into your business with the Google Cloud Machine Learning platform For structured data

For unstructured data

Classification and regression

Image and video analytics

Recommendation Text analytics Anomaly detection ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 173

Cloud Vision API ●  Analyze images with a simple REST API ○  Logo detection, label detection, etc ●  With the Cloud Vision API, you can: ○  ○  ○  ○ 

Gain insight from images Detect inappropriate content Analyze sentiment Extract text


Cloud Speech API ●  Recognizes over 80 languages and variants ●  Can return text in real time ●  Highly accurate, even in noisy environments ●  Access from any device ●  Powered by Google’s machine learning ●  Used by Google Assistant


Cloud Natural Language API ●  Uses machine learning models to reveal structure and meaning of text. ●  Extract information about items mentioned in text documents, news articles, and blog posts. ●  Analyze text uploaded in request or integrate with Cloud Storage.


Cloud Translation API ●  Translate arbitrary strings between thousands of language pairs ●  Programmatically detect a document’s language ●  Support for dozens of languages


Cloud Video Intelligence API ●  Annotate the contents of videos ●  Detect scene changes ●  Flag inappropriate content ●  Support for a variety of video formats


Dialogflow API (previously called API.AI) BETA ●  Build chatbots capable of natural and rich interactions between users and your business. ●  Uses ML to recognize the intent & context of a conversation, allowing your interface to provide an efficient and accurate response. ●  Use cases:

○  Customer Service ○  Commerce ○  IOT Devices ○  Enterprise Productivity ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 179

Cloud Job Discovery API PRIVATE BETA ●  Power your job site with Machine Learning ●  Improve job engagement and candidate conversion.


Agenda

1


2


3

Quiz


Quiz When would you use Cloud Dataproc?

Name two use cases for Cloud Dataflow.

Name three use cases for the Google machine learning platform.



You can use it to migrate on-premises Hadoop jobs to the cloud. You can also use it for data mining and analysis of cloud-based data.







ETL, orchestration






ETL, orchestration


Fraud detection, sentiment analysis, content personalization


Resources

Google Big Data Platform https://cloud.google.com/products/big-data/ Google Machine Learning Platform https://cloud.google.com/products/machinelearning/


Cloud OnBoard

Igniting Your Career with Google Cloud Training & Certification


Google Cloud training - A roadmap for learners Introduction to GCP

Next Level Training

GCP Fundamentals - Core Infrastructure -

IaaS Architecting with GCP: Core Infrastructure Architecting: Design & Process Develop & Deploy Windows Applications on GCP

GCP Fundamentals - Data and ML 7 Qwiklabs - GCP Essentials GCP for AWS Professionals G Suite Admin Fundamentals

Advanced Advanced Training

8 Qwiklabs - Cloud Architecture

Getting Started with Kubernetes and Google Container Engine Coming Soon Advanced Data Engineering Advanced Machine Learning

Data and Machine Learning Data Engineering on GCP From Data to Insights with GCP 8 Qwiklabs - Data Engineering 8 Qwiklabs - Scientific Data Processing PaaS Developing Applications with GCP 7 Qwiklabs - Developing Applications ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 188

Find more training courses

cloud.google.com/training ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 189

Google Cloud Self Paced Qwiklabs

google.qwiklabs.com ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 190

After Cloud OnBoard

Continue Learning with Qwiklabs Receive a follow up email 1 day after the event

Create Qwiklabs account

● 

Google Chrome is recommended

● 

Use the email address with which you registered for the class

● 

You must confirm your account before logging in

Open your email and confirm account

Open incognito browser tab

Return to Qwiklabs and log in


Make Certification Your Goal

Getting your Google cloud certification shows the world that you can design, develop, manage and administer application infrastructure and data solutions on Google Cloud technology. Gain industry recognition Validate your technical expertise Take your career to a level beyond your wildest dreams

cloud.google.com/certification/ ©Google Inc. or its affiliates. All rights reserved. Do not distribute. 192

Join the GDG Cloud Bangkok community!

or bit.ly/gdgcloudbkk


1 2 3 4 5 6

7

https://cloud.google.com/training/

8

9 10 11 12 13 14 15 16 17 18