Teaching Network Infrastructure, Database Concepts, and Web-Based Application Development using Cloud Services: An Introduction David Rilett John Russo
“The most radical and transformative of inventions are those that empower others to unleash their creativity - to pursue their dreams.” Jeff Bezos, Letter to Shareholders, 2012
Forbes #12
Foundational Level June 10,2014 AWS Summits 2014
Getting Started with AWS Session – Technical Audience Level: Foundational • Security • Credential management • Provisioning • Managing • Amazon EC2 • Amazon EBS • Amazon S3 • Elastic Load Balancing • Auto Scaling • Amazon CloudWatch • Amazon RDS. • Virtual Private Network (VPN) connection between your datacenter and your VPC https://aws.amazon.com/aws-summit-2014/amsterdam/
Intermediate: HA Architecting Highly Available Applications on AWS Audience Level: Intermediate • High availability options for workloads • Architectural design patterns for automating fault tolerance • Architectural design patterns for high availability • Application security
Intermediate: Big Data Store, Manage and Analyze Big Data in the Cloud Audience Level: Intermediate • Big Data scenarios that merge structured and unstructured data into a data warehouse for analysis • Gather and store large quantities of data • Amazon Elastic Map Reduce to process raw data using the massive scale of Amazon EC2 • Amazon Redshift to analyze data and make it accessible to business users. AWS Data Pipeline and Amazon Kinesis.
Advanced:DevOps Taking AWS Operations to the Next Level Audience Level: Advanced • Change management of infrastructure and applications • Configuration management of your infrastructure • AWS CloudFormation • Chef • AWS SDKs to automate the provisioning and configuration of your AWS infrastructure resources and application • Amazon SimpleWorkflow to create, build, and deploy an end-toend automation system to enable hands off failure recovery • Puppet
Use Case: "VPC with a Single Public Subnet Only" Host a simple, public-facing website Host a basic web application, such as a blog or simple website Secure the website by creating security group rules which allow the webserver to respond to inbound HTTP and SSL requests from the Internet while simultaneously prohibiting the webserver from initiating outbound connections to the Internet.
http://docs.aws.amazon.com/AmazonVPC/latest /UserGuide/VPC_Scenario1.html
Use Case: "VPC with Public and Private Subnets" • Multi-tier web applications that strictly enforce access and security restrictions between your webservers, application servers, and databases. • Launch webservers in a publicly accessible subnet • Launch application servers and databases in non-publically accessible subnets. • Application servers and databases can’t be directly accessed from the Internet, but access the Internet via a NAT instance to download patches, for example. • Control access between the servers and subnets using inbound and outbound packet filtering by network access control lists and security groups. http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario2.html
"VPC with Public and Private Subnets" Extended for Security: Bastion Server • In this case there are multiple instances in the private subnet. • The NAT blocks SSH to private subnet. • A Bastion or jump server can SSH to private network. • Bastion Server access is limited by ACL.
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo-vpc-bh.html
Use Case: "VPC with Public and Private Subnets and Hardware VPN Access" • Host scalable web applications in the AWS cloud that are connected to your local datacenter. • Create a VPC where instances in subnet one are webservers to communicate with the Internet. • Subnet two has application servers to communicate with databases on your corporate network with IPsec VPN connection between VPC and your local network to secure communication between cloud application servers and local. • Webservers and application servers in VPC implement Amazon EC2 elasticity and Auto Scaling.
Use Case: "VPC with a Private Subnet Only and Hardware VPN Access" • Move local applications to the cloud, launch additional webservers, or add more compute capacity to your network by connecting your VPC to your local network. • Because your VPC can be hosted behind a local firewall, IT resources can move into the cloud without changing how users access them.
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Scenario3.html
Use Case: Disaster Recovery • Backup mission-critical data from your datacenter to a small number of Amazon EC2 instances with Amazon Elastic Block Store (EBS) volumes • Import your virtual machine images to Amazon EC2. • In the event of a disaster in your own datacenter, quickly launch replacement compute capacity in AWS to ensure business continuity. • When the disaster is over, send mission-critical data back to your datacenter and terminate the Amazon EC2 instances. • All the benefits of a disaster recovery site at a fraction of the normal cost with no tapes etc. http://aws.amazon.com/disaster-recovery/
NYC AWS Session “Levels” July 10, 2014
Introductory – “I’m new to this.” Gives an overview of the a service or feature with the assumption that attendees are new to the topic. These sessions highlight basic use cases, features, functions, and benefits. Advanced – “I get it and want to learn more.” Advanced sessions will dive deeper into the topic. Code may be shared but will not be the primary focus of the session.
Expert – “I live and breathe this stuff.” Much of these sessions are code-based, covering best practices, advanced tricks, and future developments.
Introductory Amazon Elastic Compute Cloud
Amazon EC2
AMI
Elastic IP
Amazon Simple Storage Service
Amazon S3
Amazon SimpleDB
Amazon SimpleDB
Intermediate Amazon Elastic Block Store
Amazon ElastiCache
Amazon Glacier
AWS Elastic Beanstalk
AWS Identity and Access Management
Advanced Amazon Route 53
Amazon Route 53
Elastic Load Balancing
Amazon Simple Queue Service
route table
AWS CloudFormation Amazon CloudWatch
Amazon Relational Amazon DynamoDB Database Service
Expert Amazon Virtual Private Cloud
Amazon VPC
router
Auto Scaling
Internet gateway
customer gateway
Amazon Redshift
virtual private gateway
VPN connection
AWS OpsWorks
Visual Studio 2013 AWS Plugin and Git AWS Explorer
VS2013
Git Team Explorer
AWS Solution Calculator (AWS Plugin)
Introductory Lesson Plans • • • • • •
Use free private GitHub accounts to teach programming courses. Create EC2 instances for each student to learn any OS. Create individual EC2 instances for LAMP stack (MySQL dbase-backed web site). Add NoMachine or Remote Desktop above to work in GUI. Use Oracle for database concepts on individual EC2. Configure course instance for programming specific subject (i.e.BioPerl for bioinformatics). • Configure course instance with AMI for specific subject (i.e. CloudBioLinux). • Scripts traded from GitHub. • Use micro-instances and free tier to limit costs.
Intermediate Lesson Plans • Combine Github and AWS instance for group project. • Use CloudFormation to configure “VPC with a Single Public Subnet Only” manually on VPC. • Use Cisco Cloud Services Router AMI to teach IOS XE command line etc. • Build "VPC with Public and Private Subnets" from template, make changes to NAT and security settings and test. Add and configure subdomain and submit CloudFormation script and test report. • Scripts traded from GitHub. • Use AWS educator grants to defray cost. AWS grant.
Advanced Lesson Plans • Build "VPC with Public and Private Subnets" and add and Bastion server and submit CloudFormation script and test report. • Send any CloudFormation script by GitHub have students build and test and submit revised script and test report. • Students build any VPC project, reverse engineer to script and submit on Git. • Pull any student script or branch into your Git repo on Visual Studio 2013 or Eclipse, build to test and deploy to test. • Create advanced database project and test (i.e. web site with stored procedures). • Scripts traded from GitHub. • Use AWS educator grants to defray cost. AWS grant.
Expert Lesson Plans • Build "VPC with Public and Private Subnets" and add load balancing to front end. Submit reverse engineered script and report. (HA) • Build "VPC with Public and Private Subnets" and add cache to backend end. Submit reverse engineered script and test report. • Build "VPC with Public and Private Subnets" and add VPN to local network and block SSH on NAT. Submit reverse engineered script and report. • Replace CF “user data” configuration with CHEF scripting on above. • Use CF “user data” to install Puppet and configure and test Puppet Master reconfiguration of servers. • Build "VPC with Public and Private Subnets" and add load balancing to front end. Submit reverse engineered script and report. • Build a multi-tiered web site configured by CHEF and Puppet that has. Submit reverse engineered script and report. • Build "VPC with Public and Private Subnets" and add auto scaling. Submit reverse engineered script and test report. • Scripts traded from GitHub. • Use AWS educator grants to defray cost. AWS grant.
Collaboration GitHub and CloudFormation are tremendous innovations for collaboration. As engineering schools trend more towards interdisciplinary projects and external collaborations, tools such as these must play a central role in projects. For networking, the CloudFormation JSON scripts become a project deliverable, particularly for external projects. Once a custom template is created, nearly anyone can instantiate the network and its components using their own AWS credentials and account. These scripts can be delivered and updated on Git. The skills needed for creating and administrating these projects are Git (strong learning curve for developers), JSON (easy for those familiar with Javascript), Ruby, networking fundamentals (TCP stack, CIDR rules, NAT, firewalls, port filtering, access control lists), web security, application configuration, operating systems (Ubuntu, CentOS, Windows), databases (Oracle, NoSQL, MySQL etc.), bash shell and powershell, AWS, VPC, and CloudFormation. There are also tools for big data analytics such as Redshift and Map/Reduce but these may be more relevant to large, commercial projects, unless big data is the project. There are other systems such as Microsoft Azure and OpenStack.
Git and GitHub • “Git is a distributed revision control and source code management (SCM) system with an emphasis on speed. Git was initially designed and developed by Linus Torvalds for Linux kernel development in 2005.” Wikipedia • Git keeps copies of repositories locally and keeps history with pointers to a tree-like structure. • The GitHub site combines social networking with Git. Free access is public, paid access can be private and GitHub provides free private accounts for educators for lab work. Repos can be cloned, branched, pushed, pulled and merged and GitHub provides the communication backbone for pull requests and general collaboration. “Throughout my entire college career, all programming assignments have had to be submitted through BlackBoard. To be as diplomatic as possible, BlackBoard does a lot of things to help students & teachers. Unfortunately it does not do a single one of them well. The normal way to submit coding assignments is to have homework assigned through BlackBoard and to have to code up the solutions and then zip them up and submit them using the web interface. The best part about using GitHub for homework submissions is that it is incredibly easy to give feedback and help the students through learning the material.” Josh Davis, TA
Git and GitHub
Git includes “diff” for comparing versions on GitHub, in local GUI and in bash shell.
Cloud Formation Template CloudFormation scripts and automates cloud deployment of servers, databases, firewalls, routers, applications and networks generally. Existing AWS VPC’s can be captured as scripts (reverse engineered) The process uses JSON templates. • 1. 2. 3. 4. 5. 6. • •
There are six top level JSON objects: AWS Template Format Version Description Parameters Mappings Resources Outputs A top-level object for the Resources object is required. Must declare at least one resource.
http://aws.amazon.com/cloudformation/aws-cloudformation-templates/
CloudFormation (JSON)
JSON (Used much like XML, perhaps faster) • JSON is built on the following two structures: • A collection of name/value pairs. In various languages, this is realized as an object, record, struct, dictionary, hash table, keyed list, or associative array. • An ordered list of values. In most languages, this is realized as an array, vector, list, or sequence. • These are universal data structures. Virtually all modern programming languages support them in one form or another. • In JSON, they take on these forms: • An object is an unordered set of name/value pairs. An object begins with { (left brace) and ends with } (right brace). Each name is followed by : (colon) and the name/value pairs are separated by , (comma). • An array is an ordered collection of values. An array begins with [ (left bracket) and ends with ] (right bracket). Values are separated by , (comma). • A value can be a string in double quotes, or a number, or true or false or null, or an object or an array. These structures can be nested. • A string is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes. A character is represented as a single character string. A string is very much like a C or Java string. • A number is very much like a C or Java number, except no octal and hexadecimal formats. • JSON is a lightweight, plain-text alternative to XML used for client-server communication in web apps. source: json.org
CloudFormation + Chef • Repeatable builds • Known configuration • Source control of infrastructure
"files" : { "/etc/chef/solo.rb" : { "content" : { "Fn::Join" : ["\n", [ "log_level :info", "log_location STDOUT", "file_cache_path \"/var/chef-solo\"", "cookbook_path \"/var/chef-solo/cookbooks\"", "json_attribs \"/etc/chef/node.json\"", "recipe_url \"https://s3.amazonaws.com/cloudformation-examples/wordpress.tar.gz\"" ]] },
In “UserData” section of the JSON template after yum and pull on Ruby gems etc.”
Ruby • Interpretive programming: No compilation ; you edit and send your program to the interpreter. • Dynamic programming: Almost everything done at runtime. Types and class and method definitions are determined at runtime. • Write programs within programs and execute them. • Familiar syntax: Java, Perl, Python, C/C++ "ish" Simple factorial function: def factorial(n) if n == 0 return 1 else return n * factorial(n-1) end end • Iterators: loop abstraction is built in, blocks can be attached to method call which can call back the block. Array with each method example: ary = [1,2,3,4,5] ary.each do |i| puts i*2 end # prints 2,4,6,8,10 for each line • More: ...
LearnToProgram Poignant Guide to Ruby
Puppet Puppet is an open source, configuration management framework that knows how to configure all machines to a predetermined state automatically once provided the required “manifest” set. Puppet can automate the entire infrastructure lifecycle: discovery, provisioning, operating system and application configuration, management and reporting. • Use to configure multiple machines similarly. • Use to set up infrastructure that requires dynamic scaling-up and scalingdown of machines to a pre-built configuration. • Use to control all configured machines so a centralized change gets propagated to all automatically according to type etc.
Puppet Labs Learn
Database JP
new!
AWS Simple Icons: Database
Database
JP
Amazon DynamoDB
DynamoDB
table
item
items
attribute
attributes
global secondary index new!
Amazon Relational Database Service
Amazon RDS
RDS DB instance
RDS DB RDS DB instance standby instance read (Multi-AZ) replica
MySQL DB instance new!
AWS Simple Icons: Database
Oracle DB instance new!
MS SQL instance new!
Database
PostgreSQL instance
SQL master
SQL slave
PIOP
JP
Database
JP
Sources: Cisco Cloud Services Router (IOS XE command line etc.) Josh Davis on GitHub
ComS 342: Principles of Programming languages at Iowa State during Spring 2014. GitHub Educator Discount
Github for Windows
JSON
Ruby Intro
David Rilett
[email protected] John Russo
[email protected]
copyright 2014 David Rilett
