13 procfs, sysfs are not containerized. /proc/meminfo will show the free memory of the host, not the container. Legacy c
Docker At Scale Pitfalls And How To Debug Them © 2016 Mesosphere, Inc. All Rights Reserved.
1
Jan Schlicht Distributed Systems Engineer Apache Mesos contributor Marathon contributor
[email protected]
© 2016 Mesosphere, Inc. All Rights Reserved.
2
DEFINITIONS CONTAINER
Runs a task isolated from other running tasks CONTAINER IMAGE
Template of the container environment CONTAINERIZER
Mechanism in Mesos to run containers
© 2016 Mesosphere, Inc. All Rights Reserved.
3
WHY CONTAINERS?
Less overhead compared to VMs Easy deployment Easy packaging Smaller disk footprint
© 2016 Mesosphere, Inc. All Rights Reserved.
4
COMPLEXITY AT SCALE
Mesos moves fast (and tries not to break things) Docker moves fast (and tries not to break things) The Linux kernel moves fast (and tries not to break things) Bringing it all together… at scale… might break things
© 2016 Mesosphere, Inc. All Rights Reserved.
5
CONTAINER DEPLOYMENT
Docker client
Container images
Containers
Mesos agent
Docker images
Containers
Docker daemon
© 2016 Mesosphere, Inc. All Rights Reserved.
6
DOCKER vs MESOS CONTAINERIZER
DOCKER CONTAINERIZER
MESOS CONTAINERIZER
Provides many useful Docker features
Runs containers without daemon
Complexity by running Docker commands
Support for new features
Docker daemon can hang
Container image support rather new
No feature parity (yet)
© 2016 Mesosphere, Inc. All Rights Reserved.
7
CONTAINERS AT SCALE
© 2016 Mesosphere, Inc. All Rights Reserved.
8
HANGING DOCKER DAEMON
Can have many causes Beware when scaling tasks May result in an agent not shutting down To debug, test if the Docker daemon is still responsible Mesos containerizer not affected
© 2016 Mesosphere, Inc. All Rights Reserved.
9
CONTAINER IMAGES Container Container Layer
Docker uses layers to store images efficiently and to share layers Instead of copying the whole filesystem, union mounts are used Having many layers can result in performance drawbacks
Container Image
Layer 3
Layer 2
Layer 1
© 2016 Mesosphere, Inc. All Rights Reserved.
10
INODE EXHAUSTION
Running out of disk space when there’s plenty left Check if there are Inodes left (df -i) Use fewer layers or squash images XFS dynamically allocates Inodes Kernel >= 4.0? Use Docker’s overlay2 driver, Mesos’ overlay backend
© 2016 Mesosphere, Inc. All Rights Reserved.
11
SECURITY
Containers reduce the attack surface of applications But the attack surface is larger than VMs procfs, sysfs are not containerized
© 2016 Mesosphere, Inc. All Rights Reserved.
12
MEMORY IN CONTAINERS
procfs, sysfs are not containerized /proc/meminfo will show the free memory of the host, not the container Legacy code may not know this Always specify the heap size if JVM is run inside a container Good practice: Disable swap
© 2016 Mesosphere, Inc. All Rights Reserved.
13
NETWORKING
HOST: no separate network stack, worse security BRIDGED: isolated network stack, worse performance, may cause problems at scale Other options (e.g. CNI) may only work with certain containerizers
© 2016 Mesosphere, Inc. All Rights Reserved.
14
CONCLUSIONS
A container isn’t a virtual machine Know your code Run a recent (TESTED!) combination of Kernel, Docker, Mesos … and update as often as possible
© 2016 Mesosphere, Inc. All Rights Reserved.
15
THANK YOU!
© 2016 Mesosphere, Inc. All Rights Reserved.
16
REFERENCES
“Understanding and Hardening Linux Containers” (NCC Group) https://docs.docker.com/engine/userguide/storagedriv er/selectadriver/
© 2016 Mesosphere, Inc. All Rights Reserved.
17
![[PDF] Learning Theories Simplified: ...and how to apply them to ...](https://m.moam.info/img/260x300/pdf-learning-theories-simplified-and-how-to-apply-_6478454f097c4737708c9ebb.jpg)
