A reference architecture for the container ecosystem Madiha H. Syed
Eduardo B. Fernandez
Florida Atlantic University
[email protected]
Florida Atlantic University
[email protected]
ABSTRACT Containers have gained immense popularity as a portable and lightweight virtualization solution. They facilitate application development, deployment and distribution across computing environments. Their success is also attributed to the support they offer for DevOps teams and for applications developed using a microservices architecture style. Containers are not the only components in the environment but work closely with other components for managing and supporting them, forming an ecosystem. Architectural modeling can be used as a powerful tool to represent ecosystems which helps understand, build and secure such complex systems. We describe in this paper several models we have created for container ecosystem components. These models are abstract, and they help generalize the systems to handle complexity and heterogeneity; they provide a common vocabulary and build holistic and unified views of the systems. The use of UML for modeling improves precision. This can lead to better implementations with respect to reliability, security and interoperability compared to ad hoc methods. A reference architecture will not just facilitate the work of developers and security engineers but also of anyone who aims to ensure compliance, privacy, safety, reliability and/or governance for container ecosystems and we show how to build one. We also describe relationships between container, cloud and IoT ecosystems. This paper is part of our work on developing a security reference architecture for container ecosystems.
CCS CONCEPTS • Software and its engineering → Design patterns • Computer systems organization → Cloud computing
KEYWORDS Containers; container manager; container ecosystem; Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
[email protected]. ARES 2018, August 27–30, 2018, Hamburg, Germany © 2018 Association for Computing Machinery. ACM ISBN 978-1-4503-64485/18/08…$15.00 https://doi.org/10.1145/3230833.3232854 architectural modeling; software ecosystems; patterns, reference architecture ACM Reference format:
M. H. Syed, E. B. Fernandez, 2018. A reference architecture for the container ecosystem. In Proceedings of ARES conference, Hamburg, Germany, August, 2018 (ARES 2018). 6 pages. DOI: 10.1145/3230833.3232854
1 INTRODUCTION Cloud computing has been using Virtual Machines (VM) as a virtualization solution but recently containers have gained popularity as a more portable and lightweight option. Containerbased virtualization and operating system level virtualization run applications in an isolated execution environment (i.e. container) sharing a host operating system, binaries, and libraries. Software containers although not new, have become very important to support convenient, secure, and low overhead applications. Cloud-based systems involve a multitude of applications and services that may be deployed over clusters of hosts. This makes it very complex to create, manage, and maintain the infrastructure for deploying services on them and orchestrate them. Many human resources will be required if this work is done in non-automatic fashion by operation teams. Containers facilitate the work of DevOps teams by allowing the use of code to automatically perform infrastructure management tasks. A desired state is specified by changing configurations and this state is then automatically applied to the environment by the container manager, which optimizes resource utilization by offering grouping, load balancing, auto-scaling and auto-healing features. Use of containers and container cluster management can significantly reduce the human effort required to handle these large and complex systems. This has led to the large popularity of containers in contrast to VMs where most of these tasks have to be performed manually. Another reason for the popularity of containers is their support for microservice architectures. A microservice architecture allows development of an application as a collection of small services which run in their own separate process and communicate with each other using lightweight mechanisms, such as HTTP resource API [1]. These services are independently deployed by using container clusters that can be centrally managed. Microservice architectures need lightweight mechanisms, small independently deployable services, scalability and portability [2]. These requirements can be met by using containers. Containers which started as a virtualization solution and one of the units in cloud ecosystems, have a number of associated systems which work to support and manage them. They collectively constitute an ecosystem. Containers and container managers are important components of this container ecosystem. Several
ARES’18, August 2018, Hamburg, Germany
container manager frameworks are available which orchestrate and manage distributed environments comprising groups or clusters of software containers. Some significant examples include Google Kubernetes [3], Docker Swarm [4] and Mesosphere Marathon [5]. A conceptual view, based on abstract models, can lead to system development that follows the principles of software engineering and can thus avoid the numerous pitfalls of bad implementations that lead to unreliable and insecure systems as well as lack of interoperability. A common set of terms and models can be also a good support for standardization, a highly desirable goal in ecosystems with many heterogeneous components. A variety of projects are developing aspects of container ecosystems, building specific units or developing ways to build these systems. As these are independent projects, they use different ways to develop their systems. This has resulted, as indicated by [6], in a lack of clear separation of functionalities and an inconsistent terminology in the variety of projects building architectures, tools, or applications using containers. To add some structure and generality in terms and definitions we need to use abstraction and we propose the use of patterns and reference architectures (RAs) for this purpose. We try here to define a common vocabulary of container-related terms as well as indicating their corresponding abstract models and their inter-relationships. A pattern is an encapsulated solution to a recurrent problem in a given context; an RA is an abstract and generic architecture. We use ecosystems as guidelines for creating models and we describe these ecosystems using pattern diagrams. Pattern diagrams indicate how patterns or ecosystems relate to each other by showing the contribution they make to the pointed pattern or ecosystem. An advantage of using pattern diagrams is that they provide a perspective of how the different units or products interact with each other. We have found them very useful to study security relationships [7]. We indicate when patterns already exist for some of these components or if it is possible to find a more general pattern that can be specialized to describe them. We consider containers in isolation and groups or clusters of containers. There is some existing work on container ecosystems. The survey in [6] focused on containers and their clusters and described each architectural unit functions and their relationships to other units. We try here to add more generality to that work. Our contributions include: A pattern diagram describing ecosystems that relate to each of the components and groups of containers. The identification of existing patterns for containers and clusters of containers as well as missing patterns. Definition of architectural dependencies in container ecosystems. A partial reference architecture for container ecosystem, which shows our way to build such architectures. Section 2 describes architectural modeling and section 3 presents our existing and current work on software ecosystems, Section 4 presents ecosystems of containers and describes stakeholders, use cases and existing container and container cluster patterns. Section 5 lists some values of these architectural models. 2
M. H. Syed and E. B. Fernandez
We have included related work and discussion in section 6. Section 7 considers conclusions and future work.
2
ARCHITECTURAL MODELING
Architectural models using patterns can be used to describe ecosystems and their components. A pattern is a solution to a recurrent problem in a specific context. Patterns can be used to design and analyze complex systems, to capture design decisions, and to evaluate new or existing systems [8]. They encapsulate the experience and knowledge of designers, provide a larger unit of reuse, and a communication vocabulary for designers. Design and architectural patterns are used to build software that is flexible and extensible [8], and security patterns can be used to build secure systems by describing ways to control specific threats, fix a vulnerability, or provide a security attribute [9]. In particular, security patterns can suggest well-known solutions to designers without much security experience. In order to describe attacks, we used another type of pattern: A misuse pattern describes, from the point of view of an attacker, a generic way of performing an attack that takes advantage of the specific vulnerabilities of some environment or context; they define the environment where the attack is performed, countermeasures to prevent it, and provide forensic information in order to trace the attack once it happens [10]. This systematic and structured representation of threats is important to classify and unify them as well as to find countermeasures against them. Both security and misuse patterns aid designers in building secure systems [9]. Usually, a template with predefined sections is used to describe patterns. This structured description facilitates the work of both writers and users of the pattern. We use the POSA (Pattern-oriented software architecture) template [8]. Pattern descriptions can include formal languages in addition to modelling techniques such as UML diagrams. An intent section presents a summary of the solution provided by the pattern. Environment is part of context section which is the general condition to which the pattern applies. A problem section describes the problem being solved and the forces or concerns that guide the solution. Next is the solution which explains the recommended way to solve the problem in the given context and forces. The solution section may use UML modeling and diagrams such as class, sequence, state, and activity. An implementation section provides guidelines for implementing the pattern. A known uses section lists three or more real situations where the patterns have been used. A section on consequences includes both advantages and liabilities of the solution. There is also a section on related patterns which includes other patterns that solve a similar problem or are complementary to this pattern. While patterns were initially limited to software, now their use has been extended to other aspects of computing including hardware and other physical components of the ecosystems. Patterns can be simple or compound, which are made up of other patterns. For the sake of brevity, we have omitted detailed descriptions which can be found in the corresponding papers.
A reference architecture for the container ecosystem
In addition to patterns we will also use Reference Architectures (RAs) to describe the software ecosystem. An RA is a generic and abstract software architecture that applies to a particular domain and does not contain implementation details. It specifies the components of the system, their individual functionalities and their mutual interaction [11]. An RA can be considered as a compound pattern and its components described as patterns. These architectural models can be used to analyze the security of systems and to identify threats. Security patterns can then be added to the models to handle the identified threats. This results in a security RA or SRA. Similarly, safety patterns or compliance patterns can also be created to produce safe or complying RAs. Avgeriou’s approach [11] for building reference architectures proposes to define stakeholders, describes viewpoints to address the stakeholders’ needs, describes views, architectural patterns, quality attributes such as security, modifiability, etc., and implementation constraints as well as standards and regulations. The paper discusses also an evaluation of the approach and a complete example. Another approach, proposed by Nakagawa et al. [17] builds a reference model (metamodel) for RAs, so its scope is broader. This model can be used to build RAs, similarly to Avgeriou’s, but also to analyze RAs, compare RAs, and to define Software Product Lines. This work can be used to add more detail to each item and produce a better guidance to build RAs and SRAs. It, however, ignores UML and patterns, which are very important to build complex systems. Both approaches say very little about NFRs, which are fundamental for high quality architectures.
3
ECOSYSTEMS
Ecosystems can be defined around functions (use cases) or specific devices. For example, smart phones are associated to laptops, tablets, watches, TV sets, and similar. In this case the same function may appear in different devices. Our models describe functions not devices. We first represented cloud ecosystems in the form a pattern diagram that shows the units and their interaction with each other. This was followed by detailed modeling of the components. Most of the components of this system have been already modeled as patterns and reference architectures using UML models. However, this work is just a step in the architectural representation of the cloud and related ecosystems. It cannot be considered as complete or final as the ecosystems can evolve over time. New patterns can be added for newly identified components as we have been doing for several components described in the following sections. Complete patterns are not included in this paper as we just intend to highlight their main functions, the complete descriptions can be found in their corresponding publications listed in the references. The objective is to describe every participant of the ecosystems as patterns which consequently will contribute towards a holistic and unified view of the complete ecosystem.
ARES’18, August 2018, Hamburg, Germany
Figure 1: Cloud and Container Ecosystems Systems of ecosystems are emerging as what started as a single component in our representation of cloud ecosystem has now evolved into an ecosystem on its own, examples include containers and IoT. This calls for the need to study not only the security of these ecosystems individually but also as part of system of ecosystems, as security is not composable. Even if we ensure that our ecosystems are secure in isolation, issues can arise from the interaction of one system with another. Fig. 1 shows cloud, IoT and container ecosystems and their mutual interactions. Fog Computing, represented as part of IoT ecosystem in fig. 1 is an intermediate platform that provides computation, storage, and networking services between end devices and the cloud [12]. It has emerged as an extension of the cloud for supporting IoT. It is clear that ecosystems may overlap, due to the different roles of some of the products/systems/units in some ecosystems. For example, containers are part of cloud ecosystems but since they can be used in operating systems, they are also part of that ecosystem. We show the relationships between some ecosystems in Fig. 2, where the circles represent ecosystems instead of patterns. The units used in ecosystems can be described based on their roles; for example, some of the units in container frameworks can be considered as independent units and associated to either operating systems or cloud ecosystems. Fig. 2 shows this overlap.
3
ARES’18, August 2018, Hamburg, Germany
M. H. Syed and E. B. Fernandez
Figure 4: The Container Stack vs the VM Stack
4 Figure 2: Ecosystem overlaps The IoT ecosystem has an obvious connection to the cloud ecosystem; however, the IoT deals with a variety of devices not directly related to clouds, e.g. traffic lights. Both cloud and IoT ecosystems can run applications using containers. Fig. 3 is a metamodel for the architectural dependencies between associated systems in this environment. Containers can be used to build clouds and IoT systems; also clouds (IaaS) and IoT systems can use both VMs and containers for running applications. It also shows that VMs and OSs can provide execution environments for containers. In turn, IaaS can contain OSs. Also, the cloud itself could be built out of containers.
THE CONTAINER ECOSYSTEM
Fig. 1 shows how containers fit into the cloud ecosystem. The cloud ecosystem includes the Cloud RA as its hub. Its components are IaaS (which can be considered to define another ecosystem), PaaS, and SaaS, which provide the cloud services. The VM Execution Environment [13] creates and manages virtual machines, the Container Engine [14] creates and manages containers. Fig. 4 shows a typical stack for software containers. It is compared with the stack of VMs to highlight the difference between these two virtualization technologies. We have identified several patterns for the container ecosystem. In this section we describe the patterns for software container and the container manager (which is used when cluster of containers are being used). We also present a partial reference architecture for container ecosystems. To complete the RA we need to add its stakeholders and use cases, as well as descriptions of these use cases including sequence diagrams.
4.1
Stakeholders and Use cases
We first identify the stakeholders for the container ecosystem. These include any entities that are concerned with the ecosystem or exhibit interest in it [11]. Stakeholders can be users, acquirers, developers or maintainers of the system. In this case its clients, system developers, system administrators, software development companies, cloud service providers. Use cases for the ecosystem include creation of a container image, launch a container, container deployment, etc. Container clusters are managed by grouping closely-related containers into sets, named Coherent Units (CUs) and use cases for the manager include creation and termination of CUs, Schedule CUs, Assign workloads to a CU, CU Replication, load balancing in a cluster of containers, and others.
4.2 Figure 3: Architectural dependencies
4
The Software Container Pattern [14]
4.2.1 Intent. A Software Container provides an execution environment for applications sharing a host operating system, binaries, and libraries with other containers. Containers are lightweight, portable and extensible. 4.2.2 Solution. Provide a runtime environment that can support the isolated execution of applications on a shared Host OS, this is a Software Container (Fig. 5). Multiple processes can share
A reference architecture for the container ecosystem
ARES’18, August 2018, Hamburg, Germany
one container in special circumstances [15]. They also may share binaries and libraries with other containers. Containers provide isolated execution and extensible services to the application.
4.3
The Container Manager Pattern [15]
Groups or clusters of containers need to be orchestrated and managed by a higher-level controller, composed of several appropriate units (Orchestrator, Scheduler,…). We need then another pattern to consider these aspects, a Container Manager, composed of patterns for container management functions. 4.3.1 Intent. A container manager acts as a high-level controller to orchestrate and manage clusters of containers across distributed hosts. It provides functions for organizing, deploying, managing, and securing container clusters. 4.3.2 Solution. Add a central container manager to coordinate the work of the containers which implement the parts of an application. This manager orchestrates the execution and interaction of these groups/clusters of containers, and can control their properties. Fig. 5 shows a partial reference architecture (RA) for a container ecosystem using both the software container pattern (in shaded outline) and the container manager pattern. A Container controls a set of Applications (App) sharing a Host OS that provides a set of Resources. The client interacts with the container, which represents the application running in it. The application interacts with the container, which provides an interface to the Host. The Container provides a set of services to the applications. Container Images are stored in an Image Repository. A Container Manager acts as a high-level controller that can orchestrate and manage the execution of containers across multiple Hosts that form a Cluster. A container manager manages a diverse set of containers and their mutual dependencies and interactions by grouping closely-related containers into sets, named Coherent Units (CUs), which are managed as a single entity. Each CU is assigned an IP address (it can change when they change hosts). CUs have the same lifecycle and the same environment as they share resources on the same host. The Scheduler assigns CUs to hosts which meet requirements such as sufficient resources. The Overlay Network allows CUs to communicate with each other within and across hosts by providing a consistent networking interface. The Discovery Service allows processes and services in a cluster to locate each other so they can collaborate. The State Storage stores information about cluster states and provides this data for scheduling and orchestration. The Resource Monitor collects and provides health and resource allocation/usage data for each host.
Figure 5: Partial Reference Architecture for Container Ecosystems
5
VALUE OF THE CONTAINER RA MODEL
The representation of complex systems in the form of reference architectures, has several advantages. It not only helps understand the systems involved but also maps their mutual interactions and dependencies. This leads to a holistic and unified view of the system. We use UML models to represent the reference architecture which provides more precision as compared to other presentations like block diagrams. Several systems work together to enable the use of containers in cloud and microservices environment. These tools and platforms provide many useful services including automation of management and orchestration; however, this also adds complexity to the system. The supporting systems also require administration and management which itself can be a daunting task for system administrators. It may require technical training for the developers, system architects and administrators to get familiar with the new software platforms. Holistic architectural models can be useful for gaining understanding of the system and their limitations and can help utilize the systems to their full potential. Moreover, there are several variants of the systems involved developed by different providers, each using their own terminologies and descriptions adding to the overall confusion. This heterogeneity can be addressed by a reference architecture which focuses on general categories of the system by grouping them on the basis of similarities in structure and functionality. This can lead to development of standardized abstract representational models of the ecosystem components. 5
ARES’18, August 2018, Hamburg, Germany
These holistic and unified models can help us build and configure the systems for better design. It can help us identify and remove flaws, missing components as well as redundancies in the model. In addition to concerns about functionality, complex systems are also required to be secure and reliable (and other NFRs) in most circumstances. Security, compliance, privacy, safety, reliability and governance can all be improved by the use of these models to provide understanding of the underlying system which can be built upon as a base.
6
RELATED WORK AND DISCUSSION
We have used as main reference the work of [6], which is very complete and provides a valuable overview of the components related to containers. We consider our work a complement to that paper intended to provide a more conceptual view of container architecture. As [6] says, there is no explicit work on classification and conceptual modeling of containers, although there are many papers on specific aspects of containers. NIST has published an application container security guide and security assurance requirements for Linux application container deployments. However, while this guide includes descriptions and block diagrams it lacks detailed architectural models. In addition, there is not much work on threat modeling of containers.
7
CONCLUSIONS AND FUTURE WORK
We have extended the holistic conceptual view of containers started by [6] and made it more abstract. Expressing the units of an architecture as patterns makes explicit their abstract properties and the way they relate to other units. Starting a new architecture from a set of patterns provides a way to visualize the structure and interactions of its units, fundamental to add quality factors such as security or reliability. This conceptual clarity is also useful for understanding existing systems and to create standards. The architectural representation of containers and container managers enables us to study their ecosystem components as well as their mutual interactions. This can be useful to systematically analyze threats and propose countermeasures. These can be represented using security and misuse patterns thus providing a comprehensive, holistic and unified view of the system. As a part of our ongoing work we are extending the reference architecture presented in section 3 to add security and misuse patterns, thereby developing a security reference architecture for container ecosystems, in the style of [18]. We need to create a catalog of patterns for containers including design and security patterns as well as a catalog of misuse patterns describing their threats. The goal is to facilitate the work of system architects and developers that are using container-based systems, help make the ecosystem easier to understand for them and consequently make it easier to perform security analysis of the system.
6
M. H. Syed and E. B. Fernandez
REFERENCES [1] [2]
[3] [4] [5] [6]
[7] [8] [9] [10]
[11] [12] [13] [14] [15] [16] [17] [18]
J. Lewis and M. Fowler. 2014. Microservices. Retrieved March 2, 2017 from https://martinfowler.com/articles/microservices.html E. Ekici. 2014. Microservices Architecture, Containers and Docker. Retrieved April 15, 2017 from https://www.ibm.com/developerworks/community /blogs/1ba56fe3-efad-432f-a1ab-58ba3910b073/entry/microservices_ architecture_containers _and_docker?lang=en.. Kubernetes. 2017. Production-Grade Container Orchestration. Retrieved February 2. 2017 from https://kubernetes.io/ Swarm. 2017. Docker Swarm. Retrieved February 2. 2017 from https://www.docker.com/products/docker-swarm Mesosphere. 2017. Meet Marathon: Production-ready container orchestration at scale. Retrieved February 2. 2017 from https://mesosphere.com/blog/2016/02/17/marathon-production-ready-containers/ D. Ernst, D. Bermbach and S. Tai. 2016. Understanding the container ecosystem: A taxonomy of building blocks for container lifecycle and cluster management. Retrieved February 2. 2017 from http://www.ise.tuberlin.de/fileadmin/fg308/publications/2016/Ernst_WoC_containertaxonomy.pdf E. B. Fernandez, N. Yoshioka, H. Washizaki, and M. H. Syed. 2016. Modeling and security in cloud ecosystems. J. Future Internet 2016, 8(2), 13; doi:10.3390/fi8020013 F. Buschmann, R. Meunier, H. Rohnert, P. Sommerlad and M. Stal. 1996. Pattern-Oriented Software Architecture: A System of Patterns, Volume 1. Wiley & Sons, Inc. E. B. Fernandez. 2013. Security patterns in practice: Building secure architectures using software patterns. Wiley Series on Software Design Patterns. E. B. Fernandez, J. Pelaez and M. Larrondo-Petrie. 2007. Attack patterns: A new forensic and design tool. In Procs. of the Third Annual IFIP WG 11.9 Int. Conf. on Digital Forensics, Orlando, FL, Jan. 29-31, 2007. Chapter 24 in Advances in Digital Forensics III, P. Craiger and S. Shenoi (Eds.), Springer/IFIP, 2007, 345357. DOI: https://doi.org/10.1007/978-0-387-73742-3_24 P. Avgeriou, 2003. Describing, Instantiating and Evaluating a Reference Architecture: A Case Study. J. Enterprise Architect, Fawcette Tech. Publications, Jun. 2003. M. H. Syed, E. B. Fernandez and M. Ilyas. 2016. A pattern for fog computing. Pattern Languages of Programming (VikingPLoP 2016), 7th-10th April 2016, Leerdam, Netherlands M. H. Syed and E. B. Fernandez. 2016. A pattern for a virtual machine environment. 23rd Conference on Pattern Languages of Programs (PLoP 2016), Champaign, IL, October 24-26, 2016 M. H. Syed and E. B. Fernandez. 2015. The Software Container pattern. 22nd Conference on Pattern Languages of Programs (PLoP 2015), Pittsburgh, PA, October 24-26, 2015 M. H. Syed and E. B. Fernandez. 2017. The Container Manager Pattern. 22nd European Conference on Pattern Languages of Programs (EuroPLoP 2017), Germany, July 12-16, 2017 Docker. Docker (April 2015). Retrieved April 10, 2015 from http://www.docker.com/ E.Y. Nakagawa, F. Oquendo, M. Beecker. 2012. RAModel: A Reference Model for Reference Architectures. European Conference on Software Architecture, 2012 E.B. Fernandez, R. Monge, and K. Hashizume. 2016. Building a security reference architecture for cloud systems. In J. Requirements Engineering. DOI: 10.1007/s00766-014-0218-7, June 2016, Volume 21, Issue 2, 225-249.
