Software Provisioning Inside a Secure Environment ...

Software Provisioning Inside a Secure Environment as Docker Containers using S TROLL File-system Abdulrahman Azab

Diana Domanska

Biomedical Informatics Group, Department of Informatics Research Support Services Group, University Center for Information Technology, University of Oslo Email: [email protected]

Research Group for Biomedical Informatics, Department of Informatics University of Oslo Email: [email protected] Department of Modelling and Computer Graphics, Department of Informatics University of Silesia Email: [email protected]

Abstract—TSD (Tjenester for Sensitive Data), is an isolated infrastructure for storing and processing sensitive research data, e.g. human patient genomics data. Due to the isolation of the TSD, it is not possible to install software in the traditional fashion. Docker containers is a platform implementing lightweight virtualization technology for applying the build-oncerun-anyware approach in software packaging and sharing. This paper describes our experience at USIT (The University Centre of Information Technology) at the University of Oslo With Docker containers as a solution for installing and running software packages that require downloading of dependencies and binaries during the installation, inside a secure isolated infrastructure. Using Docker containers made it possible to package software packages as Docker images and run them smoothly inside our secure system,TSD. The paper describes Docker as a technology, its benefits and weaknesses in terms of security, demonstrates our experience with a use case for installing and running the Galaxy bioinformatics portal as a Docker container inside the TSD, and investigates the use of S TROLL file-system as a proxy between Galaxy portal and the HPC cluster.

I. I NTRODUCTION The TSD (Services for Sensitive Data) is a system and an isolated network infrastructure for providing storage and processing of sensitive research data. This infrastructure provides a huge storage space and a HPC cluster for data storage and processing. The infrastructure is divided from the inside into project areas; each is associated with a specific research project who is a member of the TSD. Each project owns an amount of storage space, and all projects can get access to the HPC cluster for running computationally intensive analyses. Projects are totally isolated from each other, and access from the outside world is very restricted and is carried out through two factor authentication for both interactive login and file import/export. The TSD is secure, but its isolation makes it sometimes difficult installing software packages especially those who require fetching package dependencies from remote public repositories during the installation. One solution for this problem is to install the software on a VM outside the TSD which has unrestricted access to the Internet for downloading software dependencies, import the VM image inside the TSD project area. But the security rules and regulations of the

TSD do not currently allow such scenarios. Docker [1] is a new tool that automates the deployment of applications inside software containers. It offers an environment as close in the functionality as possible to a VM but without the overhead of running a whole separate kernel and simulating the hardware. A Docker image can be built from a human readable Docker script. This feature gives an advantage for Docker over the VM technology in terms of security since looking at the Docker script prior to building the image would make it possible to easily discover any security vulnerabilities/threats the image could cause. Using the build-once-run-anywhere approach, Docker has gained interest of different scientific communities, including life-sciences. It has been demonstrated that Docker containers have only a minor impact on the performance of common genomic pipelines [15]. Although Docker is not yet mature and is having some security and operational issues, it still has the advantage, from our experience, over VMs in terms of security and flexibility. Our experience installing Galaxy portal [9] inside the TSD as a Docker container [1] proves that Docker is a very useful and practical tool for deploying software packages inside isolated, closed, and secure systems. This paper describes our experience building Docker images and installing them inside the TSD. The paper is organized as follows: Section II gives a brief introduction about the TSD as an infrastructure and a service. Section III describes the scenarios for software provisioning inside the TSD, and the issues and obstacles with these scenarios. Section IV describes Docker as a technology, illustrates the security benefits and weaknesses of Docker containers, and describes our proposed procedures for a secure use of Docker containers. Section V describes the use case for installing and running Galaxy portal as a Docker container inside the TSD. Section VI describes the use of S TROLL filesystem to submit jobs from Galaxy to the HPC cluster. II. TSD (S ERVICES FOR S ENSITIVE DATA ) Project Services Sensitive Data (TSD) 2.0 [21] has been created by USIT (The University Centre of Information Technology) at The University of Oslo, to offer a service to

researchers in Norway for storing and processing sensitive data, including health data. A. Background In recent years, several research groups at the Oslo University Hospital, OUS, and at other universities and university promoted the need for e-Infrastructure for sensitive research data that is of such a size that one cannot use the same systems which have been used before to handle. These are typically high resolution MR images, DNA data, a number of different registers of sensitive data, as well as video of patients where there is often talk about many terabytes of data. Additional requirements have signed up for a service for web-based collection of sensitive information, such as dietary habits coupled with address, salary, disease profile and more. All these requirements have resulted in project services for Sensitive Data 1.0 (TSD 1.0) which is now continued as TSD 2.0 to encompass far more users and enhanced functionality without sacrificing safety. B. Architecture TSD is an isolated network infrastructure which is composed of two collections of nodes, Fig. 1. The first is the HPC compute cluster, Colossus [22], which is SLURM based and relies on a fast parallel file-system, BeeGFS [23]. The second collection of nodes is for hosting the user-space and a collection of VMs which are used as front-ends between the users and the system, and relying on HNAS file-system. The user area is divided into projects. Each project is isolated in a separate subnet, and is accessible by a set of users who have been registered to this specific project. A typical project area contains one Windows Server VM, and a collection of RHEL VMs which are shared among the project users. In addition there is a shared project area, in which software packages and data files (which are NOT classified as sensitive data) are kept. All TSD users have read access to the shared area, while only a few users have write access.

Fig. 1. TSD 2.0 Architecture

pXX/fx/import through the file-lock daemon which is running on an internal VM, tsd-fx02. Similarly, for exporting a file from the project working area to outside the TSD, the file is placed in pXX/fx/export which inside the project working area. The file is then transferred through the file-lock protocol to the intermediary project area pXX/export, where it can be downloaded by the user via SFTP. Project administrators have both import and export rights, while regular users have only import but not export rights.

C. Interactive Login Users can login to their project areas in the TSD from the outside world through a two factor authentication system. Three pieces of information must be provided in order to login: username, password, and one-time code (from a Google authenticator which implements TOTP security tokens from RFC 6238). D. Data Transfer Importing and exporting file into/out of a project area is performed via SFTP interactive sessions which are established through the same two factor authentication mechanism, described in II-C. The file transfer process, presented in Fig.2, takes place in two steps. For importing a file into own TSD project, pXX, the file is uploaded via SFTP to pXX/import which is an intermediary area on HNAS file-system through an external server, tsd-fx01. Files are then transferred from the intermediary area to a directory in the working project area,

Fig. 2. TSD 2.0 File Transfer Protocol

III. TSD S OFTWARE P ROVISIONING Getting software installed and usable by TSD users can be performed in one of three scenarios: • A user could import a software package through the file-lock protocol, described in II-D, into his/her home directory or the project area (which is shared among the project members), install, and run the software. • Software packages which are useful for users from different projects and can be executed from the commandline can be installed as UNIX modules, and be visible on both the user area and the HPC cluster area. This is performed as follows: A user sends a request for installing a software package to the TSD helpdesk (tsd-drift), A TSD administrator takes the case, validates the software, and install it as a module on the system.

•

•

Another alternative for software packages of a shared interest is to have them installed on the shared project area. Other software packages, which require specific resource configurations and/or specific underlying engines e.g. Docker and .Net Framework (for Windows), can be installed on one of the project VMs, i.e. the Windows VM or one of the Linux VMs depending on the associated platform.

A. Problem statement Many software packages require access to the Internet during the installation process for downloading and updating package dependencies. Some of these packages are popular for use by many researchers for data analysis, and hence necessary to be installed inside the TSD. As already described in II-B, the TSD is an isolated infrastructure, so it does not support access to the world-wide web for downloading stuff from public repositories/websites. One solution for this problem is to install the software on a machine/VM outside the TSD which has unrestricted access to the Internet for downloading software dependencies, import the VM image inside the TSD project area, and start it so that users can use the installed software. But the security rules and regulations of the TSD do not allow such scenario, i.e. installing a VM in the unsecure world, importing it, and starting it inside the secure environment. Another solution is to study the installation wizard/script of the desired software package, document all the package dependencies which are needed and downloaded during the installation process, manually download all these packages to a local path, and modify the installation wizard/script to fetch those dependencies from the local path instead of the original Internet external links. The whole thing can be then packaged in e.g. a TAR ball, imported and installed inside the TSD project area. The problem with this solution that it takes a lot of time and effort, especially for software packages that need a huge number of dependency packages in different stages of the installation process. So there was a need for another solution using which a software package can be installed outside the TSD, the installation can be packaged in a way which makes it possible to run it inside the secure environment without a need to install an entire VM to reach this target.

feature of Docker containers, and Linux containers in general, is that they offer an environment as close in the functionality as possible to a VM but without the overhead of running a whole separate kernel and simulating the hardware [2]. Fig. 3 presents an overview on the architectural difference between Docker and the VM technology. Virtual machines have a full OS (guest OS) with its own memory management installed with virtual device drivers, and controlled by a hypervisor (also called Virtual Machine Monitor, VMM), which works on the top of the host OS.

Fig. 3. Docker Containers VS Virtual Machines [3]

Docker is a new tool which automates the deployment of applications inside software containers. It provides an additional layer of abstraction and automation of operatingsystem-level virtualization on Linux [1].

Docker containers are executed and controlled by the Docker engine (Docker daemon), instead of the hypervisor. Since it does not include a full guest OS, a Docker container is smaller and faster to start up than a VM. A Docker container instead mounts a separate root file-system, which contains the directory structure of the Unix-like OS plus all the configuration files, binaries and libraries required to run user applications. The boot file-system which contains the bootloader and the kernel is not included in the container, but a container uses the host’s boot file-system [3]. When Docker mounts the container’s root file-system, it starts read-only, as in a traditional Linux boot, but then, instead of changing the file system to read-write mode, it uses union mount [5] to add a read-write file system over the read-only file system. It is also possible to have multiple read-only file systems stacked on top of each other, see Fig. 4. Each of those file-systems can be considered as a layer [4]. Based on the above description, one could start with e.g. a Debian base image (which contains only the Debian root file-system), and run it from the Docker engine on a machine with e.g. RHEL kernel. Then install a piece of software, e.g. emacs, on the top of this base image, and deploy this as a new image (containing both Debian root file-system and emacs installed). Using this new image as a base image, one could install another piece of software, e.g. Apache server, and make another image which has both emacs and Apache installed on Debian, and so on.

A. Technology Overview

B. Building a Docker Image

Docker is not a new technology itself, but is a high-level tool which is built on the top of LXC Linux containers API, and provides additional functionality. Other examples of Linux containers are LXD, CGManager, and LXCFS [2]. The main

Building own Docker image is performed in one of two scenarios, presented in Fig. 5: • Interactive building which is carried out by starting a base image as a container (via ’Docker run’), running the

IV. D OCKER

3) Commit the changes made in 2 and build a new image ’azab/bowtie2’ (an image is usually named as: /:). 4) Start the new image as a container and run ’bowtie2 −−version’ to validate the installation. 5) Push the new image to the Docker hub to be publically available.

Fig. 4. Docker File-system Layers [4]

•

commands to install the desired software on the running container, then committing the changes creating a new image on the local Docker repository. Building from a Docker file which is carried out by writing a Docker file. A Docker file is a script file which contains all the commands one would normally execute manually in order to build a Docker image, and is written in a Docker specific format. The Docker file starts by loading a base image, followed by the list of Docker formatted commands to install the desired software. The image is built by calling the docker build command having the Docker file as an argument. The building process will go step by step, executing the commands successively.

Fig. 5. Building a Docker Image

1) Example: Bowtie Docker: A practical example on interactive building is presented in Fig. 6. It describes how to build a Docker image for Bowtie2 [6] using a Ubuntu 14.04 base image with the following procedure: 1) Start a Ubuntu 14.04 base image (automatically fetched from the Docker hub upon the execution of the ’docker run’ command if it does not already exist locally) as a container. 2) After logging in the container ’2a896c8cdd83’ as root, run the command sequence for installing Bowtie2, then exit from the session with the container.

Fig. 6. Bowtie2 as a Docker Image

C. Docker in a Secure Environment Docker is a platform for providing smooth and easy packaging, distribution, and management of Linux apps, the technology itself is not yet mature enough though to be fully reliable in terms of security. The following two subsections describe some security benefits and weaknesses of Docker containers. 1) Security Benefits of Docker Containers: Docker containers offer benefits in terms of isolation between applications [7]: 1) Docker creates a set of namespaces and control groups for each container. Processes running within a container cannot influence, or even see, processes running in another container, or in the host system. 2) Each container has a separate network stack. Unless the host system is setup to allow containers to interact with each other through their respective network interfaces, no interactions can happen between containers. 3) The use of Linux control groups ensure that each container gets a fair share of memory, CPU, disk I/O. A single container cannot bring the system down by exhausting one of those resources. 4) Containers make it easier to control which data and software components are installed, through the use of scripted instructions in setup files, i.e. Docker files. 2) Security Weaknesses of Docker Containers: Running and installing Docker on a host machine/VM has security issues: 1) The Docker daemon/engine always requires root privileges, and currently always runs as root [7]. 2) Docker allows users, upon running/starting a container, to mount directories from the host on the container; and it allows you to do so without limiting the access rights of the container [7]. This means that any user who has

access to the Docker command can mount ’any’ host directory on the container, including the root directory. 3) Docker does not, so far, provide each container with a separate user namespace, which means that it offers no user ID isolation. A process running in the container with, e.g. UID 451, will have the privileges of UID 451 on the underlying host as well. Along these lines, a process running as root (UID 0) in a container will have root-level privileges on the underlying host when interacting with the kernel [8]. 4) The isolation provided by Docker is not as robust as the segregation established by hypervisors for virtual machines. Docker Containers so far dont have any hardware isolation, which is the case for VMs. D. Proposals for a Secure Use of Docker Containers Here we provide proposals to overcome the security weaknesses and make use of the security benefits of Docker containers. 1) Secure Building of Docker Images: For a secure use of Docker containers, first of all, we need to assure that all what is installed and stored on our Docker images is secure. Sec. IV-B described two scenarios for building a Docker image; one is from a Docker script. This scenario gives advantage for Docker over VM technology in terms of security. It is not trivial to write one script for installing a whole VM, while it is possible and is a standard in case of Docker. Looking at the Docker script prior to building the image would makes it possible to easily discover any security vulnerabilities/threats the image would introduce prior to the building process. Fig. 7 presents a proposed procedure for secure building of a Docker image to run inside a secure environment: •

•

•

•

The user/developer initially prepares a Docker script file which includes all necessary instructions to build an image for a specific software package, or a collection of software packages. This Docker file can then be scanned by the system’s security officer, and approved for building in case of no security threats. A system administrator can then start the process of building the Docker image from the Docker file including any necessary package/dependency download. In case of any embedded downloads, the building process needs to be carried out on a machine/VM which have Internet access, i.e. outside the secure environment (for the case of the TSD or a similar system). The Docker image can then be packaged as a tarball, imported into the secure environment, e.g. using the filelock protocol, and be added to a local Docker repository.

This procedure guaranties that it is safe to run containers from the built image, since the building process is done using an approved Docker file, and by a system administrator, instead of asking the user/developer to build the image then importing it inside the system as a black-box.

Fig. 7. Secure Building Procedure of Docker images

2) Secure Running of Docker containers: When Docker is installed, it creates a Linux group ’docker’. The root user, the admin users, in addition to other users which are added to the docker group by the root, all will be able to run the ’docker’ command and hence will be able to do anything Docker can do. To overcome the security vulnerabilities, described in sec. IV-C2:1-3, only trusted users should be able to directly start/run Docker container. It is also not practical to limit access to Docker only to admin users, e.g. ’wheel’ group members, and not to regular users. This would make it almost useless to install Docker on the system. Alternatively, regular users should be able to start and interact with containers, but with restricted permissions. For example, the main security threat could come from mounting of host directories inside Docker containers, sec. IV-C2. This can be avoided by allowing users to mount only specific directories. This can be carried out as follows: 1) Create a script, e.g. named ’docker−run’ on the root directory, which gets all ’docker’ command parameters, checks that it does NOT contain the ’−v’ option (which mounts a volume in the container), and if it does, it must be mounting permissible directories. Otherwise, the command should fail. The contents of this script could be further developed for different systems based on different security regulations. 2) This script should be owned by the root user (or any other user who is a member of the ’docker’ group). To allow non-admin users to run this script, create a Linux group ’docker−safe’, and then create a file inside ’/ etc / sudoers . d /’ with the following contents: Cmnd\_Alias DOCKERSAFE = /docker-run %docker-safe ALL=(ALL) NOPASSWD: DOCKERSAFE This will allow any user who is a member of the ’dockersafe’ group to run the ’docker-run’ script as root using the sudo command as follows: $sudo docker-run At the same time, regular users will not have direct access to the ’docker’ command since they are not members of the ’docker’ group.

V. U SE C ASE : G ALAXY P ORTAL INSIDE THE TSD AS A D OCKER C ONTAINER This section describes our experience installing Galaxy bioinformatics portal [9] and [10] inside a TSD project area.

new tool, or even more, upon adding/producing/updating any dataset. Alternatively, the installation could be separated into two parts, presented in Fig. 9: •

A. Galaxy Bioinformatics Portal Galaxy is an open platform for data intensive biomedical research. It is a very useful, rich, efficient, and user friendly web-based wrapper mainly for bioinformatics tools. Fig. 8 presents how Galaxy portal wraps and provides access to tools using a web interface. For wrapping a tool, e.g. SMALT [12], a simple entry point script, e.g. a python script, should be written in order to simplify the interface and minimize the complexity of providing command-line parameters, in case of any. To include SMALT as a Galaxy tool and to build a web form for user interaction with this tool, an XML based tool description needs to be written. This xml ’smalt wrapper.xml’ mainly describes the shape and format of the input and output datasets.

•

Galaxy portal core and runtime environment files are stored inside the image and available inside the container during the runtime. This is obvious since Galaxy core is expected to change only to update to a new Galaxy release when there is one. Tools and datasets are configured to be placed outside the container in a directory located on the host and mounted as ’/ export ’ in the container. Upon starting the container, it scans the /export directory. If it is empty, i.e. first run, it moves the tools and data directories to / export and places symbolic links to them in the original places. Otherwise, i.e. it is not empty, it starts using tools and data files from / export.

Fig. 9. Galaxy Docker with Mounted Export Directory

Fig. 8. Galaxy Tool Wrapping

Galaxy can be installed and used in different ways. One is the free public server at UseGalaxy.org (also known as Main) that can be used by anyone. Another option is to install own Galaxy server for public or restricted use. Other options are described in ref. [11]. Galaxy attracted interest of a huge portion of the bioinformatics community including our TSD bioinformatics project users, and thus it has been a requirement to have a Galaxy portal up and running inside the TSD to be used by bioinformatics related projects. Since bioinformatics analyses are usually computational intensive, as it processes huge datasets, Galaxy has the option of being installed on the top of a HPC cluster.

The advantage of this approach is that one Galaxy Docker image can be used to run on the top of different export directories each with a different set of tools and datasets. Practically this is very useful in the TSD setting, since Galaxy is required by many projects with a different set of tools each. Another advantage of this approach, presented in Fig. 10, is that it is easy to get a new Galaxy image (built on a new Galaxy release) and run it as a container on the top of an export directory which is generated by an old container. This would save the effort of reinstalling and committing all custom tools on the new image.

B. Galaxy as a Docker Container A creative idea, developed by one of our collaborators, is to make a Galaxy Docker image and run Galaxy as a Docker container [13]. An important issue is that Galaxy is not a simple binary which takes a set of inputs, executes, and produces a set of outputs. It is a rich web portal application which includes hundreds of wrapped tools and stores a lot of datasets, so called History Elements [9]. It is not practical in this case to have the whole Galaxy, runtime, tools, and files inside a Docker container. This would make it necessary to commit the changes in the container as a new image upon adding any

Fig. 10. Running a New Galaxy Container on the Export Directory of an Old Galaxy Container

C. Running Galaxy Docker inside the TSD This section describes our particular experience for having Galaxy up and running inside the TSD as a Docker container. 1) Installation: Galaxy is a web portal by design. It is not mainly designed to be installed inside an isolated environment, e.g. the TSD. The installation process of Galaxy includes fetching a lot of packages, e.g. python eggs, from the Internet throughout the installation. Sec. III-A described the issues of having such a piece of software installed inside the TSD. The Galaxy Docker with the export directory option has been considered as a practical solution to this problem. Following the secure building procedure, described in sec. IV-D1, we managed to get Galaxy up and running inside the TSD project P77 using the following procedure, described in Fig. 11: 1) We Built a Galaxy Docker image (using an approved Docker file) on one of our University servers, insilico, saved the image as a tarball. 2) The tarball was imported inside P77 in the TSD through the file-lock, and loaded it as a new image inside our local TSD Docker repository, and started as a container using a very simple export directory with only the basic Galaxy tools installed, for testing. 3) Then we installed the project specific Galaxy tools and workflows on the Galaxy instance that located on the external server, to be placed on the associated export directory. This was necessary since many of those tools are installed from remote repositories, i.e. toolsheds, and require Internet access to be installed. 4) This /export directory was packaged in a tarball. 5) Then we imported this tarball inside P77 through the file-lock, unpacked it on a VM, and started a new container on this VM using the image imported in step 2 on the top of the imported export directory. A very important part of step 4: While composing the tarball, file attributes (e.g. ownerships and access permissions) MUST be preserved, otherwise the unpackaged export directory tools wont be able to run elsewhere. This is due to the fact that those files were created by a user on the Docker container, and the same user will attempt to access them from a similar container on another machine. If the files ownerships and permissions change, this attempt would fail and the Galaxy container won’t be able to start. 2) Managing Access to Reference Data: Galaxy is a bioinformatics based portal. A lot of bioinformatics analyses use a reference genome [14] as input. Reference genomes are usually stored in large files (which are either original, i.e. .fasta and .fastq, or tool specific binary files, i.e. index files). This means that many Galaxy tools would need access to these files. And since Galaxy is needed by different projects, many Galaxy containers which are running on different VMs inside different projects would need to access to the same set of large files. As described in sec. II-B, the TSD shared project area is meant for this purpose. Therefore we have stored all reference data files in the shared project area, and configured the tools of

Fig. 11. Packaging Externally Built Galaxy Docker Image and Export Directory For Running inside the TSD

all Galaxy instances (which use reference genomes) to access these files. Fig. 12 illustrates this solution.

Fig. 12. The use of the Shared Project Area for Storing Reference Data

3) Managing Access to Project Data: Other than shared data, there are the project specific datasets which are the input and output files for running the analysis tools by the project users. These are known in Galaxy as history elements. The obvious place to store history elements is the storage area of the project. One issue with this solution is that the storage area of each project is accessible and seen only by the registered user of this particular project, i.e. pXX−. When Galaxy runs as a Docker container, the Docker daemon runs as the root of the container, which will also be mapped to the root of the VM on which Docker is running. In the TSD, root users of VMs in a particular project do NOT have access to the project storage area since they are not considered as registered project users. Root squashing [20] is used to enforce this policy. One way to resolve this issue is to store the Galaxy history elements on the local VM disk, but this is not practical in the long run since the local disk space of a VM is very limited. A more practical solution, which we implemented, is to do the following: 1) From the Galaxy VM inside the TSD project, get the UID of a registered project user (e.g. pXX−uXX) using

which Galaxy can access the project stored area. 2) In the prior to the building process of the Galaxy image, described in sec. V-C1:1, configure the UID of the internal user which will run the Galaxy daemon inside the container (the ’galaxy’ user) to the UID of the user pXX−uXX, and then go through the image building and importing procedure. 3) When the Galaxy container starts on the VM, the galaxy user of the container will be seen as pXX−uXX inside the VM and the TSD system, and hence will have the privilege to access the project storage area. This solution has issues. First, it is a sort of a hack which uses one of the security vulnerabilities (i.e. the lack of UID isolation) in Docker in order to reach the target. Second, it makes it necessary to build a separate Galaxy Docker image for each TSD project; each is configured with a different UID of the Galaxy user. Despite having these issues, this is the most practical and working alternative so far, from our view, considering the current structure of Docker and the TSD system.

which is also configured as S TROLL virtual-path and where all Galaxy history elements are placed. This directory resides on the project storage area, /tsd/p65/data. For a Galaxy tool to have its jobs allocated on Colossus, the job has to be configured to use the S TROLL job-runner. Job execution takes place as follows: • S TROLL job-runner pulls jobs from the Galaxy job queue, creates a task directory with the job script and data files in the S TROLL virtual-path, then runs a file-system write() command in the config sub-directory of the task [18]. • S TROLL file-system submits the the job to colossus and puts the write() on hold until the job is completed. • Once the job is completed, S TROLL places the output files in the task directory and releases the write() command output. • When the write() command output is released, S TROLL job-runner reports job success/failure to Galaxy

VI. S UBMITTING J OBS FROM G ALAXY TO THE HPC CLUSTER USING S TROLL FILE - SYSTEM A. Problem statement Galaxy portal can be configured to execute tools as jobs on a compute cluster, through a job runner [24]. This functionality is possible when the machine/VM hosting Galaxy mounts a shared file-system with the cluster. This is not the case in TSD, due to security reasons. Colossus has a separate file-system which not mounted on the TSD user VMs. To resolve this issue, we needed to have an intermediate software/daemon that packages Galaxy jobs and submit them to Colossus. Galaxy does support the functionality of submitting to a remote cluster, though Pulsar [25]. But Pulsar offers slow file transfer using only HTTPS, which is not proper for transferring large data files between two machines in the same room (the case in TSD). In addition, Pulsar doesn’t support submission of workflows to the remote cluster and is not even configured to support a containerized Galaxy instance. For these reasons we decided to use S TROLL file-system.

Fig. 13. Submitting Galaxy jobs to Colossus through S TROLL file-system

D. Performance evaluation B. S TROLL file-system S TROLL file-system [18] is a universal interface for seamless deployment of compute tasks to a variety of HPC and HTC platforms. S TROLL allows job submission, monitoring, and administration through simple read() and write() file-system commands. The interface is implemented as a user space synthetic file-system, so that it operate as a proxy between the application and the compute platform. S TROLL supports both simple jobs and workflows [19]. C. Galaxy and S TROLL file-system For enabling a containerized Galaxy to submit jobs to Colossus through S TROLL, we have created S TROLL jobrunner. The architecture is presented in fig. 13 for TSD project p65. The Galaxy Docker container has a mounted directory

To evaluate the performance of our solution, we tested the execution of a number of concurrent Bowtie jobs for sequence alignment of FASTQ files into SAM files using the following parameters: bowtie -p 16 -S -v 2 -m 1 hg19 INPUT.fastq > OUTPUT.sam • -p 16: Launch 16 parallel search threads. • -S: Output in SAM format. • -v 2: No more than 2 mismatches. • -m 1: Suppress all alignments for a read if more than 1 reportable alignment exist. In our experiment we tested the submission of 5, 10, 20, 30, and 40 concurrent Bowtie jobs. The experiment was performed using three methods.

1) sbatch array: In this method we used the traditional manual way which is mostly used by TSD users, submitting an array of the SLURM sbatch command to run Bowtie jobs from the project area to colossus. The job script is manually created. 2) Galaxy with S TROLL: Here we used the architecture presented in fig. 13 for launching concurrent runs of the Galaxy Bowtie tool wrapper using S TROLL file-system to colossus. 3) Galaxy CloudMan on EC2: In this method, we created a Galaxy CloudMan [26] SLURM cluster on Amazon EC2 [27] with 16 small instances, and launched a number of concurrent Bowtie Galaxy jobs on that cluster of virtual machines. Fig. 13 presents the results. We measured the total allocation time for all concurrent jobs using under different submission load. The total allocation time is calculated from submitting the concurrent jobs until all jobs start running. E. Discussion

Total allocation time (minutes)

In fig. 13, it is clear that the sbatch array method causes a lot of pre-processing overhead due to the fact that the next job is submitted when the current job is allocated. This overhead is negligible for HPC jobs where the actual runtime is hours, but not applicable for HTC jobs. The CloudMan method introduces very little overhead due to the fact that Galaxy has a dedicated cluster of VMs and the connection between VMs within the same VPC in EC2 is very fast. S TROLL introduces an overhead which close to that of Galaxy CloudMan despite the fact that S TROLL submits from Galaxy to colossus cluster not to a dedicated cloud cluster. This is due to the fact that S TROLL creates a separate submission thread for each job. We conclude that using S TROLL file-system as a proxy between Galaxy Docker and colossus eliminates the burden of manual submission, and introduces a much smaller overhead.

40

sbatch array Galaxy+S TROLL Galaxy CloudMan

30 20 10 0 10

20

30

40

# concurrent jobs Fig. 14. Allocation time for concurrent Galaxy Bowtie jobs using: array of sbatch on colossus, Galaxy on colossus using S TROLL, and Galaxy CloudMan on EC2

VII. C ONCLUSION This paper describes a first experience in the TSD, at the University of Oslo, using Docker containers for software provisioning inside a secure environment for processing sensitive data. Galaxy bioinformatics portal was presented as a use case. Docker is not yet mature in terms of security because of what we have created a wrapper for the Docker command. The use of S TROLL file-system as a proxy to submit jobs from Docker containers to the secure HPC cluster has been described, and the evaluation results show that S TROLL is very efficient in terms of low computational overhead in addition to eliminating the burden of job scripting. VIII. F UTURE WORK DIRECTIONS We are planning to support Docker on the compute cluster, colossus, for running SLURM jobs as Docker containers. To do so, we need to further improve the docker-safe script so that it could read the cgroups configuration of the SLURM job, and assign similar resource shares to the running container. We are also planning to make use of the user namespaces support on Docker 1.10 to prevent any container from running as the root of the host. To implement this, we need to update the Linux kernel on the compute nodes to 3.12+. In addition, we are planning to make better use of the TSD Galaxy, by wrapping docker images as Galaxy tools, benefiting from ongoing efforts [16]. IX. ACKNOWLEDGMENTS This work is funded by the European ELIXIR [28] project, the Norwegian node, and the nordic Tryggve [29] project which is part of NeIC (Nordic e-Infrastructure Collaboration) [30], in addition to USIT (The university Center for Information Technology) at the University of Oslo. R EFERENCES [1] Merkel, D. ”Docker: lightweight linux containers for consistent development and deployment”. Linux Journal 2014.239; 2. 2014. [2] ”Infrastructure for container projects”. https://linuxcontainers.org/. Retrieved 2015-04-22. [3] ”File System”. http://docs.docker.com/terms/filesystem/. Retrieved 201504-22. [4] ”Layers”. https://docs.docker.com/terms/layer/. Retrieved 2015-04-22. [5] Pendry, Jan-Simon; Marshall Kirk McKusick. ”Union Mounts in 4.4BSDLite”. Proceedings of the USENIX Technical Conference on UNIX and Advanced Computing Systems: 2533. December, 1995. [6] Langmead B, Salzberg S. ”Fast gapped-read alignment with Bowtie 2”. Nature Methods. 2012, 9:357-359. [7] ”Docker Security”. http://docs.docker.com/articles/security/. Retrieved 2015-04-22. [8] Lenny Zeltser. ”Security Risks and Benefits of Docker Application Containers”. https://zeltser.com/security-risks-and-benefits-of-dockerapplication/. Retrieved 2015-04-22. [9] Goecks, J, Nekrutenko, A, Taylor, J and TheGalaxy Team. ”Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences”. Genome Biol.2010 Aug 25;11(8):R86. [10] Giardine B, Riemer C, Hardison RC, Burhans R, Elnitski L, Shah P, Zhang Y, Blankenberg D, Albert I,Taylor J, Miller W, Kent WJ, Nekrutenko A. ”Galaxy: a platform for interactive large-scale genome analysis.” Genome Research. 2005 Oct; 15(10):1451-5. [11] ”Big Picture/Choices”. https://wiki.galaxyproject.org/BigPicture/Choices Retrieved 2015-04-22.

[12] Ponstingl H, Ning Z. ”SMALTa new mapper for DNA sequencing reads”. F1000 Posters. 2010; 1:313. [13] Bj¨orn A. Grning. ”Galaxy Docker Image”. https://github.com/bgruening/docker-galaxy-stable. Retrieved 201504-22. [14] MHC Sequencing Consortium (1999). ”Complete sequence and gene map of a human major histocompatibility complex”. Nature 401 (6756): 921923. [15] Di Tommaso, P, Palumbo, E et al. ”The impact of Docker containers on the performance of genomic pipelines”. PeerJ 3. 2015: e1273. [16] Moreews, F, Sallou, O, Bras Y. et al. ”A curated Domain centric shared Docker registry linked to the Galaxy toolshed”. Galaxy Community Conference. 2015. [17] Bui, T. ”Analysis of Docker security”. arXiv preprint arXiv:1501.02967. 2015. [18] Azab, A, Meling, H. ”Stroll: A Universal Filesystem-Based Interface for Seamless Task Deployment in Grid Computing”. Proceedings of the 12th IFIP WG 6.1 International Conference on Distributed Applications and Interoperable Systems (DAIS). 2012. pp 162-176 [19] Azab, A, Meling, H. ”Building composite grid tasks using STROLL filesystem”. Proceeding of The International Conference on IT Convergence and Security. IEEE conference proceedings. ISBN 978-1-4799-2845-3. [20] Olexiy, T, Tonkonog, D. ”Starting Share Files with NFS”. Linux Journal. Jan 01, 2002. [21] ”Tjenester for Sensitive Data 2.0 (TSD 2.0)”. http://www.uio.no/tjenester/it/forskning/sensitiv/mer-om/. [22] ”Colossus: the Computing Cluster for TSD”. http://www.uio.no/english/services/it/research/hpc/colossus/. [23] ”BeeGFS: The Parallel Cluster File System”. http://www.beegfs.com/content/. Retrieved March 16, 2016. [24] ”Running Galaxy Tools on a Cluster”. https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster. Retrieved March 16, 2016. [25] ”Distributed job execution application built for Galaxy”. https://github.com/galaxyproject/pulsar. Retrieved March 16, 2016. [26] Afgan, E, Baker, D et al. ”Galaxy CloudMan: delivering cloud compute clusters”. Proceedings of the 11th Annual Bioinformatics Open Source Conference (BOSC) 2010. [27] Juve, G, Deelman, E. ”Scientific workflow applications on Amazon EC2”. 5th IEEE International Conference on E-Science Workshops. 2009. [28] ”ELIXIR: A distributed infrastructure for life-science information”. https://www.elixir-europe.org/. [29] ”NeIC Tryggve: Collaboration on Sensitive Data”. https://wiki.neic.no/wiki/Tryggve. [30] ”NeIC: Nordic e-Infrastructure Collaboration”. https://neic.nordforsk.org/.