Mesos
Mesos is a fault-tolerant cluster manager which consists of a master daemon that manages the slave daemons running on each node in a cluster. The master also manages tasks running on these slaves. The master manages the physical resources across these tasks by accepting resource offers from slave. A framework running on top of Mesos consists of two components: a scheduler that registers with the master and an executor process that is launched on slave nodes to run the framework’s tasks. Mesos uses ZooKeeper as its configuration manager to coordinate activities across a cluster. More information on Mesos can be found at http://mesos.apache.org
.Our goal was to explore ways of assigning fine grain resources (like ports) to a specific slave and run a Docker daemon on each of this slave for isolation. We used Marathon (https://github.com/mesosphere/marathon
) as our scheduler.
Here is our deployment
We have two hosts running Linux Ubuntu; the master and the marathon scheduler runs in 10.13.216.113 and the slaves in 10.13.216.214. There are two Docker daemons running on this 10.13.216.214 as well. We would like to use marathon to schedule a task which will launch a web application called “outyet” inside Docker.
Starting two Docker daemons in the same host
We wanted to deploy our application “outyet” on separate Docker daemons for each slave in 10.13.216.214.Before we explain our motivation as to why we need two separate Docker daemons and how we start multiple daemons, we need to understand the internals of Docker. A good explanation of Docker architecture and its components can be found at https://docs.docker.com/introduction/understanding-docker/
.- Docker uses a client-server architecture, the client is the CLI to connect to the server which is the Docker daemon.
- A Docker client can connect to a Docker daemon running locally or to a remote host.
- The Docker images, registries and containers are the main components of a Docker daemon.
- A Docker image is a template which contains the operating system, the web server etc for an application.
- A Docker container is created from the Docker image, a Docker container container is the runtime component of Docker.
- Docker registry hold images, a registry could be public or private. When a Docker client issues the run command, Docker checks the presence of the image in its local registry and if not available pulls it from the configured public or private registry.
- A Docker image is read-only; when Docker runs a container from an image, it adds a read-write layer on top of the image in which it can run the application for the image.
- Isolation : Docker takes advantages of a technology called namespaces to provide isolation; so when we run a container Docker creates a namespace for that container. By running a dedicated Docker daemon for each Mesos slave, we add another level of isolation, so each Docker daemon has its own set of constraints, resources and data.
- Multi-Tenancy : Isolation and Multi-Tenancy go hand in hand; isolation ensures good Multi-Tenancy. Since each tenant runs on its own Docker daemon, tenants do not have access to each other file system which adds to another layer of security.
- Microservices : A tenant can have its own registry and can restrict access to its registry from other tenants as each Docker daemon can have it own private or public registry.
Starting Docker Daemons
As we have seen from the Docker architecture, running two Docker daemons on the same host requires its own set of sockets to bind, files and directories to persist its filesystem and registries. Docker provides additional flags like host, graph and pidfile to facilitate running multiple Docker daemons on the same host.
sudo docker --daemon=true --host=unix:///var/run/docker_a.sock --graph=/var/lib/docker_a --pidfile=/var/lib/docker_a_pid
The --host flag is the socket to which the daemon has to bind to; if we are running multiple daemons, we need have to provide an unique socket identifier (Unix automatically creates a file descriptor for a socket when its created so that IO operations can be performed on the socket just as with normal file descriptors).
The --graph specified the path to use as the root of the Docker runtime. By default is /var/lib/docker, but if we are running multiple daemons, we have to specify a unique directory for each daemon. Each docker daemon maintains its own registry, filesystems, volume etc. in this directory.
The --pidfile is the file the process id is written to, must be unique so that one does not overwrite the other.
Similarly, let us start another Docker daemon:
sudo docker --daemon=true --host=unix:///var/run/docker_b.sock --graph=/var/lib/docker_b --pidfile=/var/lib/docker_b_pid
Starting Mesos master.
sudo <MESOS_INSTALL_DIR>/bin/mesos-master.sh --ip=10.13.216.113
--work_dir=/var/lib/mesos --quorum=1 --zk=zk://127.0.0.1:2181/mesos
--work_dir is the directory where mesos stores persistent information.
--quorum Number of replicas.
--zk zookeeper URL, we need this as we are running a cluster.
Starting Mesos slave.
Now let us start Mesos slaves in our slave node 10.13.216.214
sudo ./mesos-slave.sh --log_dir=/var/log/mesos1 --port=5051
--hostname=sclq214-1 --work_dir=/tmp/mesos1
--master=10.13.216.113:5050 --containerizers="docker"
--executor_registration_timeout="5mins"
--resources='ports:[21000-24000]'
--docker_socket="/var/run/docker_a.sock" --ip=10.13.216.214
--attributes='slave:a'
Please note that the support for specifying a docker socket (--docker_socket) will not be available until Mesos .25 version.
--log_dir : The directory where Mesos slave writes its logs.
--port: Port to listen on
--hostname: The hostname the slave should report. If left unset, the hostname is resolved from the IP address that the slave binds to.
--work_dir: Directory path to place Mesos work directories
--master: The IP address of the master or masters
--containerizers: To run the slave to enable the Docker Containerizer, you must launch the slave with “docker” as one of the containerizers option.
--executor_registration_timeout: Amount of time to wait for an executor to register with the slave before considering it hung and shutting it down
--resources: The Mesos slave offers this (21000-24000) range of ports.
--docker_socket: The socket of Docker daemon which the executor of the slave will connect to for executing the task.
--ip: IP address of the slave.
--attributes: Attributes that can be passed to the slave.
Similarly let us start another Mesos slave in the same node
sudo ./mesos-slave.sh --log_dir=/var/log/mesos2 --port=5052
--hostname=sclq214-2 --work_dir=/tmp/mesos2
--master=10.13.216.113:5050 --containerizers="docker"
--executor_registration_timeout="5mins"
--resources='ports:[21000-24000]'
--docker_socket="/var/run/docker_b.sock" --ip=10.13.216.214
--attributes='slave:b'
Executing a Marathon application in Mesos
For this work we chose a sample Docker web application called "outyet" (http://open.mesosphere.com/intro-course/ex12.html). The "outyet" image has to be built and loaded in each of the docker daemon; more instruction is available in the above link.
sudo docker -H unix:///var/run/docker_a.sock build -t outyet .
sudo docker -H unix:///var/run/docker_b.sock build -t outyet .
sudo docker -H unix:///var/run/docker_b.sock build -t outyet .
curl -i -H 'Content-Type: application/json' -d @outyet.json localhost:8080/v2/apps
{
"id": "outyet",
"cpus": 0.2,
"mem": 20.0,
"instances": 1,
"constraints": [["slave", "CLUSTER", "a"]],
"container": {
"type": "DOCKER",
"docker": {
"image": "outyet",
"network": "BRIDGE",
"portMappings": [{ "containerPort": 8080, "hostPort": 21001, "servicePort": 0, "protocol": "tcp" }]
}
}
}
constraints: The constraint here forces Mesos to execute the task in "slave" "a"; the slave was started with these attributes.
Marathon Console
We can view the status of our apps ("outyet") from the Marathon console.
How do we test if the app has been deployed and if the app is running?
To test the app ("outyet"), we first need the port where the app is running. We can use the following command to figure this out.
sudo docker --host unix:///var/run/docker_a.sock ps
CONTAINER ID IMAGE COMMAND CREATED
STATUS PORTS NAMES
58c253680cd7 outyet "go-wrapper run" 2 weeks ago Up 2 weeks 0.0.0.0:23705->8080/tcp mesos-20150814-181831-1909984522-5050-12612S9.23dd487c-01de-4d02-8e20-0deaca20c193
58c253680cd7 outyet "go-wrapper run" 2 weeks ago Up 2 weeks 0.0.0.0:23705->8080/tcp mesos-20150814-181831-1909984522-5050-12612S9.23dd487c-01de-4d02-8e20-0deaca20c193
The output will look like:
<!DOCTYPE html><html><body><center>
<h2>Is Go 1.4 out yet?</h2>
<h1>
<a href="https://go.googlesource.com/go/+/go1.4">YES!</a>
</h1>
</center></body></html>
<h2>Is Go 1.4 out yet?</h2>
<h1>
<a href="https://go.googlesource.com/go/+/go1.4">YES!</a>
</h1>
</center></body></html>
Conclusion and Future work
In this exercise we were able to demonstrate how we can assign fine grained resources like ports to a Mesos slave and run a simple Docker web application image ("outyet") using a Mesos Docker executor and scheduling it with a Marathon scheduler. We also demonstrated how we can run multiple Docker daemons on a single host and how we could map each Docker daemon to a Docker executor thereby providing Isolation and Multi-Tenancy in a Mesos/Docker environment.We should be able to extend this approach to provision a ScaleIO volume to a Docker container and guard-rail them using ScaleIO's "Protection Domains" thereby improving the characteristics of Isolation and Multi-Tenancy to Docker.
