Monthly Archives: January 2017

Stressor – The Container

I have been working on the docker auto-scaling stuff, and got the need of container which could stress the CPU

stress

In Linux, there is stress utility which would stress the CPU, simple and sweet. Now, I need to put this stress utility on container and all get to go. Quick google got me few containers on docker hub, which were already build for this job, Great!

But, I had problems with these. First, they would start stressing the CPU as soon they start, this is not what I need in my scenario. Secondly,  I can’t fire command remotely, i.e. out container and docker host, I don’t have control on when to start, how much CPU to stress and how much time I should let them run

There is another utility, lookbusy, which let me control how much CPU percentage I want to stress. It was important for me, a utility which gives me control on the CPU percentage, say 70% load unlike stress utility, where I need to do trial and error to find what number would stress my CPU to 70%

Second, I had these container running behind a load balancer. I got an idea, where I could simply develop python Flask web app. This would serve Web UI where I could mention the percentage of CPU and time to stress, and in the hood it would use lookbusy to stress the CPU. This away, even without accessing the docker host, I can stress host CPU remotely from a browser.

I created a flask app which would stress my CPU and containerize it. I named it stressor. You can get the flask code from Github.

You can start the container with following command

docker run -p 80:5000 -name stressor sbrakl/stressor

Here, if port 80 is used by other application, you chose whatever available on your machine, say 5000.

This container run flask app on port 5000.

Now, this come in three flavours

flavours-logo

flaskappwithWerkzeug

Flask stress app running on Werkzeug web server. It good for light weight concurrent loads, but bad for 5+ concurrent load

flaskappwithSSL

Same as flaskappwithWerkzeug, but configure to run on SSL. It useful in scenarios, where you need to configure containers behind load balancer. This will test load balancer for SSL traffic.

flaskappwithuwsgi

Flask app is configure to run on the uwsgi and nginx webserver. It configure to run 16 concurrent request.

You can find more information about configuration on Github repository

Advertisements

Container scaling via python script

If you need to monitor docker containers, there are plenty of tools available in market. Some are paid, some are free but you need to do your homework to find which suits you the best.

If you are interest in tools, let me name the few cAdvisor, influx DB, Prometheus.io, Sematext, Universal Control Plane. This is not the definitive list, but go to start with.

For me, it was more of Do-It-Yourself project, where I just needed simple script to monitor CPU and do some scale action base on monitor CPU.

Docker provides an command stats, which provides the stats of container running

docker stats display a live stream of the following container(s) resource usage statistics:

  • CPU % usage
  • Memory usage, limit, % usage
  • Network i/o
  • Disk i/o

The stats are updated every second. Here is a sample output:

CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O               BLOCK I/O
4827f0139b1f        10.94%              706.2 MB / 1.045 GB   67.61%              299.7 kB / 2.473 MB   456 MB / 327.3 MB

I was planing to exploit more this command for CPU monitoring.

Docker engine provides Docker Remote API, which is REST API and can be use communicate to the engine. Being REST API, I can call this in any of my favorite language I want.

Since, I was more in for the scripting, python become the prefer choice of language.

I began my search with python libraries which can connect to Docker. This could be very frustrating. When you search Google, various results show up which refer to different version of docker client, but none EXPLICITLY mentions it. It took couple of days for me to figure it out.

Let me point out few examples

There is tutorial at http://containertutorials.com/py/docker-py.html which says

pip install docker-py

>>> from docker import client
>>> cli = Client(base_url='unix://var/run/docker.sock')

There is official docker client from docker, which says as follow

pip install docker

>>> import docker 
>>> client = docker.DockerClient(base_url='unix://var/run/docker.sock')

Now,  here you see, there are two different API to instantiate client.

Advice would be to read the documentation from Github site. They would be the latest

Coming back to problem, get container stats using Docker python client. I have written my pet shop code to get the docker container CPU using Python client. In the remainder of this article, I would be referencing my script code which you can get from
https://github.com/sbrakl/dockercpumonitor

When I was developing this script, I was writing against stand alone docker engine v 1.11, but I intend to run against docker engine 1.11 with swarm 1.25 on TLS.  I had write a method in the clientConn.py – GetDockerClient, where I can connect to local as well as swarm instance by passing the environment parameter. It interesting to know, how to connect to remote TLS enable docker host by passing client certificates.

If you use docker-compose, there would be problem to get the container name. Compose formats container name with the prefix folder name where compose file resides and suffix of the count of containers. i.e. container name like ‘coolweb’ will translate to ‘foldername_coolweb_1′.  utility.py contains the getContainerInComposeMode method, which get container in by formatting container name with compose pattern.

I know, you would be thinking, code isn’t in best form, but it more of jugglery to get it done rather than making it master piece for the world to see.

Moving forward, of getting the docker stat. It come with another surprise, docker python API doesn’t have stats() method on client object. In fact it has stats() method on the container object. So, basically it means, you can’t get stats like you get with docker stats command which gives stats for all the container running on the docker host. Bad! Even people over the internet express their frustration about docker-py  like in this blog.

Holding our focus, moving back to code to get stats about the container. In the utility.py, get_CPU_Percentage method, you will get code get container stats

# 'con' is container which you need to monitor
constat = con.stats(stream=False)

stream=false mean, you will get stats for just one time, and not a continues stream object.

It will give back json object something like this below. It large object, but I have just highlighted the CPU related stuff

{
    "read": "2016-02-07T13:26:56.142981314Z",
    "precpu_stats": {
        "cpu_usage": {
            "total_usage": 0,
            "percpu_usage": null,
            "usage_in_kernelmode": 0,
            "usage_in_usermode": 0
        },
        "system_cpu_usage": 0,
        "throttling_data": {
            "periods": 0,
            "throttled_periods": 0,
            "throttled_time": 0
        }
    },
    "cpu_stats": {
        "cpu_usage": {
            "total_usage": 242581854769,
            "percpu_usage": [242581854769],
            "usage_in_kernelmode": 33910000000,
            "usage_in_usermode": 123040000000
        },
        "system_cpu_usage": 3367860000000,
        "throttling_data": {
            "periods": 0,
            "throttled_periods": 0,
            "throttled_time": 0
        }
    },
    "memory_stats": {
        ...
        },
        "failcnt": 0,
        "limit": 1044574208
    }

precpu_stats are CPU stats before point of reference, say 10 sec. cpu_stats are stats at point in time. If you look more into get_CPU_Percentage method, it juggles from the JSON object, get relevant variables and computes the percentage CPU for the container.

Once I get the CPU in percentage, I have put it in array at the interval of 2 sec. It a fix width array with 5 slots. So, it hold only last 5 reading, i.e. last 10 secs reading in term of time

Then I compute the mean of of array to get mean CPU, so it rules out uneven CPU spikes. I take this mean CPU against CPU threadhold, i.e. 50%. If mean CPU is more than 50%, it will trigger scale out action. If it’s less than 50, it will trigger scale down action

The entire logic to scaling up and down with cooling time in between is in ScaleContaienr method of utility.py.

These all methods are called from main.py, which runs the code in loop.

That’s it. It brings me to the end of Do-It-Yourself project of docker scaling. I know, it’s not the ultimate script when it come to container scaling.

Client Auth with Interlock and Nginx

I had the requirement of setting Interlock + Nginx where backend expects client authentication. If you have directly landed up here, to get the context about service discovery and interlock and Nginx, read my previous post.

Note: This topic applies to Interlock 1.3 and Docker 1.11. If you are using docker > 1.12, I recommended to use inbuilt docker load balancer, which ships with swarmkit.

Problem Definition:

Setup client authentication certificates with Interlock + Nginx

Why it a problem: 

Interlock controls the Nginx configuration file. You can’t directly modify the Nginx configuration file, as Interlock would be overwrite when a container starts or dies.

Interlock allows certain data labels which allows you to configure the Nginx configuration file. Read Interlock data label section of previous post for more info.

There are data label to set SSL certificate, set SSL only, set SSL backend, etc. But, there aren’t any labels to set SSL proxy certificate. I had eve raise an issue, to found it not supported.

No data label to configure client authentication certificates is the problem

Possible Solution

If you need to set client authentication certificates with Nginx, serverfault threads hints how to do

backend {
 server some-ip:443;
}

server {
 listen 80;


   location / {
      proxy_ssl_certificate certs/client.crt;
      proxy_ssl_certificate_key certs/client.key;

      proxy_pass https://backend;
   }
}

Now, I need to find a way with Interlock, where I could get control of template it uses for configuring the interlock

Hint’s from the interlock docs, where it shows configuration variable TemplatePath  in the toml configuration. It allows us to give the template, which it would use with variable substitution to create final Nginx config.

Again, I can get the example of this template file in interlock docs.

I found this template file a perfect opportunity to modify the template to include client auth certificates in template and use.

 location / {
 # Added by Shabbir 9th Dec 2016, For Client Authentication
 
 proxy_ssl_certificate /certs/client.crt;
 proxy_ssl_certificate_key /certs/client.key;
 proxy_ssl_password_file /certs/pass.txt;
 # Change End
 
 {{ if $host.SSLBackend }}proxy_pass https://{{ $host.Upstream.Name }};{{ else }}proxy_pass http://{{ $host.Upstream.Name }};{{ end }}
 }

This certificate needs to be present on the machine where Nginx container would be launch, and they are added to container via volume mounts.

Here, the extract of docker-compose file, which configures nginx container

nginx:
   image: nginx:latest
   entrypoint: nginx
   networks:
     - common     
   ports:
     - 8443:8443
     - 8009
   depends_on:
     - interlock
   command: -g "daemon off;" -c /etc/nginx/nginx.conf
   restart: unless-stopped
   labels:
       interlock.ext.name: nginx
   environment:
       - "constraint:com.function==interlock"
   volumes:
       - ~/myapp/certs:/certs
       - ~/myapp/logs:/var/log/nginx

This is how I solve the issue of client authentication, but this technique could be use to configure interlock for all the unsupported Nginx scenarios like tcp pass through, etc

 

 

 

Service discovery sauce with Interlock and Nginx

Poster Information: If you need to know more about the Service discovery, what options are available and how things change in docker 1.12, check out my service discovery post.

This article applies to docker standalone swarm compare to docker 1.12 swarmkit, where docker engine has integrated swarm mode.

Note: This point forward, in this article, I am referring to swarm, I am referring to standalone swarm and not swarmkit one.

I started the work in Docker 1.11 era, when the swarm was a separate tool from the docker engine, and you need to launch a couple of swarm containers to setup the swarm cluster.

IMHO, interlock + Nginx is poor man’s tools in terms of service discovery. There would be better options available, but for me, it all started with taking a look at swarm at scale example at the docker site. They have shown how to use interlock with Nginx for load balancing and service discovery. Not knowing much on service discovery, and having working example demonstrate was good enough for me to engage with interlock.

Interlock is swarm event listener tool, which listens to the swarm events and performs a respective action on the extension. As of current (Dec 2016), it supports only two extensions, Nginx and HAProxy. Both acts as service discovery and load balancer. There are another extension planned called beacon, which would be used for monitoring and autoscaling perhaps, but now seem to be abandon, thanks to docker 1.12 swarmkit

In simple terms, there are three actors in the play. Swarm manager, Interlock and Nginx acting as a load balancer. Best part, all three runs as the docker container. It means no installation/configuration at host VM and easy to set them up.

Interlock listens to swarm master for start or stop container events. When it hear something about it, it updates the Nginx config. Below animated diagrams explains it better

Interlock.gif

Interlock play in action

Now, we know “what” part of the Interlock, let move towards the “how” part. Unfortunately, there aren’t much documentation on the interlock available on the net. Interlock QuickStart guide provides some clue, but it missing the enterprise part. It doesn’t guide much if you are using docker-compose with the multi-host environment.

For later part, you can draw some inspiration from the Docker at scale, and there is obsolete lab article which shows how to use interlock with docker-compose, but the interlock commands are obsolete for 1.13 (was latest in Oct 2016) version.

I am not planning to write entire to-do article for interlock, but intent to hints some useful when running is multi-host docker cluster with interlock.

The first part is docker swarm cluster, articles like Docker at scale and codeship used docker-machine to create an entire cluster on your desktop/laptop.  I have been more privilege to use R&D cloud account and use Azure Cloud to create my Docker cluster. You use tools like UCP, docker machine, ACS, docker cloud and many other are there in the market or just create cluster manually handheld. It doesn’t matter, where you run your cluster and how did you create it, as long you have working swarm cluster, you are good to play with Interlock and Nginx

Another piece of advice, while setting up swarm cluster, it not mandatory, when good practice to have a dedicated host for interlock and Nginx container. You can see swarm at scale article, where they use docker engine labels to tag particular host for tagging.

If you are docker-machine, you can give docker engine labels similar to below

Docker engine labels.png

And in the docker-compose.yml file, you would specify the contrains that container would be load at com.function=interlock

interlock-compose

Interlock in docker-compose.yml file

Now, in order to prepare interlock sauce, you need following ingredients to set it right

  1. Interlock Config (config.toml)
  2. Interlock data labels
  3. (Optional) SSL certificates

Interlock Configuration File

Interlock uses a configuration store to configure options and extensions. There are three places where this configuration can be saved

1) File,  2) Environment variable or 3) Key value store

I find it convenient to store it in a file. This file by Interlock convention is named as config.toml.

Content: This file contains key-value options which are use by interlock and it’s extension. For more information, you can see https://github.com/ehazlett/interlock/blob/master/docs/configuration.md

Location: If you are running Swarm on multi-host, this file needs to be present  on a VM which will host interlock container. You can then mount this file to a container by volume mapping. See docker-compose file above for more info

TLS Setting: If you are running Swarm on TLS, you need to set TLSCACert , TLSCert, TLSKey variable in toml file. For info, read setting Docker on TLS and Swarm on TLS.

TLSCACert = “/certs/ca.pem”
TLSCert = “/certs/cert.pem”
TLSKey = “/certs/key.pem”

Plus, this certificate needs to be present on a VM which will host interlock container. You can then mount this certificates via volume mount in the compose file. See docker-compose file above for example

PollInterval: If your interlock is not able to connect to Docker swarm, try setting the PollInterval to 3 seconds. In some environments, the event stream can be interrupted and hence Interlock need to rely on pooling mechanism

Interlock Data Label

Now, we have just setup Interlock with Nginx. If you have carefully observe the config.toml file, nowhere we have given which container we need to load balance. Then, how would interlock get this information from?

This brings us to the Interlock Data Labels. It the set of labels you pass to the container, which when Interlock inspect, would know, which containers it needs to load balance.

Here below example shows how to pass Interlock label along with other container labels.

interlockdata

Example of Interlock Data labels in docker-compose.yml

You can get more information about the data labels at https://github.com/ehazlett/interlock/blob/master/docs/interlock_data.md

There is another example from Interlock repo itself, where it how to launch interlock with Nginx as load balancer in docker swarm via compose.

https://github.com/ehazlett/interlock/blob/master/docs/examples/nginx-swarm/docker-compose.yml

(Optional) SSL certificates

As seen the above Interlock label, there are lot of interlock variable related to SSL.

To understand better, we will enumerate to different combinations with SSL, we can setup load balancer

I) NO SSL

You can have flow something like this

OnlyHTTP.png

HTTP Only

Here, we are not using SSL at all.

II) SSL Termination at the Load Balancer 

Or you if you planning to use Nginx as the frontend internet facing load balancer, you should do something like this

HTTPS Termination.png

SSL Termination at the load balancer

III) Both lags with SSL

In my case, there was compliance requirement where all the traffic, internal or external needs to be SSL/TLS. So, I need to do something like this

HTTPS Only.png

HTTPS only traffic

For Case II and III, you need to set interlock SSL related data label. Let me give quick explanation of important ones

interlock.ssl_only : It you want your load balancer to list to HTTPS traffic only, set this to true. If false, the interlock configure Nginx to listen to both, HTTP and HTTPS. If true, then it set redirection rule in HTTP to redirect it to HTTPS

interlock.ssl_cert: This needs to be the X509 certificate path which load balancer will use to server frondend traffic. This certificate Common Name equal to load balancer name.  Plus, in multi-host environment, this certificates needs to be present on the machine which launch the Nginx container. You can then mount this file to a container by volume mapping. See docker-compose file above for more info

interlock.ssl_cert_key: Private key from X509 certificate. Same goes with key, it needs to be on the VM which will run Nginx container.

If your backend requires certificate client authentication, as it was in my case, then interlock has no support for it. But, there is a hack to SSL proxy the certificates. But, that for the another post.

Hope, information I share with you was useful. If you want any help, do write in comments below