Thursday, August 25, 2022

Docker 101 - Docker Compose with Environment Variables

 


Let's modify the previous exp 'Flask + Redis' to make environment variables to include some sensitive data (password)



Flask + Redis (require password)





docker-compose.yml
version: "3.9"
services:
  web:
    build:
      context: ./
      dockerfile: Dockerfile
    ports:
      - "5000:5000"
    volumes:
       - .:/code
     environment:
       # Setup a sensitive environment variable
       - REDIS_PASSWORD=my_pwd
     networks:
       - my-bridge
   redis-server:
     image: "redis:alpine"
     # Run server with password
     command: redis-server --requirepass my_pwd
     networks:
       - my-bridge
 networks:
   my-bridge:

First, we need to override the command of 'redis-server' service to make sure the redis server is protected by password.

Second, in 'web' service, we define redis password as an environment variable called 'REDIS_PASSWORD' for app.py to use.

app.py
import time
import os
import redis
from flask import Flask

app = Flask(__name__)

# get redis passwrod from environment variables
redis_password = os.environ["REDIS_PASSWORD"]
# Using StrictReis with password
cache = redis.StrictRedis(host="redis-server", port=6379,
password=redis_password)


def get_hit_count():
    retries = 5
    while True:
        try:
            return cache.incr("hits")
        except redis.exceptions.ConnectionError as exc:
            if retries == 0:
                raise exc
            retries -= 1
            time.sleep(0.5)


@app.route("/")
def hello():
    count = get_hit_count()
    return "Hello World! I have been seen {} times.\n".format(count)

In order to connect to a redis server with password, we need to use StrictRedis.
And we pass the redis password by reading environment variable we set in container instead of hard-code it in app.py.

Then after running 'docker compose up', we can see the same result like the previous exp.

But if we share this docker-compose.yml file to others, then they will know our redis credential.

In order to deal with this situation, docker compose allow us to setup default values for environment variables in .env file. Reference

First of all, create a .env file under the parent directory of docker-compose.yml.

.env
REDIS_PASSWORD=my_pwd

docker-compose.yml
version: "3.9"
services:
  web:
    build:
      context: ./
      dockerfile: Dockerfile
    ports:
      - "5000:5000"
    volumes:
       - .:/code
     environment:
# Bind shell/env_file environment variables to containers'
       - REDIS_PASSWORD=${REDIS_PASSWORD}
     networks:
       - my-bridge
   redis-server:
     image: "redis:alpine"
     # Run server with password
# (which is bound to shell/env_file environment variables )
     command: redis-server --requirepass ${REDIS_PASSWORD}
     networks:
       - my-bridge
 networks:
   my-bridge:

In docker-compose.yml, we can substitute variables by shell/env_file environment variables.

Then we can see the same result in browser.

Also, we can use the following command to see the value after impetrated by .env.

Ex
$ docker compose convert

Result:
name: compose-env-exp-2
  services:
    redis-server:
        command:
        - redis-server
        - --requirepass
        - my_pwd
        image: redis:alpine
        networks:
          my-bridge: null
    web:
        build:
          context: /home/fcheng/Exp/compose-env-exp-2
          dockerfile: Dockerfile
        environment:
          REDIS_PASSWORD: my_pwd
        networks:
          my-bridge: null
        ports:
        - mode: ingress
          target: 5000
          published: "5000"
          protocol: tcp
        volumes:
        - type: bind
          source: /home/fcheng/Exp/compose-env-exp-2
          target: /code
          bind:
            create_host_path: true
    networks:
      my-bridge:
        name: compose-env-exp-2_my-bridge

Also, we can define multiple .env file such as dev.env, staging.env and prod.env and using --env-file option to load it when running 'docker compose up'

For example, let's create a file called prod.env.

prod.env
REDIS_PASSWORD=my_prod_pwd

And we can compare the result between the following two commands.

Ex
$ docker compose convert

Result:
  name: compose-env-exp-2   services:   redis-server: command: - redis-server - --requirepass - my_pwd image: redis:alpine networks: my-bridge: null   web: build: context: /home/fcheng/Exp/compose-env-exp-2 dockerfile: Dockerfile environment: REDIS_PASSWORD: my_pwd networks: my-bridge: null ports: - mode: ingress target: 5000 published: "5000" protocol: tcp volumes: - type: bind source: /home/fcheng/Exp/compose-env-exp-2 target: /code bind: create_host_path: true   networks:   my-bridge: name: compose-env-exp-2_my-bridge

Ex
$ docker compose --env-file prod.env convert

Result:
name: compose-env-exp-2 services: redis-server: command: - redis-server - --requirepass - my_prod_pwd image: redis:alpine networks: my-bridge: null web: build: context: /home/fcheng/Exp/compose-env-exp-2 dockerfile: Dockerfile environment: REDIS_PASSWORD: my_prod_pwd networks: my-bridge: null ports: - mode: ingress target: 5000 published: "5000" protocol: tcp volumes: - type: bind source: /home/fcheng/Exp/compose-env-exp-2 target: /code bind: create_host_path: true networks: my-bridge: name: compose-env-exp-2_my-bridge

Saturday, August 13, 2022

Docker 101 - What is Docker Compose?



Before introducing Docker Compose, let's use the following example to see how to create multiple containers to build a web server with a database.



Flask + Redis





First of all, let's create an user-define bridge network.



Ex: 
$ docker network create -d bridge my-network

Result:
b91e945e5f1b3ef128fbb275d6f9430eca3aaba9af0a55cf4bb123a24d28b889

Ex: 
$ docker network ls

Result:
NETWORK ID     NAME         DRIVER    SCOPE
5b906668a27e   bridge       bridge    local
  f419db6a9e21   host         host      local
  b91e945e5f1b   my-network   bridge    local
  0175d39fdfe6   none         null      local


Secondly, let's create an Redis Server Container (by the official image)



Ex: 
$ docker image pull redis:alpine

Result:
alpine: Pulling from library/redis
  213ec9aee27d: Already exists
  c99be1b28c7f: Pull complete
  8ff0bb7e55e3: Pull complete
  6d80de393db7: Pull complete
  8dbffc478db1: Pull complete
  7402bc4c98a0: Pull complete
  Digest: sha256:dc1b954f5a1db78e31b8870966294d2f93fa8a7fba5c...
  Status: Downloaded newer image for redis:alpine
  docker.io/library/redis:alpine

Ex: 
$ docker image ls

Result:
REPOSITORY   TAG          IMAGE ID       CREATED        SIZE
  python       3.7-alpine   cd2a4f346519   30 hours ago   45.6MB
  redis        alpine       9192ed4e4955   19 seconds ago 28.5MB

Ex: Then run an redis:alpine container using my-network bridge
$ docker container run -d \
--name redis-server \
--network my-network redis:alpine

Result:
337382d14e9b5af557b9414ba540a45a53d0559e8dee0ad09e2f7c82a99e49a3


In the end, let's create a Flask Web Server Container (By our Dockerfile)



app.py
import time
import redis
from flask import Flask

app = Flask(__name__)
cache = redis.Redis(host="redis-server", port=6379)


def get_hit_count():
    retries = 5
    while True:
        try:
            return cache.incr("hits")
        except redis.exceptions.ConnectionError as exc:
            if retries == 0:
                raise exc
            retries -= 1
            time.sleep(0.5)


@app.route("/")
def hello():
    count = get_hit_count()
    return "Hello World! I have been seen {} times.\n".format(count)


Dockerfile
FROM python:3.7-alpine
WORKDIR /code
ENV FLASK_APP=app.py
ENV FLASK_RUN_HOST=0.0.0.0
RUN apk add --no-cache gcc musl-dev linux-headers
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
EXPOSE 5000
COPY . .
CMD ["flask", "run"]

requirements.txt 
flask
redis

Ex: Build an image based on Dockerfile
$ docker image build -t my-flask .
$ docker image ls

Result:
REPOSITORY   TAG          IMAGE ID       CREATED          SIZE
  my-flask     latest       61471500f3d3   19 seconds ago   182MB
  python       3.7-alpine   cd2a4f346519   31 hours ago     45.6MB
  redis        alpine       9192ed4e4955   1 hours ago     28.5MB

Ex: 
$ docker container run -d \
--network my-network \
--name flask-server \
--env REDIS_HOST=redis-server \
-p 5000:5000 my-flask

Result:
06f350b4c12431665b3a59581fff835bf663123b32f9b70419d844bd127433e5


Once it is all set, then if we keep refreshing our browser, then we can see the count is increasing.



From the example above, we notice that if we want to build a simple backend service, then we need lots of steps to make it happen. 

    1. Create networks
    2. Prepare images
    3. Create containers

Docker Compose is invented to reduce those steps.


Docker Compose



Compose is a tool for defining and running multi-container Docker applications.
With Compose, you use a YAML file to configure your application’s services.
Then, with a single command, you create and start all the services from your configuration.

A sample docker-compose.yml:
version: "3.9" # optional since v1.27.0
services: # put containers inside this services block
    web: # container_1 (its name is web)
        build: .
# Using the image built by local Dockerfile

        environment:
# docker container run --env FLASK_DEBUG=1 my-image

        command:
# docker container run -it my-image echo 'Hi there'

        networks:
# docker container run --network my-bridge my-image

        ports:
# docker container run -p 5000:5000 my-image
            - "5000:5000"

        volumes:
# docker container run \
# --mount type=bind,srouce=.,target=/code my-image
            - .:/code

        depends_on:
# control the order of service startup and shutdown
            - redis-server

    redis-server: # container_2 (its name is redis-server)
        image: redis # name of image

volumes:
# docker volume create

networks:
# docker network create my-bridge


Let's follow the official document to build 'Flask and Redis' by docker compose instead  of launching multiple containers by ourselves. Reference

docker-compose.yml
version: "3.9"
services:
    web:
        build:
            context: ./
# Change Dockerfile to Dockerfile.exp for demo purpose
            dockerfile: Dockerfile.exp
        ports:
            - "5000:5000"
        volumes:
            - .:/code
        environment:
            FLASK_DEBUG: 1
        networks:
            - my-bridge
    redis-server:
        image: "redis:alpine"
        networks:
            - my-bridge
networks:
    my-bridge:

Ex:
$ docker image ls

Result:
REPOSITORY   TAG          IMAGE ID       CREATED      SIZE
python       3.7-alpine   cd2a4f346519   4 days ago   45.6MB
  redis        alpine       9192ed4e4955   6 days ago   28.5MB

Ex:
$ docker network ls

Result:
NETWORK ID     NAME      DRIVER    SCOPE
  72bf167e3207   bridge    bridge    local
  f419db6a9e21   host      host      local
  0175d39fdfe6   none      null      local

Ex:
$ docker volume ls

Result:
DRIVER    VOLUME NAME

Ex:
$ docker container ls -a

Result:
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Ex:
$ docker compose up -d

Result:
[+] Building 0.2s (11/11) FINISHED
=> [internal] load build definition from Dockerfile.exp                
=> => transferring dockerfile: 295B                                    
=> [internal] load .dockerignore                                        
=> => transferring context: 2B                                          
=> [internal] load metadata for docker.io/library/python:3.7-alpine    
=> [1/6] FROM docker.io/library/python:3.7-alpine                      
=> [internal] load build context                                        
=> => transferring context: 809B                                        
=> CACHED [2/6] WORKDIR /code                                          
=> CACHED [3/6] RUN apk add --no-cache gcc musl-dev linux-headers      
=> CACHED [4/6] COPY requirements.txt requirements.txt                  
=> CACHED [5/6] RUN pip install -r requirements.txt                    
=> [6/6] COPY . .                                                      
=> exporting to image                                                  
=> => exporting layers                                                  
=> => writing image sha256:8ec105e7b2595c073fdadf3744651bf07761629fba248.
=> => naming to docker.io/library/compose-exp_web                      

Use 'docker scan' to run Snyk tests against images to find
vulnerabilities and learn how to fix them
[+] Running 3/3
⠿ Network compose-exp_my-bridge         Created                        
⠿ Container compose-exp-redis-server-1  Started                        
⠿ Container compose-exp-web-1           Started

Then after hitting '0.0.0.0:5000', then we can run the following command to see some logs.

Ex:
$ docker compose logs container web

Result:
172.19.0.1 - - [16/Aug/2022 08:56:36] "GET / HTTP/1.1" 200 -

Ex:
$ docker image ls

Result:
REPOSITORY        TAG          IMAGE ID       CREATED          SIZE
  compose-exp_web   latest       96e3212ff705   16 minutes ago   182MB
  python            3.7-alpine   cd2a4f346519   4 days ago       45.6MB
  redis             alpine       9192ed4e4955   6 days ago       28.5MB

Ex:
$ docker compose ps

Result:
NAME                         COMMAND                  SERVICE          
  compose-exp-redis-server-1   "docker-entrypoint.s…"   redis-server        
  compose-exp-web-1            "flask run"              web                

STATUS               PORTS
running             6379/tcp
running             0.0.0.0:5000->5000/tcp, :::5000->5000/tcp

NOTE: compose-exp is our project directory.

Ex:
$ docker network ls

Result:
NETWORK ID     NAME                    DRIVER    SCOPE
  72bf167e3207   bridge                  bridge    local
  770579e9fc81   compose-exp_my-bridge   bridge    local
  f419db6a9e21   host                    host      local
  0175d39fdfe6   none                    null      local

Ex:
$ docker volume ls

Result:
DRIVER    VOLUME NAME
  local     41935893e2f387d51dcc56a2eea8ca638c4ab813d26aa0ae...