dockerfile tips
caching
For an efficient use of the caching mechanism , we need to place the instructions for layers that change frequently after the ones that incur less changes. application’s dependencies change less frequently than the Python code. Therefore we copy the dependencies file and install them and then we copy the source code
# copy the dependencies file to the working directory
COPY requirements.txt .
# install dependencies
RUN pip install -r requirements.txt
# copy the content of the local src directory to the working directory
COPY src/ .
BuildKit’s new caching
you can cache a directory across builds with Buildkit(normally each build is its own self-contained little filesystem)
# syntax = docker/dockerfile:1.2
FROM python:3.9-slim-buster
COPY requirements.txt .
#caching the /root/.cache directory, since that is also where Pipenv and Poetry will store their files; pip uses ~/.cache/pip by default
RUN --mount=type=cache,target=/root/.cache \
pip install -r requirements.txt
# ... etc. ...
during docker build you need to activate buildkit with
export DOCKER_BUILDKIT=1
the limitation with buildkit is that BuildKit caching would not work in cloud docker building, since cloud CI service starts with a new environment every time
Multi-stage builds
using multi-stage builds can strip the final application image of all unnecessary files and software packages and deliver only the files needed to run our Python code.
# first stage
FROM python:3.8 AS builder
COPY requirements.txt .
# install dependencies to the local user directory (eg. /root/.local)
RUN pip install --user -r requirements.txt
# second unnamed stage
FROM python:3.8-slim
WORKDIR /code
# copy only the dependencies installation from the 1st stage image
COPY --from=builder /root/.local/bin /root/.local
COPY ./src .
# update PATH environment variable
ENV PATH=/root/.local:$PATH
CMD [ "python", "./server.py" ]
Compose file tips
to start up all services docker-compose up -d to stop and remove all project containers docker-compose down to rebuild images docker-compose build to check logs docker-compose logs app
Network separation
to avoid all service in same default network, we need to define separate networks for unrelevant components
Docker Volumes
To persist DB data between different containers, we can exploit named volumes.
Docker Secrets
version: "3.7"
services:
db:
image: mysql:8.0.19
command: '--default-authentication-plugin=mysql_native_password'
restart: always
secrets:
- db-password
volumes:
- db-data:/var/lib/mysql
networks:
- backend-network
environment:
- MYSQL_DATABASE=example
- MYSQL_ROOT_PASSWORD_FILE=/run/secrets/db-password
app:
build: app
restart: always
secrets:
- db-password
networks:
- backend-network
- frontend-network
web:
build: web
restart: always
ports:
- 80:80
networks:
- frontend-network
volumes:
db-data:
secrets:
db-password:
file: db/password.txt
networks:
backend-network:
frontend-network:
Applying Code Updates & debugging
update code updates
during debugging, we can bind-mount the source code to container, so that we can debug code without rebuild image and redeploy container
app: build: app restart: always volumes: - ./app/src:/code
A reloader watches all the source code files and automatically restarts the server when detects that a file has changed. To enable the debug mode we only need to set the debug parameter as below:
#server.py
server.run(debug=True, host='0.0.0.0', port=5000)
debugging
- map locally the port to the debugger app: build: app restart: always volumes: - ./app/src:/code ports: - 5678:5678
-
import dubugger modul vs code support ptvsd
import ptvsd ptvsd.enable_attach(address=(‘0.0.0.0’, 5678))
-
create remote launch
#in launch.json file { “version”: “0.2.0”, “configurations”: [ { “name”: “Python: Remote Attach”, “type”: “python”, “request”: “attach”, “port”: 5678, “host”: “localhost”, “pathMappings”: [ { “localRoot”: “${workspaceFolder}/app/src”, “remoteRoot”: “/code” } ] } ] }
docker volume
docker is based on Union File System (changes lives on docker layer, if container is deleted, changes are lost), docker volume can help persistent data storage, sharing data betwenn docker containers and seperation of data and container
init volume
if no host path is given, default path is /var/lib/docker. the mount path will be created there
#bind-mounts /tmp/data1 from the host on to docker /tmp/data2
docker run -itd --volume /tmp/data1:/tmp/data2 --name container-test ubuntu bash
#mount volume without specifying host path
docker run -v /some/dir
#mount volume using Dockerfile
VOLUME /tmp/data1
create volume
docker volume create portainer_data
inspect volume
#inspect setting of the volume
docker inspect --format='' container-test | python -m json.tool
data sharing
docker run --name my_container -v /some/path ...
docker run --volumes-from my_container --name my_container2 ubuntu /bin/bash
data-only container
data container does not need to be run to be accessable, so it does not take extra space #there is already volume definition in postgres image docker run –name dbdata postgres echo “Data-only container for postgres” #backup data in data container to tar docker run –rm –volumes-from dbdata -v $(pwd):/backup debian tar cvf /backup/backup.tar /var/lib/postgresql/data
permission
file changes after VOLUME command in Dockerfile do not change the Volume
#do change before VOLUME
FROM debian:wheezy
RUN useradd foo
RUN mkdir /data && touch /data/x
RUN chown -R foo:foo /data
VOLUME /data
delete volumes
using docker rm without -v will not delete volume. volume will be delete when no container is using it. bind-mounts volume refering to host folder will never be deleted #to delete volume docker rm -v #or using –rm when doing docker run docker run –rm
docker-compose
docker-compose is a tool to define and run multi-docker application. Containers are defined in service tag in yaml file
#execuate docker-compose
docker-compose up
#run docker-compose in background
docker-compose up -d
#not using default docker-compose.yml file
docker-compose -f server.yml up -d
volume
mount volume from host to container (HOST:CONTAINER) or（HOST:CONTAINER:ro) can rebuild images if change code is needed
volumes:
data-volume:
- ../src:/opt/src
build
image: webapp:tag
build:
context: ./dir
dockerfile: Dockerfile-alternate
args:
buildno: 1
command
default command to run when docker starts
command: bundle exec thin -p 3000
links
access service in other containers
web:
links:
- db
- db:database
external_links
access service in containers outside docker-compose
external_links:
- redis_1
- project_db_1:mysql
expose
expose port to linked service
expose:
- "3000"
ports
expose container port to host
ports:
- "3000"
- "3000-3005"
- "8000:8000"
- "9090-9091:8080-8081"
restart
restart policy if error occurs
restart: "no"
restart: always
restart: on-failure
restart: unless-stopped
environment
set environment variable, if no value (only key) is given, the value on host will be taken
environment:
RACK_ENV: development
SHOW: 'true'
SESSION_SECRET:
pid
container and host can share pid space
pid: "host"
dns
config dns server
dns: 8.8.8.8
dns:
- 8.8.8.8
- 9.9.9.9 ## docker manager tool portainer.io ### installation
sudo docker volume create portainer_data
sudo docker run -d -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer
url
http://dockerIP:9000
