Copr docker-compose without supervisord
A couple of years ago we decided to containerize our Copr development environment to make the onboarding of new contributors easier, and to have a unified development environment for all of our team members. Several improvements have happened since then and the original blog post Copr stack dockerized! isn’t up-to-date anymore. We are going to make that right.
Just a quick side note about what happened since the original Copr stack dockerized! blog post:
- We don’t use
supervisordinside the containers anymore. Instead, we spawn more containers than before, each of them running just a single process
- Within containers, we don’t run processes as root anymore but use specific users instead
- Copr backend got reworked to use resalloc, therefore a
resalloccontainer was added
- We try to maintain compatibility with
podman, which might be the next step for our development environment
Getting started with Copr development should be as easy as possible. Make sure
you have Docker properly configured, and
command installed. Then simply build, and run the stack. Once it’s up and
running, the database needs to be initialized.
$ docker-compose up -d $ docker exec -it copr_frontend_1 bash [copr-fe@frontend /]$ init-database.sh
At this moment, you should be able to open http://127.0.0.1:5000/ in a web browser, log-in, create a project, and successfully build a package.
How can I start everything?
docker-compose up -d
How can I see running containers?
Why doesn’t some container start as expected?
docker-compose logs --follow
How can I open a shell in the container?
docker exec -it <name> bash
How can I open a root shell in the container?
docker exec --user root -it <name> bash
How can I throw away all changes, that I made inside the container?
docker-compose up -d --force-recreate <service>
How can I drop a container and image for one service?
dco rm --stop <name> && docker rmi copr_<name>
How can I drop the whole docker-compose environment?
docker-compose down --rmi 'all'
Running services from git
Probably everyone has his own preferred way of testing changes. My workflow is described in the previous blog post. Here we can see its updated version.
$ docker-compose -f docker-compose.yaml -f docker-compose.shell.yaml up -d frontend $ docker exec -it copr_frontend_1 bash [copr-fe@frontend /]$ PYTHONPATH=/opt/copr/frontend/coprs_frontend /opt/copr/frontend/coprs_frontend/manage.py runserver -p 5000 -h 0.0.0.0 --without-threads --no-reload
$ docker-compose -f docker-compose.yaml -f docker-compose.shell.yaml up -d distgit $ docker exec -it copr_distgit_1 bash [root@distgit /]# PYTHONPATH=/opt/copr/dist-git /usr/sbin/runuser -u copr-dist-git -g copr-dist-git -- /opt/copr/dist-git/run/importer_runner.py
Backend has multiple containers, so it depends on what you changed. For build dispatcher:
$ docker-compose -f docker-compose.yaml -f docker-compose.shell.yaml up -d backend-build $ docker exec -it copr_backend-build_1 bash [root@backend /] # PYTHONPATH=/opt/copr/backend /usr/sbin/runuser -u copr -g copr -- /usr/bin/copr-run-dispatcher builds
$ docker-compose -f docker-compose.yaml -f docker-compose.shell.yaml up -d backend-action $ docker exec -it copr_backend-action_1 bash [root@backend /] # PYTHONPATH=/opt/copr/backend /usr/sbin/runuser -u copr -g copr -- /usr/bin/copr-run-dispatcher actions
$ docker-compose -f docker-compose.yaml -f docker-compose.shell.yaml up -d backend-log $ docker exec -it copr_backend-log_1 bash [root@backend /] # PYTHONPATH=/opt/copr/backend /usr/sbin/runuser -u copr -g copr -- /opt/copr/backend/run/copr_run_logger.py
In production, we spawn new builder instances in Amazon AWS, but this is
simplified for the development environment. Locally, we run a
container and use it for all builds without recycling. This is the easiest way
to debug the
copr-rpmbuild client tool.
$ docker exec -it copr_builder_1 bash [root@builder /]# PYTHONPATH=/opt/copr/rpmbuild/ /opt/copr/rpmbuild/main.py --chroot fedora-rawhide-x86_64 --task-url http://frontend:5000/backend/get-build-task/123-fedora-rawhide-x86_64
$ docker exec -it copr_database_1 bash bash-4.2$ psql coprdb
Permission denied for openid_store
copr-frontend from git, the
contains files that were created within the frontend container. The
problem is accessing them from the host system or a new container
when the old one is dropped. Those don’t have the user and group
PermissionError: [Errno 13] Permission denied: "/opt/copr/frontend/data/openid_store/associations/https-id.fedoraproject.org-b'ahW3p5yqmHART1i9_lWSDz825NY'-b'VOcKQaT5MFkyT4oYgexUGGSA8zI'"
It is safe to simply drop all generated data
$ docker exec --user root -it copr_frontend_1 bash [copr-fe@frontend /]$ rm -rf /opt/copr/frontend/data/
Outdated database schema
This might happen for various reasons, for example when trying to run
copr-frontend from git for the first time. Fix it by running
migrations from the git repository.
$ docker exec -it copr_frontend_1 bash [copr-fe@frontend /]$ cd /opt/copr/frontend/coprs_frontend/ [copr-fe@frontend coprs_frontend]$ alembic-3 upgrade head
Alternatively, for non-git
copr-frontend, you might want to run
Some dependencies are not installed
When running a service from git, some dependencies might be missing. The most
comfortable way to install them is to upgrade the relevant Copr package. The
following example is for
copr-frontend but it can be done the
same way for every other service.
$ docker exec --user root -it copr_frontend_1 bash [copr-fe@frontend /]$ dnf install tito [copr-fe@frontend frontend]$ cd /opt/copr/frontend/ [copr-fe@frontend frontend]$ dnf builddep copr-frontend.spec [copr-fe@frontend frontend]$ tito build --rpm --test --install --rpmbuild-options=--nocheck