From 0e1b793e8c3b7fa353136e2a0a0528b02067698f Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Sat, 14 Oct 2023 23:39:02 +0300 Subject: [PATCH 01/18] start rootless, pt 1 --- docs/getting-started.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index eb00f281..5b8fc72a 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -88,10 +88,18 @@ Confirm that Docker installed correctly by opening a terminal and running `docke To avoid writing sudos you may consider [adding yourself to docker group](https://docs.docker.com/install/linux/linux-postinstall/) -Keep in mind that if you do so, you can now run containers without sudo and containers give you super user access to the computer. +Keep in mind that if you do so, you can now run containers without sudo and containers give you or anyone who gains access to your account super user access to the computer. ::: +### Rootless Docker + +Instead of above installation, on Linux you can run Docker as a non-root user. This requires that your system has [certain programs and configurations set up in advance](https://docs.docker.com/engine/security/rootless/#prerequisites) by the system administrator. + +If your system is set up correctly, you can possibly run locally available installation script dockerd-rootless-setuptool.sh or download it from [https://get.docker.com/rootless](https://get.docker.com/rootless). Script will inform you of any missing requirements if there are any. + +Do note that while running Docker rootless does limit some security risks to your system, it just adds one hurdle for potential malicious attacker (and in any case, system staying secure might not sound so great if you end up "only" losing your user data). + ## Deadline The sign up for ECTS credits and the course ends 18.6.2023! After that course is locked and submissions can no longer be made or credits earned. As the certificate is received through submissions, you have to submit everything before the course ends. More details under completion and after each part. From ab5f796acc98575e5b455bf9f2451d62150581c5 Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Mon, 16 Oct 2023 15:39:05 +0300 Subject: [PATCH 02/18] grammarfix --- docs/getting-started.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 5b8fc72a..d7499b57 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -96,7 +96,7 @@ Keep in mind that if you do so, you can now run containers without sudo and cont Instead of above installation, on Linux you can run Docker as a non-root user. This requires that your system has [certain programs and configurations set up in advance](https://docs.docker.com/engine/security/rootless/#prerequisites) by the system administrator. -If your system is set up correctly, you can possibly run locally available installation script dockerd-rootless-setuptool.sh or download it from [https://get.docker.com/rootless](https://get.docker.com/rootless). Script will inform you of any missing requirements if there are any. +If your system is set up correctly, you can possibly run locally available installation script dockerd-rootless-setuptool.sh or download it from [https://get.docker.com/rootless](https://get.docker.com/rootless). The script will inform you of any missing requirements if there are any. Do note that while running Docker rootless does limit some security risks to your system, it just adds one hurdle for potential malicious attacker (and in any case, system staying secure might not sound so great if you end up "only" losing your user data). From 8a497c23b548ea17df038d4ee3e7e5ff4a10e132 Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Mon, 16 Oct 2023 17:23:32 +0300 Subject: [PATCH 03/18] cant stat --- docs/part-1/section-3.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/part-1/section-3.md b/docs/part-1/section-3.md index 8a4e09b3..2a09cd1c 100644 --- a/docs/part-1/section-3.md +++ b/docs/part-1/section-3.md @@ -201,6 +201,12 @@ If you're now getting "/bin/sh: ./hello.sh: not found" and you're using Windows ::: +:::can't stat + +If you are running rootless docker and build process gives can't stat error, you may try removing old images. [This issue](https://github.com/docker/for-linux/issues/380) may be relevant. + +::: + Now executing the application is as simple as running `docker run hello-docker`. Try it! During the build we see that there are multiple steps with hashes and intermediate containers. The steps here represent the layers so that each step is a new layer to the image. From 91fc6123f59e0ee96d91ab281a09c2affa8686d1 Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Mon, 16 Oct 2023 17:23:53 +0300 Subject: [PATCH 04/18] typofix, clarify --- docs/part-1/section-3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/part-1/section-3.md b/docs/part-1/section-3.md index 2a09cd1c..8cbd611e 100644 --- a/docs/part-1/section-3.md +++ b/docs/part-1/section-3.md @@ -363,7 +363,7 @@ Try `docker run devopsdockeruh/simple-web-service:alpine hello`. The application In this exercise create a Dockerfile and use FROM and CMD to create a brand new image that automatically runs `server`. -The Docker documentation [CMD](https://docs.docker.com/engine/reference/builder/#cmd) says a bit indirectly that if a image has ENTRYPOINT defined, CMD is used to define it the default arguments. +The Docker documentation [CMD](https://docs.docker.com/engine/reference/builder/#cmd) says a bit indirectly that if an image has ENTRYPOINT defined, CMD is used to define the default arguments for it. Tag the new image as "web-server" From f72025c8130afb2c0e954cb3df6803ceee0f7ba8 Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Wed, 18 Oct 2023 23:12:33 +0300 Subject: [PATCH 05/18] fix tip --- docs/part-1/section-3.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/part-1/section-3.md b/docs/part-1/section-3.md index 8cbd611e..c85c463c 100644 --- a/docs/part-1/section-3.md +++ b/docs/part-1/section-3.md @@ -201,9 +201,9 @@ If you're now getting "/bin/sh: ./hello.sh: not found" and you're using Windows ::: -:::can't stat +:::tip can't stat -If you are running rootless docker and build process gives can't stat error, you may try removing old images. [This issue](https://github.com/docker/for-linux/issues/380) may be relevant. +If you are running rootless docker and build process gives can't stat error, you may try removing old images. For more information, reading [this issue](https://github.com/docker/for-linux/issues/380) may be relevant. ::: From 016b5922c629ff3f87575a8724125da341594d69 Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Mon, 23 Oct 2023 14:25:29 +0300 Subject: [PATCH 06/18] mention ports<1024 --- docs/part-1/section-5.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/part-1/section-5.md b/docs/part-1/section-5.md index 1956d001..3d13cfd4 100644 --- a/docs/part-1/section-5.md +++ b/docs/part-1/section-5.md @@ -37,7 +37,7 @@ The details on how programs communicate are not detailed in this course. Courses - Sending messages: Programs can send messages to [URL](https://en.wikipedia.org/wiki/URL) addresses such as this: http://127.0.0.1:3000 where http is the [_protocol_](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol), 127.0.0.1 is a ip address, and 3000 is a [_port_](). Note the ip part could also be a [_hostname_](https://en.wikipedia.org/wiki/Hostname): 127.0.0.1 is also called [_localhost_](https://en.wikipedia.org/wiki/Localhost) so instead you could use http://localhost:3000. -- Receiving messages: Programs can be assigned to listen to any available port. If a program is listening for traffic on port 3000, and a message is sent to that port, it will receive it (and possibly process it). +- Receiving messages: Programs can be assigned to listen to any available port. If a program is listening for traffic on port 3000, and a message is sent to that port, it will receive it (and possibly process it). Port numbers below 1024 may be unavailable for users with normal privileges. The address _127.0.0.1_ and hostname _localhost_ are special ones, they refer to the machine or container itself, so if you are on a container and send message to _localhost_, the target is the same container. Similarly, if you are sending the request from outside of a container to _localhost_, the target is your machine. From afebd353066a4d06b33a4e535e237ab91041a307 Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Mon, 23 Oct 2023 14:26:15 +0300 Subject: [PATCH 07/18] mention manual compose install --- docs/part-2/section-1.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/part-2/section-1.md b/docs/part-2/section-1.md index e16a937e..f4533f2e 100644 --- a/docs/part-2/section-1.md +++ b/docs/part-2/section-1.md @@ -6,6 +6,12 @@ Even with a simple image, we've already been dealing with plenty of command line Next we will switch to a tool called [Docker Compose](https://docs.docker.com/compose/) to manage these. Docker Compose used to be a separate tool but now it is integrated to Docker and can be used like the rest of the Docker commands. +:::tip Rootless Docker + +If you are using Docker without root privileges, your system might not have Docker Compose installed. If this is the case, you can [install it manually yourself](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually). + +::: + Docker Compose is designed to simplify running multi-container applications to using a single command. Assume that we are in the folder where we have our Dockerfile with the following content: From aec05f474063a532bdce9da24c3234785ba018c4 Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Mon, 23 Oct 2023 14:27:29 +0300 Subject: [PATCH 08/18] mention mesos and nomad, typofix --- docs/part-3/section-5.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/part-3/section-5.md b/docs/part-3/section-5.md index f66695a8..00ff65ca 100644 --- a/docs/part-3/section-5.md +++ b/docs/part-3/section-5.md @@ -8,9 +8,9 @@ In the scope of this course, we cannot go into how to use the tools in this sect **Docker swarm mode** is built into Docker. It turns a pool of Docker hosts into a single virtual host. You can read the feature highlights [here](https://docs.docker.com/engine/swarm/). You can run right away with `docker swarm`. Docker swarm mode is the lightest way of utilizing multiple hosts. -**Kubernetes** is the de facto way of orchestrating your containers in large multi-host environments. The reason being it's customizability, large community and robust features. However the drawback is the higher learning curve compared to Docker swarm mode. You can read their introduction [here](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/). +**Kubernetes** is commonly default way of orchestrating your containers in large multi-host environments. The reason being it's customizability, large community and robust features. However the drawback is the higher learning curve compared to Docker swarm mode. You can read their introduction [here](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/). Other similar technologies exist, for example Apache Mesos and HashiCorp Nomad. -It is always good to remember that a singe tool is rarely an optimal solution for all the possible scenarios. In a 2-3 host environment for a hobby project the gains from Kubernetes might not be as large compared to a environment where you need to orchestrate hundreds of hosts with multiple containers each. +It is always good to remember that a single tool is rarely an optimal solution for all the possible scenarios. In a 2-3 host environment for a hobby project the gains from Kubernetes might not be as large compared to a environment where you need to orchestrate hundreds of hosts with multiple containers each. You can get to know Kubernetes with [k3s](https://k3s.io/) a lightweight Kubernetes distribution which you can run inside containers with [k3d](https://github.com/rancher/k3d). This is a great way to get started as you don't have to worry about complicated setup or any credit limits that the cloud providers alvays have. From 20c5c00a051fecae7b7ef22764e6b2ec80b3ec56 Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Mon, 27 Jan 2025 22:23:17 +0200 Subject: [PATCH 09/18] hashes for parts as root inside container --- docs/part-1/section-4.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/part-1/section-4.md b/docs/part-1/section-4.md index 0a6c23a0..df2e0ab4 100644 --- a/docs/part-1/section-4.md +++ b/docs/part-1/section-4.md @@ -16,8 +16,8 @@ $ docker run -it ubuntu:22.04 ..and, as we already know, curl is not installed - let's add `curl` with `apt-get` again. ```console -$ apt-get update && apt-get install -y curl -$ curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp +# apt-get update && apt-get install -y curl +# curl -L https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o /usr/local/bin/yt-dlp ``` At some point, you may have noticed that _sudo_ is not installed either, but since we are _root_ we don't need it. @@ -25,21 +25,21 @@ At some point, you may have noticed that _sudo_ is not installed either, but sin Next, we will add permissions and run the downloaded binary: ```console -$ chmod a+rx /usr/local/bin/yt-dlp -$ yt-dlp +# chmod a+rx /usr/local/bin/yt-dlp +# yt-dlp /usr/bin/env: 'python3': No such file or directory ``` Okay, [documentation](https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#dependencies) mentions that Python 3.8 or later is needed to run yt-dlp. So let us install that: ```console -$ apt-get install -y python3 +# apt-get install -y python3 ``` We can now try to run the app again: ```console -$ yt-dlp +# yt-dlp Usage: yt-dlp [OPTIONS] URL [URL...] From 4213968f9661d73bb7e5417efb4022f5e44b79e6 Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Tue, 28 Jan 2025 14:03:31 +0200 Subject: [PATCH 10/18] rootless updates + podman --- docs/getting-started.md | 18 ++++++++++++++++-- docs/part-1/section-2.md | 8 +++++++- docs/part-1/section-3.md | 12 ++++++++++-- docs/part-1/section-4.md | 8 +++++++- docs/part-1/section-6.md | 11 +++++++++++ 5 files changed, 51 insertions(+), 6 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index 0a5a77c1..f5a2acad 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -103,12 +103,26 @@ Keep in mind that if you do so, you can now run containers without sudo and cont ### Rootless Docker -Instead of above installation, on Linux you can run Docker as a non-root user. This requires that your system has [certain programs and configurations set up in advance](https://docs.docker.com/engine/security/rootless/#prerequisites) by the system administrator. +Instead of above installation, on Linux you can run Docker as a non-root user. This requires that your system has [certain programs and configurations set up in advance](https://docs.docker.com/engine/security/rootless/#prerequisites) by the system administrator. Since running Docker in rootless mode limits risks for system security, convincing your IT admin to cooperate to the extent required for rootless Docker may be viable alternative if you do not have administrative privileges yourself. -If your system is set up correctly, you can possibly run locally available installation script dockerd-rootless-setuptool.sh or download it from [https://get.docker.com/rootless](https://get.docker.com/rootless). The script will inform you of any missing requirements if there are any. +If your system is set up correctly, you can possibly run locally available installation script dockerd-rootless-setuptool.sh or download it from [https://get.docker.com/rootless](https://get.docker.com/rootless). The script will inform you of any missing requirements, if there are any. The script will also tell you how to control docker service, and what settings to set or change in your environment (PATH, DOCKER_HOST). + +See also [known limitations](https://docs.docker.com/engine/security/rootless/#known-limitations). Do note that while running Docker rootless does limit some security risks to your system, it just adds one hurdle for potential malicious attacker (and in any case, system staying secure might not sound so great if you end up "only" losing your user data). +### Podman + +Alternatively, your system may already have [Podman](https://podman.io/) installed (or you can [install it yourself](https://podman.io/docs/installation)). Podman is another containerization framework, functioning as a drop-in replacement for Docker to a high degree - generally you can just replace command docker with command podman. If you intend to work later with Kubernetes, Podman also offers some extra conveniences, but on the other hand Podman does not support Docker Swarm. + +This course should be doable with `alias docker=podman`. Easiest way to make this permanent is `echo "alias docker=podman" >> .bashrc`. Podman does not use default registry, while Docker uses docker.io by default. You will need to remember define registry with your commands (eg, `docker run docker.io/nginx`) or you need to configure your default registry (`mkdir $HOME/.config/containers && echo "unqualified-search-registries = ['docker.io']" >> $HOME/.config/containers/registries.conf`) + +If `podman info | grep graphDriverName` tells you are using vfs as your storage driver, prepare for _very_ slow and large builds. Requesting (or installing yourself) and then using fuse-overlayfs or native overlay (in recent Ubuntu and Debian package containers-storage) would be a good idea. (Or possibly changing default configuration with $HOME/.config/containers/storage.conf might be enough. [source](https://blog.abysm.org/2023/06/switching-system-wide-default-storage-driver-from-vfs-to-overlayfs-for-podman-on-debian-bookworm/)) + +Some of the Podman output may slightly differ from example Docker output in the material. + +***TODO*** + ## Deadline The sign up for ECTS credits and the course ends 16.6.2024! After that course is locked and submissions can no longer be made or credits earned. As the certificate is received through submissions, you have to submit everything before the course ends. More details under completion and after each part. diff --git a/docs/part-1/section-2.md b/docs/part-1/section-2.md index 27f4a69e..97cc0804 100644 --- a/docs/part-1/section-2.md +++ b/docs/part-1/section-2.md @@ -77,6 +77,12 @@ $ docker attach looper Now you have process logs (STDOUT) running in two terminals. Now press control+c in the attached window. The container is stopped because the process is no longer running. +:::tip Podman + +If you are using Podman, depending on your version, control+c with --no-stdin mentioned below might not work. + +::: + If we want to attach to a container while making sure we don't close it from the other terminal we can specify to not attach STDIN with `--no-stdin` option. Let's start the stopped container with `docker start looper` and attach to it with `--no-stdin`. Then try control+c. @@ -140,7 +146,7 @@ $ docker exec -it looper bash From the `ps aux` listing we can see that our `bash` process got PID (process ID) of 64. -Now that we're inside the container it behaves as you'd expect from Ubuntu, and we can exit the container with `exit` and then either kill or stop the container. +Now that we're inside the container it behaves as you'd expect from Ubuntu, and we can exit the container with `exit` or control+d and then either kill or stop the container. Our looper won't stop for a SIGTERM signal sent by a stop command. To terminate the process, stop follows the SIGTERM with a SIGKILL after a grace period. In this case, it's simply faster to use kill. diff --git a/docs/part-1/section-3.md b/docs/part-1/section-3.md index 211cb1de..75917a18 100644 --- a/docs/part-1/section-3.md +++ b/docs/part-1/section-3.md @@ -26,6 +26,12 @@ $ docker search hello-world ... ``` +:::tip Podman + +If you are using Podman, use --compatible to see stars, official and automated columns. For explanation, see `man podman-search`. + +::: + Let's examine the list. The first result, `hello-world`, is an official image. [Official images](https://docs.docker.com/docker-hub/official_images/) are curated and reviewed by Docker, Inc. and are usually actively maintained by the authors. They are built from repositories in the [docker-library](https://github.com/docker-library). @@ -40,7 +46,7 @@ There are also other Docker registries competing with Docker Hub, such as [Quay] `docker pull quay.io/nordstrom/hello-world` -So, if the host's name (here: `quay.io`) is omitted, it will pull from Docker Hub by default. +So, if the alternative registry's name (here: `quay.io`) is omitted, it will pull from Docker Hub by default. NOTE: Trying the above command may fail giving manifest errors as the default tag latest is not present in quay.io/nordstrom/hello-world image. Specifying a correct tag for a image will pull the image without any errors, for ex. `docker pull quay.io/nordstrom/hello-world:2.0` @@ -208,7 +214,9 @@ If you're now getting "/bin/sh: ./hello.sh: not found" and you're using Windows :::tip can't stat -If you are running rootless docker and build process gives can't stat error, you may try removing old images. For more information, reading [this issue](https://github.com/docker/for-linux/issues/380) may be relevant. +If you are running rootless docker and build process gives error saying "can't stat...", you may try removing old images. + +If you do get this error, this will be a significant problem with some later exercises. It may be worthwhile to check if you could do the course with standard Docker or with Podman instead of rootless Docker. ::: diff --git a/docs/part-1/section-4.md b/docs/part-1/section-4.md index df2e0ab4..ba4d4c79 100644 --- a/docs/part-1/section-4.md +++ b/docs/part-1/section-4.md @@ -259,11 +259,17 @@ $ docker diff determined_elion Let's try `docker cp` command to copy the file from the container to the host machine. We should use quotes now since the filename has spaces. ```console -$ docker cp "determined_elion://mydir/Welcome to Kumpula campus! ｜ University of Helsinki [DptFY_MszQs].mp4" . +$ docker cp "determined_elion:/mydir/Welcome to Kumpula campus! ｜ University of Helsinki [DptFY_MszQs].mp4" . ``` And now we have our file locally and we can watch it if the machine has a suitable player installed. Sadly, the use of `docker cp` is not proper to fix our issue. In the next section, we will improve this. +:::tip Podman + +If you are using Podman, you may also need to escape `[` in file name. (ie `podman cp "determined_elion:/mydir/Welcome to Kumpula campus! ｜ University of Helsinki \[DptFY_MszQs].mp4" .`) + +::: + ## Improved curler With `ENTRYPOINT` we can make the curler of the [Exercise 1.7.](/part-1/section-3#exercises-17---18) more flexible. diff --git a/docs/part-1/section-6.md b/docs/part-1/section-6.md index 3370d590..6505ca16 100644 --- a/docs/part-1/section-6.md +++ b/docs/part-1/section-6.md @@ -156,6 +156,17 @@ TIPS: ::: +:::tip can't stat + +If you are running rootless docker and have been struggling this far with build process giving error "can't stat...", following exercise 1.14 will present a major challenge. + +If this error is still a problem, possible workarounds: + - Jump to next section ("Publishing projects"), and upload your images to registry. Pulling one image and building another may work. + - Part 2 of this course, using docker-compose, may allow building several images at the same time. + +::: + + :::caution Mandatory Exercise 1.14: Environment Start both the frontend and the backend with the correct ports exposed and add [ENV](https://docs.docker.com/reference/dockerfile/#env) to Dockerfile with the necessary information from both READMEs From e9765f332d6cad79f1290eacd9b5a2db34829e82 Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Tue, 28 Jan 2025 14:08:06 +0200 Subject: [PATCH 11/18] will -> may --- docs/part-1/section-3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/part-1/section-3.md b/docs/part-1/section-3.md index 75917a18..c21b500b 100644 --- a/docs/part-1/section-3.md +++ b/docs/part-1/section-3.md @@ -216,7 +216,7 @@ If you're now getting "/bin/sh: ./hello.sh: not found" and you're using Windows If you are running rootless docker and build process gives error saying "can't stat...", you may try removing old images. -If you do get this error, this will be a significant problem with some later exercises. It may be worthwhile to check if you could do the course with standard Docker or with Podman instead of rootless Docker. +If you do get this error, this may be a significant problem with some later exercises. It may be worthwhile to check if you could do the course with standard Docker or with Podman instead of rootless Docker. ::: From 45b8bc59cde732fb54c5cc88a0f1b20e4ab9a1bd Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Tue, 28 Jan 2025 14:25:04 +0200 Subject: [PATCH 12/18] docker compose for podman --- docs/part-2/section-1.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/part-2/section-1.md b/docs/part-2/section-1.md index a7b1dd6e..a3776b4a 100644 --- a/docs/part-2/section-1.md +++ b/docs/part-2/section-1.md @@ -12,6 +12,14 @@ If you are using Docker without root privileges, your system might not have Dock ::: +:::tip Podman + +Podman has native alternative for Docker Compose, Podman Compose, which you may or may not already have installed. Podman can also work with Docker Compose (which you may or may not already have installed), and should do so by default if both Podman Compose and Docker Compose are available. + +For purposes of this course, it is better to use Docker Compose. If you do not already have it installed, you can install it [manually yourself](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually) (or possibly from your operating system packages). + +::: + Docker Compose is designed to simplify running multi-container applications using a single command. Assume that we are in the folder where we have our Dockerfile with the following content: From dbeea80e6b8f8fee64a73a923ee480a3c53c9cbb Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Tue, 28 Jan 2025 16:14:35 +0200 Subject: [PATCH 13/18] more on docker compose for podman --- docs/part-2/section-1.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/part-2/section-1.md b/docs/part-2/section-1.md index a3776b4a..ef430b15 100644 --- a/docs/part-2/section-1.md +++ b/docs/part-2/section-1.md @@ -14,9 +14,13 @@ If you are using Docker without root privileges, your system might not have Dock :::tip Podman -Podman has native alternative for Docker Compose, Podman Compose, which you may or may not already have installed. Podman can also work with Docker Compose (which you may or may not already have installed), and should do so by default if both Podman Compose and Docker Compose are available. +For Podman, native way to orchestrate containers is Kubernetes. For small scale orchestration Podman supports Podman Compose and Docker Compose, which you may or may not already have installed. Since Docker Compose is reference for everyone else, latest versions of Podman should use it by default if both Podman Compose and Docker Compose are available. You can read more about relationships between these projects [here](https://www.redhat.com/en/blog/podman-compose-docker-compose). -For purposes of this course, it is better to use Docker Compose. If you do not already have it installed, you can install it [manually yourself](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually) (or possibly from your operating system packages). +For purposes of this course, it is better to use Docker Compose. If you do not already have it installed, you can install it manually yourself [as a plugin (?needs testing?)](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually), or [standalone version](https://docs.docker.com/compose/install/standalone/) (or possibly from your operating system packages, for example `apt install --no-install-recommends docker-compose podman-docker`). Depending on your software versions and how you installed it, you may have to call `docker-compose` instead of `docker compose`. + +For docker-compose to work with Podman, you need a socket: `systemctl --user enable --now podman.socket`, check path with `podman info | grep -i remotesocket -A2` and then export required environment variable: for example `export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock` + +If necessary, Podman Compose can be found [here](https://github.com/containers/podman-compose). Read README.md for installation instructions. It is more lightweight and does not require socket, but might not be sufficient for this course. Do note that even if you install both, you should only use one of them, they are not interchangeable. ::: From dbb46a44af5d6c094c7d5978c4d3807b124097af Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Thu, 30 Jan 2025 04:54:21 +0200 Subject: [PATCH 14/18] part 2, 1st half --- docs/part-1/section-6.md | 4 ++-- docs/part-2/section-1.md | 6 ++++-- docs/part-2/section-2.md | 25 +++++++++++++++++++++++++ 3 files changed, 31 insertions(+), 4 deletions(-) diff --git a/docs/part-1/section-6.md b/docs/part-1/section-6.md index 6505ca16..b2002043 100644 --- a/docs/part-1/section-6.md +++ b/docs/part-1/section-6.md @@ -160,9 +160,9 @@ TIPS: If you are running rootless docker and have been struggling this far with build process giving error "can't stat...", following exercise 1.14 will present a major challenge. -If this error is still a problem, possible workarounds: +If this error is still a problem for you, possible workarounds: - Jump to next section ("Publishing projects"), and upload your images to registry. Pulling one image and building another may work. - - Part 2 of this course, using docker-compose, may allow building several images at the same time. + - Part 2 of this course, using docker-compose, may allow building several images at the same time, see exercise 2.3. ::: diff --git a/docs/part-2/section-1.md b/docs/part-2/section-1.md index ef430b15..5937e7e9 100644 --- a/docs/part-2/section-1.md +++ b/docs/part-2/section-1.md @@ -14,14 +14,16 @@ If you are using Docker without root privileges, your system might not have Dock :::tip Podman -For Podman, native way to orchestrate containers is Kubernetes. For small scale orchestration Podman supports Podman Compose and Docker Compose, which you may or may not already have installed. Since Docker Compose is reference for everyone else, latest versions of Podman should use it by default if both Podman Compose and Docker Compose are available. You can read more about relationships between these projects [here](https://www.redhat.com/en/blog/podman-compose-docker-compose). +For small scale orchestration Podman supports Podman Compose and Docker Compose, which you may or may not already have installed. Since Docker Compose is reference for everyone else, latest versions of Podman should use it by default if both Podman Compose and Docker Compose are available. You can read more about relationships between these projects [here](https://www.redhat.com/en/blog/podman-compose-docker-compose). -For purposes of this course, it is better to use Docker Compose. If you do not already have it installed, you can install it manually yourself [as a plugin (?needs testing?)](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually), or [standalone version](https://docs.docker.com/compose/install/standalone/) (or possibly from your operating system packages, for example `apt install --no-install-recommends docker-compose podman-docker`). Depending on your software versions and how you installed it, you may have to call `docker-compose` instead of `docker compose`. +For purposes of this course, it is better to use Docker Compose. If you do not already have it installed, you can install it manually yourself as a [standalone version](https://docs.docker.com/compose/install/standalone/) (or possibly from your operating system packages, for example `apt install --no-install-recommends docker-compose podman-docker`). Depending on your software versions and how you installed it, you may have to call `docker-compose` instead of `docker compose` (or `podman compose`). For docker-compose to work with Podman, you need a socket: `systemctl --user enable --now podman.socket`, check path with `podman info | grep -i remotesocket -A2` and then export required environment variable: for example `export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock` If necessary, Podman Compose can be found [here](https://github.com/containers/podman-compose). Read README.md for installation instructions. It is more lightweight and does not require socket, but might not be sufficient for this course. Do note that even if you install both, you should only use one of them, they are not interchangeable. +If you are not forced to use Compose, check also [pods](https://developers.redhat.com/blog/2019/01/15/podman-managing-containers-pods) (see also man podman-pod), [quadlets](https://www.redhat.com/en/blog/quadlet-podman) (Podman 4.4+) or Kubernetes. (This is not relevant for this course.) + ::: Docker Compose is designed to simplify running multi-container applications using a single command. diff --git a/docs/part-2/section-2.md b/docs/part-2/section-2.md index 5823ee27..21485d59 100644 --- a/docs/part-2/section-2.md +++ b/docs/part-2/section-2.md @@ -144,6 +144,12 @@ $ docker compose port --index 3 whoami 8000 0.0.0.0:32768 ``` +:::tip Podman + +If you are using Podman, `podman ps` will list also ports. + +::: + We can now curl from these ports: ```console @@ -186,6 +192,14 @@ $ curl localhost:80 ``` +:::tip Rootless Docker and Podman + +If you are using rootless Docker, or Podman (Podman runs by default also rootless), you may be unable to use port numbers below 1024. If this is the case, you could for example use 8888:80 for port in your docker-compose.yml and then `curl localhost:8888`. + +Note also that for both rootless Docker and for Podman, socket path and possibly name is different. You can see your socket information with `echo $DOCKER_HOST`. + +::: + It's "working", but the Nginx just doesn't know which service we want. The `nginx-proxy` works with two environment variables: `VIRTUAL_HOST` and `VIRTUAL_PORT`. `VIRTUAL_PORT` is not needed if the service has `EXPOSE` in it's Docker image. We can see that `jwilder/whoami` sets it: - Note: Mac users with the M1 processor you may see the following error message: `runtime: failed to create new OS thread`. In this case you can use the Docker Image `ninanung/nginx-proxy` instead which offers a temporary fix until `jwilder/nginx-proxy` is updated to support M1 Macs. @@ -276,3 +290,14 @@ This exercise was created with [Sasu Mäkinen](https://github.com/sasumaki) Please return the used commands for this exercise. ::: + +:::tip Rootless Docker and Podman + +If you are using rootless Docker, or Podman (Podman runs by default also rootless), you may be unable to use port numbers below 1024. If this is the case, above exercise may require extra effort. + +Like earlier, you could for example use 8888:80 for port in your docker-compose.yml. However, calculator/src/commons.js has hardcoded address (with default port, ie 80) which will be a problem. You will have to edit address to include suitable port, and then build calculator image yourself, instead of using provided one. + +Note again that for both rootless Docker and for Podman, socket path and possibly name is different. You can see your socket information with `echo $DOCKER_HOST`. + +::: + From e8c762c67c24b04d36a99cebc0ee0886fb2e6ada Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Thu, 30 Jan 2025 05:20:06 +0200 Subject: [PATCH 15/18] minor cleanup --- docs/getting-started.md | 4 +--- docs/part-1/section-3.md | 2 +- docs/part-1/section-6.md | 5 ++--- docs/part-2/section-1.md | 4 ++-- 4 files changed, 6 insertions(+), 9 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index f5a2acad..6c189ada 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -113,7 +113,7 @@ Do note that while running Docker rootless does limit some security risks to you ### Podman -Alternatively, your system may already have [Podman](https://podman.io/) installed (or you can [install it yourself](https://podman.io/docs/installation)). Podman is another containerization framework, functioning as a drop-in replacement for Docker to a high degree - generally you can just replace command docker with command podman. If you intend to work later with Kubernetes, Podman also offers some extra conveniences, but on the other hand Podman does not support Docker Swarm. +Alternatively, your system may already have [Podman](https://podman.io/) installed (or you can [install it yourself](https://podman.io/docs/installation)). Podman is another containerization framework, functioning as a drop-in replacement for Docker to a high degree - often you can just replace command docker with command podman. If you intend to work later with Kubernetes, Podman also offers some extra conveniences, but on the other hand Podman does not support Docker Swarm. This course should be doable with `alias docker=podman`. Easiest way to make this permanent is `echo "alias docker=podman" >> .bashrc`. Podman does not use default registry, while Docker uses docker.io by default. You will need to remember define registry with your commands (eg, `docker run docker.io/nginx`) or you need to configure your default registry (`mkdir $HOME/.config/containers && echo "unqualified-search-registries = ['docker.io']" >> $HOME/.config/containers/registries.conf`) @@ -121,8 +121,6 @@ If `podman info | grep graphDriverName` tells you are using vfs as your storage Some of the Podman output may slightly differ from example Docker output in the material. -***TODO*** - ## Deadline The sign up for ECTS credits and the course ends 16.6.2024! After that course is locked and submissions can no longer be made or credits earned. As the certificate is received through submissions, you have to submit everything before the course ends. More details under completion and after each part. diff --git a/docs/part-1/section-3.md b/docs/part-1/section-3.md index c21b500b..86873322 100644 --- a/docs/part-1/section-3.md +++ b/docs/part-1/section-3.md @@ -216,7 +216,7 @@ If you're now getting "/bin/sh: ./hello.sh: not found" and you're using Windows If you are running rootless docker and build process gives error saying "can't stat...", you may try removing old images. -If you do get this error, this may be a significant problem with some later exercises. It may be worthwhile to check if you could do the course with standard Docker or with Podman instead of rootless Docker. +If you keep getting this error with future exercises, this may be a significant problem with some later exercises. It may be worthwhile to check if you could do the course with standard Docker or with Podman instead of rootless Docker. ::: diff --git a/docs/part-1/section-6.md b/docs/part-1/section-6.md index b2002043..a72ce948 100644 --- a/docs/part-1/section-6.md +++ b/docs/part-1/section-6.md @@ -158,15 +158,14 @@ TIPS: :::tip can't stat -If you are running rootless docker and have been struggling this far with build process giving error "can't stat...", following exercise 1.14 will present a major challenge. +If you are running rootless docker and have been struggling this far with build process sometimes giving error "can't stat...", following exercise 1.14 will present a major challenge. If this error is still a problem for you, possible workarounds: - - Jump to next section ("Publishing projects"), and upload your images to registry. Pulling one image and building another may work. + - Jump to next section ("Publishing projects"), and upload your images to registry. Pulling one image and building another one may work. - Part 2 of this course, using docker-compose, may allow building several images at the same time, see exercise 2.3. ::: - :::caution Mandatory Exercise 1.14: Environment Start both the frontend and the backend with the correct ports exposed and add [ENV](https://docs.docker.com/reference/dockerfile/#env) to Dockerfile with the necessary information from both READMEs diff --git a/docs/part-2/section-1.md b/docs/part-2/section-1.md index 5937e7e9..814859e1 100644 --- a/docs/part-2/section-1.md +++ b/docs/part-2/section-1.md @@ -18,9 +18,9 @@ For small scale orchestration Podman supports Podman Compose and Docker Compose, For purposes of this course, it is better to use Docker Compose. If you do not already have it installed, you can install it manually yourself as a [standalone version](https://docs.docker.com/compose/install/standalone/) (or possibly from your operating system packages, for example `apt install --no-install-recommends docker-compose podman-docker`). Depending on your software versions and how you installed it, you may have to call `docker-compose` instead of `docker compose` (or `podman compose`). -For docker-compose to work with Podman, you need a socket: `systemctl --user enable --now podman.socket`, check path with `podman info | grep -i remotesocket -A2` and then export required environment variable: for example `export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock` +For docker-compose to work with Podman, you need a socket: `systemctl --user enable --now podman.socket`, check path with `podman info | grep -i remotesocket -A2` and then export required environment variable: for example `export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock`. If you want to make Compose always available, add these to your .bashrc. -If necessary, Podman Compose can be found [here](https://github.com/containers/podman-compose). Read README.md for installation instructions. It is more lightweight and does not require socket, but might not be sufficient for this course. Do note that even if you install both, you should only use one of them, they are not interchangeable. +If necessary, Podman Compose can be found [here](https://github.com/containers/podman-compose). Read README.md for installation instructions. It is more lightweight and does not require socket, but has not been tested with this course. Do note that even if you install both, you should only use one of them, they are not interchangeable. If you are not forced to use Compose, check also [pods](https://developers.redhat.com/blog/2019/01/15/podman-managing-containers-pods) (see also man podman-pod), [quadlets](https://www.redhat.com/en/blog/quadlet-podman) (Podman 4.4+) or Kubernetes. (This is not relevant for this course.) From 2885fd6d8c10cf0c0f15960b233fbb850b2a4976 Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Thu, 30 Jan 2025 16:39:37 +0200 Subject: [PATCH 16/18] match redmine version to one in earlier material --- docs/part-2/section-3.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/part-2/section-3.md b/docs/part-2/section-3.md index 74c13e9a..da8308fe 100644 --- a/docs/part-2/section-3.md +++ b/docs/part-2/section-3.md @@ -162,7 +162,7 @@ services: volumes: - database:/var/lib/postgresql/data redmine: - image: redmine:4.1-alpine + image: redmine:5.1-alpine environment: - REDMINE_DB_POSTGRES=db - REDMINE_DB_PASSWORD=example From 2b1bb82a6db21ec465fd8e5d64646f808023f635 Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Thu, 30 Jan 2025 20:33:17 +0200 Subject: [PATCH 17/18] part 2, end --- docs/part-2/section-3.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/part-2/section-3.md b/docs/part-2/section-3.md index da8308fe..05012691 100644 --- a/docs/part-2/section-3.md +++ b/docs/part-2/section-3.md @@ -260,6 +260,14 @@ TIPS: ::: +:::tip Rootless Docker and Podman + +In case you do not have root permissions (ie, are using rootless Docker and possibly Podman), following exercise 2.7 is very likely to create a directory which (including contents) you can not remove yourself, due to ownership and permissions. As such, convenient location mentioned in the exercise is *not* `./database` (at least initially), instead you might consider `/tmp/database` (which might get emptied automatically by your operating system, either periodically or during system boot). Note there is no `.` in front of `/tmp/database`. + +Read carefully "Arbitrary --user Notes" from Postgres image documentation for hints how to actually solve this problem. This may require building your own image based on Postgres image. + +::: + :::info Exercise 2.7 Postgres image uses a volume by default. Define manually a volume for the database in a convenient location such as in `./database` so you should use now a [bind mount](https://docs.docker.com/storage/bind-mounts/). The image [documentation](https://github.com/docker-library/docs/blob/master/postgres/README.md#where-to-store-data) may help you with the task. @@ -414,3 +422,9 @@ Nmap done: 1 IP address (1 host up) scanned in 1.28 seconds ``` ::: + +:::tip Rootless Docker and Podman + +If nmap gives an error about not being permitted to open raw socket, you may want to check in what context [nmap documentation](https://nmap.org/book/man-misc-options.html) mentions raw socket privileges. + +::: From d8fbd1cc841afc1bc3400ade16e38b1e4544e488 Mon Sep 17 00:00:00 2001 From: Terho Uotila Date: Thu, 30 Jan 2025 22:39:56 +0200 Subject: [PATCH 18/18] part 3 --- docs/part-3/section-2.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/part-3/section-2.md b/docs/part-3/section-2.md index c5a8d268..ee4cff15 100644 --- a/docs/part-3/section-2.md +++ b/docs/part-3/section-2.md @@ -220,3 +220,9 @@ Submit the Dockerfile and the final version of your script. See [Part 1](/part-1/section-4) for more. ::: + +:::tip Rootless Docker and Podman + +If you are using Podman or rootless Docker, remember your socket path from earlier parts of this material. + +:::