Skip to content

[BUG] compose watch stops services using links and depends_on with restart: true #12814

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
clue opened this issue May 8, 2025 · 1 comment
Assignees
Labels

Comments

@clue
Copy link

clue commented May 8, 2025

Description

Compose watch mode may currently stop services that link other services without ever restarting:

  1. The idea is that a service (think a front proxy) links any another service (think a backend).
  2. Compose watch mode will correctly identify that the backend service needs to be recreated/restarted (e.g. using action: rebuild).
  3. It will correctly stop the front service before stopping the backend service.
  4. It will correctly restart the backend, but fails to restart the front service afterwards.
  5. The application is now in a broken state ("half up"), the backend can no longer be reached without a front proxy.

I've specifically encountered this when using legacy links between services, but the same behavior can be reproduced when using depends_on with restart: true. It's my understanding legacy links would basically act like depends_on with restart: true, despite restart: false being the default otherwise.

I think it's debatable if the front proxy service needs to be restarted in this example. In either case, it should either be restarted completely or not restarted at all. I don't think stopping the service without restarting it again would ever be intended behavior.

The problem occurred as part of a bigger application using multiple services, but I've been able to reproduce this in a most basic compose.yaml file with no external dependencies except a public image. The resulting file may look a bit contrived, but I hope this should be enough to reproduce this without any specific service configurations.

Possibly related, I've checked docker compose up -a reports the affected services as "exited". Using docker compose restart backend does restart the backend and the affected services as expected. However, the running docker compose up --watch session will not pick up the affected services again. This appears to work fine if I do not trigger a rebuild first.

I've checked for similar reports and stumbled upon #11695 and #11813, but believe they may point to different, but possibly related, issues.

The recent changes regarding watch mode and watching multiple services as implemented in #12469 and others have been a real game-changer for me personally and a larger number of projects I'm working on, so keep up the great work! 👍

Steps To Reproduce

Basic reproducer:

services:
  nok:
    image: httpd:2.4-alpine
    links:
      - backend
  backend:
    build:
      dockerfile_inline: |
        FROM httpd:2.4-alpine
    develop:
      watch:
        - path: compose.yaml
          action: rebuild

Reproducer showing two broken services + 2 working ones as possible workarounds:

services:
  nok:
    image: httpd:2.4-alpine
    links:
      - backend
  same:
    image: httpd:2.4-alpine
    depends_on:
      backend:
        condition: service_started
        restart: true
  ok:
    image: httpd:2.4-alpine
    depends_on:
      - backend
  alright:
    image: httpd:2.4-alpine
    depends_on:
      backend:
        condition: service_started
        restart: false
  backend:
    build:
      dockerfile_inline: |
        FROM httpd:2.4-alpine
    develop:
      watch:
        - path: compose.yaml
          action: rebuild

Start watch mode in foreground:

docker compose up --watch

Change compose.yaml file to trigger rebuild:

touch compose.yaml

Log output shows backend service is correctly rebuilt, but affected front proxy services stop:

           ⦿ Rebuilding service(s) ["backend"] after changes were detected...
           ⦿ service(s) ["backend"] successfully built
nok-1      | [Thu May 08 13:30:52.184586 2025] [mpm_event:notice] [pid 1:tid 123450472430344] AH00492: caught SIGWINCH, shutting down gracefully
same-1     | [Thu May 08 13:30:52.189356 2025] [mpm_event:notice] [pid 1:tid 130455114877704] AH00492: caught SIGWINCH, shutting down gracefully
nok-1 exited with code 0
same-1 exited with code 0
backend-1  | [Thu May 08 13:30:53.516278 2025] [mpm_event:notice] [pid 1:tid 128908057438984] AH00492: caught SIGWINCH, shutting down gracefully
backend-1 exited with code 0
backend-1 has been recreated

You can see both nok and same are stopped without restarting.

Compose Version

Docker Compose version v2.36.0

Docker Environment

Client:
 Version:    26.1.3
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc.)
    Version:  v2.36.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 33
  Running: 11
  Paused: 0
  Stopped: 22
 Images: 1537
 Server Version: 26.1.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-59-generic
 Operating System: Ubuntu 24.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 30.54GiB
 Name: me
 ID: 519acebf-0404-4fc7-a89d-afcb3bcf9b15
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

@ndeloof
Copy link
Contributor

ndeloof commented May 14, 2025

links indeed involves a dependency with target service, which can be visualized by running docker compose config:

  nok:
    depends_on:
      backend:
        condition: service_started
        restart: true
        required: true
    image: httpd:2.4-alpine
    links:
      - backend

There's something obviously wrong here, thanks for providing a reproduction example I can use to investigate this bug

@ndeloof ndeloof self-assigned this May 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants