Process down (docker-swarm)

Modified on Mon, 23 Jun, 2025 at 9:08 AM

• Note: Applicable only for systems running the Docker Swarm model

Symptoms:
- The process monitor on the swarm-manager reports a service down.
Root Cause:
- Disk full on the node causes the Docker service on that node to go down.
- Docker service crashes and restarts, causing the node to lose connection with the manager.
How to check:
- On swarm-manager:
  - ```
  docker node ls
```
  Check the list of nodes.
- ```
docker service ls
```
    Check the list of services and their up/down status.
  - ```
  docker service inspect <service_name>
```
  Check which node the service is running on via the "Placement" field.
- On docker-node:
  - ```
  docker ps -a
```
  Check the containers running on the node.
- ```
service docker status
```
    Check the service status and uptime.
  - Note: Sometimes the Docker service and containers run normally, but the swarm-manager still shows the node as down.
Resolution steps:
- First, try to restart the service:
  - ```
  docker service update --replicas=0 phishing_centrel && docker service update --replicas=5 phishing_centrel
```
  Replace phishing_centrel with the name of the service reported as down.
- ```
docker service ls
```
    Check if the service has the desired number of replicas.
  - If waiting does not resolve the issue, proceed with the next steps.
- On docker-node:
  - ```
  service docker restart
```
- ```
docker swarm leave (-f)
```
- On swarm-manager:
  - ```
  docker node rm <node_name>
```
- ```
docker swarm join-token worker
```
    Copy the output and run it on the docker-node to rejoin it to the swarm.
- Check the services and nodes again to ensure everything is normal.
- Check the logs of the process you just brought up:
```
docker logs -ft <container_name>
```