Overlay Networks

Overlay networks are an abstraction that make it possible for nodes on otherwise disconnected networks to communicate as if they are on the same network. Turns out their is a way to do this with docker networking so you can run your services from a docker compose on different machines without them having to know about it. The services can all continue to talk to each other using the docker DNS names without concern for where the other services are actually located!

Setting up a Docker Swarm

In order to set up an overlay network, even for non-swarm containers, you first have to create a swarm with each of the machines you want to deploy to. Luckily this is pretty simple, assuming you already have docker on each machine.

Step 1:

First run this on the machine you want to be the manager node: docker swarm init --data-path-port <port>

  • --data-path-port <port> : In order to get the networking between the nodes working in my case, I had to override the default port as it was conflicting with other ports on the machines for some reason, you may find it works without this.

Note: If you have trouble getting the swarm to work, there are a few ports that docker swarm operates on. Make sure these are open in any firewall between your machines as mentioned here: Required Ports

Step 2:

The previous command will output a new command including a token. Copy and paste this command into your other nodes/machines to join them to the swarm.

On the manager node, you can run docker node ls to list the nodes in the swarm. You should see each machine you added.

Step 3:

Once you have a docker swarm set up, any overlay network created on the manager will be accessible from any of the worker nodes. I will show an example of how I integrated the creation of this overlay network and the connection of the nodes on the other machine in the following section.

Docker Compose

I used a single docker compose file to deploy different sections to different machines using profiles. You could also create different compose files. Learn More about Compose Profiles

I created two profiles, manager and worker, and put each service in the profile where I wanted them to run:

Profiles

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

services:
  node2:
  ...
    profiles:
      - manager
    networks:
      - w-overlay

  node3:
    ...
    profiles:
      - worker
    networks:
      - w-overlay-worker

Node 2 runs on the manager, node 3 runs on the worker

Github Action Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

 deploy-application:
    # I have a self hosted runner on each machine with a label to identify which machine its on
    runs-on: [self-hosted, office5] 
    steps:
      - name: checkout repo
        uses: actions/checkout@v3

      - name: redeploy application
        run: |
          cd ops/prod
          docker compose pull
          docker compose --profile manager up -d --build           
		# only runs services with the profile 'manager'

  deploy-other-node:
    runs-on: [self-hosted, office4]
    steps:
      - name: checkout repo
        uses: actions/checkout@v3

      - name: redeploy application
        run: |
          cd ops/prod
          docker compose pull
          docker compose --profile worker up -d --build           
		# only runs services with the profile 'worker'

Compose Network Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

services:

...

networks:
  w-overlay:
    name: w-overlay
    driver: overlay
    attachable: true
  w-overlay-worker:
    name: w-overlay
    driver: overlay
    external: true

The w-overlay is created on the manager as an ‘overlay’ network. attachable: true is required for standalone containers to be able to ‘attach’ to the network.

The w-overlay-worker is used in nodes deployed to the worker machine. external: true tells this node that the network should already exist, and not to work about creating it. This will connect to the network created by the manager.

References: