Drone CI failing on clone step

Question

I'm running on a machine in docker-compose the following:

• gitlab
• drone (server)
• drone (agent)

and when I trigger a build (or triggered by git push), drone keeps on failing on the issue:

git init
Initialized empty Git repository in /drone/src/.git/
git remote add origin http://my-git/amaziagur/location-service.git
git fetch --no-tags origin +refs/heads/master:
fatal: unable to access 'http://my-git/amaziagur/location-service.git/': Couldn't resolve host 'my-git'
exit status 128

here is the docker-compose.yml:

version: '2'
services:
  #PROXY
  gitlab:
    image: 'gitlab/gitlab-ce:9.1.0-ce.0'
    restart: always
    hostname: 'my-git'
    links:
      - postgresql:postgresql
      - redis:redis
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        postgresql['enable'] = false
        gitlab_rails['db_username'] = "gitlab"
        gitlab_rails['db_password'] = "gitlab"
        gitlab_rails['db_host'] = "postgresql"
        gitlab_rails['db_port'] = "5432"
        gitlab_rails['db_database'] = "gitlabhq_production"
        gitlab_rails['db_adapter'] = 'postgresql'
        gitlab_rails['db_encoding'] = 'utf8'
        redis['enable'] = false
        gitlab_rails['redis_host'] = 'redis'
        gitlab_rails['redis_port'] = '6379'
        external_url 'http://my-git'
        gitlab_rails['gitlab_shell_ssh_port'] = 30022
    ports:
      # both ports must match the port from external_url above
      - "80:80"
      # the mapped port must match ssh_port specified above.
      - "30022:22"
  # the following are hints on what volumes to mount if you want to persist data
    volumes:
     - /data/gitlab/config:/etc/gitlab:rw
     - /data/gitlab/logs:/var/log/gitlab:rw
     - /data/gitlab/data:/var/opt/gitlab:rw

  postgresql:
    restart: always
    image: postgres:9.6.2-alpine
    environment:
      - POSTGRES_USER=gitlab
      - POSTGRES_PASSWORD=gitlab
      - POSTGRES_DB=gitlabhq_production
  # the following are hints on what volumes to mount if you want to persist data
    volumes:
     - /home/foresight/postgresql:/var/lib/postgresql:rw

  redis:
    restart: always
    image: redis:3.0.7-alpine
  # DRONE
  drone-server:
    image: drone/drone:0.7.3
    ports:
      - "8000:8000"
    networks:
      - drone
      - gitlab
    links:
     - gitlab
    volumes:
      - /home/drone:/var/lib/drone/
    environment:
      #@@@@@
      DRONE_OPEN: "true"
      DRONE_HOST: "http://10.0.0.200:8000"
      DRONE_ADMIN: amaziagur
      DRONE_GITLAB: "true"
      DRONE_GITLAB_URL: "http://10.0.0.200"
      DRONE_GITLAB_CLIENT: "secret"
      DRONE_GITLAB_SECRET: "secret"
      DRONE_SECRET: "my_secret"
      #@@@@@@@
  drone-agent:
    image: drone/drone:0.7.3
    command: agent
    depends_on:
      - drone-server
    networks:
      - drone
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      DRONE_SERVER: ws://drone-server:8000/ws/broker
      DRONE_DEBUG: "true"
      DRONE_SECRET: "our_secret_4ever_and_ever"

networks:
  drone:
    driver: bridge
  gitlab:
    driver: bridge

Both are installed on the same machine, and I can't figure out what I'm doing wrong, what is the reason that the drone keeps not recognizing the git host.

I already added the mapping in my local /etc/hosts and on /etc/resolve.conf tips that I found around the web. Can someone help?

Answer 1

Brad Rydzewski's comment got me on the right track. He said:

when you configure gitlab you need to use an IP address or domain name. You cannot use the internal docker hostname (e.g. my-git) because when drone spawns pipeline containers, these containers will be on different networks, and not be able to resolve my-git

This was precisely correct. Here is the final configuration that worked for me:

version: '2'
services:
    web:
      image: 'gitlab/gitlab-ce:latest'
      restart: always
      environment:
        GITLAB_OMNIBUS_CONFIG: |
          external_url '<IP address or domain name>'
          gitlab_rails['gitlab_shell_ssh_port'] = 22
          # Add any other gitlab.rb configuration here, each on its own line
      ports:
        - '80:80'
        - '443:443'
        - '22:22'
      volumes:
        - '/srv/gitlab/config:/etc/gitlab'
        - '/srv/gitlab/logs:/var/log/gitlab'
        - '/srv/gitlab/data:/var/opt/gitlab'

    drone-server:
      image: drone/drone:0.7.3
      links:
        - web
      ports:
        - "8000:8000"
      volumes:
        - /home/drone:/var/lib/drone/
      environment:
        DRONE_OPEN: "true"
        DRONE_HOST: "<IP address or domain name> : <port>"
        DRONE_ADMIN: <admin users - should be git user names>
        DRONE_GITLAB: "true"
        DRONE_GITLAB_URL: "<IP address or domain name>"
        DRONE_GITLAB_CLIENT: "<Application Id in gitlab>"
        DRONE_GITLAB_SECRET: "<Secret in gitlab>"
        DRONE_SECRET: "<secret drone>"
    drone-agent:
      image: drone/drone:0.7.3
      command: agent
      depends_on:
        - drone-server
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
      environment:
        DRONE_SERVER: ws://drone-server:8000/ws/broker
        DRONE_DEBUG: "true"
        DRONE_SECRET: "<secret drone>"

Answer 2

As this is the highest ranking question for this issue I thought to round this up.

The previously mentioned issue has the response you need, but it isn't immediately clear, as it is in the github mentions log and the originally linked issue was closed without a solution.

---

Solution

You can pass DRONE_RUNNER_NETWORKS= to the agent (or the server if you are not using agents). When Drone spawns containers it will attach all containers to the specified network.

Source: https://discourse.drone.io/t/gitea-drone-traefik-agent-cannot-reach-git-repo/4170/2?u=bradrydzewski

Note that you also want to add the networks to your runner in docker-compose: https://docs.docker.com/compose/networking/#specify-custom-networks - otherwise the initial pings will fail and no further runners will be spawned.

An example for a full drone docker runner:

drone-runner-docker:
  volumes:
      - '/var/run/docker.sock:/var/run/docker.sock'
  environment:
      - DRONE_RPC_PROTO=http
      - DRONE_RPC_HOST=172.20.0.19
      - DRONE_RPC_SECRET=SECRET
      - DRONE_RUNNER_CAPACITY=2
      - DRONE_RUNNER_NAME=drone-runner-docker
      - DRONE_RUNNER_NETWORKS=droneio_network,another_network
  networks:
      droneio_network:
      another_network:
  ports:
      - '3000'
  restart: always
  image: 'drone/drone-runner-docker:latest'

Refer to the drone docs for explanation of the variables.

Make sure the names you provide in the environment for DRONE_RUNNER_NETWORKS are the actual network names from docker network ls. (if spawned via docker-compose they usually have the folder name infront, e.g. foldername_default)

Answer 3

TL;DR

The main reason, is because the actual cloning step is driven by a adhoc separate docker container, that is in its own docker network. So it can't resolve the name my-git, and even if it would be able to resolve it, it wouldn't be able to reach it.

The expected workaround fails:

At first, you can notice that your error is about resolving my-git. This has to be tackled on the docker git container instance that actually does the clone, not in any other place. This means to modify the /etc/hosts of this docker instance. This can be done in your .drone.yml, by replacing the cloning step by yours, and using docker-compose's extra_hosts feature (cf docs). This is how it is done:

kind: pipeline
name: default

clone:
  disable: true

steps:
- name: myclone
  image: docker:git
  ## If you wanted to solve resolving part
  extra_hosts:
    - "my-git:10.0.0.200"
  ## Alas this will also be needed, and is NOT supported yet
  # networks:
  #  - gitlab
  commands:
    ## this will show you if ``my-git`` is resolved.
    - ping -c 1 my-git
    - git clone http://my-git/amaziagur/location-service.git/

##
## ... your normal steps follows ...
##

Alas, if your container will be able to resolve now my-git to the actual IP, it will still not be able to reach it, as it was spawned in a custom network that is not connected to the other.

To do that, we would need to be able to specify also the network in which the docker instance is supposed to be connected to, which is usually done through networks key in docker-compose (as you can check in the docker-compose reference about specifying custom networks).

Alas, this is not supported in drone currently and is a bug currently tracked through this issue. It is still not resolved even in 1.0.0-rc.5 as of this writing.

Please note, that if the networks key would be working as expected, the extra_hosts will be superfluous.