Dockerfile issue - Why is the binary dlv not being found - No such file or directory

Question

I had a docker file that was working fine. However to remote debug it , I read that I needed to install dlv on it and then I need to run dlv and pass the parameter of the app I am trying to debug. So after installing dlv on it and attempting to run it. I get the error

exec /dlv: no such file or directory

This is the docker file

    FROM golang:1.18-alpine AS builder

# Build Delve for debugging
RUN go install github.com/go-delve/delve/cmd/dlv@latest

# Create and change to the app directory.
WORKDIR /app
ENV CGO_ENABLED=0


# Retrieve application dependencies.
COPY go.* ./
RUN go mod download

# Copy local code to the container image.
COPY . ./


# Build the binary.
RUN go build -gcflags="all=-N -l" -o fooapp

# Use the official Debian slim image for a lean production container.
FROM debian:buster-slim

EXPOSE 8000 40000

RUN set -x && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
    ca-certificates && \
    rm -rf /var/lib/apt/lists/*

# Copy the binary to the production image from the builder stage.
#COPY --from=builder /app/fooapp /app/fooapp #commented this out  

COPY --from=builder /go/bin/dlv /dlv

# Run dlv as pass fooapp as parameter
CMD ["/dlv", "--listen=:40000", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "/app/fooapp"]

The above results in exec /dlv: no such file or directory

I am not sure why this is happening. Being new to docker , I have tried different ways to debug it. I tried using dive to check and see if the image has dlv on it in the path /dlv and it does. I have also attached an image of it

Answer 1

You built dlv in alpine-based distro. dlv executable is linked against libc.musl:

# ldd dlv 
        linux-vdso.so.1 (0x00007ffcd251d000)
        libc.musl-x86_64.so.1 => not found

But then you switched to glibc-based image debian:buster-slim. That image doesn't have the required libraries.

# find / -name libc.musl*                                        
<nothing found>

That's why you can't execute dlv - the dynamic linker fails to find the proper lib.

You need to build in glibc-based docker. For example, replace the first line

FROM golang:bullseye AS builder

BTW. After you build you need to run the container in the priviledged mode

$ docker build . -t try-dlv
...
$ docker run --privileged --rm try-dlv
API server listening at: [::]:40000
2022-10-30T10:51:02Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)

In non-priviledged container dlv is not allowed to spawn a child process.

$ docker run --rm try-dlv
API server listening at: [::]:40000
2022-10-30T10:55:46Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
could not launch process: fork/exec /app/fooapp: operation not permitted

Really Minimal Image

You use debian:buster-slim to minimize the image, it's size is 80 MB. But if you need a really small image, use busybox, it is only 4.86 MB overhead.

FROM golang:bullseye AS builder

# Build Delve for debugging
RUN go install github.com/go-delve/delve/cmd/dlv@latest

# Create and change to the app directory.
WORKDIR /app
ENV CGO_ENABLED=0

# Retrieve application dependencies.
COPY go.* ./
RUN go mod download

# Copy local code to the container image.
COPY . ./

# Build the binary.
RUN go build -o fooapp .

# Download certificates
RUN set -x && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
    ca-certificates 

# Use the official Debian slim image for a lean production container.
FROM busybox:glibc

EXPOSE 8000 40000

# Copy the binary to the production image from the builder stage.
COPY --from=builder /app/fooapp /app/fooapp 
# COPY --from=builder /app/ /app

COPY --from=builder /go/bin/dlv /dlv

COPY --from=builder /etc/ssl /etc/ssl

# Run dlv as pass fooapp as parameter
CMD ["/dlv", "--listen=:40000", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "/app/fooapp"]
# ENTRYPOINT ["/bin/sh"]

The image size is 25 MB, of which 18 MB are from dlv and 2 MB are from Hello World application.

While choosing the images care should be taken to have the same flavors of libc. golang:bullseye links against glibc. Hence, the minimal image must be glibc-based.

But if you want a bit more comfort, use alpine with gcompat package installed. It is a reasonably rich linux with lots of external packages for just extra 6 MB compared to busybox.

FROM golang:bullseye AS builder

# Build Delve for debugging
RUN go install github.com/go-delve/delve/cmd/dlv@latest

# Create and change to the app directory.
WORKDIR /app
ENV CGO_ENABLED=0

# Copy local code to the container image.
COPY . ./

# Retrieve application dependencies.
RUN go mod tidy

# Build the binary.
RUN go build -o fooapp .

# Use alpine lean production container.
# FROM busybox:glibc
FROM alpine:latest

# gcompat is the package to glibc-based apps
# ca-certificates contains trusted TLS CA certs
# bash is just for the comfort, I hate /bin/sh
RUN apk add gcompat ca-certificates bash

EXPOSE 8000 40000

# Copy the binary to the production image from the builder stage.
COPY --from=builder /app/fooapp /app/fooapp 
# COPY --from=builder /app/ /app

COPY --from=builder /go/bin/dlv /dlv

# Run dlv as pass fooapp as parameter
CMD ["/dlv", "--listen=:40000", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "/app/fooapp"]
# ENTRYPOINT ["/bin/bash"]

Answer 2

The no such file or directory error indicates either your binary file does not exist, or your binary is dynamically linked to a library that does not exist.

As said in this answer, delve is linked against libc.musl. So for your delve build, you can disable CGO since that can result in dynamic links to libc/libmusl:

# Build Delve for debugging
RUN CGO_ENABLED=0 go install github.com/go-delve/delve/cmd/dlv@latest
...

This even allows you later to use a scratch build for your final target image and does not require you to install any additional packages like musl or use any glibc based image and require you to run in privileged mode.

Answer 3

TL;DR

Run apt-get install musl, then /dlv should work as expected.

Explanation

Follow these steps:

1. docker run -it <image-name> sh
2. apt-get install file
3. file /dlv

Then you can see the following output:

/dlv: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, Go BuildID=xV8RHgfpp-zlDlpElKQb/DOLzpvO_A6CJb7sj1Nxf/aCHlNjW4ruS1RXQUbuCC/JgrF83mgm55ntjRnBpHH, not stripped

The confusing no such file or directory (see this question for related discussions) is caused by the missing /lib/ld-musl-x86_64.so.1.

As a result, the solution is to install the musl library by following its documentation.

My answer is inspired by this answer.