GCE "create-with-container --container-mount-disk" flag mounts disk as read-only

Question

I am trying to use a Percona Docker image for MongoDB on GCE, however I'm running into an issue with Mongo saying the mounted path is read-only. I looked around as much as I could, but im stumped at what could be the issue.

gcloud compute instances create-with-container mongo-svr \
--create-disk name=disk-1,size=1GB \
--container-mount-disk mount-path="/data/mongodb",mode=rw \
--container-image=docker.io/percona/percona-server-mongodb:4.2

I used the above command and it creates my instance. I then SSH into the server, connect to the running mongo instance to shutdown, then I run: docker exec -it [NAME] mongod --configsvr --replSet rs0 --dbpath=/data/mongodb --bind_ip localhost

This spits out an error stating:

CONTROL  [initandlisten] options: { net: { bindIp: "localhost" }, replication: { replSet: "rs0" }, sharding: { clusterRole: "configsvr" }, storage: { dbPath: "/data/mongodb" } }
STORAGE  [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /data/mongodb, terminating

At this point, I've been recreating instances with different params, but nothing has worked so far. Anyone have an idea what I'm missing?

Updated with command output

gcloud compute instances create-with-container mongo-config-f --zone us-central1-f --create-disk name=disk-1,size=1GB --container-mount-disk mount-path="/data/mongodb" --container-image=docker.io/percona/percona-server-mongodb:4.2 --machine-type=f1-micro
WARNING: Default device-name for disk name [disk-1] will be [disk-1] because it is being mounted to a container with [`--container-mount-disk`]
Created [https://www.googleapis.com/compute/v1/projects/[PROJECT_NAME]/zones/us-central1-f/instances/mongo-config-f].
NAME            ZONE           MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
mongo-config-f  us-central1-f  f1-micro                   xx.xx.xx.xx  xx.xx.xx.xx     RUNNING

Answer 1

I've tried to replicate your issue on my test project and found that:

• persistent disk was created and mounted in read-write mode as expected;

  bash-4.2$ mount 
  ...
  /dev/sdb on /data/mongodb type ext4 (rw,relatime)
  

• docker runs containers inside our VM properly;

• the cause of the error while running docker exec -it [NAME] mongod --configsvr --replSet rs0 --dbpath=/data/mongodb --bind_ip localhost is permissions inside mongodb container:

  bash-4.2$ ls -l /data/        
  ...
  drwxr-xr-x 3 root    root 4096 Feb 19 15:33 mongodb
  

As a workaround commands could be executed with root permissions:

$ docker exec -it --user root klt-mongo-svr-upd-wowt mongod --configsvr --replSet rs0 --db path=/data/mongodb

Please find more details and my steps below:

1. create VM:

   $ gcloud compute instances create-with-container mongo-svr \
   --create-disk name=disk-1,size=1GB \                                                                                 
   --container-image docker.io/percona/percona-server-mongodb:4.2 \
   --container-mount-disk mount-path="/data/mongodb"                               
   WARNING: Default device-name for disk name [disk-1] will be [disk-1] because it is being mounted to a container with [`--container-mount-disk`]
   Created [https://www.googleapis.com/compute/v1/projects/test-prj/zones/europe-west3-a/instances/mongo-svr].
   NAME           ZONE            MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
   mongo-svr-upd  europe-west3-a  n1-standard-1               10.156.0.9   35.XXX.155.XXX  RUNNING
   

2. SSH to instance;

3. check if container is running:

   $ docker ps
   CONTAINER ID        IMAGE                                                                COMMAND                  CREATED              STATUS              PORTS               NAMES
   dfad9c10235d        percona/percona-server-mongodb:4.2                                   "/entrypoint.sh mong…"   About a minute ago   Up About a minute                       klt-mongo-svr-upd-wowt
   bbe02c8e8621        gcr.io/stackdriver-agents/stackdriver-logging-agent:0.2-1.5.33-1-1   "/entrypoint.sh /usr…"   About a minute ago   Up About a minute                       stackdriver-logging-agent
   

   everything looks good at this point;

4. try to run command as user:

    $ docker exec -it klt-mongo-svr-upd-wowt mongod --configsvr --replSet rs0 --dbpath=/data/mongodb --bind_ip localhost
   

   and observe the same error:

   2020-02-19T15:37:53.176+0000 I  STORAGE  [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /data/mongodb, terminating
   

   here key read-only directory: /data/mongodb;

5. check mounts and permissions inside of the container:

   $ docker exec -it klt-mongo-svr-upd-wowt /bin/bash
   bash-4.2$ mount 
   ...
   /dev/sdb on /data/mongodb type ext4 (rw,relatime)
   ...
   

   as we expected disk was created and mounted in read-write mode to the container

   bash-4.2$ ls -l /data/        
   total 8
   drwxr-xr-x 4 mongodb root 4096 Feb 19 15:36 db
   drwxr-xr-x 3 root    root 4096 Feb 19 15:33 mongodb
   bash-4.2$ 
   

   but to work with /data/mongodb you need root permissions;

6. try to run command as root:

   $ docker exec -it --user root klt-mongo-svr-upd-wowt mongod --configsvr --replSet rs0 --dbpath=/data/mongodb
   2020-02-19T15:45:24.970+0000 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
   2020-02-19T15:45:24.973+0000 I  CONTROL  [initandlisten] MongoDB starting : pid=119 port=27019 dbpath=/data/mongodb 64-bit host=mongo-svr-upd
   2020-02-19T15:45:24.974+0000 I  CONTROL  [initandlisten] db version v4.2.2-3
   2020-02-19T15:45:24.974+0000 I  CONTROL  [initandlisten] git version: 2cdb6e50913583f627acc5de35dc4e04dbfe196f
   2020-02-19T15:45:24.974+0000 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017
   2020-02-19T15:45:24.974+0000 I  CONTROL  [initandlisten] allocator: tcmalloc
   2020-02-19T15:45:24.974+0000 I  CONTROL  [initandlisten] modules: none
   2020-02-19T15:45:24.974+0000 I  CONTROL  [initandlisten] build environment:
   2020-02-19T15:45:24.975+0000 I  CONTROL  [initandlisten]     distarch: x86_64
   2020-02-19T15:45:24.975+0000 I  CONTROL  [initandlisten]     target_arch: x86_64
   2020-02-19T15:45:24.975+0000 I  CONTROL  [initandlisten] options: { replication: { replSet: "rs0" }, sharding: { clusterRole: "configsvr" }, storage: { dbPath: "/data/mongodb" } }
   2020-02-19T15:45:24.976+0000 I  STORAGE  [initandlisten] Detected data files in /data/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
   ...
   

   and it's working with root permissions.