System clock in Docker (MacOS) jumps back in time

71 Views Asked by At

I am running Postgres in Docker on MacOS. My application relies on Postgres generating timestamps rapidly one after another. I encountered some issues, which turned out to be due to Postgres generating timestamps in the past. E.g. the following were created in the order shown, using the CLOCK_TIMESTAMP() function in Postgres:

Timestamp 1: 2023-10-29 16:02:06.887574+00
Timestamp 2: 2023-10-29 16:02:06.872373+00
Timestamp 3: 2023-10-29 16:02:06.891193+00

But the timestamps are out of order: Timestamp 2 < Timestamp 1 < Timestamp 3.

This would happen seemingly random (every few thousand requests to the db server).

However, the above issue does not occur, when using a Postgres server running locally (not in Docker).

This makes me believe that Docker is here to blame. It appears as if the system clock used by Docker (which is used by the Postgres Docker Container to generate timestamps) sometimes shifts back in time. I found the following article on the Docker blog, which is suggesting that this "jump back in time" indeed does happen, but that the "VM clock" should always be running slower than the "system clock", meaning that this should never result in an actual jump back in time... But for me the "VM clock" seems to be running faster, as I am seeing this jump back in time.

This github issue, seems to be running into the same problem as me, but it got closed due to being stale.

My question thus is, is my suspicion above correct that the issue I'm facing is due to a "jump back in time"? Or might there be something else going on?

The below Node.js code snippet, when run long enough, will trigger the issue I'm facing:

const { Client } = require('pg');

const test = async () => {
  const client = new Client({
    host: 'localhost',
    port: 5432,
    database: 'postgres',
  });

  await client.connect();

  let prevClockTimestamp;
  let iteration = 0;
  // eslint-disable-next-line no-constant-condition
  while (true) {
    const {
      rows: [{ clockTimestamp }],
    } = await client.query(
      'SELECT CLOCK_TIMESTAMP()::text AS "clockTimestamp"'
    );

    if (clockTimestamp < prevClockTimestamp) {
      console.log(clockTimestamp, prevClockTimestamp);
      await client.end();
      throw new Error('Gotcha...');
    }

    prevClockTimestamp = clockTimestamp;
    iteration += 1;

    if (iteration % 1000 === 0) {
      console.log(iteration);
    }
  }
};

test();

Node version is v18.18.2 Docker Desktop version is 4.15.0 (93002) Postgres version is 16.0 (runs in Docker) MacOS version is 10.15.7

0

There are 0 best solutions below