DocumentDB start_at_operation_time params gives "modifyChangeStreams has not been run for this collection/database/cluster"

Question

stream = coll.watch(full_document="updateLookup",start_at_operation_time=x)

throws

pymongo.errors.OperationFailure: modifyChangeStreams has not been run for this collection/database/cluster, full error: {'ok': 0.0, 'operationTime': Timestamp(1704966523, 1), 'code': 136, 'errmsg': 'modifyChangeStreams has not been run for this collection/database/cluster'}

however, stream = coll.watch(full_document="updateLookup") works correctly indicating modifyChangeSteams was enabled

DocumentDB is 4.0, start at operation time is expected to work

The collection has data from 2019. I've tried start_at_operation_time timestamps ranging from then to 2024 - all with the same error

Only relevant resource where issue existed (but unsolved) -

• https://www.mongodb.com/community/forums/t/issue-around-usage-of-startatoperationtime-in-change-streams/150049

Answer 1

I am also facing the same issue. Though my data is recent, and I am developing something from scratch, it worked for the first time, but I did dropped my Db collections, as it contained lot of test data, when I was setting up Change Streams Event Driven Flow, after that I am getting this error, even when I tried to disable streams from Cloud9, and re-enable it. I tried to purge all the data and delete cloudformation stack, Lambda ESM trigger and also re-enabling the change streams, but it didn't worked for me.

I have enabled streams as well with this command -

db.adminCommand({modifyChangeStreams: 1, database: "dbName",collection: "colName",enable: true});

After that I tried to list all streams, then also I am getting that streams have been enabled -

rs0 [direct: primary] dbName> db.runCommand({ aggregate: 1, pipeline: [ { $listChangeStreams: 1 }], cursor: {} });

Got this output from there -

{waitedMS: Long('0'),cursor: {firstBatch: [ { database:'dbName',collection: 'colName' } ],
id: Long('0'),
ns: 'dbName.$cmd'},

Do let me know, in case you find the reason and solution for this. Thanks

---

Proposed Solution -

I tried figuring out and messing with the params, to understand the reason why this is happening, I was not able to understand the reason for the same, but I was able to get a workaround of this issue, I'll share with you my serverless Event Source Mapping of the Lambda -

MyLambdaESM:
  Type: AWS::Lambda::EventSourceMapping
  Properties:
    BatchSize:
      Type: Number
      Default: 100
      Description: Batch Size Lambda ESM
    DocumentDBEventSourceConfig: 
      CollectionName: 'colName'
      DatabaseName: 'dbName'
      FullDocument: 'UpdateLookup'
    Enabled: true
    EventSourceArn: "arn:aws:rds:ap-south-1:${ACCOUNTId}:cluster:cluster-name"
    FunctionName: 
      Ref: AWS::Lambda::Function.changeStreamsHandler
    FunctionName: !GetAtt changeStreamsHandler.Arn
    MaximumBatchingWindowInSeconds: 5
    MaximumRetryAttempts: 2
    SourceAccessConfigurations: 
      -   Type: 'BASIC_AUTH'
          URI: 'arn:aws:secretsmanager:ap-south-1:${ACCOUNTId}:secret:secretKeyValue'
    StartingPosition: 'LATEST'

This configuration solves the issue that I was facing -

PROBLEM: modifyChangeStreams has not been run for this collection/database/cluster

I have come to the conclusion that we need to set the Starting Position of the change streams to 'Latest', what earlier I was using was -

StartingPosition: 'TRIM_HORIZON'

When I was using it earlier, I tried changing my databaseCluster, deleting serverless template, deleting cloudformation stack, even attaching the trigger manually with StartingPosition as TRIM_HORIZON, but nothing worked.

I did not found the solution to as to why, my trigger does not give any error when I use StartingPosition: 'LATEST'

Please anyone feel free to add - on to what I have found and let's help each other in this community.

Thanks for the read. Do upvote the solution if you liked it. :)