Rally Lookback API doesn't retrieve records newer than 1 week

Question

I'm running some queries with Rally Lookback API and it seems that revisions newer than 1 week are not being retrieved:

λ date
Wed, Nov 28, 2018  2:26:45 PM

using the query below:

{
    "ObjectID": 251038028040,
    "__At": "current"
}

results:

{
    "_rallyAPIMajor": "2",
    "_rallyAPIMinor": "0",
    "Errors": [],
    "Warnings": [
        "Max page size limited to 100 when fields=true"
    ],
    "GeneratedQuery": {
        "find": {
            "ObjectID": 251038028040,
            "$and": [
                {
                    "_ValidFrom": {
                        "$lte": "2018-11-21T14:44:34.694Z"
                    },
                    "_ValidTo": {
                        "$gt": "2018-11-21T14:44:34.694Z"
                    }
                }
            ],
            "_ValidFrom": {
                "$lte": "2018-11-21T14:44:34.694Z"
            }
        },
        "limit": 10,
        "skip": 0,
        "fields": true
    },
    "TotalResultCount": 1,
    "HasMore": false,
    "StartIndex": 0,
    "PageSize": 10,
    "ETLDate": "2018-11-21T14:44:34.694Z",
    "Results": [
        {
            "_id": "5bfe7e3c3f1f4460feaeaf11",
            "_SnapshotNumber": 30,
            "_ValidFrom": "2018-11-21T12:22:08.961Z",
            "_ValidTo": "9999-01-01T00:00:00.000Z",

            "ObjectID": 251038028040,
            "_TypeHierarchy": [
                -51001,
                -51002,
                -51003,
                -51004,
                -51005,
                -51038,
                46772408020
            ],

            "_Revision": 268342830516,
            "_RevisionDate": "2018-11-21T12:22:08.961Z",
            "_RevisionNumber": 53,
        }
    ],
    "ThreadStats": {
        "cpuTime": "15.463705",
        "waitTime": "0",
        "waitCount": "0",
        "blockedTime": "0",
        "blockedCount": "0"
    },
    "Timings": {
        "preProcess": 0,
        "findEtlDate": 88,
        "allowedValuesDisambiguation": 1,
        "mongoQuery": 1,
        "authorization": 3,
        "suppressNonRequested": 0,
        "compressSnapshots": 0,
        "allowedValuesHydration": 0,
        "TOTAL": 93
    }
}

Having in mind that this artifact have, as for now, 79 revisions with the latest revision pointing to 11/21/2018 02:41 PM CST as per revisions tab at Rally Central. One other thing is that if I run the query a couple of minutes later the ETL date seems to be updating, as some sort of indexing being run:

{
    "_rallyAPIMajor": "2",
    "_rallyAPIMinor": "0",
    "Errors": [],
    "Warnings": [
        "Max page size limited to 100 when fields=true"
    ],
    "GeneratedQuery": {
        "find": {
            "ObjectID": 251038028040,
            "$and": [
                {
                    "_ValidFrom": {
                        "$lte": "2018-11-21T14:45:50.565Z"
                    },
                    "_ValidTo": {
                        "$gt": "2018-11-21T14:45:50.565Z"
                    }
                }
            ],
            "_ValidFrom": {
                "$lte": "2018-11-21T14:45:50.565Z"
            }
        },
        "limit": 10,
....... rest of the code ommited.

Is there any reason why Lookback API shouldn't processing current data instead of one week of difference between records?

Answer 1

It appears that your workspace's data is currently being "re-built". The _ETLDate is the date of the most-current revision in the LBAPI database and should eventually catch up to the current revision's date.