Google App Engine Search API Not Including Time in DateField Sort

Question

I have set of search documents that have a DateField that I would like to sort by. The values in this field also contain the time. When I try to sort descending by this field, I'm getting the dates to sort correctly, but it seems as though the time is ignored ie:

{
  "results": [
    {
      "photo_create_date": "2016-01-04T16:51:39.096000",
    },
    {
      "photo_create_date": "2016-01-04T17:55:36.483000",
    },
    {
      "photo_create_date": "2016-01-04T22:46:37.141000",
    },
    {
      "photo_create_date": "2016-01-04T16:51:13.450000",
    },
    {
      "photo_create_date": "2016-01-04T22:44:10.289000",
    },
    {
      "photo_create_date": "2016-01-04T22:36:28.252000",
    },
    {
      "photo_create_date": "2015-12-30T18:06:34.511000",
    }
  ]
}

Any idea how to fix this or is this a limitation of the GAE search API?

Answer 1

Seeing as though, this seems to be a bug I had to roll my own solution. Here is what I used:

create_date_aware = pytz.utc.localize(item.create_date)
epoch_datetime = datetime.datetime(1970, 1, 1, tzinfo=pytz.utc)
create_timestamp = (create_date_aware - epoch_datetime).total_seconds()

All you have to do is store this value as a NumberField and it works out pretty well. Some credit goes to this question in figuring out how to do this:

python - datetime with timezone to epoch

Here is the bug I filed with Google:

https://code.google.com/p/googleappengine/issues/detail?id=12650&thanks=12650&ts=1452021081

They rejected this as works as intended, so I created a feature request:

https://code.google.com/p/googleappengine/issues/detail?id=12651&thanks=12651&ts=1452038680

Answer 2

This is the documented behaviour.

While it looks like you have a work around already, you have a few options:

• Store as a numeric as seconds since epoch
• Store as a numeric as millis since epoch, however you will need to divide by at least 1000 to be able to fit in the numeric range available in search index
• Use multiple fields (date, time, possibly timezone)

If you need to retain timezones, you will need to use a multifield approach, with a canonical form in seconds/millis since epoch for comparison.