MarkLogic Optic API: Dynamic update support for Template View Configuration

Question

<column>
  <name>daysFromACertainDate</name>
  <scalar-type>int</scalar-type>
  <val>fn:floor((fn:current-dateTime() - xs:dateTime(./updateDate)) div xs:dayTimeDuration('P1D'))</val>
  <nullable>true</nullable>
  <invalid-values>ignore</invalid-values>
</column>

When using calculations and transformations via Template View Configuration XML, are the values calculated for dateTime transformations tied to the exact time the TDE configuration is persisted into the database? (via tde:template-insert()) This seems to be the case, as this field does not update on it's own when running optic plans against them daily. We were hoping that this column calculation would be dynamic and not have to reinsert the template configuration XML on a daily basis to get these column values to update...

For our use case, we leverage this template view and the Optic API to serve data to a UI. We do sorting and filtering all within the Optic plans as well, serving a JSON representation of each plan's result set to UI itself, which is displayed in a tabular list view. With these requirements transforming fields to our plan's result set are not an option.

Is there any way we can get this calculation to be dynamic using existing Optic or TDE functionality? We are on MarkLogic 10-8.3.

Answer 1

In addition to Mads' suggestion to use query-based views, here is a description of how your example works. TDE is applied at indexing time. All the expressions in the val elements are evaluated at that time. Indexing happens when you insert a fragment (a new document or an update to an existing document.) But there are nunerous occasions when reindexing occurs: new TDE or new version of a TDE that applies to a document, new range index, explicit reindexing, etc.

The following TDE exposes some kind of ID, then a datetime taken from the document itself, supposed to be the ingestion timestamp, and the current timestamp when the TDE is applied:

import module namespace tde = "http://marklogic.com/xdmp/tde" at "/MarkLogic/tde.xqy";

let $tde :=
  <template xmlns="http://marklogic.com/xdmp/tde">
    <context>/test</context>
    <rows>
      <row>
        <schema-name>test</schema-name>
        <view-name>test</view-name>
        <columns>
          <column>
            <name>id</name>
            <scalar-type>string</scalar-type>
            <val>id</val>
          </column>
          <column>
            <name>ingested</name>
            <scalar-type>dateTime</scalar-type>
            <val>ingested</val>
          </column>
          <column>
            <name>indexed</name>
            <scalar-type>dateTime</scalar-type>
            <val>fn:current-dateTime()</val>
          </column>
          <!--column>
            <name>extra</name>
            <scalar-type>string</scalar-type>
            <val>'Hello, world!'</val>
          </column-->
        </columns>
      </row>
    </rows>
  </template>
return
(:
  tde:validate($tde)
  tde:node-data-extract( 
    <test>
      <id>ant</id>
      <ingested>{fn:current-dateTime()}</ingested>
    </test>,
    $tde)
:)
  tde:template-insert('/test/tde.xml', $tde)

If you ingest a few documents like this (do not use a loop, for the timestamp to be different in each document):

xdmp:document-insert(
  '/test/ant.xml',
  <test>
    <id>ant</id>
    <ingested>{fn:current-dateTime()}</ingested>
  </test>)

If you SQL select everything in the view, you will get something like the following:

id     ingested                       indexed
ant    2022-10-01T08:14:48.996588Z    2022-10-01T08:14:48.996588Z
bear   2022-10-01T08:14:56.524203Z    2022-10-01T08:14:56.524203Z
cat    2022-10-01T08:15:13.608067Z    2022-10-01T08:15:13.608067Z
dog    2022-10-01T08:15:19.887738Z    2022-10-01T08:15:19.887738Z

The ingested and indexed are the same, both calls to fn:current-dateTime() happen in the same transaction, and so return the same value.

But if you insert a new version of the TDE, for instance by uncommenting the field extra in the example above, and inserting it again, this will trigger reindexing of the /test documents. If you select again the view, you will get this:

id     ingested                       indexed                        extra
ant    2022-10-01T08:14:48.996588Z    2022-10-01T08:19:45.672175Z    Hello, world!
bear   2022-10-01T08:14:56.524203Z    2022-10-01T08:19:45.709324Z    Hello, world!
cat    2022-10-01T08:15:13.608067Z    2022-10-01T08:19:45.709324Z    Hello, world!
dog    2022-10-01T08:15:19.887738Z    2022-10-01T08:19:45.709324Z    Hello, world!

You can see that the indexed value has been updated. The value in the document has not changed, but the expression in the TDE has been evaluated again.

Now in your initial case, you want to introduce a value from query time. This is something TDE cannot achieve on its own. But you can expose updateDate as a dateTime and use it in your query. Modifying your TDE every day is not something I would recommend, as it will trigger reindexing of all the documents it applies to.