I'm using StreaMSets Data Collector to download from a Microsoft dataverse API, which uses pagination, supplying a next-page link in the record.
I'm using an HTTP Processor stage with Pagination = Link in Response Field.
It works fine when there are more pages, but when there are no more data the field is simply not there and I get "Link field '/EntityData/'@odata.nextLink'' does not exist in record"; nothing I've tried will get around this.
${record:exists("/EntityData/'@odata.nextLink'")} in the Stop Condition should work.
StreaSets Data Collector v3.22.0
It seems like a bug in StreamSets but nothing like that's been fixed according to the Release Notes up to the latest version.
Can anyone advise of a solution?
I checked ${record:exists} function in Stop Condition in the later version of SDC, and it works fine. Must've been fixed since SDC 3.22 (which is quite old).