StormCrawler - Metadata fields not being persisted

Question

I have a topology with a spout that emits a tuple to the status stream and is picked up by the StatusUpdaterBolt, which in turn write data to an elasticsearch index.

The spout emits a tuple with a Metadata object that contains certain metadata (eg: crawler).

This is not being written to the status index.

The config looks something like this:

bolts:
  - id: "myspout"
    className: com.mycompany.MySpout
    parallelism: 8
  - id: "status"
    className: com.digitalpebble.stormcrawler.elasticsearch.persistence.StatusUpdaterBolt
    parallelism: 4

streams:
  - from: "myspout"
    to: "status"
    grouping:
      type: FIELDS
      args: ["url"]
      streamId: "status"

The Metadata object is built like this:

Metadata metadata = new Metadata();
...
metadata.setValue("crawler", "mycrawl");

and then is emitted:

collector.emit(new Values(url, metadata));

Why would the custom properties not get written to the status index?

Versions:

storm: 2.4.0 stormcrawler: 2.8

Answer 1

As per the documentation here: https://github.com/DigitalPebble/storm-crawler/wiki/MetadataTransfer

It's important to specify what fields you want transferred/persisted into the status index. If you don't, it won't get persisted.

In your example:

metadata.persist:
  - crawler

Note: If you were using parsefilters to extract Outlinks, you'd also need to include:

metadata.transfer:
  - crawler

if you wanted it on new documents generated by outlink identification.