LSHModel on spark structured streaming

Question

Apparently, the LSHModel of MLLib from spark 2.4 supports Spark Structured Streaming (https://issues.apache.org/jira/browse/SPARK-24465).

However, it's not clear to me how. For instance an approxSimilarityJoin from MinHashLSH transformation (https://spark.apache.org/docs/latest/ml-features#lsh-operations) could be applied directly to a streaming dataframe?

I don't find more information online about it. Could someone help me?

Answer 1

You need to

1. Persist the trained model (e.g. modelFitted) somewhere accessible to your Streaming job. This is done outside of your streaming job.

modelFitted.write.overwrite().save("/path/to/model/location")

2. Then load this model within you Structured Streaming job

import org.apache.spark.ml._
val model = PipelineModel.read.load("/path/to/model/location")

3. Apply this model to your streaming Dataframe (e.g. df) with

model.transform(df)

// in your case you may work with two streaming Dataframes to apply `approxSimilarityJoin`.

It might be required to get the streaming Dataframe into the correct format to be used in the model prediction.