Get all records within a window in spark structured streaming

18 Views Asked by At

I've a streaming application in structured streaming , I want to get all the records within a defined window (this is a event time window), when I am trying to use groupby(window) and use aggregate as count , I'm not able to fetch all the columns.

        val windowSpec = window($"timestamp", " minute","1 minutes")

        // Perform the grouping and aggregation
        val windowedDF = timestampedDF
                .withColumn("window", windowSpec)
                .groupBy($"window")
                .count()

How can I get all the records within a defined window and without using an agg. functions ?

I tried Using flatMapGroupsWithState with forEachBatch this solution but stuck in same I need to use forEachBatch for some computations on Micro batches.

0

There are 0 best solutions below