How to iterate over a huge amount of records with scala sorm

Question

I want to iterate over all records of a specific table in sorm, but I want to do it in a way that it is memory efficient.

The code that I use today is:

Db.query[Items].whereEqual("title", someTitle).fetch.foreach { webCount =>
          //do something
}

The problem is that this code first loads all records, before going into each item. Is there any way to stream the records?

Answer 1

Ideally such functionality would require support for database cursors, but it's not implemented.

However this is resolvable with manual batching:

val results : Stream[ Items ] = {
  val batchSize = 256
  Stream
    .from(0)
    .map(Db.query[Items].whereEqual.limit(batchSize).offset(_ * batchSize).fetch)
    .takeWhile(_.nonEmpty)
    .flatten
}

Of course, you can wrap this pattern in a utility function or an implicit transformation.