How to make a large number of GET requests with StandaloneAhcWSClient

Question

How to make a very large number of simple get requests with StandaloneAhcWSClient in Scala? (it's the default http client bundled with Play2 framework). In my case, I got ~100K GET requests to make to an external API. Future.traverse() does not cut it, is there a better approach, maybe a way to process the list of url sort of like a stream?

Here's the code I have now: https://scastie.scala-lang.org/HgrIyR23TmG12j3MzMCxUw

It works up to a certain number of urls in the list, but breaks with a large number with an exception java.lang.IllegalStateException: failed to create a child event loop

Answer 1

Here's what I ended up with:

import play.api.libs.json.JsValue
import akka.actor.ActorSystem

import scala.concurrent.{Await, Future}
import scala.concurrent.duration.Duration
import scala.concurrent.ExecutionContext.Implicits.global
import play.api.libs.ws.StandaloneWSClient
import akka.stream.ActorMaterializer
import play.api.libs.ws.ahc.StandaloneAhcWSClient
import play.api.libs.ws.JsonBodyReadables._
import play.api.libs.json._

implicit val system: ActorSystem = ActorSystem()
system.registerOnTermination {
  System.exit(0)
}
implicit val materializer: ActorMaterializer = ActorMaterializer()

def getAllRecApiResponses(urls: List[String])(
  implicit actorSystem: ActorSystem,
  materializer: ActorMaterializer): List[JsValue] = {
  implicit val wsClient: StandaloneWSClient = StandaloneAhcWSClient()

  val res: Future[List[JsValue]] = Future.traverse(urls)(urlString => {
    wsClient.url(urlString).get().map(_.body[JsValue]).recover {
      case ex: Exception => {
        println( s"Url call returned exception for url $urlString: $ex" )
        JsNull
      }
    }
  }) andThen { case _ => wsClient.close() }

  Await.result(res, Duration.Inf)
}

val result = getAllRecApiResponses(List.fill(10)("https://jsonplaceholder.typicode.com/todos/1"))

result foreach println

With following build.sbt:

scalaVersion := "2.11.12"

val liftVersion = "2.6"

libraryDependencies ++= Seq(
  "com.typesafe.play" %% "play-ahc-ws-standalone" % "2.0.3",
  "com.typesafe.play" %% "play-ws-standalone-json" % "2.0.3",
  "com.typesafe.play" %% "play-json" % "2.7.2"
)