I'm trying to use a prepared statement to query with an IN condition with a large list (thousands) of values like this:

   val srcIdLookupFuture = Future[List[List[Any]]] {
      if (srcIdList.nonEmpty) { //srcIdList is like 15k items
        val sql = """select my_cols
                   | from my_table a
                   | join my_other_table b
                   | on a.join_key = b.join_key
                   | where my_id_column in (?);""".stripMargin
        val conn: Connection = connectionPool.getConnection
        val pstmt: PreparedStatement = conn.prepareStatement(sql)
        val chunks: List[List[String]] = srcIdList.sliding(1000, 1000).toList
        try {        
            chunks.par.map(chunk => {
              pstmt.setArray(1, conn.createArrayOf("VARCHAR", chunk.toArray))
              val rs: ResultSet = pstmt.executeQuery()
              getResults(rs)
            }).flatten.toList      
        } finally {
          pstmt.close()
          conn.close()
        }

      } else List(List())

    }

Note that my list of "ids" I want to look up is very large, aprox. 20k. So I chunk it up into 1000 id chunks. But of course the last chunk or or the first will not have 1k values so i cant do something like this PreparedStatement with list of parameters in a IN clause

The way the code written now I get error ERROR: operator does not exist: character varying = character varying[] so i need some other solution. My understanding is that preparing the statement each time is a waste of using a prepared statement (I don't care about sql injection here) I'd also be curious to know if using a prepared statement with a large IN condition is even a good idea? or if I should stick to statement.

0

There are 0 best solutions below