I'm trying to use a prepared statement to query with an IN condition with a large list (thousands) of values like this:
val srcIdLookupFuture = Future[List[List[Any]]] {
if (srcIdList.nonEmpty) { //srcIdList is like 15k items
val sql = """select my_cols
| from my_table a
| join my_other_table b
| on a.join_key = b.join_key
| where my_id_column in (?);""".stripMargin
val conn: Connection = connectionPool.getConnection
val pstmt: PreparedStatement = conn.prepareStatement(sql)
val chunks: List[List[String]] = srcIdList.sliding(1000, 1000).toList
try {
chunks.par.map(chunk => {
pstmt.setArray(1, conn.createArrayOf("VARCHAR", chunk.toArray))
val rs: ResultSet = pstmt.executeQuery()
getResults(rs)
}).flatten.toList
} finally {
pstmt.close()
conn.close()
}
} else List(List())
}
Note that my list of "ids" I want to look up is very large, aprox. 20k. So I chunk it up into 1000 id chunks. But of course the last chunk or or the first will not have 1k values so i cant do something like this PreparedStatement with list of parameters in a IN clause
The way the code written now I get error ERROR: operator does not exist: character varying = character varying[] so i need some other solution. My understanding is that preparing the statement each time is a waste of using a prepared statement (I don't care about sql injection here)
I'd also be curious to know if using a prepared statement with a large IN condition is even a good idea? or if I should stick to statement.