Passing a list into pd.read_sql

24 Views Asked by At

Really hoping I can explain my problem properly - I think it's a simple problem but convoluted in it's execution due to the data I'm dealing with.

What I'm trying to do is use the contents of a Python column (account numbers stored as text) to lookup corresponding values in an Oracle database (and pull in additional data from the Oracle db). I'm very nearly there, just having one last small issue which involves passing a list of size n into my pd.read_sql statement.

To give a little bit of context to the steps I've taken so far:

  1. I take the contents of the df column and convert it to a list: accs_to_lookup = df['account_no'].tolist()

  2. Then I split the list into chunks of 1000, as this seems to be the lookup limit (error message DatabaseError: ORA-01795: maximum number of expressions in a list is 1000)

n = 1000
def create_chunks(l, n): 
    for i in range(0, len(accs_to_lookup), n):  
        yield l[i:i + n]

accs_to_lookup = list(create_chunks(accs_to_lookup, n))

**Important to note that this is a repeatable process and accs_to_lookup length will vary week to week. **

  1. I then feed the accs_to_lookup chunks into a pd.read_sql statement. I wont put the entire SQL statement up as it's long, involves CTEs and technically works fine. The issue I have is in the WHERE element which needs honing. The way I'm dealing with the accs_to_lookup length variation currently is manually e.g. where the list length is 2, I manually create 2 x variables and pass them into the read_sql statement
temp1 = (accs_to_lookup[0])
temp2 = (accs_to_lookup[1])

followed by:

pd.read_sql(SELECT columns FROM table WHERE refno IN
                ({",".join(["'" + str(x) + "'"  for x in temp1])})
            OR a.fullref IN
               ({",".join(["'" + str(x) + "'"  for x in temp2])})
           )

So I'm hoping to make that aspect dynamic e.g. replace temp1, temp2, tempn with a statement that works out how many chunks there are on the fly rather than hardcoding it in each time.

Like I said above, hoping I've explained my problem properly but happy to give more info if you've struggled to follow (and kudos if you've made it this far!)

Gratefully appreciate any help received :)

0

There are 0 best solutions below