I am loading data via python from SQL to MariaDB and my python script runs fine, it import the data and loads all of it. When I check the table to see how much data is there by doing SELECT COUNT(*) FROM TABLE then it shows only 120,000 records. When I open a new SQL script in DBeaver, and run the same command, it shows the table only has 80,000 records.
It’s the same table, same database, same everything. I think the issue is when I made a copy of the connection in DBeaver. I made a copy of the connection I am currently on then I deleted it when I was done using it so now I am only working with one copy of the database and the connection to MariaDB itself. Since then, it’s like splitting the load and I don’t know why. I deleted one copy, is there somewhere else I need to delete it?
It’s so stupid, I do right click database -> SQL editor -> New SQL script then I enter SELECT COUNT(*) FROM TABLE and it returns 120,000 record.
Then I do right click database -> SQL editor -> New SQL script then I enter SELECT COUNT(*) FROM TABLE and it returns 80,000 records in a new SQL script window. I have no idea what the issue is, there is only 1 table and 1 database in my whole DBeaver.
This is a probably transaction-commit issue. In your python program be sure to to call .commit() every few thousand records while you're loading, and also at the end of your program before you close the connection to your MariaDB server.