I am currently gathering data every hour and then importing into my mysql table hosted through clearDB (on heroku).
The problem I am running into is I keep hitting my max number of connections (40), eventhough I am only establishing a few connections while running an insert query.
When reviewing the clearDB dashboard right after i receive the connection limit error i typically see significantly less active connections than what my limit is.
Here is a simplified version of what I am doing.. I collect the data (not shown), assign to an object and then loop through an array of objects that represent a table in mysql. I have tried using connection pooling but found that i hit my limit of 40 even faster. I have also experimented with using connection.end() but no matter where I placed it in the function I get an error that the connection closed prior to the insert query.
My dataArray has 12 objects that it loops through, how am i using 40 connections running this function once?
thanks for any insight!
Error Message:
code: 'ER_USER_LIMIT_REACHED',
errno: 1226,
sqlMessage: "User 'xxxxxxx' has exceeded the 'max_user_connections' resource (current value: 20)",
sqlState: '42000',
fatal: true
const connection = mysql.createConnection({
host: 'us-cluster-xxxx-xxxxxxx',
user: 'xxxxxx',
password: 'xxxxx',
database: 'heroku_xxxxxxx'
});
const insertDataPromise = (query, values) => {
return new Promise((resolve, reject) => {
connection.query(query, values, (error, results) => {
if (error) {
reject(error);
} else {
resolve(results);
}
});
});
};
const getTheData = async () => {
const dataArray = [
{
locationName: 'The Bay - 463044',
tableName: 'the_bay_wind_wave_data',
},
{
locationName: 'Big Shoal - 43331',
tableName: 'big_shoal_wind_wave_data',
},
{
locationName: 'Silva Strait - 43303',
tableName: 'silva_strait_wind_wave_data',
},
......
]
const locationVariables = {
direction: 'SW',
speed: 12,
gustSpeed: 20,
waveHeight: 2,
wavePeriod: 1.2,
barometer: 112,
airTemp: 56,
waterTemp: 58,
};
try {
dataArray.forEach((buoy) => {
const insertDataQuery = `
INSERT INTO ${buoy.tableName}
(direction, speed, gustSpeed, waveHeight, wavePeriod, barometer, airTemp, waterTemp)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
`;
insertDataPromise(insertDataQuery, [direction, speed, gustSpeed, waveHeight, wavePeriod, barometer, airTemp, waterTemp])
})
// Close the connection after all operations.. this ends the connection before the loop completes so I removed it
// connection.end();
console.log(`Data inserted successfully at ${currentPDTTime}.`);
} catch (err) {
console.error('Error inserting data:', err);
} finally {
// Close the connection after all operations.. this ends the connection before the loop completes so I removed it
// connection.end();
}
};