clearDB max connection limit issue

70 Views Asked by At

I am currently gathering data every hour and then importing into my mysql table hosted through clearDB (on heroku).

The problem I am running into is I keep hitting my max number of connections (40), eventhough I am only establishing a few connections while running an insert query.

When reviewing the clearDB dashboard right after i receive the connection limit error i typically see significantly less active connections than what my limit is.

Here is a simplified version of what I am doing.. I collect the data (not shown), assign to an object and then loop through an array of objects that represent a table in mysql. I have tried using connection pooling but found that i hit my limit of 40 even faster. I have also experimented with using connection.end() but no matter where I placed it in the function I get an error that the connection closed prior to the insert query.

My dataArray has 12 objects that it loops through, how am i using 40 connections running this function once?

thanks for any insight!

Error Message:

  code: 'ER_USER_LIMIT_REACHED',
  errno: 1226,
  sqlMessage: "User 'xxxxxxx' has exceeded the 'max_user_connections' resource (current value: 20)",
  sqlState: '42000',
  fatal: true
const connection = mysql.createConnection({
  host: 'us-cluster-xxxx-xxxxxxx',
  user: 'xxxxxx',
  password: 'xxxxx',
  database: 'heroku_xxxxxxx'
});

const insertDataPromise = (query, values) => {
  return new Promise((resolve, reject) => {
    connection.query(query, values, (error, results) => {
      if (error) {
        reject(error);
      } else {
        resolve(results);
      }
    });
  });
};

const getTheData = async () => {

  const dataArray = [
    {
      locationName: 'The Bay - 463044',
      tableName: 'the_bay_wind_wave_data',
    },
    {
      locationName: 'Big Shoal - 43331',
      tableName: 'big_shoal_wind_wave_data',
    },
    {
      locationName: 'Silva Strait - 43303',
      tableName: 'silva_strait_wind_wave_data',
    },
    ......
  ]

    const locationVariables = {
      direction: 'SW',
      speed: 12,
      gustSpeed: 20,
      waveHeight: 2,
      wavePeriod: 1.2,
      barometer: 112,
      airTemp: 56,
      waterTemp: 58,
    };

    try {

    dataArray.forEach((buoy) => {

      const insertDataQuery = `
        INSERT INTO ${buoy.tableName}
        (direction, speed, gustSpeed, waveHeight, wavePeriod, barometer, airTemp, waterTemp)
        VALUES (?, ?, ?, ?, ?, ?, ?, ?)
      `;

      insertDataPromise(insertDataQuery, [direction, speed, gustSpeed, waveHeight, wavePeriod, barometer, airTemp, waterTemp])
    })
    // Close the connection after all operations.. this ends the connection before the loop completes so I removed it
    // connection.end();
    console.log(`Data inserted successfully at ${currentPDTTime}.`);
  } catch (err) {
    console.error('Error inserting data:', err);
  } finally {
    // Close the connection after all operations.. this ends the connection before the loop completes so I removed it
    // connection.end();
  }
};
0

There are 0 best solutions below