I have smart meter data for approximately 6,000,000 meter series. For each meter, we have data for three different schemes: log, daily, and 30min. Should I create three separate tables for these schemes? Our current goal is to identify anomalous meters by analyzing information from all three types of data. For example, we want to select meters that have had a signal greater than X for N consecutive days between dates a and b.
- How can I partition the data to achieve the fastest query speed at the lowest cost?
- Should I store all numerical values in the measure_value column and all non-numerical values in the dimension column?
- I have already conducted some experiments using Node.js and the
aws-sdkto write flattened data into a Timestream database. Is there a tool that can easily flatten large amounts of JSON data into the proper Timestream format for batch processing and conversion into CSV format? I believe that formatting nested JSON and appending it requires significant computing power, so I may need to usepandasto accomplish this task.
This is scheme for one of the data type.
/** @typedef {object} json
* @property {string} type
* @property {string} version
* @property {object} meter
* @property {string} meter.identifier
* @property {string} meter.serial
* @property {object} payload
* @property {string} payload.Type
* @property {string} payload.Version
* @property {string} payload.Result
* @property {null} payload.Reference
* @property {string} payload.ReadDateTime
* @property {object} payload.Meter
* @property {string} payload.Meter.Identifier
* @property {string} payload.Meter.Serial
* @property {string} payload.Meter.MeterPhaseCount
* @property {string} payload.Meter.PlantNumber
* @property {object} payload.Configuration
* @property {string} payload.Configuration.LoadSurvey
* @property {string} payload.Configuration.IntervalLength
* @property {object[]} payload.Configuration.Channels
* @property {string} payload.Configuration.Channels.Source
* @property {string} payload.Configuration.Channels.Type
* @property {object[]} payload.Channels
* @property {object[]} payload.Channels.Intervals
* @property {string} payload.Channels.Intervals.Time
* @property {string} payload.Channels.Intervals.Value
*/