Detect changes in a list of variances

Question

I'm working on calculating the average distance between an object and ultrasonic sensor. The ultrasonic sensor is connected with arduino but the data is being transferred to python. My detection goes like this

• Ultrasonic sensor is giving normal data with less fluctuating variance
• Object goes in front of Ultrasonic sensor
• The variance shoots up and starts an upwards movement
• After some time, variance starts decreasing whether the object is in front of Ultrasonic sensor or not

Now what I want to understand is how can I save or know at what moment the variance shot up. So I'll start calculating the average of distance from the time when variance shot up.

I've attached the graph and numbers it gave me. Also, my code is following:

import matplotlib.pyplot as plt
from serial import Serial
import numpy as np

try:
    ard = Serial(port='COM6', baudrate=9600)
    print("connect")
except:
    print("Port is not conneced")
    exit(0)


x = []
y = []
variance = []
mean = []
data = 0
no_of_samples = 0


for i in range(300):

    data = float(ard.readline().decode().strip())

    if data > 400:
        data = 400

    x.append(no_of_samples)
    y.append(int(data))
    no_of_samples += 1

    variance.append(round(np.var(y), 3))
    mean.append(round(np.mean(y), 3))

    print(f"Mean: {mean[-1]}\tVariance: {variance[-1]}\tDistance: {y[-1]}\tSample #: {x[-1]}")


plt.xlabel('No of Samples')
plt.ylabel('Variance')
plt.plot(x, variance)


plt.draw()
plt.show(block=True)

Numbers

Graph

I've tried multiple approaches like calculating the average of variance but all in vain. I want to calculate the average difference that starts when the variance rises instantly.

Answer 1

I think that you are addressing the well-known "Change point detection (CPD)" problem, it consists of finding abrupt changes in one-dimensional data. (I hope the concrete name helps you to find more resources)

I suggest ruptures package in Python. There you will find suitable algorithms to detect such change depending on your use case. Here I put a brief example with the Binary Segmentation method. First blue shade is before the change, first red shade is after the change and so on.

import ruptures as rpt

data = np.array([200, 201, 200, 202, 203, 200, 11, 12, 13, 201, 203, 203, 200, 199])
model = rpt.Binseg(model='l2').fit(data)
pred = model.predict(n_bkps=2) # number of change points to detect
rpt.show.display(data, pred, figsize=(10, 6))[0].savefig('fig.png')

Here is a notebook with additional good examples: Kaggle: Change Point Detection. I hope this notebook help you to debug and find the most suitable technique.