how to visualize readible big datasets with matplotlib?

Question

when trying to visualize big datasets in python, the plot and date becomes unreadible. How to make sure the data always keeps readible? All solutions are welcome including making use of other packages.

My code:

import json
import matplotlib.pyplot as plt

# Load the JSON data from the file
with open('sorted_data.json', 'r') as file:
    sorted_data = json.load(file)

# Create a list of dates and corresponding values for each number
dates = list(sorted_data.keys())[:150]
numbers = list(sorted_data.values())[:150]

# Set a larger figure size
plt.figure(figsize=(10, 6))  # Adjust the width and height as needed

# Create a scatter plot for each date
for i in range(len(dates)):
    date = dates[i]
    number_values = list(numbers[i].values())
    plt.scatter([date]*7, list(numbers[i].values()), label=date)

# Adding labels and title
plt.xlabel('Dates')
plt.ylabel('Values')
plt.title('Visualization of Sorted JSON Data')
plt.xticks(rotation=45)  # Rotate the x-axis labels for better visibility
plt.legend()  # Show the legend

# Display the plot
plt.show()

Json sample:

{
    "2000-01-01": {
        "n1": 9,
        "n2": 19,
        "n3": 22,
        "n4": 39,
        "n5": 41,
        "n6": 42,
        "n7": 17
    },
    "2000-01-05": {
        "n1": 9,
        "n2": 13,
        "n3": 14,
        "n4": 22,
        "n5": 23,
        "n6": 39,
        "n7": 18
    },...
}

Output:

Trying to visualize big datasets in python and i expect it to be always readible

Answer 1

The problem is not what matplotlib does, but rather what you ask it to do.

When importing the data, you treat the date strings as string, which makes them categorical entries. There is nothing matplotlib can do to keep the plot readable.

What you probably want to do is to plot the data on a time axis. To do that, parse the strings into datetime objects on import.

dates = [dt.datetime.strptime(k,'%Y-%m-%d') for k in sorted_data.keys()]

The second thing is that you request an individual scatter plot for every date. Rather than doing that, you probably actually want to do it for every key in your json.

Do do that we need to re-structure the input data to list for the individual n1, n2,... values. We can store them in dictionaries for convinience.

dates_dict = {}
numbers_dict = {}
for date,vs, in sorted_data.items():
    for k,v in vs.items():
        if not k in numbers_dict.keys():
            # create lists on first call
            dates_dict[k] = []
            numbers_dict[k] = []
        dates_dict[k].append(dt.datetime.strptime(date,'%Y-%m-%d'))
        numbers_dict[k].append(v)

# Create a scatter plot for each date
for key in numbers_dict.keys():
    plt.scatter(dates_dict[key], numbers_dict[key], label=key)

The plot will then have one color for every key: