How to train a model with CSV files of multiple patients?

Question

I have CSV files for 167 patients, they have the same number and type of columns but the number of rows in each CSV file varies. I want to train a CNN or LSTM but without combining all the CSVs. I want to train the model using each file separately because it's important for me to distinguish between each patient, so I don't combine all the files. Each file has a target column which is the values of a signal that I need to predict. It's a regression problem. Each file has about 50 columns, meaning 50 features. Could you help me? I don't know how to do this in python.

I combined all the files and trained my model with 80% of the data, but then I realized that it's incorrect to combine the data from all the patients. Each patient has a unique behavior in their characteristics.

Answer 1

I suggest you import them into a dataframe with pandas and then use dataframe.shape to get the number of rows and columns of each of them. Then you make the number of rows an input to your program.

You can also export your pandas dataframe to a numpy array using pandas' to_numpy() method, so it should fit nicely into your data import routine.