how shuffle is apply with tf.data.experimental.make_csv_dataset

21 Views Asked by At

Small question, but high priority in my situation. How shuffle is apply with tf.data.experimental.make_csv_dataset. For exemple, I want to load 20000 .csv (each is data of a different city) in my dataset:

dataset = tf.data.experimental.make_csv_dataset(
        file_pattern='*.csv',
        num_parallel_reads=4,
        batch_size=512,
        num_epochs=10,
        label_name='streamflow',
        select_columns=keep_columns,
        shuffle_buffer_size=10000,
        header=True,
        field_delim=','
    )

1- shuffle the orther of read of the csv list only

2- shuffle the orther of sample inside each .csv read

3- shuffle all sample of all csv like it was an single big csv

Thank for your time

0

There are 0 best solutions below