Validation of object within a list in Parallel C#

238 Views Asked by At

I'm implementing an import functionality, wherein I need to read the data from the user uploaded CSV or Excel file and run a few validations and sanitize the data before writing the data to DB.

I'm able to get the data from the file to a list of objects, with the following structure:

public class Order
{
    public string Sku { get; set; }

    public decimal Cost { get; set; }

    public DateTime OrderFulfillmentStartDate { get; set; }

    public DateTime OrderFulfillmentEndDate { get; set; }

    public string ValidationErrors{ get; set; }
}

Following are the validations that need to happen within the objects in the list, a few examples below:

  1. No two orders with the same SKU and OrderFulfillmentStartDate, OrderFulfillmentEndDate are allowed.
  2. No two orders with the same SKU and overlapping OrderFulfillmentStartDate, OrderFulfillmentEndDate are allowed.

etc.

The way I implemented it:

  1. During the first encounter of a distinct record (passing all the validations AND "ValidationErrors" == string.empty), I'm adding the record to a temp list.
  2. During the next iteration, I'm validating the currently processing record with the record present in the temp list, if the validation fails, I'm populating the "ValidationErrors" field and adding the temp list.

E.g:

enter image description here

Now coming to the crux of the issue: The size of the data can be around One Million rows.

When I implemented the validations sequentially using foreach loop, the validation process is taking more than 8 hours. Having said that, I believe doing the validation in parallel would cut down the time needed drastically.

I did try implementing the logic using Parallel.ForEach and Partitioner concept. The processing did quicken up, but I'm not sure, how I can keep a temp list that can be used/updated by multiple threads in the ForEach loop, during the validation.

Is there is a better or quicker approach to achieve what I'm trying to do here? Please do let me know about it.

Thank You!

0

There are 0 best solutions below