Download huge number of images in the most efficient and fast way

Question

In my app, I am needing to download about 6,000 images. I realize this is a lot, but it is needed.

Currently I am using doing the following:

NSArray *photos = @[hugeAmountOfPhotoObjects];
for (ZSSPhoto *photo in photos) {
    [self downloadImageWithURL:photo.mobileURL progress:^(double progress) {

    } completion:^(UIImage *image) {

        // Save the image

    } failure:^(NSError *error) {

    }];
}

...

- (void)downloadImageWithURL:(NSURL *)url progress:(void (^)(double progress))progress completion:(void (^)(UIImage *image))completion failure:(void (^)(NSError *error))failure {

    NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:url];
    [request setTimeoutInterval:600];
    self.operationQueue.maxConcurrentOperationCount = NSOperationQueueDefaultMaxConcurrentOperationCount;
    AFHTTPRequestOperation *requestOperation = [[AFHTTPRequestOperation alloc] initWithRequest:request];
    requestOperation.responseSerializer = [AFImageResponseSerializer serializer];
    [requestOperation setCompletionBlockWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject) {
        completion(responseObject);
    } failure:^(AFHTTPRequestOperation *operation, NSError *error) {
        [self processOperation:operation error:error failure:failure];
    }];
    [requestOperation setDownloadProgressBlock:^(NSUInteger bytesRead, long long totalBytesRead, long long totalBytesExpectedToRead) {
        double percentDone = (double)totalBytesRead / (double)totalBytesExpectedToRead;
        progress(percentDone);
    }];
    [self.operationQueue addOperation:requestOperation];

}

The problem here is that it takes forever to download using this method, and some of my users are reporting crashing because of high memory usage.

Is there a better method that I could be using to download such a large number of image files?

Answer 1

You could try this somewhat recursively

NSMutableArray *undownloaded;

- (void) startDownload {

    undownloaded = [photos mutableCopy]; //get a list of the undownloaded images

    for(int i = 0; i < 3;i++) //download 3 at a time
        [self downloadImage];
}


- (void) downloadImage {

    if(undownloaded.count > 0){

        ZSSPhoto *photo = undownloaded.firstObject;
        [undownloaded removeObjectAtIndex:0];

        [self downloadImageWithURL:photo.mobileURL progress:^(double progress) {

        } completion:^(UIImage *image) {

            // Save the image

            [self downloadImage];

        } failure:^(NSError *error) {

            [self downloadImage];
            //[undownloaded addObject:photo]; //insert photo back into the array maybe to retry? warning, could cause infinite loop without some extra logic, maybe the object can keep a fail count itself

        }];
    }
}

warning: untested code, may need some tweaking

Answer 2

The speed problem can be solved (the speed will increase, but it might still be slow) with multithreading, downloading all the images at the same time instead of one per time. However, the memory problem is a bit more complicated.

ARC will release all the images after everything is finished, but right before that you gonna have 6,000 images in the device memory. You could optimize the images, reduce their resolution or download them in steps, like Google Images do (you download the images that will be visible at first, then when the user scrolls down you load the images in the new visible area; downloading the images only when they are needed).

Considering that you are downloading enough images to give a memory problem, you will probably take a lots of space in your user's device if you download all of them, and the 'steps' solution may solve that as well.

Now, let's suppose you must download all of them at the same time and space isn't a problem: I suppose that if you put the downloadImageWithURL:progress: method inside a concurrent queue, the images are gonna be freed from memory just after saving (it's just a supposition). Add this to your code:

dispatch_queue_t defaultPriorityQueueWithName(const char* name)
{
    dispatch_queue_t dispatchQueue = dispatch_queue_create(name, DISPATCH_QUEUE_CONCURRENT);
    dispatch_queue_t priorityQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0);
    dispatch_set_target_queue(dispatchQueue, priorityQueue);
    return dispatchQueue;
}

And change your code to that:

dispatch_queue_t threadItemLoadImage = defaultPriorityQueueWithName("DownloadingImages");
NSArray *photos = @[hugeAmountOfPhotoObjects];
for (ZSSPhoto *photo in photos) 
{
    dispatch_async(threadItemLoadImage, ^
    {
        [self downloadImageWithURL:photo.mobileURL progress:^(double progress) {

        } completion:^(UIImage *image) {

            // Save the image

        } failure:^(NSError *error) {

        }];
    });
}

You will need to remove setDownloadProgressBlock: in case it updates some view, since they will be downloaded simultaneously. Also, a warning: totalBytesExpectedToRead not always will be correctly retrieved at first, containing 0, which might make your app crash for dividing by zero as well in some rare occasions. In future cases, when you need to use setDownloadProgressBlock:, check totalBytesExpectedToRead value before doing that division.