How to properly separate API calls from File Storage Service in a Clean Architecture Pattern?

Question

I am working on a .NET Core project in which I need to implement a Clean Architecture Pattern. The Task is to:

1. make an HTTP request to a third-party API service
2. read the response content as a Stream
3. save the Stream Content to File Storage

In order to solve this problem, I created two classes in an infrastructure layer:

1. ApiService.cs

        public async Task GetDataAsync(string url, Func<Stream, Task> func)
        {
            using (HttpResponseMessage httpResponseMessage = await httpClient.GetAsync(url))
            {
                if (httpResponseMessage.IsSuccessStatusCode)
                {
                    using (Stream stream = await httpResponseMessage.Content.ReadAsStreamAsync())
                    {
                        await func(stream);
                    }
                }
            }
        }

2. FileStorageService.cs

        public async Task CreateFileAsync(string path, Stream content)
        {
            using (FileStream fileStream = new(path, FileMode.Create))
            {
                await content.CopyToAsync(fileStream);
            }
        }

There is also a third method in Core Layer that combine and process the above two methods:

        public async Task DownloadData(string url, string path)
        {
            await apiService.GetDataAsync(url, async stream =>
            {
                await fileStorageService.CreateFileAsync(path, stream);
            });
        }

Even though the above code works there is an obvious direct dependency of File Storage Service on API Service. My Expectations are to have a clean separation of File and API service in a way that two of those do not need to know about each other. I would like to be able to have the option to implement Pipeline pattern in the future (if needed) and to be able to define the above code somehow in a way:

string url = "<some_url>";
Stream result = await apiService.GetDataAsync(url);

string path ="<some_path>";
await fileStorageService.CreateFileAsync(path, result);

I was trying to replace Func<> parameter with the Stream return type in API Service Method, but I face a challenge in properly dealing with using statements in HttpResponseMessage and Stream.

I will appreciate it a lot if you have some suggestions on how to solve the above problem in order to:

1. clearly separate API Service from File Service so that one does not know about another
2. properly dispose of the Using statements in HttpResponseMessage and Stream from apiService.GetDataAsync method without (if possible) using Func<>

Thanks in advance.

Answer 1

I'm not sure if you've considered the class route, but my preferred way of dealing with this is to have a StorageFile and StorageFolder framework that mimics what UWP/Metro/etc. have had from the beginning. In fact there's a very old but still open issue in the dotnet/runtime Github repo.

The idea is simply to have abstract base classes - StorageFile, StorageFolder, and their parent StorageObject - that get implemented for each storage platform you want to use, from the local file system to Azure blobs to anything else.

These would be the basic definitions:

public abstract class StorageObject 
{
    public abstract string Name { get; }
    public abstract Task RenameAsync();
    public abstract Task<StorageProperties> GetPropertiesAsync();   // would retrieve metadata like creation/access dates, size, etc.
    public abstract Task DeleteAsync();
    public abstract Task CopyAsync (StorageFolder destination);
    public abstract Task<IEnumerable<StorageObject>> GetChildrenAsync();
}

public abstract class StorageFolder : StorageObject
{
    public abstract int Files { get; }
    public abstract int Folders { get; }
    public abstract Task<StorageFile> GetFileAsync (string filename, bool createIfNotExists);
    public abstract Task<StorageFolder> GetSubfolderAsync (string subfolderName, bool createIfNotExists);

    public virtual Task<IEnumerable<StorageFile>> FindFilesAsync (string criteria, bool includeSubfolders) { ... } // default implementation would use GetChildrenAsync but be overridable for more efficient implementations
    public virtual Task<IEnumerable<StorageFolder>> FindSubfoldersAsync (string criteria, bool includeSubfolders) { ... } // same
}

public abstract class StorageFile : StorageObject
{
    public abstract Task<Stream> OpenReadAsync(bool shared = false);
    public abstract Task<Stream> OpenWriteAsync(bool shared = false); 
}

So a basic (incomplete) implementation of LocalStorageFile, for example, would look like this:

public class LocalStorageFile : StorageFile 
{
   private readonly string _path;

   public LocalStorageFile (string path)
   {
      _path = path;
   } 

   public override Task<Stream> OpenReadAsync(bool shared = false)
   {
      var str = File.Open(_path, FileMode.Open, FileAccess.Read, shared ? FileShare.Read : FileShare.None);
      return Task.FromResult(str);
   }

   // ...
}

Then you would consume it as such:

LocalStorageFile f = new LocalStorageFile("c:\\mypath\\file.bin");
using (var str = await f.OpenStreamForReadAsync())
{
   // do your thing
}

Most importantly, all the rest of your code references the base classes - StorageFile, etc., and so becomes completely agnostic to the implementation. And by dealing with Stream - another highly abstract and extensible class - you don't have to do anything special other than Dispose it. Of course if you have something highly specialized you can subclass Stream too and return whatever you want from OpenReadAsync and OpenWriteAsync.

Answer 2

After discussion in comments I believe the problem lies in the fact that the stream is being disposed by the apiService and then the fileStorageService throws a disposed object exception.

A possible solution for this problem is removing the using statement inside the apiService and returning the stream.

    public async Task<Stream> GetDataAsync(string url)
    {
        using (HttpResponseMessage httpResponseMessage = await httpClient.GetAsync(url))
        {
            if (httpResponseMessage.IsSuccessStatusCode)
            {
                return await httpResponseMessage.Content.ReadAsStreamAsync();
            }
        }
    }

Then you could use it in the following way:

string url = "<some_url>";
Stream result = await apiService.GetDataAsync(url);

using(result)
{
    string path ="<some_path>";
    await fileStorageService.CreateFileAsync(path, result);
}