How to store temporary data in an Azure multi-instance (scale set) virtual machine?

Question

We developed a server service that (in a few words) supports the communications between two devices. We want to make advantage of the scalability given by an Azure Scale Set (multi instance VM) but we are not sure how to share memory between each instance.

Our service basically stores temporary data in the local virtual machine and these data are read, modified and sent to the devices connected to this server.

If these data are stored locally in one of the instances the other instances cannot access and do not have the same information. Is it correct?

If one of the devices start making some request to the server the instance that is going to process the request will not always be the same so the data at the end is spread between instances.

So the question might be, how to share memory between Azure instances?

Thanks

Answer 1

Depending on the type of data you want to share and how much latency matters, as well as ServiceFabric (low latency but you need to re-architect/re-build bits of your solution), you could look at a shared back end repository - Redis Cache is ideal as a distributed cache; SQL Azure if you want to use a relation db to store the data; storage queue/blob storage - or File storage in a storage account (this allows you just to write to a mounted network drive from both vm instances). DocumentDB is another option, which is suited to storing JSON data.

Answer 2

You could use Service Fabric and take advantage of Reliable Collections to have your state automagically replicated across all instances.

From https://azure.microsoft.com/en-us/documentation/articles/service-fabric-reliable-services-reliable-collections/:

The classes in the Microsoft.ServiceFabric.Data.Collections namespace provide a set of out-of-the-box collections that automatically make your state highly available. Developers need to program only to the Reliable Collection APIs and let Reliable Collections manage the replicated and local state.

The key difference between Reliable Collections and other high-availability technologies (such as Redis, Azure Table service, and Azure Queue service) is that the state is kept locally in the service instance while also being made highly available.

Reliable Collections can be thought of as the natural evolution of the System.Collections classes: a new set of collections that are designed for the cloud and multi-computer applications without increasing complexity for the developer. As such, Reliable Collections are:

• Replicated: State changes are replicated for high availability.
• Persisted: Data is persisted to disk for durability against large-scale outages (for example, a datacenter power outage).
• Asynchronous: APIs are asynchronous to ensure that threads are not blocked when incurring IO.
• Transactional: APIs utilize the abstraction of transactions so you can manage multiple Reliable Collections within a service easily.

Working with Reliable Collections -
https://azure.microsoft.com/en-us/documentation/articles/service-fabric-work-with-reliable-collections/