I have an API hosted on Azure. Load tests were executed for that API in local JMeter and the same jmx script is exported to Azure load test resource and performed another load test.
When I look into the test results, the response times for the run on local JMeter (X sec) is different from the test on the Azure load test resource (Y sec). Both test scenarios and workload is same.
Also when I look into the DataDog, the same API have different response time (Z sec). Mostly this is how the response time looks when compared. X < Z < Y or Z < X <Y.
Why there is significant amount of differences in the response time even for the same API.Does the Azure one considers the client side metrics too? All the other components like css, java script, images are not present in the script.
DataDog gives you server processing time only, it doesn't include the time for the request to get to the server and to the response to travel back
JMeter gives you perceived system experience, to wit end-to-end (time to prepare the request, time to establish TCP connection, time for SSL handshake, sending the request, receiving the response). The differences between Azure and local might be connected with the you and Azure region geo-location and network latency. You can roughly estimate it by comparing Latency and Elapsed Time which gives you the time required for downloading the response.
In general it doesn't really matter what are the numbers as you cannot expect to be the same response times with 1 ms precision, it matters what do trends look like and what is relationship between number of users and response time or number of users and number of requests per second or when errors start occurring or what is the first bottleneck.
More information: