How to get bulk reports via Twilio API?

Question

I'm going to get reports over call/message on Twilio via API. They have a limitation of 1000 reports on a single page. The requested GET query is something like this https://api.twilio.com/2010-04-01/Accounts/{Account_Sid}/Calls.json?PageSize=1000. I want to get them and show them to the user through a pie chart in Asp.net Core MVC. It returns a next_page_uri property for each request that you make and you'll get the reports for the next 1000 if you make a request to that. I tried it with Postman and it works fine, but I failed to get it worked via HttpClientFactory in Asp.net Core MVC. I think I should iterate through a collection of next_page_uri and make a GET request for every one, but It's not a collection.

try
            {
                var request = new HttpRequestMessage(HttpMethod.Get, $"https://api.twilio.com/2010-04-01/Accounts/{_configuration.GetSection("TwilioAuth:ACCOUNT_SID").Value}/Messages.json?DateSent={report.Date}&PageSize=1000");
                var client = _httpClientFactory.CreateClient("MyClient");
                HttpResponseMessage response = await client.SendAsync(request);
                if (response.StatusCode == HttpStatusCode.OK)
                {
                    var jsonResponse = await response.Content.ReadAsStringAsync();
                    model = JsonConvert.DeserializeObject<ReportMessageModel>(jsonResponse);
                    foreach (var message in model.messages)
                    {
                        if (message.direction == "outbound-api")
                        {
                            outboundMessages.Add(message.status);
                            ViewBag.messageCount = outboundMessages;
                            if (message.status == "delivered")
                            {
                                delivered.Add(message.status);
                                ViewBag.delivered = delivered.Count;
                            }
                            else if (message.status == "undelivered")
                            {
                                undelivered.Add(message.status);
                                ViewBag.undelivered = undelivered.Count;
                            }
                        }
                    }
                }
                if (model.next_page_uri != null)
                {
                    var newRequest = new HttpRequestMessage(HttpMethod.Get, $"https://api.twilio.com{model.next_page_uri}");
                    HttpResponseMessage newResponse = await client.SendAsync(newRequest);
                    if (newResponse.StatusCode == HttpStatusCode.OK)
                    {
                        var jsonResponse = await newResponse.Content.ReadAsStringAsync();
                        model = JsonConvert.DeserializeObject<ReportMessageModel>(jsonResponse);
                        foreach (var message in model.messages)
                        {
                            if (message.direction == "outbound-api")
                            {
                                outboundMessages.Add(message.status);
                                ViewBag.messageCount = outboundMessages;
                                if (message.status == "delivered")
                                {
                                    delivered.Add(message.status);
                                    ViewBag.delivered = delivered.Count;
                                }
                                else if (message.status == "undelivered")
                                {
                                    undelivered.Add(message.status);
                                    ViewBag.undelivered = undelivered.Count;
                                }
                            }
                        }
                    }

                }

            }

Answer 1

Since you are using C#, why not use the C# SDK and let it handle the MessageResource paging for you?

// Find your Account SID and Auth Token at twilio.com/console
// and set the environment variables. See http://twil.io/secure
string accountSid = Environment.GetEnvironmentVariable("TWILIO_ACCOUNT_SID");
string authToken = Environment.GetEnvironmentVariable("TWILIO_AUTH_TOKEN");

TwilioClient.Init(accountSid, authToken);

var messages = MessageResource.Read(dateSent: {report.Date});

    foreach(var record in messages)
    {
       Console.WriteLine(record.Sid);
    }

An alternative, is to use the Usage Records:

 var records = RecordResource.Read(
        category: RecordResource.CategoryEnum.Sms,
        startDate: {report.Date},
        endDate: {report.Date}
    );

The above will return data quickly but only be a total count which won't allow for complex graphics.

Also, as an aside, I think you will find the MessageResource.Read approach not efficient (slow for end-user reporting). A more efficient approach is to use Twilio's webhooks to collect the data in your own database. Then you can query your database however and how often you prefer. Twilio describes this in the REST API Best Practices document:

If you are performing a large amount of GET requests, consider implementing webhooks aka StatusCallBack requests for the resource endpoint(s) your account is utilizing. The information contained in the responses posted to your servers will often remove the need to perform any future polling GET requests.

If you are frequently fetching the same data from Twilio, we recommend moving the data from Twilio to your own servers. Once data has been successfully moved, delete data stored on Twilio servers if you no longer need it. Not only will this will reduce costs, this is also a generally recommended business practice for privacy, security, and compliance.