LinqToTwitter List DMs since a specified date

Question

Is it possible to list Direct Messages after a specified date? If I have a large number of Direct Messages, I'll reach the rate limit quickly if I have to page through many results. I'd like to track the last time I queried for DirectMessageEventsType.List and limit the next query to only messages sent/received after that date.

Answer 1

As you might already know, the only parameters in the Twitter API for listing DMs are count and cursor. That said, here's a work-around that might be handy. It involves a LINQ to Objects query after the LINQ to Twitter query:

using System;
using System.Linq;
using System.Collections.Generic;
using System.Threading.Tasks;
using LinqToTwitter;
using LinqToTwitter.OAuth;
using System.Diagnostics;

namespace TwitterDMListDate
{
    class Program
    {
        static async Task Main()
        {
            TwitterContext twitterCtx = await GetTwitterContext();

            int count = 50; // set to a low number to demo paging
            string cursor = "";
            List<DMEvent> allDmEvents = new();

            bool isPastCreatedAt = false;
            DateTime lastCreatedAt = DateTime.UtcNow.AddHours(-1);

            // you don't have a valid cursor until after the first query
            DirectMessageEvents dmResponse =
                await
                    (from dm in twitterCtx.DirectMessageEvents
                     where dm.Type == DirectMessageEventsType.List &&
                           dm.Count == count
                     select dm)
                    .SingleOrDefaultAsync();

            isPastCreatedAt = CheckPastCreatedAt(dmResponse, lastCreatedAt);

            allDmEvents.AddRange(dmResponse?.Value?.DMEvents ?? new List<DMEvent>());
            cursor = dmResponse?.Value?.NextCursor ?? "";

            while (!string.IsNullOrWhiteSpace(cursor) && !isPastCreatedAt)
            {
                dmResponse =
                    await
                        (from dm in twitterCtx.DirectMessageEvents
                         where dm.Type == DirectMessageEventsType.List &&
                               dm.Count == count &&
                               dm.Cursor == cursor
                         select dm)
                        .SingleOrDefaultAsync();

                allDmEvents.AddRange(dmResponse?.Value?.DMEvents ?? new List<DMEvent>());
                cursor = dmResponse?.Value?.NextCursor ?? "";

                isPastCreatedAt = CheckPastCreatedAt(dmResponse, lastCreatedAt);
            }

            if (!allDmEvents.Any())
            {
                Console.WriteLine("No items returned");
                return;
            }

            Console.WriteLine($"Response Count: {allDmEvents.Count}");
            Console.WriteLine("Responses:");

            allDmEvents.ForEach(evt =>
            {
                DirectMessageCreate? msgCreate = evt.MessageCreate;

                if (evt != null && msgCreate != null)
                    Console.WriteLine(
                        $"DM ID: {evt.ID}\n" +
                        $"From ID: {msgCreate.SenderID ?? "None"}\n" +
                        $"To ID:  {msgCreate.Target?.RecipientID ?? "None"}\n" +
                        $"Message Text: {msgCreate.MessageData?.Text ?? "None"}\n");
            });
        }

        static bool CheckPastCreatedAt(DirectMessageEvents dmResponse, DateTime lastCreatedAt)
        {
            return
                (from dm in dmResponse.Value.DMEvents
                 where dm.CreatedAt <= lastCreatedAt
                 select dm)
                .Any();
        }

        static async Task<TwitterContext> GetTwitterContext()
        {
            var auth = new PinAuthorizer()
            {
                CredentialStore = new InMemoryCredentialStore
                {
                    ConsumerKey = "",
                    ConsumerSecret = ""
                },
                GoToTwitterAuthorization = pageLink =>
                {
                    var psi = new ProcessStartInfo
                    {
                        FileName = pageLink,
                        UseShellExecute = true
                    };
                    Process.Start(psi);
                },
                GetPin = () =>
                {
                    Console.WriteLine(
                        "\nAfter authorizing this application, Twitter " +
                        "will give you a 7-digit PIN Number.\n");
                    Console.Write("Enter the PIN number here: ");
                    return Console.ReadLine() ?? string.Empty;
                }
            };

            await auth.AuthorizeAsync();

            var twitterCtx = new TwitterContext(auth);
            return twitterCtx;
        }
    }
}

The two variables that make this work are isPastCreatedAt and lastCreatedAt. The isPastCreatedAt flags the condition where a set of DMs (from a query of size count) contain one or more DMs that are older than a certain date. The date that qualifies isPastCreatedAt is lastCreatedAt. Essentially, we don't want to continue querying once we have tweets older than lastCreatedAt because subsequent queries are guaranteed to return all the tweets older than lastCreatedAt.

The demo sets lastCreatedAt to an hour earlier. Notice that it's using UTC time, because that's the time that Twitter uses. In your application, you should keep track of what this time is, re-setting it to the CreatedAt property of the oldest tweet received since the last set of queries.

After each LINQ to Twitter query for DMs, there's a call to CheckPastCreatedAt. This is where isPastCreatedAt gets set. Inside of the method is a LINQ to Objects query. It queries the list of DMs that LINQ to Twitter just returned, checking to see if any DMs contain a date earlier than the lastCreatedAt date. It uses the CreatedAt property and, as its name suggests, is the reason for the naming of the variables and method in this demo to ensure the query doesn't excessively waste rate limit. It also used the Any operator, which is much more efficient than Count, but I digress.

Notice that the while statement in Main adds an additional condition to what you are probably using to iterate through the cursor: && !isPastCreatedAt. That's what keeps you from going too far and wasting rate limit.

There's only one more thing you need to do when this is done - filter out the DMs that you've already received to avoid duplicates. However, my gut feeling is that you already know that.

While this wasn't the answer you might have hoped for, it does outline an important pattern for anyone working with the Twitter API, whether LINQ to Twitter or any one of the other excellent libraries out there. That is, to make the query with what the Twitter API gives you and then filter the results locally. While some other libraries provide additional abstractions that sometimes feel like they help, LINQ to Twitter takes a more raw approach to stay closer to the Twitter API itself. For in situations like this, it's good to know intuitively about what is crossing the wire so you can reason about additional solutions that meet your needs.