Spring Integration Mail inbound duplication with Outlook

Question

I'm currently working on a project that will poll an email inbox for a daily report.

I have a working implementation using spring-integration-mail with an imapInboundAdapter when hitting an email user hosted on Amazon Workmail.

When there is a single email in the inbox that is unread and unflagged it receives that email and properly emits only a single message.

However, when I changed the email server to one hosted by outlook I instead receive the email twice in the same poll.

As far as I can tell, it seems like the seen / flag status of the email isn't updated until the second polling attempt when on outlook, but is updated on the firstattempt on workmail.

The second attempt on outlook retrieves an email that should already have been processed.

IntegrationFlows
    .from(
        Mail.imapInboundAdapter(format("imaps://%s:%s/INBOX", source.getHost(), source.getPort()))
            .javaMailAuthenticator(authenticator)
            .maxFetchSize(10),
        e -> e.poller(... maxMessagesPerPoll = -1, cronTrigger = "*/15 * * ? * *" ...)
    ).log(INFO, m -> "Received email: " + m)

---

This is the logging that occurs when hitting workmail:

com.sun.mail.imap.connectionpool         : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool         : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap                        : IMAPProtocol noop
com.sun.mail.imap.connectionpool         : releaseStoreProtocol()
com.sun.mail.imap.connectionpool         : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool         : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool         : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool         : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver    : opening folder [imaps://[SNIPPED]@imap.mail.us-west-2.awsapps.com:993/INBOX]
com.sun.mail.imap                        : connection available -- size: 1
com.sun.mail.imap.messagecache           : create cache of size 8
o.s.integration.mail.ImapMailReceiver    : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver    : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
com.sun.mail.imap.messagecache           : create message number 8
o.s.integration.mail.ImapMailReceiver    : found 1 new messages
o.s.integration.mail.ImapMailReceiver    : Received 1 messages
o.s.integration.mail.ImapMailReceiver    : USER flags are not supported by this mail server. Flagging message with system flag
com.sun.mail.imap.connectionpool         : connection pool current size: 0   pool size: 1
com.sun.mail.imap.connectionpool         : connection pool current size: 0   pool size: 1
com.sun.mail.imap                        : added an Authenticated connection -- size: 1
o.s.i.mail.MailReceivingMessageSource    : received mail message [org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage@a101a27]
o.s.integration.handler.LoggingHandler   : Received email: GenericMessage [payload=org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage@a101a27, headers={id=20fb1886-eeda-c1e2-7ce0-8a9aae4f7ebc, timestamp=1624590243455}]
com.sun.mail.imap.connectionpool         : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool         : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap                        : IMAPProtocol noop
com.sun.mail.imap.connectionpool         : releaseStoreProtocol()
com.sun.mail.imap.connectionpool         : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool         : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool         : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool         : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver    : opening folder [imaps://[SNIPPED]@imap.mail.us-west-2.awsapps.com:993/INBOX]
com.sun.mail.imap                        : connection available -- size: 1
com.sun.mail.imap.messagecache           : create cache of size 8
o.s.integration.mail.ImapMailReceiver    : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver    : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
o.s.integration.mail.ImapMailReceiver    : found 0 new messages
o.s.integration.mail.ImapMailReceiver    : Received 0 messages
com.sun.mail.imap.connectionpool         : connection pool current size: 0   pool size: 1
com.sun.mail.imap.connectionpool         : connection pool current size: 0   pool size: 1
com.sun.mail.imap                        : added an Authenticated connection -- size: 1

---

This is the logging that occurs when hitting outlook:

com.sun.mail.imap.connectionpool         : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool         : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap                        : IMAPProtocol noop
com.sun.mail.imap.connectionpool         : releaseStoreProtocol()
com.sun.mail.imap.connectionpool         : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool         : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool         : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool         : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver    : opening folder [imaps://[SNIPPED]@outlook.office365.com:993/INBOX]
com.sun.mail.imap                        : connection available -- size: 1
com.sun.mail.imap.messagecache           : create cache of size 2
o.s.integration.mail.ImapMailReceiver    : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver    : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
com.sun.mail.imap.messagecache           : create message number 2
o.s.integration.mail.ImapMailReceiver    : found 1 new messages
o.s.integration.mail.ImapMailReceiver    : Received 1 messages
o.s.integration.mail.ImapMailReceiver    : USER flags are not supported by this mail server. Flagging message with system flag
com.sun.mail.imap.connectionpool         : connection pool current size: 0   pool size: 1
com.sun.mail.imap.connectionpool         : connection pool current size: 0   pool size: 1
com.sun.mail.imap                        : added an Authenticated connection -- size: 1
o.s.i.mail.MailReceivingMessageSource    : received mail message [org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage@5dffc30b]
o.s.integration.handler.LoggingHandler   : Received email: GenericMessage [payload=org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage@5dffc30b, headers={id=9d158e52-d46e-fd82-38b7-b438a4899a1e, timestamp=1624589763164}]
com.sun.mail.imap.connectionpool         : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool         : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap                        : IMAPProtocol noop
com.sun.mail.imap.connectionpool         : releaseStoreProtocol()
com.sun.mail.imap.connectionpool         : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool         : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool         : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool         : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver    : opening folder [imaps://[SNIPPED]@outlook.office365.com:993/INBOX]
com.sun.mail.imap                        : connection available -- size: 1
com.sun.mail.imap.messagecache           : create cache of size 2
o.s.integration.mail.ImapMailReceiver    : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver    : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
com.sun.mail.imap.messagecache           : create message number 2
o.s.integration.mail.ImapMailReceiver    : found 1 new messages
o.s.integration.mail.ImapMailReceiver    : Received 1 messages
o.s.integration.mail.ImapMailReceiver    : USER flags are not supported by this mail server. Flagging message with system flag
com.sun.mail.imap.connectionpool         : connection pool current size: 0   pool size: 1
com.sun.mail.imap.connectionpool         : connection pool current size: 0   pool size: 1
com.sun.mail.imap                        : added an Authenticated connection -- size: 1
o.s.i.mail.MailReceivingMessageSource    : received mail message [org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage@59c04c6c]
o.s.integration.handler.LoggingHandler   : Received email: GenericMessage [payload=org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage@59c04c6c, headers={id=002e6b83-caf5-86cd-b090-44aa346df119, timestamp=1624589766210}]
com.sun.mail.imap.connectionpool         : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool         : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap                        : IMAPProtocol noop
com.sun.mail.imap.connectionpool         : releaseStoreProtocol()
com.sun.mail.imap.connectionpool         : getStoreProtocol() borrowing a connection
com.sun.mail.imap.connectionpool         : getStoreProtocol() - connection available -- size: 1
com.sun.mail.imap.connectionpool         : getStoreProtocol() -- storeConnectionInUse
com.sun.mail.imap.connectionpool         : releaseFolderStoreProtocol()
o.s.integration.mail.ImapMailReceiver    : opening folder [imaps://[SNIPPED]@outlook.office365.com:993/INBOX]
com.sun.mail.imap                        : connection available -- size: 1
com.sun.mail.imap.messagecache           : create cache of size 2
o.s.integration.mail.ImapMailReceiver    : attempting to receive mail from folder [INBOX]
o.s.integration.mail.ImapMailReceiver    : This email server does not support RECENT or USER flags. System flag 'Flag.FLAGGED' will be used to prevent duplicates during email fetch.
o.s.integration.mail.ImapMailReceiver    : found 0 new messages
o.s.integration.mail.ImapMailReceiver    : Received 0 messages
com.sun.mail.imap.connectionpool         : connection pool current size: 0   pool size: 1
com.sun.mail.imap.connectionpool         : connection pool current size: 0   pool size: 1
com.sun.mail.imap                        : added an Authenticated connection -- size: 1

Answer 1

The issue turned out to be directly related to the windows outlook mail client.

If you add an account with imap in the outlook client it will cause duplication to occur if you mark an email as unread through the client (Possibly folder handling related).

I'm not sure of the exact cause, but adding the account as a full outlook account (instead of via imap) resolved the issue.

Otherwise you can mark emails as unread in the web browser instead of in the outlook client.

Answer 2

Sounds like that setFlag() is an async for Outlook. You may consider to not fetch all the messages at once - .maxFetchSize(1). Another way is to use a custom SearchTermStrategy or selector to skip messages which you have read already.