Two groups accidentally created while fetching pending messages after backup restore

[gerryfrancis]

• Operating System (Linux/Mac/Windows/iOS/Android):
  Android 11.

• Delta Chat Version:
  1.44.0 (nightly built 2024-03-26).

• Expected behavior:
  When pending messages are fetched from the Inbox and Sent IMAP folders after a backup has been restored, emails that were sent by MUAs, e.g. Webmail clients, and contain Re: in the first place of the subject, e.g. in replies, do not trigger creating a new group named Re: [Subject].

• Actual behavior:
  When pending messages are fetched from the Inbox and Sent IMAP folders after a backup has been restored, emails that were sent by MUAs, e.g. Webmail clients, and contain Re: in the first place of the subject, e.g. in replies, trigger creating a new group named Re: [Subject].

• Steps to reproduce the problem:
  -- Create a backup of your Delta Chat account.
  -- Un- and reinstall Delta Chat, but do not launch the app, yet.
  -- Receive a classic email that has been addressed to you and a second recipient. (The email should remain in the Inbox folder of your IMAP email account.)
  -- Reply to that email by using a MUA client, e.g. Webmail. (Your email should be stored in the Sent folder with Re: [Subject] as the subject.)
  -- Launch Delta Chat and restore the backup of your account. (Result: Two groups are created, named Re: [Subject] (which contains the reply message) and [Subject] (which contains the first message), while pending messages are being fetched.)

• Screenshots:
  

• Logs:
  N/A.

[gerryfrancis]

FYI, it happens with version 1.45.0 (beta), too.

[iequidoo]

A fix i can imagine here is to sort by INTERNALDATE not only messages from a particular folder, but from all folders, before passing them to receive_imf_inner(). The problem is that folders are processed by different threads, so it will be actually not sorting, but some kind of synchronisation using e.g. tokio::sync::Notify + Mutex. Not too difficult to implement, but still quite a big change so it's a question whether fixing this is worth it. Maybe also a topo-sorting using the References header is a way.

[iequidoo]

Another possible fix is to leave this unordered processing of folders as is, but search Message-ID in the already known references, this way we can insert an unordered ("missed previously") message to the already existing chat and if it becomes the first message, perform some special actions like chat renaming (but this is optional, the main thing is assigning all messages to the same chat)