Recursively fetch all messages in a conversation
Currently, if the backend receives a new post from a follower, but fails to fetch other messages in a thread from remote servers, these other messages never make it into the DB. It will not retry or queue up fetches it needs to do.
An example can be seen on my own instance.
https://pleroma.dead10ck.com/notice/AMDNYR5vLOk0HTAqS8
It looks as if this user just made a solitary post, but if you go to the source server,
https://mastodon.social/@Gargron/108769773321577551
From the logs, we can see that it timed out while fetching the other posts:
Aug 05 10:24:11 dead10ck.com pleroma[180018]: 10:24:11.255 [error] Error while fetching https://one.darnell.one/users/darnell/statuses/108769046062132141: {:error, :timeout}
Aug 05 10:24:11 dead10ck.com pleroma[180018]: 10:24:11.255 [warn] Couldn't fetch "https://one.darnell.one/users/darnell/statuses/108769046062132141", error: nil
So the real issue seems to be that the backend never tries to backfill when it encounters an error during the recursive fetch. It just gives up forever.
I was thinking, since you can search for these individual posts and get them added to your DB that way, it might make sense to recursively fetch all messages in a thread basically any time you open a message in a conversation. It probably wouldn't be that expensive since you only have to fetch from the DB first to see if it's there before doing requests to the remote server.
Or alternatively and perhaps more complicated, a kind of "job system" could be introduced that queues up these fetches, and in the event of failure, the job just stays persisted until it's succeeded or crossed some timeout threshold