Recursively fetch all messages in a conversation #1180

Closed
opened 2022-08-07 06:13:08 +00:00 by dead10ck · 1 comment
Member

Currently, if the backend receives a new post from a follower, but fails to fetch other messages in a thread from remote servers, these other messages never make it into the DB. It will not retry or queue up fetches it needs to do.

An example can be seen on my own instance.

https://pleroma.dead10ck.com/notice/AMDNYR5vLOk0HTAqS8

It looks as if this user just made a solitary post, but if you go to the source server,

https://mastodon.social/@Gargron/108769773321577551

From the logs, we can see that it timed out while fetching the other posts:

Aug 05 10:24:11 dead10ck.com pleroma[180018]: 10:24:11.255 [error] Error while fetching https://one.darnell.one/users/darnell/statuses/108769046062132141: {:error, :timeout}
Aug 05 10:24:11 dead10ck.com pleroma[180018]: 10:24:11.255 [warn] Couldn't fetch "https://one.darnell.one/users/darnell/statuses/108769046062132141", error: nil

So the real issue seems to be that the backend never tries to backfill when it encounters an error during the recursive fetch. It just gives up forever.

I was thinking, since you can search for these individual posts and get them added to your DB that way, it might make sense to recursively fetch all messages in a thread basically any time you open a message in a conversation. It probably wouldn't be that expensive since you only have to fetch from the DB first to see if it's there before doing requests to the remote server.

Or alternatively and perhaps more complicated, a kind of "job system" could be introduced that queues up these fetches, and in the event of failure, the job just stays persisted until it's succeeded or crossed some timeout threshold

Currently, if the backend receives a new post from a follower, but fails to fetch other messages in a thread from remote servers, these other messages never make it into the DB. It will not retry or queue up fetches it needs to do. An example can be seen on my own instance. https://pleroma.dead10ck.com/notice/AMDNYR5vLOk0HTAqS8 It looks as if this user just made a solitary post, but if you go to the source server, https://mastodon.social/@Gargron/108769773321577551 From the logs, we can see that it timed out while fetching the other posts: ``` Aug 05 10:24:11 dead10ck.com pleroma[180018]: 10:24:11.255 [error] Error while fetching https://one.darnell.one/users/darnell/statuses/108769046062132141: {:error, :timeout} Aug 05 10:24:11 dead10ck.com pleroma[180018]: 10:24:11.255 [warn] Couldn't fetch "https://one.darnell.one/users/darnell/statuses/108769046062132141", error: nil ``` So the real issue seems to be that the backend never tries to backfill when it encounters an error during the recursive fetch. It just gives up forever. I was thinking, since you can search for these individual posts and get them added to your DB that way, it might make sense to recursively fetch all messages in a thread basically any time you open a message in a conversation. It probably wouldn't be that expensive since you only have to fetch from the DB first to see if it's there before doing requests to the remote server. Or alternatively and perhaps more complicated, a kind of "job system" could be introduced that queues up these fetches, and in the event of failure, the job just stays persisted until it's succeeded or crossed some timeout threshold
Member

This would probably be better suited as a backend feature request, as retrying fetches via an OBAN job would be backend's job more than frontend's. Waiting for input from others before closing.

This would probably be better suited as a [backend feature request](https://git.pleroma.social/pleroma/pleroma/-/issues), as retrying fetches via an OBAN job would be backend's job more than frontend's. Waiting for input from others before closing.
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
pleroma/pleroma-fe#1180
No description provided.