Hackney: Bad connection pool handling in outgoing federation #7834

Open
opened 2026-02-23 15:23:09 +00:00 by phnt · 2 comments
Owner

Backend version: current develop (1c685ea41a) probably all the way to 2.10 and possibly older
OS: RHEL8
PostgreSQL version: 12
Install type: RPM package built from source

Somewhere probably in the 2.10 release something broke Hackney's connection pooling for outgoing federation or the update to Hackney 1.25 broke it.

When federating, Hackney can't probably acquire a slot in the outgoing federation pool and instead of waiting for the timeout to acquire a slot, probably gives up too quickly. Result is something like this in logs after making a post:

Logs
14:29:40.840 [error] Publisher failed to inbox https://shitposter.world/inbox {:error, :checkout_timeout}
14:29:40.841 [error] Publisher failed to inbox https://tsogol.tsiran.org/inbox {:error, :checkout_timeout}
14:29:40.873 [error] Publisher failed to inbox https://boymoder.biz/inbox {:error, :checkout_timeout}
14:29:40.875 [error] Publisher failed to inbox https://poa.st/inbox {:error, :checkout_timeout}
14:29:40.891 [error] Publisher failed to inbox https://minidisc.tokyo/inbox {:error, :checkout_timeout}
14:29:40.903 [error] Publisher failed to inbox https://gameliberty.club/inbox {:error, :checkout_timeout}
14:29:40.924 [error] Publisher failed to inbox https://outerheaven.club/inbox {:error, :checkout_timeout}
14:29:40.944 [error] Publisher failed to inbox https://djsumdog.com/inbox {:error, :checkout_timeout}
14:29:40.957 [error] Publisher failed to inbox https://miruku.cafe/inbox {:error, :checkout_timeout}
14:29:40.960 [error] Publisher failed to inbox https://posting.lolicon.rocks/inbox {:error, :checkout_timeout}
14:29:40.976 [error] Publisher failed to inbox https://eientei.org/inbox {:error, :checkout_timeout}
14:29:40.981 [error] Publisher failed to inbox https://annihilation.social/inbox {:error, :checkout_timeout}
14:29:40.990 [error] Publisher failed to inbox https://comp.lain.la/inbox {:error, :checkout_timeout}
14:29:41.003 [error] Publisher failed to inbox https://social.xenofem.me/inbox {:error, :checkout_timeout}
14:29:41.020 [error] Publisher failed to inbox https://netzsphaere.xyz/inbox {:error, :checkout_timeout}
14:29:41.022 [error] Publisher failed to inbox https://pleroma.soykaf.com/inbox {:error, :checkout_timeout}
14:29:41.052 [error] Publisher failed to inbox https://mk.magicka.org/inbox {:error, :checkout_timeout}
14:29:41.052 [error] Publisher failed to inbox https://mstdn.starnix.network/inbox {:error, :checkout_timeout}
14:29:41.069 [error] Publisher failed to inbox https://mai.waifuism.life/inbox {:error, :checkout_timeout}
14:29:41.086 [error] Publisher failed to inbox https://nekosat.work/inbox {:error, :checkout_timeout}
14:29:41.109 [error] Publisher failed to inbox https://detroitriotcity.com/inbox {:error, :checkout_timeout}
14:29:41.109 [error] Publisher failed to inbox https://socially.drinkingatmy.computer/inbox {:error, :checkout_timeout}
14:29:41.115 [error] Publisher failed to inbox https://aklp.club/inbox {:error, :checkout_timeout}
14:29:41.127 [error] Publisher failed to inbox https://cawfee.club/inbox {:error, :checkout_timeout}
--SNIP---
14:29:45.882 [error] Publisher failed to inbox https://fedi.layer02.net/inbox {:error, :checkout_timeout}
14:29:45.883 [error] Publisher failed to inbox https://pl.starnix.network/inbox {:error, :checkout_timeout}
14:29:45.927 [error] Publisher failed to inbox https://clubcyberia.co/inbox {:error, :checkout_timeout}
14:29:45.931 [error] Publisher failed to inbox https://hollow.raccoon.quest/inbox {:error, :checkout_timeout}
14:29:45.944 [error] Publisher failed to inbox https://56k.dile-up.nl/inbox {:error, :checkout_timeout}
14:29:45.999 [error] Publisher failed to inbox https://infosec.exchange/inbox {:error, :checkout_timeout}
14:29:46.026 [error] Publisher failed to inbox https://p.mr64.net/inbox {:error, :checkout_timeout}
14:29:46.062 [error] Publisher failed to inbox https://mstdn.io/inbox {:error, :checkout_timeout}
14:29:46.066 [error] Publisher failed to inbox https://infosec.town/inbox {:error, :checkout_timeout}
14:29:46.074 [error] Publisher failed to inbox https://fsebugoutzone.org/inbox {:error, :checkout_timeout}
14:29:46.078 [error] Publisher failed to inbox https://noauthority.social/inbox {:error, :checkout_timeout}
14:29:46.084 [error] Publisher failed to inbox https://egregoros.social/users/alice/inbox {:error, :checkout_timeout}
14:29:46.099 [error] Publisher failed to inbox https://blob.cat/inbox {:error, :checkout_timeout}
14:29:46.099 [error] Publisher failed to inbox https://quietplace.xyz/inbox {:error, :checkout_timeout}
14:29:46.121 [error] Publisher failed to inbox https://nicecrew.digital/inbox {:error, :checkout_timeout}
14:29:46.123 [error] Publisher failed to inbox https://misskey.m544.net/inbox {:error, :checkout_timeout}
14:29:46.128 [error] Publisher failed to inbox https://ryona.agency/inbox {:error, :checkout_timeout}
14:29:46.129 [error] Publisher failed to inbox https://shitpost.cloud/inbox {:error, :checkout_timeout}
14:29:46.144 [error] Publisher failed to inbox https://misskey.io/inbox {:error, :checkout_timeout}
14:29:46.144 [error] Publisher failed to inbox https://mk.absturztau.be/inbox {:error, :checkout_timeout}
14:29:46.161 [error] Publisher failed to inbox https://varishangout.net/inbox {:error, :checkout_timeout}
14:29:46.163 [error] Publisher failed to inbox https://social.freedombits.org/inbox {:error, :checkout_timeout}
14:29:46.180 [error] Publisher failed to inbox https://ak.kyaruc.moe/inbox {:error, :checkout_timeout}
14:29:46.189 [error] Publisher failed to inbox https://new.asbestos.cafe/inbox {:error, :checkout_timeout}
14:29:46.843 [error] Publisher failed to inbox https://whinge.town/inbox {:error, :checkout_timeout}

Turning off HTTP proxying for Tor federation did not help. Cannot reproduce on test instance.

Backend version: current develop (1c685ea41a771d74a1d4e1769a28911d738591b0) probably all the way to 2.10 and possibly older OS: RHEL8 PostgreSQL version: 12 Install type: RPM package built from source Somewhere probably in the 2.10 release something broke Hackney's connection pooling for outgoing federation or the update to Hackney 1.25 broke it. When federating, Hackney can't probably acquire a slot in the outgoing federation pool and instead of waiting for the timeout to acquire a slot, probably gives up too quickly. Result is something like this in logs after making a post: <details> <summary>Logs</summary> <pre> 14:29:40.840 [error] Publisher failed to inbox https://shitposter.world/inbox {:error, :checkout_timeout} 14:29:40.841 [error] Publisher failed to inbox https://tsogol.tsiran.org/inbox {:error, :checkout_timeout} 14:29:40.873 [error] Publisher failed to inbox https://boymoder.biz/inbox {:error, :checkout_timeout} 14:29:40.875 [error] Publisher failed to inbox https://poa.st/inbox {:error, :checkout_timeout} 14:29:40.891 [error] Publisher failed to inbox https://minidisc.tokyo/inbox {:error, :checkout_timeout} 14:29:40.903 [error] Publisher failed to inbox https://gameliberty.club/inbox {:error, :checkout_timeout} 14:29:40.924 [error] Publisher failed to inbox https://outerheaven.club/inbox {:error, :checkout_timeout} 14:29:40.944 [error] Publisher failed to inbox https://djsumdog.com/inbox {:error, :checkout_timeout} 14:29:40.957 [error] Publisher failed to inbox https://miruku.cafe/inbox {:error, :checkout_timeout} 14:29:40.960 [error] Publisher failed to inbox https://posting.lolicon.rocks/inbox {:error, :checkout_timeout} 14:29:40.976 [error] Publisher failed to inbox https://eientei.org/inbox {:error, :checkout_timeout} 14:29:40.981 [error] Publisher failed to inbox https://annihilation.social/inbox {:error, :checkout_timeout} 14:29:40.990 [error] Publisher failed to inbox https://comp.lain.la/inbox {:error, :checkout_timeout} 14:29:41.003 [error] Publisher failed to inbox https://social.xenofem.me/inbox {:error, :checkout_timeout} 14:29:41.020 [error] Publisher failed to inbox https://netzsphaere.xyz/inbox {:error, :checkout_timeout} 14:29:41.022 [error] Publisher failed to inbox https://pleroma.soykaf.com/inbox {:error, :checkout_timeout} 14:29:41.052 [error] Publisher failed to inbox https://mk.magicka.org/inbox {:error, :checkout_timeout} 14:29:41.052 [error] Publisher failed to inbox https://mstdn.starnix.network/inbox {:error, :checkout_timeout} 14:29:41.069 [error] Publisher failed to inbox https://mai.waifuism.life/inbox {:error, :checkout_timeout} 14:29:41.086 [error] Publisher failed to inbox https://nekosat.work/inbox {:error, :checkout_timeout} 14:29:41.109 [error] Publisher failed to inbox https://detroitriotcity.com/inbox {:error, :checkout_timeout} 14:29:41.109 [error] Publisher failed to inbox https://socially.drinkingatmy.computer/inbox {:error, :checkout_timeout} 14:29:41.115 [error] Publisher failed to inbox https://aklp.club/inbox {:error, :checkout_timeout} 14:29:41.127 [error] Publisher failed to inbox https://cawfee.club/inbox {:error, :checkout_timeout} --SNIP--- 14:29:45.882 [error] Publisher failed to inbox https://fedi.layer02.net/inbox {:error, :checkout_timeout} 14:29:45.883 [error] Publisher failed to inbox https://pl.starnix.network/inbox {:error, :checkout_timeout} 14:29:45.927 [error] Publisher failed to inbox https://clubcyberia.co/inbox {:error, :checkout_timeout} 14:29:45.931 [error] Publisher failed to inbox https://hollow.raccoon.quest/inbox {:error, :checkout_timeout} 14:29:45.944 [error] Publisher failed to inbox https://56k.dile-up.nl/inbox {:error, :checkout_timeout} 14:29:45.999 [error] Publisher failed to inbox https://infosec.exchange/inbox {:error, :checkout_timeout} 14:29:46.026 [error] Publisher failed to inbox https://p.mr64.net/inbox {:error, :checkout_timeout} 14:29:46.062 [error] Publisher failed to inbox https://mstdn.io/inbox {:error, :checkout_timeout} 14:29:46.066 [error] Publisher failed to inbox https://infosec.town/inbox {:error, :checkout_timeout} 14:29:46.074 [error] Publisher failed to inbox https://fsebugoutzone.org/inbox {:error, :checkout_timeout} 14:29:46.078 [error] Publisher failed to inbox https://noauthority.social/inbox {:error, :checkout_timeout} 14:29:46.084 [error] Publisher failed to inbox https://egregoros.social/users/alice/inbox {:error, :checkout_timeout} 14:29:46.099 [error] Publisher failed to inbox https://blob.cat/inbox {:error, :checkout_timeout} 14:29:46.099 [error] Publisher failed to inbox https://quietplace.xyz/inbox {:error, :checkout_timeout} 14:29:46.121 [error] Publisher failed to inbox https://nicecrew.digital/inbox {:error, :checkout_timeout} 14:29:46.123 [error] Publisher failed to inbox https://misskey.m544.net/inbox {:error, :checkout_timeout} 14:29:46.128 [error] Publisher failed to inbox https://ryona.agency/inbox {:error, :checkout_timeout} 14:29:46.129 [error] Publisher failed to inbox https://shitpost.cloud/inbox {:error, :checkout_timeout} 14:29:46.144 [error] Publisher failed to inbox https://misskey.io/inbox {:error, :checkout_timeout} 14:29:46.144 [error] Publisher failed to inbox https://mk.absturztau.be/inbox {:error, :checkout_timeout} 14:29:46.161 [error] Publisher failed to inbox https://varishangout.net/inbox {:error, :checkout_timeout} 14:29:46.163 [error] Publisher failed to inbox https://social.freedombits.org/inbox {:error, :checkout_timeout} 14:29:46.180 [error] Publisher failed to inbox https://ak.kyaruc.moe/inbox {:error, :checkout_timeout} 14:29:46.189 [error] Publisher failed to inbox https://new.asbestos.cafe/inbox {:error, :checkout_timeout} 14:29:46.843 [error] Publisher failed to inbox https://whinge.town/inbox {:error, :checkout_timeout} </pre> </details> Turning off HTTP proxying for Tor federation did not help. Cannot reproduce on test instance.
Author
Owner

My only suspicion so far is ef0f04ca48 that added the with_body: true option to Hackney via the AdapterHelper which would get used in outgoing federation. Which caused Hackney to not release connections when they are "done".

My only suspicion so far is https://git.pleroma.social/pleroma/pleroma/commit/ef0f04ca48e8b7e848ef5fd7cac9d9e074248d59 that added the `with_body: true` option to Hackney via the AdapterHelper which would get used in outgoing federation. Which caused Hackney to not release connections when they are "done".
Author
Owner

Never mind, also reproduced it in a different way on a test instance

Logs
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://blahaj.social/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://hessen.social/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://ruhr.social/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://flyovercountry.social/actor, :checkout_timeout
path=/users/phnt [error] Could not decode user at fetch https://freesewing.social/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://tinnies.club/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://masto.snoozetown.org/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://mas.to/actor, :checkout_timeout
path=/users/phnt [error] Could not decode user at fetch https://kanoa.de/actor, :checkout_timeout
[error] Could not decode user at fetch https://discuss.systems/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://beige.party/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://procolix.social/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://fed.interfree.ca/users/9yynj3myvdcfrg80, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://convo.casa/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://sueden.social/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://social.ericcodes.io/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://stranger.social/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://pouet.it/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://toot.coupou.fr/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://anticapitalist.party/actor, :checkout_timeout
path=/users/phnt [error] Could not decode user at fetch https://ipv6.social/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://greennuclear.online/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://ottawa.place/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://camp.smolnet.org/actor, :checkout_timeout
[error] Could not decode user at fetch https://raru.re/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://mastodon.africa/actor, :checkout_timeout
path=/users/phnt [error] Could not decode user at fetch https://troet.cafe/actor, :checkout_timeout
path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://mastodon.acc.sunet.se/actor, :checkout_timeout
Never mind, also reproduced it in a different way on a test instance <details> <summary>Logs</summary> <pre> path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://blahaj.social/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://hessen.social/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://ruhr.social/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://flyovercountry.social/actor, :checkout_timeout path=/users/phnt [error] Could not decode user at fetch https://freesewing.social/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://tinnies.club/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://masto.snoozetown.org/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://mas.to/actor, :checkout_timeout path=/users/phnt [error] Could not decode user at fetch https://kanoa.de/actor, :checkout_timeout [error] Could not decode user at fetch https://discuss.systems/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://beige.party/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://procolix.social/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://fed.interfree.ca/users/9yynj3myvdcfrg80, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://convo.casa/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://sueden.social/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://social.ericcodes.io/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://stranger.social/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://pouet.it/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://toot.coupou.fr/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://anticapitalist.party/actor, :checkout_timeout path=/users/phnt [error] Could not decode user at fetch https://ipv6.social/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://greennuclear.online/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://ottawa.place/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://camp.smolnet.org/actor, :checkout_timeout [error] Could not decode user at fetch https://raru.re/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://mastodon.africa/actor, :checkout_timeout path=/users/phnt [error] Could not decode user at fetch https://troet.cafe/actor, :checkout_timeout path=/objects/53aeda5e-a2dc-4304-96c0-079a197fbf06 [error] Could not decode user at fetch https://mastodon.acc.sunet.se/actor, :checkout_timeout </pre> </details>
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
pleroma/pleroma#7834
No description provided.