Downgrade Hackney

feld commented

2026-03-24 18:55:34 +00:00

Owner

It appears the implementation of Happy Eyeballs in 1.22.0 is the origin of
some pretty serious performance regressions that remain even in the latest
Hackney 3.0 branch.

Connection tests:

=== 1.22.0 ===
First call: 9434ms
Second call: 14ms

=== 1.21.0 ===
First call: 228ms
Second call: 16ms

Checklist

Adding a changelog: In the changelog.d directory, create a file named <code>.<type>.

It appears the implementation of Happy Eyeballs in 1.22.0 is the origin of some pretty serious performance regressions that remain even in the latest Hackney 3.0 branch. Connection tests: === 1.22.0 === First call: 9434ms Second call: 14ms === 1.21.0 === First call: 228ms Second call: 16ms ### Checklist - [ ] Adding a changelog: In the `changelog.d` directory, create a file named `<code>.<type>`.

feld added 1 commit

2026-03-24 18:55:34 +00:00

Downgrade Hackney to 1.21.0 before connection performance regressions

ci/woodpecker/pr/woodpecker Pipeline is pending

Details

d86a43afef

It appears the implementation of Happy Eyeballs in 1.22.0 is the origin of
some pretty serious performance regressions that remain even in the latest
Hackney 3.0 branch.

Connection tests:

=== 1.22.0 ===
First call:  9434ms
Second call: 14ms

=== 1.21.0 ===
First call:  228ms
Second call: 16ms

feld force-pushed hackney-downgrade from d86a43afef

ci/woodpecker/pr/woodpecker Pipeline is pending

Details

to 60b03191f0

ci/woodpecker/pr/woodpecker Pipeline failed

Details

2026-03-24 18:56:48 +00:00

Compare

feld commented

2026-03-24 18:58:57 +00:00

Author

Owner

from my testing with a simple mix task that profiles Webfinger lookups

from my testing with a simple mix task that profiles Webfinger lookups ![image](/attachments/5c3be2c5-dbd6-4a8c-ade2-a5f7b52c2e9d)

image.png

89 KiB

phnt commented

2026-03-24 19:27:00 +00:00

Owner

Summary about various recent Hackney-related issues so info isn't scattered throughout multiple issues and fedi threads:

There is a regression in Hackney somewhere between 1.22.0 and 1.25.0 (version used by Pleroma as of 6bbfba7f6e) that causes faulty {:error, :checkout_timeout} errors when processing outgoing federation jobs, or fetching/refreshing Actors. (#7834)
- Hackney 1.21.0 produced few {:error, :checkout_timeout} errors when used as client for MediaProxy; it is currently unclear if that is caused by a regression in that version of Hackney, or due to high server load.
There seems to be regression in Hackney 1.25.0 that causes Swoosh to fail sending email via Mailgun due to "incompatible ssl options" being specified possibly by Swoosh itself (#7855); Pleroma is also possibly affected by this issue when an instance admin specified custom ssl_options in the configuration file or ConfigDB. (#7824)
- The latter could not be reproduced on a local development instance.
- What version is unaffected is unknown, but based on Hackney changelog the first affected version might be 1.21.0.
Newer versions than 1.25.0 cannot be used because they all require OTP27+ and possibly changed connection pooling in an incompatible way to how Pleroma uses it.

Due to the above, 1.20.1 might be a better version to downgrade to since it possibly does not have the ssl_options regression, although I have no way to confirm that since I could not reproduce the issue and I don't have a Mailgun account to test with.

Summary about various recent Hackney-related issues so info isn't scattered throughout multiple issues and fedi threads: * There is a regression in Hackney somewhere between 1.22.0 and 1.25.0 (version used by Pleroma as of 6bbfba7f6e) that causes faulty `{:error, :checkout_timeout}` errors when processing outgoing federation jobs, or fetching/refreshing Actors. (#7834) * Hackney 1.21.0 produced few `{:error, :checkout_timeout}` errors when used as client for MediaProxy; it is currently unclear if that is caused by a regression in that version of Hackney, or due to high server load. * There seems to be regression in Hackney 1.25.0 that causes Swoosh to fail sending email via Mailgun due to "incompatible ssl options" being specified possibly by Swoosh itself (#7855); Pleroma is also possibly affected by this issue when an instance admin specified custom `ssl_options` in the configuration file or ConfigDB. (#7824) * The latter could not be reproduced on a local development instance. * What version is unaffected is unknown, but based on Hackney changelog the first affected version might be [1.21.0](https://hexdocs.pm/hackney/news.html#1-21-0-2025-02-20). * Newer versions than 1.25.0 cannot be used because they all require OTP27+ and possibly changed connection pooling in an incompatible way to how Pleroma uses it. Due to the above, 1.20.1 might be a better version to downgrade to since it possibly does not have the `ssl_options` regression, although I have no way to confirm that since I could not reproduce the issue and I don't have a Mailgun account to test with.

phnt commented

2026-03-24 19:54:25 +00:00

Owner

Also when I thought about this more. The long wait could be possibly related to the default connect_timeout in Hackney being way too long (8 seconds). Try lowering that and see if the times change and check if the host you are connecting to actually has working IPv6 besides AAAA records.

https://hexdocs.pm/hackney/1.25.0/hackney.html#request/5

Also when I thought about this more. The long wait could be possibly related to the default `connect_timeout` in Hackney being way too long (8 seconds). Try lowering that and see if the times change and check if the host you are connecting to actually has working IPv6 besides AAAA records. https://hexdocs.pm/hackney/1.25.0/hackney.html#request/5

feld commented

2026-03-24 20:09:17 +00:00

Author

Owner

@phnt wrote in #7860 (comment):

Also when I thought about this more. The long wait could be possibly related to the default connect_timeout in Hackney being way too long (8 seconds). Try lowering that and see if the times change and check if the host you are connecting to actually has working IPv6 besides AAAA records.

https://hexdocs.pm/hackney/1.25.0/hackney.html#request/5

tried playing with these Hackney timeout values when doing my testing, it didn't affect the results. And my test was against a webfinger lookup of @fakeaccount@friedcheese.us which is ~1ms away from my laptop, no AAAA records at all.

@phnt wrote in https://git.pleroma.social/pleroma/pleroma/pulls/7860#issuecomment-114822: > Also when I thought about this more. The long wait could be possibly related to the default `connect_timeout` in Hackney being way too long (8 seconds). Try lowering that and see if the times change and check if the host you are connecting to actually has working IPv6 besides AAAA records. > > https://hexdocs.pm/hackney/1.25.0/hackney.html#request/5 tried playing with these Hackney timeout values when doing my testing, it didn't affect the results. And my test was against a webfinger lookup of @fakeaccount@friedcheese.us which is ~1ms away from my laptop, no AAAA records at all.

feld force-pushed hackney-downgrade from 60b03191f0

ci/woodpecker/pr/woodpecker Pipeline failed

Details

to 7582b71f46

ci/woodpecker/pr/woodpecker Pipeline was successful

Details

2026-04-08 19:29:20 +00:00

Compare

feld commented

2026-04-08 19:29:45 +00:00

Author

Owner