Consolidate on one HTTP client stack #7944

Open
opened 2026-07-26 15:29:23 +00:00 by lambadalambda · 6 comments

While working on the current dependency security update we had to pin hackney to a fork because of a connection leak that can't be worked around from the outside (https://github.com/benoitc/hackney/issues/918). That prompted me to take stock of our HTTP client situation in general.

Some numbers from our git history: over the years we've gone through httpoison, hackney, gun and (optionally) finch, forked hackney twice, and we currently maintain about 3200 lines of workaround code plus 3600 lines of tests that only exist to pin that code. max_body, redirect handling and url encoding are each implemented twice, once per adapter family. Basically every adapter seam bug we ever had comes from having two parallel stacks.

I and Fable had a deeper look at all the available clients (mint/finch, gun, hackney 4, plus the long tail of httpc, ibrowse, fusco, buoy, katipo, erqwest). The short version: gun plus our own connection pool is the only stack that fulfills all our requirements today, most importantly SOCKS5/CONNECT proxies for onion federation. mint/finch is architecturally the nicest (whole classes of connection lifecycle bugs are impossible by construction) and akkoma has run it in production since 2022, but the stack has no SOCKS support at all, so it's not an option for us right now.

So my suggestion:

  1. Fix three known issues in the gun stack: we never set gun's flow option, so media proxy streaming has no backpressure (unbounded message queue growth with a fast origin and a slow client); gun can silently replay a request on a fresh connection during its reconnect, which is bad for signed federation posts (fixable with retry: 0); and streams aren't notified on gun_down, so readers just run into the full recv_timeout.
  2. Make gun the default again and remove the hackney path completely once the security update is merged. The 2020 reasons for switching back to hackney (OTP 23 wildcard problems etc.) are long obsolete, and the pool rework in #7934 fixed the pool problems.

What do you think?

While working on the current dependency security update we had to pin hackney to a fork because of a connection leak that can't be worked around from the outside (https://github.com/benoitc/hackney/issues/918). That prompted me to take stock of our HTTP client situation in general. Some numbers from our git history: over the years we've gone through httpoison, hackney, gun and (optionally) finch, forked hackney twice, and we currently maintain about 3200 lines of workaround code plus 3600 lines of tests that only exist to pin that code. max_body, redirect handling and url encoding are each implemented twice, once per adapter family. Basically every adapter seam bug we ever had comes from having two parallel stacks. I and Fable had a deeper look at all the available clients (mint/finch, gun, hackney 4, plus the long tail of httpc, ibrowse, fusco, buoy, katipo, erqwest). The short version: gun plus our own connection pool is the only stack that fulfills all our requirements today, most importantly SOCKS5/CONNECT proxies for onion federation. mint/finch is architecturally the nicest (whole classes of connection lifecycle bugs are impossible by construction) and akkoma has run it in production since 2022, but the stack has no SOCKS support at all, so it's not an option for us right now. So my suggestion: 1. Fix three known issues in the gun stack: we never set gun's `flow` option, so media proxy streaming has no backpressure (unbounded message queue growth with a fast origin and a slow client); gun can silently replay a request on a fresh connection during its reconnect, which is bad for signed federation posts (fixable with `retry: 0`); and streams aren't notified on `gun_down`, so readers just run into the full recv_timeout. 2. Make gun the default again and remove the hackney path completely once the security update is merged. The 2020 reasons for switching back to hackney (OTP 23 wildcard problems etc.) are long obsolete, and the pool rework in #7934 fixed the pool problems. What do you think?
Author
Owner

Some download numbers from hex, per week: mint 354k, finch 304k, hackney 293k, req 235k, gun 112k. Mint and finch both recently overtook hackney, req is the fastest growing, hackney and httpoison are living on old defaults. Gun is maintained but a niche, less than 100 hex packages depend on it, most of its downloads come from a few big users like grpc.

Another option would be to add the missing parts to mint ourselves, mainly a socks5 transport, it's not that much code. Then finch becomes usable for us and we have at least two clients to choose from. Akkoma has been running finch since 2022, so the rest of that path is known to work.

Some download numbers from hex, per week: mint 354k, finch 304k, hackney 293k, req 235k, gun 112k. Mint and finch both recently overtook hackney, req is the fastest growing, hackney and httpoison are living on old defaults. Gun is maintained but a niche, less than 100 hex packages depend on it, most of its downloads come from a few big users like grpc. Another option would be to add the missing parts to mint ourselves, mainly a socks5 transport, it's not that much code. Then finch becomes usable for us and we have at least two clients to choose from. Akkoma has been running finch since 2022, so the rest of that path is known to work.
Member

github.com/elitepleb/mint@213ad48cbe for an old mint socks poc

+1 for gun, req wraps finch wraps mint, it's a mess, pooling is written around a few hosts instead of thousands, see akkoma.dev/AkkomaGang/akkoma@690b973544

https://github.com/elitepleb/mint/commit/213ad48cbec2b09696ab7169ad1e2bb477c258af for an old mint socks poc +1 for gun, req wraps finch wraps mint, it's a mess, pooling is written around a few hosts instead of thousands, see https://akkoma.dev/AkkomaGang/akkoma/commit/690b973544f8faba75b21ba09eeb0e90c8698a0c
Owner

@lambadalambda wrote in #7944 (comment):

mint/finch is architecturally the nicest (whole classes of connection lifecycle bugs are impossible by construction) and akkoma has run it in production since 2022, but the stack has no SOCKS support at all, so it's not an option for us right now.

What's worse, it has an annoying pool management and config. Finch does what Hackney switched to as well and that is per-host connection pooling. Every new host is a new NimblePool pool with by default no idle timeout, so by default all outgoing federation requests (in the hundreds) create hundreds of pools that remain alive until Pleroma is stopped. So that is opening another can of issues that needs to be dealt with and have tests written for.

https://finch.hexdocs.pm/Finch.html#start_link/1

@lambadalambda wrote in #7944 (comment):

2. Make gun the default again and remove the hackney path completely once the security update is merged. The 2020 reasons for switching back to hackney (OTP 23 wildcard problems etc.) are long obsolete, and the pool rework in #7934 fixed the pool problems.

Agreed, I'm in favor of changing the default to Gun in the next release after this security one. I would however keep Hackney as a backup, sort of how like Gun was kept around. What I would however remove is Finch instead.

@lambadalambda wrote in #7944 (comment):

req 235k

Req is built on top of Finch, and it retains the Finch default of keeping connections alive until the end of time:
https://req.hexdocs.pm/Req.Steps.html#run_finch/1-http-1-pools

@lambadalambda wrote in https://git.pleroma.social/pleroma/pleroma/issues/7944#issue-13286: > mint/finch is architecturally the nicest (whole classes of connection lifecycle bugs are impossible by construction) and akkoma has run it in production since 2022, but the stack has no SOCKS support at all, so it's not an option for us right now. What's worse, it has an annoying pool management and config. Finch does what Hackney switched to as well and that is per-host connection pooling. Every new host is a new NimblePool pool with by default no idle timeout, so by default all outgoing federation requests (in the hundreds) create hundreds of pools that remain alive until Pleroma is stopped. So that is opening another can of issues that needs to be dealt with and have tests written for. https://finch.hexdocs.pm/Finch.html#start_link/1 @lambadalambda wrote in https://git.pleroma.social/pleroma/pleroma/issues/7944#issue-13286: > 2\. Make gun the default again and remove the hackney path completely once the security update is merged. The 2020 reasons for switching back to hackney (OTP 23 wildcard problems etc.) are long obsolete, and the pool rework in #7934 fixed the pool problems. Agreed, I'm in favor of changing the default to Gun in the next release after this security one. I would however keep Hackney as a backup, sort of how like Gun was kept around. What I would however remove is Finch instead. @lambadalambda wrote in https://git.pleroma.social/pleroma/pleroma/issues/7944#issuecomment-117042: > req 235k Req is built on top of Finch, and it retains the Finch default of keeping connections alive until the end of time: https://req.hexdocs.pm/Req.Steps.html#run_finch/1-http-1-pools
Owner

Also

@lambadalambda wrote in #7944 (comment):

url encoding are each implemented twice

Since Pleroma.Utils.URIEncoding is now a thing, that shouldn't be needed. I'm aware of only the Hackney reverse proxy disabling the Hackney URI encoding with fn uri -> uri end and the multiple calls to that module for handling encoding in multiple places like media upload, reverse proxy and Tesla middleware.

Also @lambadalambda wrote in https://git.pleroma.social/pleroma/pleroma/issues/7944#issue-13286: > url encoding are each implemented twice Since Pleroma.Utils.URIEncoding is now a thing, that shouldn't be needed. I'm aware of only the Hackney reverse proxy disabling the Hackney URI encoding with `fn uri -> uri end` and the multiple calls to that module for handling encoding in multiple places like media upload, reverse proxy and Tesla middleware.
Owner

What do you think?

I continue to be extremely happy with Gun and have run it for years now, finding it to be the fastest to federate and fastest to shrink its memory footprint afterwards.

> What do you think? I continue to be extremely happy with Gun and have run it for years now, finding it to be the fastest to federate and fastest to shrink its memory footprint afterwards.
Author
Owner

i've had it running with the whole stack of PR fixes for like a month now and afaict it 'just works'.

i've had it running with the whole stack of PR fixes for like a month now and afaict it 'just works'.
Sign in to join this conversation.
No milestone
No project
No assignees
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
pleroma/pleroma#7944
No description provided.