Distributing the job queue capacity evenly between instances
Yesterday's round of federated spam made me think of how to lessen the impact for pleroma instances. I do not think rate limiting by host is feasible on a federated scale (though we can try greylisting on first appearance), instances can have anywhere from 1 to 1 million users, so it would be hard to devise a timeout that includes both and is useful at the same time. However, one thing we probably can do is distribute the capacity of the job queue evenly. That is, if there are 4 instances pushing at full speed, only 12-13 activities will be processed from one of them at a time, provided the queue has a 50 job limit.
As for how to do this technically, starting a separate oban queue for every instance and scaling it up or down is obviously not very feasible. However, Oban already supports job tags, though currently they are used only for metadata. We can submit a patch to Oban to add an option to use these tags to plan how the jobs are scheduled.
cc @lambadalambda, @lanodan, @feld to check if this makes sense and if this is something we need in Pleroma. If it is, I will ask the Oban maintainer about implementing something this