It seems the changes in !3398 (merged) caused issues after all. I think this MR should fix it, by purging only the pleroma build directory between runs. This way pleroma still has to be rebuilt every time, but deps don't (unless mix.lock changes). A happy medium.