Hi all. Several of you have reported problems with fedia.io not federating with other instances correctly.
The cause is that rabbitmq crashed, but not all the way. It crashed to the point where new connections would timeout, but the service was still running such that it wouldn’t auto restart. I will be creating some automation to detect that proactively and restart rabbitmq if/when it happens again.
Not entirely. It looks like the rabbit issue was only impacting one of the queues (“deliver”), though I would have expected that to impact things like microblog too. All I can say with clarity is that the instance was operating in a very unhealthy state.
The queue appears like it’ll take several hours to flush, but it’s working.
i salute you!
be aware that this is fixed now