Right now, we're using mpsc's for almost everything (plus one mutex in the server). These are very nice for having simple interfaces that are easy to reason about, but they're not actually the right solution for everything we're using them for. Notably, we use them to update the sockets threads are listening on/writing to, which means if a flaky connection drops, reestablishes, drops again, and reestablishes again, etc., all before one of the relevant threads tries to use their socket end, that thread will go through and try all the dead sockets it missed before trying the latest one it should be starting with.
The likely best solution for the socket updaters is to switch to a notify plus a mutex (while the existing mutex in the server should be switched to a RW lock). We use this a lot, so it might make sense to implement our own mpsc-like interface that does what we want (a "send" operation that locks the object, updates it, unlocks it, then sends a notify, and a "read" operation that waits for a notify, locks the object, reads it, and unlocks---similar to a watch, but with a single consumer, so allowing ownership).
Right now, we're using [mpsc's](https://docs.rs/tokio/latest/tokio/sync/mpsc/index.html) for almost everything (plus one mutex in the server). These are very nice for having simple interfaces that are easy to reason about, but they're not actually the right solution for everything we're using them for. Notably, we use them to update the sockets threads are listening on/writing to, which means if a flaky connection drops, reestablishes, drops again, and reestablishes again, etc., all before one of the relevant threads tries to use their socket end, that thread will go through and try all the dead sockets it missed before trying the latest one it should be starting with.
The likely best solution for the socket updaters is to switch to a [notify](https://docs.rs/tokio/latest/tokio/sync/struct.Notify.html) plus a mutex (while the existing mutex in the server should be switched to a RW lock). We use this a lot, so it might make sense to implement our own mpsc-like interface that does what we want (a "send" operation that locks the object, updates it, unlocks it, then sends a notify, and a "read" operation that waits for a notify, locks the object, reads it, and unlocks---similar to a [watch](https://docs.rs/tokio/latest/tokio/sync/watch/index.html), but with a single consumer, so allowing ownership).
Right now, we're using mpsc's for almost everything (plus one mutex in the server). These are very nice for having simple interfaces that are easy to reason about, but they're not actually the right solution for everything we're using them for. Notably, we use them to update the sockets threads are listening on/writing to, which means if a flaky connection drops, reestablishes, drops again, and reestablishes again, etc., all before one of the relevant threads tries to use their socket end, that thread will go through and try all the dead sockets it missed before trying the latest one it should be starting with.
The likely best solution for the socket updaters is to switch to a notify plus a mutex (while the existing mutex in the server should be switched to a RW lock). We use this a lot, so it might make sense to implement our own mpsc-like interface that does what we want (a "send" operation that locks the object, updates it, unlocks it, then sends a notify, and a "read" operation that waits for a notify, locks the object, reads it, and unlocks---similar to a watch, but with a single consumer, so allowing ownership).
Progress on this in
0c63ae0da2,4874416e92,58ca61082f.We'll call this done for now with
91268cdd44