Huh, looks like I haven’t done a DNA Lounge blog post in two months. Well, there hasn’t been a lot to talk about, because December and January suck. We had few events, low attendance, and one big, expensive cancellation. Hooray.

Anyway, one good thing that has happened is that, after nearly six months of investigation, someone who is much better than I am at reading tcpdump logs has finally figured out why the video webcast kept stalling all the time. So now you should be able to watch it for hours at a time without it rebuffering, instead of minutes, which is where we had been for a while.

Our uplink from the club is MonkeyBrains, who are wireless, because nobody will sell me a fiber link. And everything about wireless is trash and always has been.

When using RTSP or RTMP for video transport, there was an unusually high number of lost and reordered packets. These are hallmarks of a wireless network (thanks again, MonkeyBrains), but they only seemed to show up under load; meaning that you can’t see the problem using mtr, speedtest, or iperf3, all of which showed 0% loss. All of the testing apps send a bunch of evenly-spaced packets of the same size, but the stream sends unpredictable bursts (because of MPEG keyframes) that lead to loss, and eventually those (successful) retransmissions kill TCP throughput. Then the video stalls.

So the fix was to switch to SRT (“Secure Reliable Transport”, not to be confused with “SubRip subtitles”), which is a UDP protocol that implements its own TCP-like retransmission within a limited window, and with bandwidth smoothing. This avoids the bursting we were seeing when RTSP would blast out many large packets at once.

It took so long to figure this out, because none of the obvious tools were indicating that anything was wrong, even though your lying eyes saw the video crapping out all the time.

“Why don’t you fix the stupid webcast already?” Because it’s fuckin’ hard, dude.

Oh yeah, also I got the payphone working again. Critical stuff here.

Anyway, come see Black Flag tonight. Is fun band, you like.

From jwz via this RSS feed