Or should I say Cooling On? The datacenter just finished transferring our hard drives for the primary video system server to its new hardware. Hopefully it stays much cooler than the previous.

Thank you guys for being patient! Let’s hope this is a start to a much more stable video experience.

Mark Vaughn

For several months we’ve been trying to track down a very elusive cause of the video system crashing. From extensive configuration changes to all servers to code patches to the primary ingest software by its vendor. Two days ago we set up hyper logging of all OS events and waited for the primary ingest server to crash again. Like clockwork, it crashed last night and shed light on a serious issue. One of the CPUs in the primary ingest server were running over 90c at less than 50% load. This caused the OS to throttle that CPU down significantly which caused the ingest server software to lock up, which caused a domino effect, thus resulting in a complete video system outage.

At 9am, July 12, 2018, the datacenter housing that server is replacing it with an entirely new server. This will cause about 15-30 minutes downtime, unfortunately. But I’d say that’s minimal compared to the frustration the video system crashes have caused us and everyone using our sites for the last several months.

Of course there’s always the possibility that this may not entirely fix the issue. But based on how the video system crashes when that CPU goes into thermal protection, we’re fairly confident that will be resolved with the new server.

Thank you all for sticking with us and being so patient. The amount of lost sleep and frustration thanks to this issue has been incredible and has taken its toll on us. (I’m personally taking a 30 hour nap after all this.) But we’re pushing forward to try and provide the best live streaming experience we possibly can.

Mark Vaughn

Roughly one month ago we had to disable our HLS video system due to DDoS attacks and abuse. Unfortunately that means you cannot watch live streams on iPhone, iPad and PS4. We are working on a new HLS video system that can mitigate the DDoS attacks and abuse, but there is currently no ETA on when it will be ready. There is currently no alternative way to watch live streams on iPhone, iPad and PS4.

Sorry 😦