Video system crashes and overheating CPUs

For several months we’ve been trying to track down a very elusive cause of the video system crashing. From extensive configuration changes to all servers to code patches to the primary ingest software by its vendor. Two days ago we set up hyper logging of all OS events and waited for the primary ingest server to crash again. Like clockwork, it crashed last night and shed light on a serious issue. One of the CPUs in the primary ingest server were running over 90c at less than 50% load. This caused the OS to throttle that CPU down significantly which caused the ingest server software to lock up, which caused a domino effect, thus resulting in a complete video system outage.

At 9am, July 12, 2018, the datacenter housing that server is replacing it with an entirely new server. This will cause about 15-30 minutes downtime, unfortunately. But I’d say that’s minimal compared to the frustration the video system crashes have caused us and everyone using our sites for the last several months.

Of course there’s always the possibility that this may not entirely fix the issue. But based on how the video system crashes when that CPU goes into thermal protection, we’re fairly confident that will be resolved with the new server.

Thank you all for sticking with us and being so patient. The amount of lost sleep and frustration thanks to this issue has been incredible and has taken its toll on us. (I’m personally taking a 30 hour nap after all this.) But we’re pushing forward to try and provide the best live streaming experience we possibly can.

Mark Vaughn

iPhone/iPad/PS4 Viewing

Roughly one month ago we had to disable our HLS video system due to DDoS attacks and abuse. Unfortunately that means you cannot watch live streams on iPhone, iPad and PS4. We are working on a new HLS video system that can mitigate the DDoS attacks and abuse, but there is currently no ETA on when it will be ready. There is currently no alternative way to watch live streams on iPhone, iPad and PS4.

Sorry 😦

Known MvnCaster Issues

If you aren’t familiar with the name MvnCaster, it is the name of our built in broadcast tool. You can find it by clicking the red Go LIVE Now button at the top of our sites (excluding iNSTAGIB.tv)

Recently we rolled out a new MvnCaster to replace the old Flash based one. Right now it’s in Experimental phase of development due to many browsers implementing WebRTC differently. So far the most stable and RFC compliant implementation is with Google Chrome. Although it does work with Firefox, too.

There are some known issues with the new MvnCaster, here is a list (in no specific order):

  • Audio playback on the Go LIVE Now page causing an echo
  • No video device (black screen) even though webcam is hooked up
  • Getting kicked off air as soon as you start broadcast (we believe this is associated with the no video device error above)
  • Doesn’t work on latest version of Opera or Microsoft Edge
  • Only works in Safari on iOS (iPhone/iPad)

These issues are being addressed and will be rolling out bug fixes to (hopefully) address these known issues. Although the Micrsoft Edge issue may be last on the list due to how broken their WebRTC implementation is.

If you have any questions or experience any issues, please visit Tech Corner and one of our awesome volunteer Tech’s will do what they can to help you!

https://vaughnlive.tv/tech_corner

Note: Tech Corner is available on all of our sites.

Flash Is Dead

We have officially shut down our Flash video servers and removed all Flash video players from our sites. If you’re a viewer and use a computer that is less than ten years old, you likely won’t notice anything.

 

Broadcasters

If you’re a broadcaster and your stream isn’t viewable anymore, you will need to update your encoder settings (or even your encoder. Note: Free version of XSplit does not work due to its audio codec.) We recommend using OBS Studio (it’s free and awesome!) [ https://blog.vaughnsoft.com/2017/06/18/obs-studio-setup-tutorial/ ]

Here are the basic encoder settings required:

  • Video codec: H.264/x264
  • Profile: Baseline, Main, or High (preferably High)
  • FPS: 30 or 60. Not 29.9, not 50. If you do not use either 30 or 60, expect your viewers to have playback issues on your stream.
  • Video resolution: Any 16:9 ratio should work (Examples: 640×360, 1280×720, 1920×1080). We suggest starting with 640×360
  • Video Bitrate: This depends on your upload speed and the video resolution used in your encoder. We usually suggest 350Kbps to 500Kbps for 640×360 to start out with. Keep in mind that the higher the video bitrate you use, the less likely folks with bad internet will be able to watch your stream.
  • Audio codec: AAC or MP3

Note: If you use a high video resolution in your encoder with a low bitrate, you are likely to have playback issues.

 

Watching Streams

There is a small group of folks that still use Windows XP. Browser development and updates for Windows XP stopped years ago, meaning the new HTML5 features that modern browsers support, are not supported in Windows XP. So what does that mean for Windows XP users? With Flash removed, they will no longer be able to watch streams on our sites. In fact, they won’t be able to watch HTML5 live streams on any site on the internet since all major websites are dropping Flash. Not to mention that Adobe (the owner of Flash) is officially ending Flash soon.

TL;DR: If you use Windows XP to watch our streams, you’re going to need to use a computer that isn’t old enough to join the military. Sorry.

What is “latency” and why does it matter?

When I say latency I am referring to the time it takes for what the broadcaster says and does to what the viewers see. You can experience latency as high as 30 seconds on platforms such as YouTube Live, Facebook Live, Twitch, and many more. This high latency is not ideal if the broadcaster is trying to interact with their audience. Imagine asking the broadcaster a question and they respond to it 15 to 30 seconds later. What a mess!

I am proud to say that after over two years of frustration and a lot of sleepless nights, our sites now have a full HTML5 live video solution that can provide 2 second latency even at 1080p60 resolution. This is a huge step forward for VaughnSoft and a big step forward in the fight for low latency live streaming. However 2 seconds is still a long time. My goal is to reach sub second latency with full HD live streaming and I will not stop until that happens.

Thank you all that have been patient and supportive during the frustration of getting this new video system dialed in. There is more work to be done!

This is only the beginning.

Mark Vaughn