For several months we’ve been trying to track down a very elusive cause of the video system crashing. From extensive configuration changes to all servers to code patches to the primary ingest software by its vendor. Two days ago we set up hyper logging of all OS events and waited for the primary ingest server to crash again. Like clockwork, it crashed last night and shed light on a serious issue. One of the CPUs in the primary ingest server were running over 90c at less than 50% load. This caused the OS to throttle that CPU down significantly which caused the ingest server software to lock up, which caused a domino effect, thus resulting in a complete video system outage.
At 9am, July 12, 2018, the datacenter housing that server is replacing it with an entirely new server. This will cause about 15-30 minutes downtime, unfortunately. But I’d say that’s minimal compared to the frustration the video system crashes have caused us and everyone using our sites for the last several months.
Of course there’s always the possibility that this may not entirely fix the issue. But based on how the video system crashes when that CPU goes into thermal protection, we’re fairly confident that will be resolved with the new server.
Thank you all for sticking with us and being so patient. The amount of lost sleep and frustration thanks to this issue has been incredible and has taken its toll on us. (I’m personally taking a 30 hour nap after all this.) But we’re pushing forward to try and provide the best live streaming experience we possibly can.
Roughly one month ago we had to disable our HLS video system due to DDoS attacks and abuse. Unfortunately that means you cannot watch live streams on iPhone, iPad and PS4. We are working on a new HLS video system that can mitigate the DDoS attacks and abuse, but there is currently no ETA on when it will be ready. There is currently no alternative way to watch live streams on iPhone, iPad and PS4.
The main reason I get from people that don’t use OBS Studio, they say there is no easy way to capture just a portion of a screen. So in a little over 3 minutes the video below will show how this is accomplished.
If you aren’t familiar with the name MvnCaster, it is the name of our built in broadcast tool. You can find it by clicking the red Go LIVE Now button at the top of our sites (excluding iNSTAGIB.tv)
Recently we rolled out a new MvnCaster to replace the old Flash based one. Right now it’s in Experimental phase of development due to many browsers implementing WebRTC differently. So far the most stable and RFC compliant implementation is with Google Chrome. Although it does work with Firefox, too.
There are some known issues with the new MvnCaster, here is a list (in no specific order):
- Audio playback on the Go LIVE Now page causing an echo
- No video device (black screen) even though webcam is hooked up
- Getting kicked off air as soon as you start broadcast (we believe this is associated with the no video device error above)
- Doesn’t work on latest version of Opera or Microsoft Edge
- Only works in Safari on iOS (iPhone/iPad)
These issues are being addressed and will be rolling out bug fixes to (hopefully) address these known issues. Although the Micrsoft Edge issue may be last on the list due to how broken their WebRTC implementation is.
If you have any questions or experience any issues, please visit Tech Corner and one of our awesome volunteer Tech’s will do what they can to help you!
Note: Tech Corner is available on all of our sites.
We have officially shut down our Flash video servers and removed all Flash video players from our sites. If you’re a viewer and use a computer that is less than ten years old, you likely won’t notice anything.
If you’re a broadcaster and your stream isn’t viewable anymore, you will need to update your encoder settings (or even your encoder. Note: Free version of XSplit does not work due to its audio codec.) We recommend using OBS Studio (it’s free and awesome!) [ https://blog.vaughnsoft.com/2017/06/18/obs-studio-setup-tutorial/ ]
Here are the basic encoder settings required:
- Video codec: H.264/x264
- Profile: Baseline, Main, or High (preferably High)
- FPS: 30 or 60. Not 29.9, not 50. If you do not use either 30 or 60, expect your viewers to have playback issues on your stream.
- Video resolution: Any 16:9 ratio should work (Examples: 640×360, 1280×720, 1920×1080). We suggest starting with 640×360
- Video Bitrate: This depends on your upload speed and the video resolution used in your encoder. We usually suggest 350Kbps to 500Kbps for 640×360 to start out with. Keep in mind that the higher the video bitrate you use, the less likely folks with bad internet will be able to watch your stream.
- Audio codec: AAC or MP3
Note: If you use a high video resolution in your encoder with a low bitrate, you are likely to have playback issues.
There is a small group of folks that still use Windows XP. Browser development and updates for Windows XP stopped years ago, meaning the new HTML5 features that modern browsers support, are not supported in Windows XP. So what does that mean for Windows XP users? With Flash removed, they will no longer be able to watch streams on our sites. In fact, they won’t be able to watch HTML5 live streams on any site on the internet since all major websites are dropping Flash. Not to mention that Adobe (the owner of Flash) is officially ending Flash soon.
TL;DR: If you use Windows XP to watch our streams, you’re going to need to use a computer that isn’t old enough to join the military. Sorry.
When I say latency I am referring to the time it takes for what the broadcaster says and does to what the viewers see. You can experience latency as high as 30 seconds on platforms such as YouTube Live, Facebook Live, Twitch, and many more. This high latency is not ideal if the broadcaster is trying to interact with their audience. Imagine asking the broadcaster a question and they respond to it 15 to 30 seconds later. What a mess!
I am proud to say that after over two years of frustration and a lot of sleepless nights, our sites now have a full HTML5 live video solution that can provide 2 second latency even at 1080p60 resolution. This is a huge step forward for VaughnSoft and a big step forward in the fight for low latency live streaming. However 2 seconds is still a long time. My goal is to reach sub second latency with full HD live streaming and I will not stop until that happens.
Thank you all that have been patient and supportive during the frustration of getting this new video system dialed in. There is more work to be done!
This is only the beginning.
It’s been 48 hours since we officially rolled out the new low latency HTML5 video system on Vaughn Live. So far everything is looking very stable on the server side and mostly stable on the player side. Here are some known issues with the new HTML5 video player:
- Stream loads, plays, pauses, then resumes playing smoothly.
- Sometimes a black screen but with audio. Pause/Play or Refresh typically fixes that. **
- Fullscreen crops the video if you have your browser zoomed in (CTRL + 0 to reset browser zoom)
- Volume defaults to 100% when unmuting despite the pre mute volume being much lower.
- Volume will show as muted but not unmute when muted speaker is clicked if you unmuted the player in another tab.
- If a stream takes several seconds to load, this is due to you landing on a video edge server that hasn’t propagated the stream yet. This happens the most for streams with few live viewers or streams that have just gone live.
- If you see a message on the video player that says “Cannot watch this stream on this player Click here to watch on the Flash player”, this is not a browser issue. This is due to the broadcaster using the Go LIVE Now broadcast tool (aka MvnCaster) or incorrect encoder settings. These streams will have to be watched on the Flash player until the broadcaster corrects their issue.
So when will the rest of our sites get this new video player? More work needs to be done on the video player side before that happens. But we are very close!
** UPDATE: Fixed one of the causes of the black screen moments ago.