Author: VaughnSoft

VaughnSoft Blog

Vaughnmas 2022 is upon us!

Watch it LIVE on Sunday December 18th, 6pm CST / 7pm EST / 4pm PST / 12am GMT

URL: https://vaughn.live/vaughnmas

Where do you send your submission?

Due to abuse of the Dropbox upload link, please send us an email to contact@vaughnsoft.com with subject “Vaughnmas 2022 Upload” and we will reply with a link for you to upload your video and/or greeting card. Thank you for your understanding!

Servers at capacity? What does that mean?

If you’ve been following the video system updates we’ve been giving, you’re aware of a problematic server company causing a lot of our buffering issues. This morning we pulled the plug on that server company entirely which has greatly increased the quality of service for everyone. The downside? When we pulled the plug on that problematic server company, we reduced our video edge outbound capacity by 10Gbps. But what point is there in having that extra 10Gbps of capacity if it’s absolute garbage and making everyone mad, including me?

That’s where the “Uh oh! Servers are at capacity” message comes in. Since we have reduced video edge capacity right now, this message will show up more frequently. This is to ensure the existing video edge capacity we have isn’t maxed out. If we didn’t have that safety measure in place it would cause buffering and crashing to happen due to outbound bandwidth being maxed out. We’ve had this system for many years, long before we offered VIP packages. Until VIP came along, you had no option to watch a stream on our platform if our servers were at capacity. But now, because VIP is on its own video system, it allows you to bypass that servers at capacity message entirely. Or you can wear out the F5 button on your keyboard.

So why don’t we replace the servers we pulled offline or add capacity? The primary reason is cost of quality. We’re making quality of service a priority rather than quantity. As a result, it is far more expensive to add capacity to maintain a high quality of service. As of right now the plan is to add an additional 5Gbps of capacity before the end of this month.

Thank you all for your patience and thank you all for being such an amazing community! And a huge shout out to our VIP members! You guys are incredible!

-Mark

Progress on video system issues

The last few weeks have been frustrating for everyone involved. But we have our new servers in place and our video ingest system is now rock solid as far as stability of it goes. Both software and bandwidth wise.

The video edge system has been a different story. We removed our servers from Chicago and Dallas due to how problematic the issues at those datacenter locations have become. From blatantly lying to us, overselling bandwidth despite selling us dedicated 10Gbps bandwidth, and outright incompetency from them messing up routing so frequently.

Our new primary edge server is located in Virginia, in the same building as our primary ingest server, but different company. We’ve had a couple hiccups with congested routes on the edge server within the first 24 hours, but they mysteriously resolved after verbally ripping into the company in question. Keep in mind, same company that we had our Chicago and Dallas locations through. Ingest is through a totally different company, surprisingly zero issues and incredible support from them when we’ve needed it.

Once the bandwidth and routing congestion issues were resolved, another weak point of our edge system showed itself: the edge software. It’s a situation of this weak point not showing itself during local testing and simulated load tests. But once it gets put in production, despite same hardware, OS version, and such, it becomes a completely different story.

This aforementioned issue with the video edge software is something that has plagued our platform for years, often going weeks without issue and then suddenly showing itself. Our first thought was DDoS attack. While we did experience multiple DDoS attacks the last few weeks, extensive monitoring and coordination from the datacenter, ruled out this being caused by any type of attack. But rather it’s being caused by an obscure memory leak that only happens under the absolute perfect circumstances, typically during high traffic hours. Adding RAM bandaids the issue but there comes a point where no amount of additional RAM is a feasible resolution for the problem.

So what are the next steps? At the moment I am gutting the video player for desktop viewers, which is where most of our video traffic comes from. Basically everything under the hood of the video player is being updated and rewritten. This will help resolve playback errors that sometime happen which result in the stream randomly stopping and the loading message appearing as it tries to get the stream back, and help with load time. The look of it won’t change. This video player update will all be finished within the next 12 hours barring any critical issues that may arise.

And what about the edge server software? That’s ultimately the issue, right? Yes it is. I’m prototyping and testing new edge software that will not only be more reliable but even reduce latency. We do not use industry standard HLS or LL-HLS for stream delivery. Those are not ideal solutions when you’re trying to keep latency very low and scale it without investing millions of dollars in infrastructure. That’s the secret to Twitch (Amazon) and YouTube Live (Google) achieving low latency through HLS and DASH. They can afford to dump hundreds of millions into hardware to absorb the very high resource cost of low latency via HLS and DASH.

If you’ve read this far, thank you! We’re just as frustrated as you are about the video issues the last few weeks, many of such issues were completely out of our hands. But rest assured we’re not stopping until they’re completely resolved, and by the looks of it, have a better streaming experience than before.

-Mark

Stream buffering issues – What’s causing them?

The last 48 hours have been frustrating for a lot of people. There has been a routing issue between the Chicago video edge location, which is where free users connect to, and specific ISPs. If you are experiencing buffering across all streams, not just one or two, then please provide the following info:

  1. The name of your ISP
  2. Your current IP address ( https://www.whatismyip.com )
  3. A screenshot of a trace route from your computer to video-cdn.vaughnsoft.net ( How to do this: https://lexisnexis.custhelp.com/app/answers/answer_view/a_id/1088078/~/running-a-traceroute )
  4. The results from our speedtest tool to Chicago: https://speedtest.vaughnsoft.net (must use this tool and select Chicago. We cannot accept any other speedtest tool results)
  5. Send this info to contact@vaughnsoft.com (subject: ISP info) or get a hold of Mark in Tech Corner ( https://vaughn.live/tech_corner) and give him the info and screenshots

A big thank you to all that have helped troubleshoot this issue, and a big thank you to those that have remained patient while we try to resolve this extremely frustrating problem.

If you are a VIP Gold member, you won’t experience this issue. If you aren’t a VIP Gold member already, become one to gain access to the much better advanced routing which eliminates 99% of stream buffering issues. Go to https://vaughn.live/howto/vip

-Mark

Chat Exploit Found & Fixed

Today a chat exploit was brought to our attention. It allowed authenticating as any username on the chat servers. By specifying a username and any chat authentication key, you could chat as anybody. That has now been fixed and unlikely to have been used/exploited by anybody outside of discovering the exploit.

This, however, did not grant access to ones account. All of that information is safe and no action on your part is needed.

Big thank you to pizzabiceps that alerted us of this exploit! We have given them one year of VIP Gold for free as a thank you.

-Mark

Thank You

From the bottom of my heart, and the bottom of my family’s heart: Thank you! To all of you that reached out to us, prayed for us, donated, bought VIP, bought emojis. A big Thank you! to all of you! You guys are amazing and we are so blessed to have such an incredible community.

Though we’re not fully out of the woods yet, you guys have lifted a big weight off of our shoulders. What happened was not expected, that’s for sure, and we’ve been working to ensure that sort of incident (hopefully) doesn’t happen again. You truly never know what to expect in this business at times. It has been gut wrenching, to say the least. But you guys made it bearable.

So, once again we thank you. We love you. We are blessed to have you as our community. And we hope to bring you an even better experience. You guys deserve it! Keep an eye out for the new chat interface, new layout, and much, much more!

-Mark, Lynne, and Mr.

P.S. – We’ve been asked numerous times how you can help. Some don’t have PayPal and others do not want to use PayPal, and we totally understand! Below are ways you can help (if you want to):

PayPal Donate
https://vaughn.live/howto/donate

Amazon Wish List
https://www.amazon.com/hz/wishlist/ls/51ROLBS87555

Send (completely anonymous) Amazon Giftcard to giftcard@vaughnsoft.com
https://www.amazon.com/Amazon-eGift-Card-Logo/dp/B07PCMWTSG/
Note: Amazon Giftcard does not share any personal info with me, not even your email address

Myself and my family need your help

So, at the end of February, one of the ad companies I heavily rely on wasn’t running campaigns on VL. I didn’t discover this error until the end of April. As a result I’ve lost a lot of money that I deeply relied on. Things have since been resolved as of last month, but I won’t see that money in my bank until September. The stress this has caused has made me physically sick. What little buffer I had in my bank is now gone.

Please pray for me and my family. And if you can, please buy VIP or donate. You can donate here: https://vaughn.live/howto/donate

-Mark

Categories and Vaughn Live

Categories are an important aspect of live streaming. It allows people to find the kind of content they’re looking for.

We are focused on IRL content and people that turn on a camera and talk. We have many amazing streamers that do this and can be found in our People / IRL category.

In the near future we will be imposing strict requirements to have your channel in the People / IRL category.

For example: you are doing an IRL stream for a few hours but switch to a music playlist and a static image on screen, you will need to move your channel to the Music Cafe category.

If you do not, you risk your channel being locked in that category and you will no longer be allowed in the People / IRL category once these category rules take effect.

-Mark

Classic Guest Chat is back?

I know what you’re thinking… “Mark, didn’t you say guest chat was terrible and never coming back?”

Yes I did, person reading this blog post. Since removing guest chat almost four years ago, a lot of you have expressed your dislike of its removal. Through many live streams and open discussions, we came to a middle ground: Guest chat can come back, but with more limitations and only available to those on the mobile app.

Fast forward to today and you’ll see some of those decisions have been modified. Because the new chat server software and chat clients are nowhere near completion, I decided to bring back the Classic Guest Chat until the new system has been completed. That means guest chat can be enabled by broadcasters in their channels and anybody can chat, no mobile app needed. Ironically the Classic Guest Chat isn’t available in the current Android app. So if you want to chat as a guest, you’ll need to do it from a browser.

But not so fast, it’s called Classic Guest Chat but some things have been changed. There are no longer infinite guest names. They are now tied to your IP address. VPNs are of course not allowed (we didn’t have a VPN filtering system in place during the original guest chat days.) And if you’re a broadcaster, you are still responsible for moderating those that chat as guests. If at any time we feel a broadcaster is not correctly moderating their guest chat, we will revoke their guest chat from their channel. Though, admittedly I don’t see that happening unless in extreme scenarios.

So there you have it, Classic Guest Chat is back on Vaughn Live (sorry Breakers.TV, no guest chat for you.) It’s off by default in all channels. Have fun and say something nice!

-Mark