Recent Chat Issues and TwitchPlaysPokemon

Feb 19 2014 · 54 comments · Events

UPDATE: I wanted to update as I buried the lead. We did find and fix a “fundamental issue” with our redis servers, as noted here. During the week, we also updated some networking configurations in relation to our chat servers that should have smoothed out service in general.

TwitchPlaysPokemon is a bonafide phenomenon. So far we’ve seen millions of unique viewers and over 100k peak concurrent viewers. It has captured the attention of the gaming community and even made its way into the mainstream press.

The unique nature and huge chat participation in the TwitchPlaysPokemon experiment has put enormous (and unforeseen) stress on our chat system. We’re always working on improving the QoS of our chat system, and this has been a wonderful learning experience for us.

Our first adjustment on Sunday was to move the channel off of our general chat servers onto a dedicated event chat server, which we typically use for large events like The International and League Championship Series (LCS). This helped, but there were some fundamental issues with our chat infrastructure that required a review.

One of our long-time engineers, Mike Ossareh, wrote about this in a couple of posts on his personal blog. They’re a very candid read on the fundamentals of our chat system.

This quote from the second post quite succinctly sums up the chat issues we’ve faced :

“When a phenomenon like TPP comes along which increases load on the system many fold, it gives us a great opportunity to discover and fix new issues and issues that only raise their head under super high load.”

We LOVE TwitchPlaysPokemon and we want to see where this grand experiment leads us all. While it plays itself out, we’ll continue to hammer away at our chat system and make sure it holds up to the load.

Tags:

Related Posts

  • Vaughn Whiskey

    Didn’t this Mike Ossareh also say “We’ve successfully resisted the urge to “improve” chat.”

    • s_carda

      damn vaughn with that insta-reply

      • Vaughn Whiskey

        Well i know Twitch is some what trying to fix thing but as a small and struggling streamer trying to become partnered at some point. Not having a working chat kills my stream and its growth. I even started streaming on YouTube again because atleast I can see comments.

        • Fabian222

          (sorry, wrong vaughn)

        • Chris Atkins

          i have to agree with this, i am only a small ish caster on twitch (about 715 follows with an average viewership of between 5 – 15) and i focus massively on chat interaction to help promote my channel and keep people involved.

          this becomes very difficult when the chat is completely unusable for anyone but MASSIVE streamers who have the rep to not need to rely on chat AS MUCH, and one off sensations like TPP or the big tourneys once every few months.

          Please fix this asap twitch, its ruining it for the little people =(

        • http://www.twitch.tv/Dulayne Dulayne

          Totally agree with you Vaughn, it gets disheartening when you see a dead chat with a small viewer count. It is so much easier to find yourself and get into your groove. When half of the time it is in the back of your mind the chat is inactive on top of the super long delay it makes it so much more difficult than it already is. (When you do get one or two people roll in at a time and they are likely to leave within 5-10 seconds, it is not fun)

    • http://ossareh.posterous.com/ Michael Ossareh

      Before committing this to the annuals of internet history I’d like to suggest reading the edit and the follow up post.

      • Vaughn Whiskey

        No, how about you use a different term them “resisted” and maybe explain yourself better. I’d like to suggest editing the edit and the follow up post.

        • http://ossareh.posterous.com/ Michael Ossareh

          I’ve taken your suggestion and edited the post again. The fact that in the very next sentence of the original I say “the correct thing to do is double down on our scalaing efforts” and that is not referenced is quite a bewilderment to me.

          • Vaughn Whiskey

            I don’t mean to come off like a complete dickweed (which i do) but you got to understand where users come from when it involves twitch and the current issues. The 1st statement wrote and read that you put did come off as a strange way to explain it. Now that it has been cleared up twice, it does make more sense now. Once again, sorry for the aggressive manner.

          • http://ossareh.posterous.com/ Michael Ossareh

            I appreciate that Vaughn, thank you.

  • Shaktros

    It’s about time you realise how bad your chat system is, even before pokemon stuff without any lol tournament on, 1/4 of my messages would not show for others without any notification/error message.

  • PCHSC2

    THE HELIX FOSSIL HAS SPOKEN

  • Sam Murray

    While I can appreciate that you can’t predict virality and the effect it will have on your systems, you can take sensible precautions against their impact. Its clear that the chat systems are woefully inadequate at the best of times when any form of large-ish event is occuring. Whenever there is any form of high-participation chat under way, the site drops messages left right and centre.

    Surely this can be resolved simply by better investing in your infrastructure and simply adding more servers to carry the load. While this does represent additional expenditure, it is balanced against the additional income garnered day-on-day by having a strong, reliable service.
    With each issue like this ongoing, Twitch establishes itself less as ‘the place to go for gaming’ and more as ‘the place we go to because well I guess we have to’. Less people talk about the service in a positive light and I’m sure I don’t need to delve into the ‘power of 10′ effect this holds.
    This alongside pushing out a ‘bad code release’ (which pangs of a lack of testing for it to fail in such a glaringly obvious way) just paints Twitch in a pretty bad light. I can’t quite grasp how and why these issues continue to plague the company other than a lack of rigorous testing and forward planning.
    If a consumer of your services is able to see this and gain this opinion, then why are assurances and signs of improvement (or at least solid planning of it) not being shown?I can appreciate operating on this scale, this quickly, can be tough. But isn’t it time to start showing your community you’re fighting their corner?

    • Fabian222

      There should be a compromise if it is possible to do so. Bigger streams like esports SHOULD get dedicated servers, but they have to pay for it.

      RTMP was giving viewer restrictions across the site as a whole, right (from what they say). Well, most RTMP reliant streams did not get the volume that those in esports got, ever. The ability to give higher volume streams their own server shows that Twitch can use this to their advantage easily.

      Implement RTMP back in but have a viewer restriction. If you wish to get passed the cap, you must pay a certain fee for HLS access. Those streams needing the cap restriction removed are guaranteed NOT a stream that needs RTMP. Those that need RTMP will have no problem with this because they won’t reach the cap. Once they do, if this actually happens, they can apply for HLS because the stream literally will be too fast for the person to keep up with in chat anyway.

      It is a win-win for literally everyone. Twitch gets money from Esports streams to get access to HLS servers thus giving them a far bigger audience and twitch getting funds to be able to afford more servers for HLS. Smaller streams get RTMP back since they dont reach a cap, which doesn’t affect them anyway. Twitch gets money.

      This is a compromise that I can’t see not being implemented unless there is a technological reason why they can’t, which can be circumvented by being transparent as literally possible. People are already paying for turbo. Give an option to really “support” the site so this idea can be implemented and people WILL help financially to do so since they already ARE showing support to the site.

  • Fabian222

    Something I have been wondering for a while, and TPP being given its own server makes me even more curious:

    Since you guys are able to give dedicated servers to huge things like TPP, why not have the site go RTMP while those bigger streams that you expect to have massive impact on your server be HLS.

    Most of those in tournament streams do not care for “instant/realtime” input, so HLS does honesty nothing negatively on the user side. Smaller streams *smaller meaning even MvG/Towellee/etc by volume compared to esports streams* could benefit from RTMP while tournament and other massive streams reaching above a certain threshold *lets say 50k for hypothetical example* can be put on HLS.

    This can also help with interactions with esports teams as you can set a sort of “viewer” restriction on RTMP streams and they would have to apply to you guys to get the HLS viewer increase. Esports helps you, you help us. Everyone wins.

    You guys did a test before with having both RTMP and HLS running simultaneously, so theoretically, this is possible right? If so, I think this would help twitch as a whole.

    If this is all dumb, apologies and thanks for at least humoring this enough for a read :)

    • Cyrus Hall

      We’re dedicating chat servers, not video servers. :-) We run exactly zero RTMP servers on the edge. Mixing HLS and RTMP is not possible without reducing our edge efficiency, which would just result in more buffering (i.e., “lag”).

      • Fabian222

        Thank you for the answer and humoring my response :)

        There is so much I would like to learn about twitch and the backend processes it has to go through. Could you guys do a blogpost sometime about the processes twitch goes through to produce streams? I’d love to learn about that sort of stuff.

        • Cyrus Hall

          Possible, but unlikely. It’s very difficult to argue that one should ever eat a 50% reduction in efficiency (actually, soon, far more than that). Given time, we will be able to reduce the latency experienced with HTTP streaming, although it’s unlikely most viewers will ever see five or six seconds again.

          I’d love to be able to explain more about how our backend works; hopefully we’ll have time at some point this year.

          • Fabian222

            Super looking forward to it!

          • Keirnoth

            Although it’s unlikely most viewers will ever see five or six seconds again.

            You should say this up front on your blog about this. It’s hurting a lot of the smaller streamers because of this and it’s driving them away from Twitch.

            Moving to HLS improves your performance with big eSportz streams that provide you your money to keep the lights on, but building goodwill with your smaller streamers is also in your best interests because at this point, I can’t recommend the Twitch service to my friends because of this delay.

          • taizun

            Recommend Twitch? We’re in 2014 – anyone who has an interest in games most likely also know about Twitch. There’s no need to recommend Twitch to anyone anymore.

          • Cyrus Hall

            Keirnoth, I hope you stick around, or come back once we’ve started to reduce the latency, and see if will be low enough. We understand that the current system doesn’t work well for some types of interaction, and we do want to get back to a place where tighter communication loops are possible. Curiously, what sort of delay do you think is acceptable?

          • Kieran Clark

            In my personal opinion, anything over 7 or 8 seconds and it becomes exponentially more difficult to interact properly.

          • HillhomeGaming

            The old delay is acceptable. Maybe even as high as 10 – 15 seconds. However I’m regularly seeing 30 – 45 seconds delay. This breaks communication and still needs further improvement :)

  • WhaleMan

    “Unforeseen” Right…a game driven by SPAMMING chat. Does no one use logic at twitch?

    • David McLeod

      Unforseen as in… it wasn’t like we knew 8 months ago that someone was working on such a broadcaster to viewer interaction mechanic. Like the rest of the world, we woke up Valentines day and discovered it.

      • Sam Murray

        Unforeseen as in virality on the internet is an unknown quantity for companies that have enough spare change kicking about in their coffers to sensibly take precautions in case of such a circumstance?

        • David McLeod

          There are a lot of assumptions here. We are far from short-sighted.

          • Sam Murray

            I don’t mean to aimlessly mudsling here but it all feels like that is part of the issue. Large jumps in viewer/chatter participation should be an expected potential outcome of expanding and being a social interaction site, no? This is hardly the first time something in relation to the site has made it to news outlets/social networking sites.
            While the chat system was under great pressure at that time (and assumably still is), it was touch and go for a period previous to that. While not the perfect system, surely that is something eased by having greater resources to work with (i.e. more capacity for chat services) as these issues don’t tend to occur during off-peak.
            I don’t claim to be any great engineer or authority on the subject (and honestly, I’d love to be further educated on where my assumptions are wrong. I hate being ignorant about things that I feel are important) but with all the eyes of the world turned onto your stage from a multitude of sources, any failures are going to be closely scrutinised.
            I think another aspect is the fact that there are reasons and excuses given, but no real apology thrown out there for Twitch as a whole. No real-time communication either. There’s a global messaging system there, why isn’t it utilised to say ‘hey sorry, we weren’t expecting this. we’re working to sort this out. our bad, don’t blame TPP as its not their fault.’ There’s a certain personal and sincere touch that seems to be missing.

  • VenerableSage

    “Our first adjustment on Sunday was to
    move the channel off of our general chat servers onto a dedicated event
    chat server, which we typically use for large events like The
    International and League Championship Series (LCS). This helped, but
    there were some fundamental issues with our chat infrastructure that
    required a review.”

    I find this hard to swallow. Any time there’s a big event (especially LoL-related), the chat *always* seems to go down. If those servers are already being used from the get-go when those events start, then that demonstrates a bit of a problem – the dedicated event server isn’t enough or simply isn’t that effective at all.

    EDIT – Though, I will admit that since early Fall, this problem hasn’t nearly been as bad as it used to be. (I usually only went on Twitch on Fridays-Sundays for my buddy’s broadcast – often co-hosting with him over Skype – but back then it was literally almost every other weekend that the chat would go down.)

    • Fabian222

      EDIT: I wrote this hastily and it may not fit with exactly what you were saying. Sorry.

      —-

      There should be a compromise if it is possible to do so. Bigger streams like esports SHOULD get dedicated servers, but they have to pay for it.

      RTMP was giving viewer restrictions across the site as a whole, right (from what they say). Well, most RTMP reliant streams did not get the volume that those in esports got, ever. The ability to give higher volume streams their own server shows that Twitch can use this to their advantage easily.

      Implement RTMP back in but have a viewer restriction. If you wish to get passed the cap, you must pay a certain fee for HLS access. Those streams needing the cap restriction removed are guaranteed NOT a stream that needs RTMP. Those that need RTMP will have no problem with this because they won’t reach the cap. Once they do, if this actually happens, they can apply for HLS because the stream literally will be too fast for the person to keep up.

      It is a win-win for literally everyone. Twitch gets money from Esports streams to get access to HLS servers thus giving them a far bigger audience and twitch being able to afford more servers for HLS. Smaller streams get RTMP back since they dont reach a cap, which doesn’t affect them anyway. Twitch gets money.

      This is a compromise that I can’t see not being implemented unless there is a technological reason why they can’t, which can be circumvented by being transparent as literally possible. People are already paying for turbo. Give an option to really “support” the site so this idea can be implemented and people WILL help financially to do so since they already ARE showing support to the site.

  • Paul

    They’re using a client which relies on SPAMMING, plus other games are spawning from it, I think it’s time to take TPP behind the barn and pull the trigger before the hellspawns that are being created do more damage to the already delicate servers.

    • Tom Wisniewski

      I’m already clicking on games only to find out it’s just TwitchPlays spam bot channels and not any real broadcasters. Future of Twitch.

  • Keins Riesel

    “We LOVE TwitchPlaysPokemon”
    Of course you do. You’d love a simple spinning rectangle which changes the color if it’d bring you viewers / press coverage.

  • http://outsanity.tumblr.com/ Outsanity

    …

  • AnEternalEnigma

    Let’s not act like the chat has been perfect before this channel started.

    The chat has been an issue LONG before TwitchPlaysPokemon became a thing. Everyday for the past month it seems, messages are getting eaten by the server, becoming unseen to the rest of the chat and the streamer.

    Add this to the idiotic 20-or-more-second streamer-to-chat delay which STILL exists after two whole months, you get a big ball of garbage that totally kills moderate-to-small-level streamers.

    These problems do not faze big eSports events because they don’t depend on chat whatsoever. Personal streamers of ANY fame, whether it be large, moderate, small, or unknown, need these things to work.

    The server has to stop eating chat messages. The delay has got to be brought back down to a reasonable level. The 5-to-6-second delay before this video system overhaul in December was acceptable. This stuff must improve or this will give the traction a competitor (looking at you, Hitbox) is looking for to get their foot in the door.

    Get it together, Twitch.

  • DemBuns

    hey well you’re fixing things, can you fix this problem for me? It’s been three months now.

    • Cyrus Hall

      Do you AVast A/V and Chrome? If so, please disable the web scanning part of AVast, as it completely breaks video playback.

      • DemBuns

        no I don’t. Also when it does connect it buffers every 5 seconds. I have 50 down 3 up.

        • Cyrus Hall

          Can you load lower resolutions?

          • DemBuns

            yah but it still buffers every 5 seconds

          • Cyrus Hall

            Honestly, internet speed has little do to with live-streaming past a point. Internet speed is like a car: there is a speed limit (end-to-end latency), and even if you can do 200 MPH (50 Mbps down), you never actually reach that speed. Bandwidth is important for large file downloads, where all the data already exists, and you can send huge portions of it at once. But with live streaming we can only send a smaller portion at once, as the rest of the data hasn’t even been created yet, and as such, bandwidth only matters up until you can support the base data rate of the stream.

            It sounds to me like you are either on a very high latency path (where do you live?) or a link between your ISP and us is very congested (what ISP?).

          • DemBuns

            Shaw

          • Cyrus Hall

            I’ve taken a look at our stats for Shaw and do not immediately see a large problem. I’ve also talked to our network operations team, so they are aware of the report. In general, we are having no problem sending data to Shaw, which would indicate the problem is somewhere between us and you. I’d give Shaw a call and ask them if there is congestion between them and us.

            BTW, ping is not a good tool here – when you ping twitch.tv, you are pining our web CDN, not our video servers. Our video servers do not respond to ICMP, so there is no way to ping them directly.

          • DemBuns

            okay, I’ll talk to them and see if i can get a solution. Is there anyway I can talk with someone like you after I call them if there’s any further issues?

          • Andrew

            Hi Cyrus, I’m having the same problem has Dem Buns. My ISP is Frontier Communications, I live in Oregon. when I run a tracert to twitch.tv, it times out after ae-1-6.bar2.sanfrancisco1.Level3.net. but maybe that’s irrelevant because it’s not the video CDN.

  • iamjasonc

    Yeah but why sacrifice hundreds of other streamers chats, and thousands of users twitch experience for one stream that makes a pokemon trainer walk in a circle?

    • Sam Murray

      Because it isn’t the fault of someone having a (frankly damned good) idea that this is happening. It is the fault of the service they use not providing contingency plans for potential high traffic numbers.

  • Ryan David Fitzy

    when will you stop the 30 sec video delay? I want to interact with the streamers I watch daily & my audience when I stream. It’s hard to do that when 30 seconds goes by, & you forget wtf they are answering or questioning something that you said half a minute ago..

  • Mik3lik3

    Does the chat system for commands work on the mobile app (o have an android)

  • thieverpedia

    This is all fine, but it doesn’t solve the problem of the imitators. The massive amount of people put together on all these other streams is still causing immense buffering and chat lag across nearly every stream for the majority of users. Might I suggest creating another server just for “Twitch Plays (Game name)”? In all honesty, I’d like to stop missing streams because of the immense buffering. This never happened before TPP.

  • Matthew Doyle

    I got, you are banned from talking in TwitchPlaysPokemon for 330 seconds, you gotta be freaking kidding me

  • http://Watonzon.Deviantart.com Timewatcher Bri

    The one part that I still have a problem with Twitch plays is the chat system. I understand that you guys want to slow down the chat system a bit but when I post up an action on the chat system the same message comes up to me and it kind of gets annoying. The message that I keep getting is that I’m using “slow chat” and that I have to wait. I understand that you guys want to give the system a chance to input the information it kind of becomes annoying that we have to wait a few 30 seconds in order to make a choice that wont impact the story until after 30 seconds.

    With all of the problems that I see with the chat system I kind of felt like as if I should of leave the site and move on to another fun site that might be worth my time….or for me to pug in my own GBA pokemon games and have me and my friends control the keyboard at my house or at the library. Not trying to sound like a jerk but just requesting that you update the chat system a bit more so that way it wont feel like I have to wait for 1000 people to make up their minds ahead of time before we actually get to the highlights of the story.