Yahoo! Sports - Yahoo! Fantasy Football Website Crash

Yahoo! Fantasy Football Website Crash

On Sunday, November 11, 2012 Yahoo! Fantasy sports experienced an outage. By Monday, November 12 the problem had been identified and service was re-established. The following letter from Ken Fuchs, Head of Yahoo! Sports – including an FAQ for consumers - was issued to users on Wednesday, November 14. See below:

Dear Yahoo! Fantasy Users,

As I mentioned in my earlier email, Yahoo! works hard to provide great experiences for our users every day. For you, we understand that Fantasy is a core part of your daily routine during this time of year. On Sunday we experienced technical issues which caused the Fantasy experience to be inaccessible. We're sorry. Our entire team worked tirelessly to re-establish the service - however, we had a significant outage. We know this created inconvenience and frustration for millions of our fans, and we feel we owe you an explanation.

So what happened? At Yahoo!, we have giant machines called "filers" that process a lot of the real-time data and stats for us and for you. We do millions of calculations every hour for our games, and normally our machines can handle this with no problem. Recently, we discovered a hardware issue in one of the filers that caused the other one to overload. We replaced some hardware, re-configured the setup, and did some testing. However this Sunday - at approximately 12:15 p.m. Eastern - the new configuration failed. This created an overload on storage capacity and took the Fantasy part of our site down.

We had dozens of engineers from various teams working together to try to determine the cause and fix it. One option was to fall back on another data center, but that would have meant shutting the game down and losing scoring data. We wanted to avoid that at all costs. Ultimately, we were able to move our mobile apps to a back-up data center, free up storage to get the PC version of the game working, and get the mobile apps up in a "read-only" state - meaning you could see scores and data, but you still couldn't set lineups and interact.

We spent Sunday night and most of Monday looking at dozens of potential causes. Monday afternoon, we stress-tested our system. Everything seemed to be in working order, so we turned on all our mobile app functions in time for Monday Night Football. Everything performed as expected and continues to do so. We'll have all hands on deck this coming Sunday to closely monitor performance and ensure we can respond quickly in case of any abnormal activity.

We know many of you have questions about your leagues and games, so we've also set up a FAQ here that will explain how we are going to handle various scenarios.

Finally, we know we failed you this weekend, and there's no way to fix that. As a token of our appreciation for your patience and continued support, we wanted to give something back. First, we're offering a complimentary football weekly Scouting Report for the remainder of the year. Hopefully this will help you down the stretch. The report includes weekly expert analysis, rankings and news to help you make smart decisions. Second, we are offering all our users a 20% discount for the rest the year at the Yahoo! Sports Store.

For any updates, we will keep you informed on @YahooSports and facebook.com/yahoosports.

We won't stop working to provide you with new and improved features that help you enjoy your Sundays.

Thanks again for playing and for your patience.

Best,

Ken Fuchs Head of Yahoo! Sports

The Street published a story about this incident, telling readers that the twitter account of Marissa Mayer, Yahoo!'s new CEO, was flooded with complaints and that Yahoo! responded by emailing an apology to its users. .