Letter from IainC on Open Beta Situation

[IainC_Goa]

Alrighty let's have a little post mortem on the situation.

Firstly let's get the obvious stuff out of the way. Yesterday was in no way, shape or form a success. A lot of things went very wrong. Funnily enough however they didn't go wrong in the way most people thought they would or for the reasons that were being thrown around.

The website and our entire platform got hammered yesterday - which we expected. We knew how many open beta keys we'd given away and what sort of traffic to expect as a result. We got a lot more than we were expecting and yet the platform held up, the site slowed down (which was inevitable with the sheer volume of traffic, even a basic HTML page would have been slow under that onslaught) but everything stayed up. This is pretty much the big ray of sunshine that we took away, our datacentre took the onslaught and survived unscathed.

The account system broke. This wasn't down to the traffic (although that was a factor), nor the fact that the page was in Flash. There was a problem with our database server which didn't surface during testing. As you can read in the news, we have been working on this all night and we expect this to be resolved soon. Ultimately the system didn't work but we're glad we found the issue now in beta rather than before the headstart or live begins. Many of you criticised the decision to bring the account pages online the same day as the beta began but this was not the cause of the problems. While yesterday was a very painful day for us, there is a silver lining because it means that our headstart and live launch will be significantly more robust.

There was some miscommunication as well which falls entirely on us. The confusion over people with war-europe accounts not needing to create a new beta account was the main issue, other than that the team who were in the office yesterday worked very hard to keep you updated. Magnus and the other CMs who were working yesterday were in the office from 7:30am through to 3am this morning. During that time our teams were working on several different situations at once and the situation was often in a state of considerable flux. We had technical teams working on various solutions, management teams dealing with our partners and, for the most part, these people were busy fixing things rather than writing status reports that were likely to be out of date by the time they hit the CM team. Despite that, Magnus and the others posted what they had, when they had it both on the official news site and around the various fan forums.

Overall then, some things went right - not many to be sure and the important things certainly didn't. We made some mistakes, we expected many problems and we found some entirely different ones. We disappointed many of you by not letting you get into the open beta when we said you would and for that I am sorry. All of us are disappointed that we couldn't bring you the service you expected. I wasn't at work yesterday, Magnus was the man on the scene covering the situation, but I did keep a close eye on forums and was in contact with the guys in the office. A lot of what was said yesterday here and on other forums was entirely out of line. Of course you were disappointed and criticism is certainly warranted but frankly many of the posts made about the situation were borderline sociopathic. If having delayed access to a beta test really drives you to such depths of anger and fury that you felt compelled to make the death threats, racial slurs and other deeply unpleasant posts then - and there is no polite way to put this - there is something wrong with you.

Today we have a lot of people at the office who are preparing to put in a long week after a long weekend to make things right. This isn't magically going to make yesterday alright but we are very much committed to making sure that you aren't disappointed again.

Edited for clarity.

[Matrim]

When are we getting servers back up?

PS. We are just angry because we don't like being lied to...and you are well aware of the fact that on these very forums hundreds if not thousands of people told you that this would happen. You personally told us that it wouldn't happen.

[grindae]

Originally Posted by IainC_Goa

Alrighty let's have a little post mortem on the situation.

Firstly let's get the obvious stuff out of the way. Yesterday was in no way, shape or form a success. A lot of things went very wrong. Funnily enough however they didn't go wrong in the way most people thought they would or for the reasons that were being thrown around.

The website and our entire platform got hammered yesterday - which we expected. We knew how many open beta keys we'd given away and what sort of traffic to expect as a result. We got a lot more than we were expecting and yet the platform held up, the site slowed down (which was inevitable with the sheer volume of traffic, even a basic HTML page would have been slow under that onslaught) but everything stayed up. This is pretty much the big ray of sunshine that we took away, our datacentre took the onslaught and survived unscathed.

The account system broke. This wasn't down to the traffic (although that was a factor), nor the fact that the page was in Flash. There was a problem with our database server which didn't surface during testing. As you can read in the news, we have been working on this all night and we expect this to be resolved soon. Ultimately the system didn't work but we're glad we found the issue now in beta rather than before the headstart or live begins. Many of you criticised the decision to bring the account pages online the same day as the beta began but this was not the cause of the problems. While yesterday was a very painful day for us, there is a silver lining because it means that our headstart and live launch will be significantly more robust.

There was some miscommunication as well which falls entirely on us. The confusion over people with war-europe accounts not needing to create a new beta account was the main issue, other than that the team who were in the office yesterday worked very hard to keep you updated. Magnus and the other CMs who were working yesterday were in the office from 7:30am through to 3am this morning. During that time our teams were working on several different situations at once and the situation was often in a state of considerable flux. We had technical teams working on various solutions, management teams dealing with our partners and, for the most part, these people were busy fixing things rather than writing status reports that were likely to be out of date by the time they hit the CM team. Despite that, Magnus and the others posted what they had, when they had it both on the official news site and around the various fan forums.

Overall then, some things went right - not many to be sure and the important things certainly didn't. We made some mistakes, we expected many problems and we found some entirely different ones. We disappointed many of you by not letting you get into the open beta when we said you would and for that I am sorry. All of us are disappointed that we couldn't bring you the service you expected. I wasn't at work yesterday, Magnus was the man on the scene covering the situation, but I did keep a close eye on forums and was in contact with the guys in the office. A lot of what was said yesterday here and on other forums was entirely out of line. Of course you were disappointed and criticism is certainly warranted but frankly many of the posts made about the situation were borderline sociopathic. If having delayed access to a beta test really drives you to such depths of anger and fury then - and there is no polite way to put this - there is something wrong with you.

Today we have a lot of people at the office who are preparing to put in a long week after a long weekend to make things right. This isn't magically going to make yesterday alright but we are very much committed to making sure that you aren't disappointed again.

Any idea why only one server is up at the moment ?

Today is shaping up to be as much of a failure as yesterday was im afraid to say

[DarkDweller]

Dear all,

I rarely address players directly, but I wanted to write you as too few of you have been able to access the open beta today.

As my team explained in the previous post, although we have opened the servers and corrected numerous issues during the last 24 hours but we have not yet solved the issue of saturated Open Beta key activations that we’re currently experiencing. I am sincerely sorry for this, and I understand all the frustration the situation has spawned.

Today was a dark day. We expected a massive influx of players, but reality was even harsher.
I personally commit that we will communicate to you as clearly as possible on the resolution of these current issues. My goal is to give Warhammer Online the success it deserves. Be ensured that all GOA teams, with Mythic’s support, are working hard to bring you the best possible game experience.

Best regards,
Ghislaine Le Rhun
GOA CEO

I would just like to comment that taking the time to communicate with the playerbase is much appreciated. An apology such as this does help somewhat to ease the tension, however I would still agree with most, that the problems currently being experienced could have been mitigated b taking a few simple steps, and perhaps by listening more to the concerns of the community.

As I said, it is encouraging to hear GOA say that they aim to increase dialogue.

[zuckas]

I thank you for the update but it's just a reiteration of what we already knew through magnus or mark. That it wasn't the traffic to the site but the traffic going from the site to the database that screwed up is unfortunate, but the effect is still the same. Had you opened earlier, this problem would have surfaced earlier because nerds like myself would have tried to register straight away anyways.
But ignoring that fact could we at least get a general ETA on when the techs expect it to be fixed? in 5h? Today? Tomorrow?

[Toshiro]

Originally Posted by IainC_Goa

Many of you criticised the decision to bring the account pages online the same day as the beta began but this was not the cause of the problems.

We have never said that this would be the cause. What we have been saying however is that it is a bad bad decision to put the AC up the same day, as it leaves you no buffert to solve any problem that always shows up.

Originally Posted by IainC_Goa

We made some mistakes, we expected many problems and we found some entirely different ones.

So you were expecting problems and still you decided to launch AC the same day as OB? Where is the planning?!

Originally Posted by IainC_Goa

If having delayed access to a beta test really drives you to such depths of anger and fury then - and there is no polite way to put this - there is something wrong with you.

You dont seem to see the core problem here. The source of the hate and anger is not really the fact that we missed a day of beta; sh*t happens! People do however get really angry when being ignored, lied to and insulted multiply times, which has happened. Not in our eyes perhaps, but in the customers eyes. Thats whats important.

Thanks for the answer though..

[Arnafein]

Originally Posted by IainC_Goa

Alrighty let's have a little post mortem on the situation.

Firstly let's get the obvious stuff out of the way. Yesterday was in no way, shape or form a success. A lot of things went very wrong. Funnily enough however they didn't go wrong in the way most people thought they would or for the reasons that were being thrown around.

The website and our entire platform got hammered yesterday - which we expected. We knew how many open beta keys we'd given away and what sort of traffic to expect as a result. We got a lot more than we were expecting and yet the platform held up, the site slowed down (which was inevitable with the sheer volume of traffic, even a basic HTML page would have been slow under that onslaught) but everything stayed up. This is pretty much the big ray of sunshine that we took away, our datacentre took the onslaught and survived unscathed.

The account system broke. This wasn't down to the traffic (although that was a factor), nor the fact that the page was in Flash. There was a problem with our database server which didn't surface during testing. As you can read in the news, we have been working on this all night and we expect this to be resolved soon. Ultimately the system didn't work but we're glad we found the issue now in beta rather than before the headstart or live begins. Many of you criticised the decision to bring the account pages online the same day as the beta began but this was not the cause of the problems. While yesterday was a very painful day for us, there is a silver lining because it means that our headstart and live launch will be significantly more robust.

There was some miscommunication as well which falls entirely on us. The confusion over people with war-europe accounts not needing to create a new beta account was the main issue, other than that the team who were in the office yesterday worked very hard to keep you updated. Magnus and the other CMs who were working yesterday were in the office from 7:30am through to 3am this morning. During that time our teams were working on several different situations at once and the situation was often in a state of considerable flux. We had technical teams working on various solutions, management teams dealing with our partners and, for the most part, these people were busy fixing things rather than writing status reports that were likely to be out of date by the time they hit the CM team. Despite that, Magnus and the others posted what they had, when they had it both on the official news site and around the various fan forums.

Overall then, some things went right - not many to be sure and the important things certainly didn't. We made some mistakes, we expected many problems and we found some entirely different ones. We disappointed many of you by not letting you get into the open beta when we said you would and for that I am sorry. All of us are disappointed that we couldn't bring you the service you expected. I wasn't at work yesterday, Magnus was the man on the scene covering the situation, but I did keep a close eye on forums and was in contact with the guys in the office. A lot of what was said yesterday here and on other forums was entirely out of line. Of course you were disappointed and criticism is certainly warranted but frankly many of the posts made about the situation were borderline sociopathic. If having delayed access to a beta test really drives you to such depths of anger and fury then - and there is no polite way to put this - there is something wrong with you.

Today we have a lot of people at the office who are preparing to put in a long week after a long weekend to make things right. This isn't magically going to make yesterday alright but we are very much committed to making sure that you aren't disappointed again.

Excuse me? Do you mean to purposefully insult us with a post like that? There is something wrong with us? I still can't even validate my OB key.

All in all, you don't seem to be taking any of this very seriously.

[Karasu]

* Edited for Content *

[hozemaat]

On a scale of 1-5, with 1 being lowest and 5 being highest, please rate the GOA OB logistical process and organization. Explain your rating.

1. no communication at all until everything really went wrong, not being able to make an account for the OB a week before the beta launches, this could of relieved allot of stress on the flash site (which is also terrible) and we could of registered we could of all been in the beta right now and we would all be claiming that GOA succeeded in their task... but no they failed...miserably.

What is your main concern regarding the GOA Open Beta? Why is this the case?

Lack of updates on the situation we have only got two short messages saying that they are working on it and that they are sorry about the whole fiasco...sorry GOA this is NOT enough!

Given the experience, what would you have done differently if you were placed in GOA's shoes for this?

Would of been able to create an account a week before OB so that we all could of gradually filled up the beta servers.
Communicate WAY MORE with the community as we are the people that are supposed to buy their game in the first place! I expect to be treated well if i am spending my hard earned money on something i'm looking forward too.

What has GOA done well regarding this process?

admitting their errors i guess, but thats it and i expect more of them.

What could GOA do better in the future to make things smoother in general? Please provide examples.

1 # Communication! as their has been a complete lack of it untill things really got bad.

Final words

GOA i hope you can fix this mess, but i really have lost all faith in you and i would rather see EA Mystic release the game 2 weeks or a month later so that they could replace GOA as this has been an absolute disaster.

[Reaver330]

Originally Posted by IainC_Goa

Alrighty let's have a little post mortem on the situation.

-cut-

Today we have a lot of people at the office who are preparing to put in a long week after a long weekend to make things right. This isn't magically going to make yesterday alright but we are very much committed to making sure that you aren't disappointed again.

Thanks for telling us stuff we knew. Now whats CURRENTLY happening?

Edit: More detail please, not "things will be up soon(tm)." I would like to know can everyone expect to get into open beta today. Show some honesty here and don't waste peoples time.

[Speeder]

If you were ready week ago you would have flow of people spreaded over 7 days not 1.

You werent prepared, thats all

[Paaaulo]

Originally Posted by IainC_Goa

If having delayed access to a beta test really drives you to such depths of anger and fury then - and there is no polite way to put this - there is something wrong with you.

i wouldnt say its the delayed access thats angered people, its the beta key stuff being put up at the last minuite then failing and that this so far seems like DAoC redux.

Also, any idea when more than one beta server will be up? and thanks for the update.

[Bant Warick]

Originally Posted by IainC_Goa

If having delayed access to a beta test really drives you to such depths of anger and fury then - and there is no polite way to put this - there is something wrong with you.

Excellent way of putting it Iain. There are too many babys here who cry just because they didn't get a lollipop. Unfortunately we just have to put up with em.
I haven't made a comment anywhere on the whole thing yet but I'm glad to see that it all works towards the big picture, the actual release!
Remember people, open beta isn't the lauch of the game, it's just a strain test!
And yesterday proved why there is an open beta!

[Ellister]

Post mortem is ok and all that... what about ETAs?

at the moment in time people have lost 1 day outta like 6.5 days (which they have paid for by preordering....)

I dont like the long week after a long weekend statement, makes me feel like no OB for 99% of people,

[Nurf]

Originally Posted by IainC_Goa

If having delayed access to a beta test really drives you to such depths of anger and fury then - and there is no polite way to put this - there is something wrong with you.

And GOA do it again...

Telling the people upset because the one thing they've been waiting for in weeks, if not months, if not even YEARS, goes wrong have something wrong with them? What gives you the right to judge people for being upset? That one statement just totally blew my respect for you.