The WAH, 6/15/09 - "This has been a test of the WAR: Age of Reckoning System. . . ."

[Anglakhel]

Lately, the Public Test Server, or PTS, has been getting a lot of publicity. Testing has been open for Patch 1.3, featuring the Land of the Dead, for several weeks and just closed earlier today in anticipation of tomorrow's release of the patch to the live servers.

We have been told many times by Mythic that the PTS is one of the best ways to share feedback with Mythic and the development team. Lately, Mythic has been hard at work trying to encourage use of the PTS and is even offering incentives, such as special ingame items, to players that spend time on the PTS and supply constructive feedback. Still, it can often feel like time spent on the PTS has no positive impact on what we see on the live servers. Testers find themselves wondering if their time spent on the PTS is meaningful, relevant and helpful.

We can look back over the course of WAR's release and even into Beta to see how Mythic has responded to the feedback of their testers, but we have several excellent examples from the testing of Patch 1.3 on the PTS. Just days ago, Mythic announced that due to player feedback from the PTS, they would be adjusting the Realm Resource Quest that will determine who controls the Land of the Dead. According to the announcement, they will be dropping the required resources to gain control of LotD from 10,000 to 5,000. Apparently, this feedback followed several PTS Events advertised on the Herald, including “The Sands Run Red” on the 9th and 10th, which was intended to test out the Land of the Dead content and gather specific data on the Resource Expedition Quest.

Mythic has often scheduled testing events for the PTS to try to generate the best feedback possible. "The Sands Run Red" is merely the latest PTS event in course of WAR's release. However, Mythic included the following note in the Herald post about the PTS event that I find extremely interesting:

Please note that this is a hands-off event. Mythic staff will be observing the natural progression of the mechanics as both sides battle it out for supremacy. So remember to build up resources by capturing Tier 4 zones, looting your enemies, and returning the resources to your Expedition Quartermaster near your Flight Master in the capital city.

Immediately, we see that Mythic seems to be attempting to gather data about how the Resource Expedition Quest will run on live servers. They want to be "hands-off" so that they can "observ[e] the natural progression of the mechanics." After the test was complete and they started evaluating feedback, it became apparent that the Resource Expedition Quest was too slow and needed to be more responsive to player actions. So, Mythic used the PTS feedback to alter the system and cut the needed resources in half. At first glance, this all looks very encouraging. We see Mythic engaging with their players and testers and taking the initiative to make system changes to improve the game.

Except for one glaring problem:

If you logged into "The Sands Run Red" event, you couldn't help but notice that the Resource Expedition Quest didn’t really do anything on either day of testing.

I’m tempted to say that the system was broken, but that wouldn't be fair since we don't really know if that is true. The simple fact is that the Resource Expedition Quest was never actually tested in a way that even comes close to simulating the live servers.

If you logged onto the PTS on either 6/9 or 6/10 you were treated to a test in paralysis. If your side already controlled the LotD, you could fly there and test out the PvE content, but if your side didn't have control, nothing you could do was going to have any bearing on the test. Order controlled the LotD for almost the entire duration of testing on 6/9 while the Expedition Resource meter went nowhere. If you logged in the next day, Destruction controlled the LotD while Order sat helplessly by, unable to affect control of the LotD.

Many of you are probably thinking that Mythic realized the problem and already adjusted the system to fix it by cutting the needed resources in half, but this fix doesn’t address the real problem. The real problem is that the set up of the PTS for this event in no way represented the dynamics of the live servers.

The problem is that adjustments are being made to the system when the system hasn't been tested. We never saw whether or not the system would flip control of the Land of the Dead in a reasonable time frame. What we saw was a "hands-off" PTS event that generated frustration in large segments of the testing population and seemed to stray far from the stated intent of the test.

Expedition Resources come from two primary sources according to the plan. First, you earn resources from killing enemy players and looting resources from their bodies. Then you can turn those resources in at a special NPC near the flightmasters in the Inevitable City and Altdorf. Secondly, resources accumulate due to the advancement of the RvR Campaign. Controlling territory and locking zones will earn Expedition Resources and increase the speed at which your faction regains control of the Land of the Dead.

The PTS, however, was completely unequipped and unprepared to test out these two dimensions of the Expedition Resources Quest. Player Kills were almost impossible. Once one side had control of the Land of the Dead, they all abandoned the RvR lakes to explore the news lands. The consequence of this was that the faction without LotD control didn’t have any enemies to kill and the Expedition Resources Quest failed to advance in any meaningful way.

Without player kills, the Expedition Resource Quest depends upon zone capture to generate resources. However, the PTS didn’t have an established group of guilds with the Rank, Resources and Standard Bearers to claim Keeps and lock down zones. If you logged in to the event and your side was locked out of the LotD, chances are your side already controlled all the Keeps and BOs accessible, but the campaign sat paralyzed due to Keeps being unclaimed. So, there was no way for us to see how quickly points are generated by the Expedition Resource Quest since the PTS wasn't prepared with guilds to properly test zone capture mechanics.

The root of the problem is not necessarily the Expedition Resource Quest. It may or may not have major issues once it is pushed to Live Servers. The problem is the mismanagement of the PTS. The problem is setting up a “hands-off” test when the appropriate preparations have not been made to attempt to simulate the conditions of the live servers.

This is a complex problem, but it isn’t a new one. We’ve seen other recent changes based upon the PTS. Just prior to this most recent announcement, we saw Mythic pull the 1.3 C&C changes from the patch and push them back to some future date. Again, we are told that feedback on the forums and through the in game surveys led to the decision, but how useful can that information be when it is being generated on the PTS which poorly simulates the live servers? If you logged into the PTS to test out the C&C changes, chances are you never saw another player unless you traveled to the Chaos Wastes where players stood near the warcamps trying to arrange duels. Most of the testing information gathered was based upon players engaging in 1v1 Duels or, if no players were available, attacking keep doors and mobs.

Granted, duels and attacking mobs can give some baseline information about how abilities, stats, and other game mechanics work. However, they are extremely poor at simulating the complex dynamics of RvR combat. These tests fail to take into account the complex interaction between the Careers and explore their synergies and dependancies. Think about this for a minute. A massive, balance altering patch was being evaluated based upon how hard people were hitting keep doors or how hard they were hitting each other in completely artificial duel situations. How does that meaningfully contribute to balance in a complex RvR game with four distinct Archetypes and twenty-four Careers? Yet we’ve seen the C&C elements of the Patch delayed because of the feedback generated on the PTS.

I'm not saying that the PTS isn't useful. The PTS is potentially an extremely powerful tool, but it requires that it be utilized effectively. One of the problems with the PTS is a lack of interest in players to become testers. The low population on the PTS simply can't be expected to accurately simulate the live servers with hundreds of players interacting with one another. Planned events and incentives help to boost the population, but increased population alone won't improve the PTS. Just packing more players onto the PTS won’t generate relevant and quality feedback. Putting more players on the PTS can actually hurt the game more than it helps. A poorly coordinated PTS event fails to generate meaningful feedback and can create false and often negative impressions in participants.

The Sands Run Red was a nightmare test, but it didn't have to be. Getting a little "hands-on" and directing the testing would have helped to better simulate the live servers and helped to improve the quality of the feedback on the Resource Expedition Quest. A simple step like creating several testing guilds, so that the RvR Campaign could be pushed, would have enabled a far more accurate test of the Resource Expedition Quest. At the very least, Mythic could have gotten valuable test data about what would happen with the quest if one faction abandons the campaign and heads to the Land of the Dead. We didn’t even get that though. We simply saw inaction. Players that wanted to be testers were extremely frustrated. The PTS was causing more harm than good. It wasn’t gathering relevant data about the control mechanisms for Land of the Dead and it was creating bad opinions in players, which translate to poor word of mouth and decreased morale.

What is most frustrating perhaps is that Mythic had to know this would be a problem. They had hosted another PTS event to test the Expedition Resource Quest a week earlier and saw the same problems with a campaign unable to be pushed. That test generated a lot of feedback about the problems with the Expedition Resource Quest and an inability to accurately test it on the PTS. Yet, they pushed ahead with a "hands-off" test of the system without making preparations to ensure that the new test would better simulate the live servers and meaningfully test the Resource Expedition Quest.

The PTS is a unique and valuable tool. It's a chance for Mythic to share ideas with the players and get feedback. It's an opportunity to find issues before they make it to the live servers. However, the PTS will do more harm than good if Mythic doesn't get better control over this resource. They need to better plan testing events and focus on systems they need to collect data on. They need to realize the significant differences between the live servers and the PTS and intelligently mold the PTS to better simulate the live servers. Most importantly, Mythic needs to make changes based upon how these new systems and mechanic adjustments will affect the live server in complex situations. If they don’t, we’ll continue to see changes made to the game that haven’t been adequately tested and we’ll continue feeling like we’re paying to beta test on live servers.

This has been a test, and only a test, of the Warhammer: Age of Reckoning System.

We now return you to your regularly scheduled programming. . . .

[Mightythews]

Blirst!

...dammit.

[Ulthien]

so basically, whereas Mythic did preempt the imbalance issues with DAoC, creating there 3 factions - they neglected flip-flop binary faction issues here on their newest brainchild.

The side that is overpowered and rules the server, will thanks to LotD become even more powerful, dislodging the weaker side from the server. Heightening the difference gap and undermining the will to continue as an underdog.

In end effect, killing the game.

[Shadowkin]

I have to say though, getting everyone to download that 900mb patch and hop on the PTS would be rather difficult - I did, and I was well pleased to get my kinky shroud bodywrap, but unless there is a SIGNIFICANT in the actual incentive to hop on the PTS, you'll never get the numbers there that will reflect Live servers.

Not everyone is as happy as I am to simply get little visual cookies.

[wpc56]

There has never been ANY reason to test for them. They FIRED their Q.A. departments, let them reap the consequences.

The number 1 complain atm is bugs, bugs, bugs with classes, domination, token, getting stuck and the GCD exploit a few weeks back.

Tons of people quit the game because of that and skull throne, what used to be top 3 populated pve server, is now a ghost town.

Hey, I'm just glad dest are zerging on ST atm... like anyone give a crap about the less populated side.

[Herzog]

I dont think that one can simulate a live environment on the PTS at all. Its way to tricky and requires to many dev ressources to even come close to. People concentrate on, and flock to, the new additions on the testserver.

This completely leaves out the dynamics of the rvr-campaign on live.
After the shine has worn off not everyone will be in the lotd, even if his faction has access. Some will stay and do rvr, and you only need one or two organized 6mans to slow down the enemies domination plan.

I would rather like to see Mythic push small adjusts to the game onto the live servers more often. Watch their impact on the live environment, and than finetune them.

While one could say this turns paying customers into beta-testers, even those that dont want, i.e. never set foot on the pts. Id feel better with testing/playing small changes that dont flip the whole game upside down, than having to chew through massive, untested chunks off content/balance changes that were optimized on the pts "only" (see aoe-fail).

[Aralaz]

One option is to clone an entire live server over to the PTS each time it opens, so that all existing guilds etc are available. This worked well on Tabula Rasa where there was never more than 4 servers and it was easy to rotate them.

To make things fair you would then open up transfers to the PTS clone server as normal (this is the step TR missed). At least then there would be a starting set of guilds already established. Also between test events the base server could be rotated just to be fair.

Another option is to do what they always said they wouldn't, and that is to open up an always-on test server like they had in DAoC.

I honestly think if they had created one from the start many Elder testers would have flocked to it, along with newcomers interested in helping the game. (Yes, I realize there are also problems inherent in this system, but I'm just trying to provide alternatives to the mess we have now).

[Kingdom]

would be nice if we also got the option to make level 40's characters on the test server.

[Shakrah]

Thank you for the feedback, informative and well documented.

[stjobe]

Originally Posted by Herzog

While one could say this turns paying customers into beta-testers

You cannot turn someone into what they already are.

This game has been beta-quality since release, and it doesn't seem to change in any significant degree with patching.

* Bugs, bugs and more bugs, small and large, irritating and gamebreaking, subtle and glaring. Bugs.
* Class imbalances. I don't think I need say more than that.
* Endgame unplayable - city stage 2 is (allegedly, never been there myself) not completable without CSR intervention.
* RvR reward system that encourages NOT participating in the campaign (i.e. warcamp RP leechers).
* RvR locked down in T4 for 3 hours when a city is attacked - if you can't get in you can't do RvR. Not a good strategy to get people to stay logged on.
* Population imbalances that severely effect the endgame.
* Unsupervised transfers that aggravate population imbalances.
* And much, much more.

Now, I must state with emphasis that I simply love what this game has the potential to be, but sadly Mythic doesn't seem up to the task to get us there. The all-or-nothing approach to class balance, the pushing of more content before the core game is playable, the band-aid fixes for server performance, all paint a picture of a company that has less than a good idea of what they're doing, and no firm direction.

Then the PTS. An exercise in futility where massive amounts of feedback from players have absolutely no bearing whatsoever on whether a feature is included in the live game or not. Where the test methodology is so flawed it's laughable. It barely does service as a "sneak preview" on upcoming content.

And to top things off, they've got a history of not communicating with or listening to us, the players (made even worse for us EU players with the glaring incompetence of the joke that is GOA). We play this game for hours on end, we know what works and what doesn't work. I'm not saying they should let the players decide everything, I'm just saying that they should make a point of listening to us.

</rant>

tl;dr version:
We're already beta testers and PTS testing is a joke. This could have been a good game.

[akalukz]

Excellent post. The most difficult part of the PTS for me is boredom. There is rarely anything going on. And if the PTS was any indication as to what LoTD will do to live servers...then we all might as well prepare for 3 months of RvE.

[PhoenixRed]

Originally Posted by stjobe

Then the PTS. An exercise in futility where massive amounts of feedback from players have absolutely no bearing whatsoever on whether a feature is included in the live game or not. Where the test methodology is so flawed it's laughable. It barely does service as a "sneak preview" on upcoming content.

While PTS testing could be better, to say they aren't bothering to listen to feedback is to give them less credit than they deserve. Expedition Resource requirements were too high and they were changed. Zone flipping was a problem and it was (mostly) fixed. And the crown jewel, combat and careers changes were a huge source of controversy and they have been pushed back.

The "we shouldn't be paying to beta test" argument is only credible to a certain point. MMOs are never finished and can never be released in a "finished" state. Regardless of whether or not you want to admit it, MMO players are always testing on live content, because there are always bugs, there are always problems, and there are always things that need fixing. What's on the developer's burden is A)Making sure major problems don't create the perception of testing content for them and B)Fixing the issues or at least presenting proof of progress. This is the area in which Mythic could probably do better, in some cases much better (see: exploit WAH from last week), but to give them zero credit or claim they don't at least try to listen to player feedback is a huge, erroneous, assumption.

In fact in some cases, listening to the players has been a problem with Mythic. It's arguable that the overcompensation of too many servers at launch was the clear result of an immense amount of players complaining about queues for open beta. "The customer is always right" is not meant to be literal.

In short, a balance needs to be achieved. Is Mythic there yet? I'd say no, and it's hurt their game population. But it isn't for lack of trying.

[quik77]

I always figured if they truely wanted to test on the PTS, they'd have rotating squads of devs playing their characters (you can rename when copied from live!) on both sides and with organized guilds at the very least, or template

In LotD I encountered exactly 3 destro ever. Hell a hall with in game training dummies (computer controlled careers to fight against) would give more data.

Most PTS are a ghost town, no working guilds, no multiple guilds, hell you'd think they'd at least turn up the rate of zone locks to 500% faster or something for a test.

PTS feels like a dead zone from AoC or leveling an alt solo in WOW when everyone else is max level.

For real info I'd do one or more of the following:

1. C&C changes
   1. Testing method: 1000% inc in xp rate, all devs and players on pts level from level 1 to 40 with a base population on both sides at all times (base pop = each dev takes an hour or more to play their game, in an unfamiliar career or side during a set time period with other devs rotating. If theres more of one side than another they roll the lower side.) No templates all chars start at 1.
   2. Each Dev blogs/twitters/logs in someway the things that need to change and the things that are good as they do this (gives you data on, we like this add more, and we hate this change it)
   3. All players follow the same method.
   4. All Devs are requested to not play their live chars and instead play on PTS during PTS periods (if they don't want to cause its dead, its dead cause they aren't on, the biggest draw for outside testers would be knowing the entire mythic dev team is rotating through the PTS)
   5. Reasons: comprehensives/regression tests of all changes, familiarize the devs with the careers they currently don't touch and/or know, and make sure the C&C changes actualy work as intended to make the game fun from 1-40 and 1-80
   6. Organized in game events while this is happening, each team lead is responsible for a different nights event, and making sure at least some devs show up (devs show up players show up), even if its just t1 rvr ffa
2. RVR changes
   1. Game is locked down to just one realm and teir at a time, and is fought to conclusion, to get out all the bugs on a zone by zone basis once and for all (in live theres still BO's I've never seen and keeps that bug out), If anything is going haywire immediate notification to all.. and synopsis afterward.
3. Endgame changes
   1. Game locked down to t4 + forts + cities only
   2. speed up timers on ZD.
   3. Devs get to lead the WB's (yes you the devs, you keep telling us L2P so time to show what you know), and each WB = own test guild so you can get the banners going similar to live.
   4. push each zone and fort and each city on both offensive and defensive at least once.

This is just some of what occurs to me off the top of my head.

and the main points being that, if players know the devs are their playing.. even if they are anonmyous, it should up the number of players participating. THis game does not work at all low population, especially with all the nonscaling mechanics, and in order to test Must have at least med/med numbers

I come from the extreme programming/agile dev + dogfooding type of viewpoint for this... but I have no idea what their actual dev method is.

[Kheradruakh]

They should introduce content like this on one live server. C&C needs to be tested but just a pure content patch could be introduced with a rollback server if things get way out of hand. No one would really be forced to go to the new zone but they could if they wanted to. One server might get shinies ahead of everyone else but that might not be a horrible thing.

[quik77]

I keep having the same feeling when playing this game on PTS

"This is it? I expected more of you! You should of done better."