What can’t you test in beta?

When I’m playing a new game or a new patch in a MMORPG and I find any bug, the first thing I think is, “How come they didn’t spot that in beta?” Closely followed in some cases by, “Wait a sec, I REPORTED that in beta! Why didn’t they fix it two months ago?” I think every gamer knows that nerdrage, the one that wonders what beta was for if not to find and exterminate bugs. The one that wonders why although we’re paying the live game, sometimes it feels as though we’re paying to be testers. The one that wonders why we settle for imperfect products in this arena where in many others, we’d be out with complaints.

Actually, in a lot of industries, they’d be jealous of games companies that are able to run betas with thousands of free testers. No matter how complex the software or the number of users, most situations can be tested if you have enough time and resources. But even so, some parts of the game are difficult (or impossible) to test until you go live. Here’s some of the reasons:

Player behaviour in live games is different from beta

1. The economy. You can test that everything is basically working. People can buy and sell things. You can measure how much gold (or whatever you call your currency) they are earning and how much they are spending. But players in a beta know that their characters are disposable. They don’t treat the economy the same way as they do in a live game. They just won’t put the same amount of effort into trying to game the system (or at least to work it as efficiently as possible).

2. Crafting. Same reason as above. For disposable characters, there’s a limit to how much effort people will put into crafting. Also, without having all the information available on external websites (which will no doubt be up shortly after launch), players will go through the learning process themselves in beta. After that, you can assume most people will look things up. Which is sad if you like to discover things for yourself, but that’s a part of MMORPGs which has pretty much drained away unless you are a the cutting edge. A developer could buck this trend by changing all the recipes after beta, but then you have the possibility to introduce new bugs.

3. Customer service under pressure. Your beta players will mostly be better disposed to the game than live ones, if only because they don’t want to be thrown out of beta. They won’t be pressuring GMs in game as much or as harshly. You also probably weeded out a lot of the inarticulate players when you got them to fill out a webform to get into the beta. These things will not be true in live, plus your customers will (hopefully) increase by at least one scale of magnitude. You can get round this by employing people who have had experience with larger games but it’s not something you can test in beta.

4. Griefing, exploiting, goldfarming. You can try to test for these by encouraging your beta testers to go for it. But it’s pretty much guaranteed that players in a live game will find things to do that you hadn’t thought of that are against the spirit (or the letter) of any player behaviour agreement that they signed. I’m hoping that Mythic is egging on the beta testers and encouraging griefing, so that they can figure out what players may do in live and how (or if) they want to deal with it.

5. (Balance.) I put this one in brackets because of course you can test it, but that doesn’t guarantee it’ll be the same in live. Balance issues are very different among a player base that knows all the best tactics and overpowered builds (from websites) than in a player base that is still learning them. The longer the beta, the more likely these issues are to come up then. But of course, every time the devs tweak a class or a game mechanic, the beta testers have to figure out how to abuse it all over again. It is really common for a game to have balance issues and beta testers to say “I told them about this X months ago!” Truth is, by the time a game gets to the end of beta, the devs probably aren’t looking for feedback that says “Redesign this class/game from the ground up,” even if it is good feedback.

Levelling Issues

If the testing is focussed on specific parts of the game (ie. testers are all told to go and test a particular career, or scenario, or keep siege) then it’s hard to test issues arising from a more varied player base.

1. Powerlevelling. A combination of some of the above issues. Beta testers aren’t as motivated to beat the system as live ones will be, because their characters are disposable. Also you may not have the different level ranges available to players that would make this possible. It may be that powerlevelling isn’t a gameplay issue. But probably devs want to know how fast it is theoretically possible to level, and if they’ve missed any exploits. You could certainly have a competition for beta testers to test levelling speed if it was important to test out those issues. But I don’t know if many devs do that.

2. Looking for Group functionality. This won’t be as tested during beta as it is in live because there isn’t the sheer variety of level range and possible activity (ie. levelling, PvE quests, scenarios, dungeons, etc) in the game yet. Dungeons in particular are often focus tested so players don’t need to use LFG functions to group up for them.

3. Mixed level groups. If there is a period of non-focussed beta towards the end of the process where every piece of the code is basically there and players are left to their own devices, you can test this. But it won’t be the same as in a live game where Ms Hardcoregamer wants to play with her boyfriend Mr OoShiny — or at least to help him out with some quests. Players will want to know how easy it is to group up with people of different (in some cases very different) levels and how it affects xp. Beta testers are less likely to play like that.


Some issues are just harder to test because of the increased complexity of having hundreds of thousands of players in live.

1. Complex timing issues (this is to do with the software, not to do with player timing). You can test a lot of these by limiting the number of people on live servers, and then running test servers at the same capacity. That should catch most of the really weird bugs. But occasionally 1000 players will all decide to jump up and down on one particular spot in one particular scenario while all messaging each other or some other inexplicable thing and manage to find an obscure bug. These bugs are really pernicious because they’re usually hard to replicate — for that reason, it’s not worth worrying too much about them.

2. Complex network issues. Running several dozen servers is very different from the handful that are used by beta testers. Mythic certainly have experience in running large MMORPGs, they should be ready for what WAR can throw at them, but they didn’t get that experience from the beta.

3. Players doing things you never thought of. With thousands of beta players, they will think up a lot of interesting stuff to do that devs hadn’t expected. But it is guaranteed that in the live game, someone will still think of some new bizarre thing to do. Humans are good like that.

4. Different server types. I’m not sure how much time there would be for testing different server rulesets. Usually the goal is to test out the core ruleset properly. There won’t be time to do the same level of testing on different rulesets, although anything obvious should be checked. So there may be issues with a new ruleset that no one expected or thought to test. Not really a complexity issue so much as a time and tester management one.


5 Responses

  1. Very well said. Now, if only the Doomsayers would take all of these kinds of things into account before posting “The End is freakin’ near.”

  2. Perfect! I beta tested WoW for a long time and Blizzard was constantly balancing the classes and removing content. It has to be hectic for them. Change no change back no tweak tweeak less more more less less take that out put that in no no yes no yes go gold no delay no gold ARGGG!! I wouldn’t want 1000+ people telling me how to do my life’s work while they look through a pin hole a mile away.

  3. You also can’t test how I’d respond to it, because I’m not in it. Nuts.

    Great read!

  4. @Syp – too right! I heard last night another friend had got into beta, so I’m miserable!

  5. […] examines what you can, and can’t, test in beta. Like, you can’t test how flavorlicious it […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: