How smart can a contribution measurement really be?

Since the dawn of time, humanity has been struggling to find better ways to measure stuff. The big questions people were asking back then are the same questions they ask now: when, where, how long, how far, how much, how big, how small, how fast,  and why is Bobina in the next cubicle paid more than me when I work twice as hard? Our obsession with measurements and metrics could even be one of the defining human characteristics. Without it, there could never be any progress, because if you can’t measure what  you are doing, how can you decide when it improves?

Now, speaking with my engineering hat on, I love metrics as long as they are useful. Useful to me means something I can use to help improve my team’s performance. I think the contribution measurements in WAR are great – I don’t always agree with them and there are definitely things which they don’t (and possibly can’t) measure but I like that they are trying.

Everyone loves feedback. Even if the feedback isn’t 100% positive, it’s always a good thing to get feedback on what you are doing. It’s nice not to feel ignored, and nice to feel that the game gives you a pat on the head from time to time. It is a basic human need and most MMOs do it very poorly. As a healer, you know when a heal landed and stopped someone else from dying, sure it would be nice if they realised it also but even a basic UI will show you the effect you have on health bars. As a tank, you can tell when a monster is hitting you, and see very clearly when you pulled one off someone who it wasn’t supposed to be hitting. As dps, it’s way tougher. You see health bars go down but you know there are other people doing damage too.

I think it’s quite telling that one of the most popular WoW addons was the damage meter, so that dps players could compare how much damage each of them was doing. Again, fulfils a basic human need — to answer the question, “how did I do?” and to boast about it to other people.

But performance metrics are victims of their own success. People obsess on beating the metrics, because they’re so easy to show to other people. And sometimes that means you can end up encouraging all sorts of strange emergent behaviour which are nothing to do with the results that you originally wanted to measure.

I also think that metrics can be useful in teaching new players how to play in groups. One thing WoW is often criticised for is that you can very easily solo to max level, without ever learning to play your class in a group. In a game where grouping often means using a very different set of abilities to soloing, that’s really hampering how easily new player get into the endgame. Meters which let you compare your damage/healing/damage taking to other players of the same class can give people a baseline at least. And there’s no real substitute for watching how someone else plays your class to help a new player learn about using specific abilities.

I think Mythic have made a brave effort with the contribution metrics in WAR and it’s because we like the feedback that people are going to obsess about it over the whole lifetime of the game. This issue will never die. And no one will ever be totally happy with the metrics.

What you can’t measure

You can’t measure anything intangible. You can’t measure how hard people were trying. You can’t measure who was good for morale. You can’t measure individual leadership skill. You can’t measure redundancy — eg. people who were prepared to switch role to be emergency tanks or healers but weren’t needed to do that. And it’s very hard to measure flexibility and efficiency. Human beings could measure those things but they’re also subject to extra bias.

The metrics are particularly bad at measuring people doing dull but tactically important tasks like defending an objective that never gets attacked. Or holding some AP back (ie. not spam HoTing everything in sight) in case an emergency heal is needed. To the metric, that’s the same as being afk or idle. And players are terrifically sensitive to the idea that afk players (or leechers) might get rewarded for doing nothing. This is mostly because it’s behaviour that none of us want to encourage. Scenarios are fun because they are fair. They’d get a lot less fair if we regularly had half the group afk at the start.

So what contribution measurements do now is very tough and ready. Raw damage. Raw healing. And a lot of crucial contribution is not measured and possibly never can be. However, those crucial contributions have a much bigger part to play in the eventual outcome than the meters would imply.

Metric Ideas

I’d like to see the reward for actually winning a scenario being more heavily weighted. After all, this is the actual goal of the scenario. It’s unproductive to see members of the losing team earn more renown/xp than the winning one because they scored more kills. This is a war, it’s all about winning. I’d increase the randomness on PQ rewards too, assuming a minimum level of contribution. It’s practically impossible for a healer to ever top the meter on a keep take at the moment, for example. They simply require more damage than healing, but that doesn’t mean people who play those classes should always be cut out of the rewards.

I think measurements of individual contribution are worthwhile and people love them. But we need better ways to measure contributions for different classes. Sure, we all know Bright Wizards can beat the damage meters. So maybe we should take that into account when measuring the contribution of a Shadow Warrior. No one class should be singled out for lower contributions across the board; or if they are, it should be a sign that they need a tweak. Add some contribution for buffs and debuffs.

And finally, we need good ways to dissuade people from afking or logging out of scenarios. That’s not an issue to be resolved purely by contribution meters. I don’t think it’s ever going to be easy to find a way to reward people for defending flags but the scenarios which depend on this as a strategy can be weighted to reward winning more highly and random killing away from the flag much less highly.

But at the end of the day, there’s a limit to how smart a contribution meter can be. Some classes will always do better than others because they’re just better designed at whatever the meter was trying to measure. Some people will drift to those classes because they obsess about meters. But lets not forget that its actually the result that we want to reward. Winning the scenario, playing well in teams and in realms, teaching new players how to group, rewarding good group play.

Links labours lost

The Economist reckons that Google’s Lively was a flop. I know I haven’t been back to it after the first excitement of playing with virtual lava lamps, there are easier ways to chat. Bruce Everiss also comments on this and discusses other ways Google could extend its business model.

The sixth guild highlights is now up on the GOA site and discusses keep taking, it also highlights the Humberton Blackguard, another guild we know of from our Dark Age of Camelot time.

It may not have escaped your notice this week that there’s some small controversy about Mythic not crediting developers that no longer work for them. The complaint was made anonymously by an ex-Mythic developer that had put in some work on Warhammer Online. The International Games Developers Association call this ‘disrespectful’. Mark Jacobs spoke to MTV Multiplayer and had the following to say:

If you really think that we’re doing something wrong, at least have the balls to stand up and go ‘Hi, my name is so-and-so…[Leaving the person out of the credits] could be a mistake. I’m not saying it’s not happening. I just don’t know who the heck this person is. So come out, stop hiding behind the anonymity of the internet and the legal shield of ‘I’m going to sue EA.’

Scott over at the Broken Toys blog adds that it’s not him. Apparently he’s been asked a lot!

Kotaku points out some of the vagaries of multi-million dollar AI systems. If you happen to be shopping in a Target (US store) and you wanted to buy World of Warcraft but find they’ve sold out, you’ll apparently be pointed to Paws & Claws Pet Vet as an alternative!

Did you miss the Spore links? We did, here’s some more news about it. Spore is out for the iPod.

New iPhone comes loaded with pictures of the girl who made it. Who knew that working on a factory line could be such fun?

Blogger xntrek writes an open letter to Mythic from Gamenesty International about the end of the closed beta.

Dev blogs – Justin Webb has posted a bunch of vids from Leipzig on his blog. Go watch them; WAR tattoos, Leipzig ladies, T-shirt delivery (did I mention I love free T-shirts!?) and Warhammer Online winning the Best Online Game title. Iain C also posts about Leipzig and mentions T -shirts all too teasingly – check out his pics from the event – they’re really good. And Josh says something about PAX, something about it being the last time WAR gets shown off before the launch.. hrrm, forgetting Games Day UK, obviously but I’ll forgive him. Oh look, Jeff Hickman said the same at the start of his exclusive PAX blogs for Ten Ton Hammer. As if it’s not bad enough that the CE headstart starts during Games Day UK!

Massively also have been talking to Mythic at PAX about the content of the next patch with lots of new features for the open beta (auction houses, graphical improvements, pathing fixes etc). They’re planning to breadcrumb their interview over several articles as per usual so keep checking back.

Mark Jacobs stops by Warhammer Alliance to give an update on the work they’re doing post-preview weekend.

Steve Jobs is still alive (cue song) despite Bloomberg publishing his obituary. Oops.

And finally, a last minute welcome to Warhammer Tank, a blog devoted to.. well, you guess!