How smart can a contribution measurement really be?

Since the dawn of time, humanity has been struggling to find better ways to measure stuff. The big questions people were asking back then are the same questions they ask now: when, where, how long, how far, how much, how big, how small, how fast,  and why is Bobina in the next cubicle paid more than me when I work twice as hard? Our obsession with measurements and metrics could even be one of the defining human characteristics. Without it, there could never be any progress, because if you can’t measure what  you are doing, how can you decide when it improves?

Now, speaking with my engineering hat on, I love metrics as long as they are useful. Useful to me means something I can use to help improve my team’s performance. I think the contribution measurements in WAR are great – I don’t always agree with them and there are definitely things which they don’t (and possibly can’t) measure but I like that they are trying.

Everyone loves feedback. Even if the feedback isn’t 100% positive, it’s always a good thing to get feedback on what you are doing. It’s nice not to feel ignored, and nice to feel that the game gives you a pat on the head from time to time. It is a basic human need and most MMOs do it very poorly. As a healer, you know when a heal landed and stopped someone else from dying, sure it would be nice if they realised it also but even a basic UI will show you the effect you have on health bars. As a tank, you can tell when a monster is hitting you, and see very clearly when you pulled one off someone who it wasn’t supposed to be hitting. As dps, it’s way tougher. You see health bars go down but you know there are other people doing damage too.

I think it’s quite telling that one of the most popular WoW addons was the damage meter, so that dps players could compare how much damage each of them was doing. Again, fulfils a basic human need — to answer the question, “how did I do?” and to boast about it to other people.

But performance metrics are victims of their own success. People obsess on beating the metrics, because they’re so easy to show to other people. And sometimes that means you can end up encouraging all sorts of strange emergent behaviour which are nothing to do with the results that you originally wanted to measure.

I also think that metrics can be useful in teaching new players how to play in groups. One thing WoW is often criticised for is that you can very easily solo to max level, without ever learning to play your class in a group. In a game where grouping often means using a very different set of abilities to soloing, that’s really hampering how easily new player get into the endgame. Meters which let you compare your damage/healing/damage taking to other players of the same class can give people a baseline at least. And there’s no real substitute for watching how someone else plays your class to help a new player learn about using specific abilities.

I think Mythic have made a brave effort with the contribution metrics in WAR and it’s because we like the feedback that people are going to obsess about it over the whole lifetime of the game. This issue will never die. And no one will ever be totally happy with the metrics.

What you can’t measure

You can’t measure anything intangible. You can’t measure how hard people were trying. You can’t measure who was good for morale. You can’t measure individual leadership skill. You can’t measure redundancy — eg. people who were prepared to switch role to be emergency tanks or healers but weren’t needed to do that. And it’s very hard to measure flexibility and efficiency. Human beings could measure those things but they’re also subject to extra bias.

The metrics are particularly bad at measuring people doing dull but tactically important tasks like defending an objective that never gets attacked. Or holding some AP back (ie. not spam HoTing everything in sight) in case an emergency heal is needed. To the metric, that’s the same as being afk or idle. And players are terrifically sensitive to the idea that afk players (or leechers) might get rewarded for doing nothing. This is mostly because it’s behaviour that none of us want to encourage. Scenarios are fun because they are fair. They’d get a lot less fair if we regularly had half the group afk at the start.

So what contribution measurements do now is very tough and ready. Raw damage. Raw healing. And a lot of crucial contribution is not measured and possibly never can be. However, those crucial contributions have a much bigger part to play in the eventual outcome than the meters would imply.

Metric Ideas

I’d like to see the reward for actually winning a scenario being more heavily weighted. After all, this is the actual goal of the scenario. It’s unproductive to see members of the losing team earn more renown/xp than the winning one because they scored more kills. This is a war, it’s all about winning. I’d increase the randomness on PQ rewards too, assuming a minimum level of contribution. It’s practically impossible for a healer to ever top the meter on a keep take at the moment, for example. They simply require more damage than healing, but that doesn’t mean people who play those classes should always be cut out of the rewards.

I think measurements of individual contribution are worthwhile and people love them. But we need better ways to measure contributions for different classes. Sure, we all know Bright Wizards can beat the damage meters. So maybe we should take that into account when measuring the contribution of a Shadow Warrior. No one class should be singled out for lower contributions across the board; or if they are, it should be a sign that they need a tweak. Add some contribution for buffs and debuffs.

And finally, we need good ways to dissuade people from afking or logging out of scenarios. That’s not an issue to be resolved purely by contribution meters. I don’t think it’s ever going to be easy to find a way to reward people for defending flags but the scenarios which depend on this as a strategy can be weighted to reward winning more highly and random killing away from the flag much less highly.

But at the end of the day, there’s a limit to how smart a contribution meter can be. Some classes will always do better than others because they’re just better designed at whatever the meter was trying to measure. Some people will drift to those classes because they obsess about meters. But lets not forget that its actually the result that we want to reward. Winning the scenario, playing well in teams and in realms, teaching new players how to group, rewarding good group play.

Advertisements

11 Responses

  1. Hello I’d like to comment on the healing contribution, because as far as my experience tells me you get more for doing both, out and out healing or out and out damage which isn’t that great doesn’t score nearly as much as a mixture.

    I do think its sad that people don’t get recognition for the boring work like flag guarding and I do believe a bigger bonus for winning and lower for kills/healing would help this.

  2. I’d love to see two extra numbers on the scenario table. Damage mitigated by buffs/debuffs I cast, and extra damage caused due to buffs/debuffs I cast.

    I don’t really care how much extra renown they’d give me, but knowing the numbers would at least let me know how useful the debuffs were being, and to help improve my performance in that area.

  3. I’d like one for damage taken.. so the tanks get some love.

  4. Damage taken is also on my wish-list for scenario summary..

    And of course some sort of reward for doing the scenario objectives and not just kill-count of stuff.

    Healing renown.. hum well..undecided about that. last night I played with an IB from the guild, so i backed him up and he did the dirty work. It ended in slightly more XP for me. Renown wise he gained 459 while I just got 199. Seems somewhat spooky to me.

  5. I’d love it to keep track of how many man-seconds I kept enemies rooted. Not very likely.

    Healing received would be very nice. It might remind Bright Wizards what utter healing sinks they are when they don’t bother to manage their conflagration.

  6. As a long-time healer, stats were always difficult to use effectively. Quality of heals is more important than quantity of heals. It’s also worth noting that quality in PVP is different from healing in PVE.

    In PVP, I like to break out the following (which I can’t see showing up on the leaderboard, but who knows…):
    – number of players healed;
    – mean, median and mode of healing done;
    – Deaths within 15 seconds of a heal cast;
    – overcast (heal points ‘wasted’ as the player was full health).

    There are other measures but those four tend to tell me how effective I’ve been. Right now it’s done by manual crunching of the log, but someday I hope to use a tool to do a lot of it automatically.

  7. The only way to really measure contribution fairly is to give rewards to the whole team equally. It won’t accurately reward contribution for an individual match/siege, but over a large number of events the better players will be rewarded appropriately.

  8. I agree with bigger renown bonuses for winning, as you mentioned, this should always be core to a WAR
    (hey that rhymes)

    i’d personally like to see a few other metrics included and removed ‘GASP’.

    Two things I’d like to see removed, at least from a contribution calculation perspective:
    1) kills – doesn’t mean anything at all, i’ve had situations where I’ve respawned and am running back to the
    middle of the fight, haven’t targetted anybody, and bang, i get a kill registered, I’m thinking this is more
    if anyone in your party helps with getting a kill. but not sure.
    2) solo kills – obviously its not a very ‘team’ based metric is it.
    seeing someone run off and trying to lodge a few of these on the scoreboard is just plain annoying.
    as is the person who finds and kills an enemy afk player and then boasts about it.
    For bragging rights purposes, by all means keep them on the scoreboard, but leave them out of the equation.

    I think you’re right in saying we need better ways to measure contribution
    However, I think refined focus on contribution by class might fit better.
    e.g. healing done will be the bread n butter of healers, dmg done = dps and dmg taken = tanks
    This could be refined further in scenarios to say: Weighted Average heals/dps/dmg taken.
    Example:
    If you’re a healer and pop into a scenario has plenty of healers on your team, but the other healers prefer
    healing themselves or dpsing, whist you’re playing the team game, well then your weighted average heals done
    is higher, and therefore you should be awarded a bonus to renown for it.
    same with dps, and damage taken, it promotes everyone to continually compete to contribute…
    I think this is a fairer mechanic, and not too difficult to implement.

    As for AFKers, i like the current mechanic Mythic has installed kickin ppl who are in the idle zone
    more than 2 minutes, but I think they could expand on that a bit more.
    e.g.
    1) If you are kicked for being idle, a penalty reduction of exp and renown can be applied to the character
    and you cannot rejoin another scenario for 30mins

    2) In flag defending scenario’s Mythic could implement random spawn boxes or so that the player has to
    continually interact with, therefore showing the person is actively defending (i.e. scouting around the
    defensive zone) rather than running to the flag and afking there. Also for picking up these boxes, much like
    PQ’s , the player will be awarded contribution points. So in a game where it is heavily skewed in your favour
    and your entire team is on offense and camping their starting zone, you still get rewarded for doing the right
    thing.

    3)You get bonus to renown/exp/gold earned during fights/kills for staying within certain distance of your group.
    Plain and Simple, you get rewarded for working as a group?

    4) exponentially increasing Bonus modifier to renown/xp for those below the level cap:
    This is more to reward those rare skilled players who stand their ground against the scenario capped players
    Although you get a bolster to your stats, the bolster doesn’t cover you to the level cap, 11, 21, 31, etc.
    and the bolster doesn’t factor in that in most cases, level capped players will have better gear/stats.

    Note: This is only scenario specific, as far as I can see, PQ’s contribution balancing would be a lot harder IMO.

  9. I’d like to see rewards for actually completeing the scenario objectives!

    People won’t carry and capture a flag unless they get something for it so encourage them to complete the thing correctly by offering rewards.

    Flag captures, time carrying the “thing” all these sorts of things should be extremely important. Scenarios are not just about killing people (though that is very fun).

    People need a nudge in the right direction to complete a scenario the way it was intended and you always do that with some kind of reward.

    As for damage meters etc yes they are helpful but don’t get hung up on them. As I ranted to some people on TS yesterday. Some people do lots of damage because they spec that way. Others do little damage but their damage is important where you need it most but they pay the price by dying a lot.

    Is one worth more than the other? No I don’t think so.

  10. I’ve got a strange idea, bear with me here.

    All players on a team earn equal renown if they have contributed enough, this is worked out by them by passing a low limit on 2 of damage/actual healing/damage mitigated/scenario points. Players in the top 3 of highest damage/actual healing/damage mitigated/scenario points contribution get a 10/5/2 %bonus. Xp and Rp are given upon completion of the scenario.
    Scenario points are only generated from objectives. Experience and renown are generated from doing things associated with the objectives and for killing players (heavily weighted towards objectives). The winning team gets renown/xp based on the number of scenario points they got, 0 = 20% of xp/rp with a sliding scale up to 500 = 100%. The losing team gets a % of the winning teams renown based on the amount of time the scenario has lasted (based on a exponential scale, the longer you stay in the bigger the percentage you get going from 0% to 50%) or based on scenario points (going from 25% to 80%) which ever is more.

    Winning team who turtles will get way less xp/rp.
    Losing team is encouraged to keep going for scenario points and holding the enemy off.
    Losing team is still rewarded if they are rolled.
    Afkers who haven’t contributed enough get nothing, rewards are much lower if your team loses fast.

    I’m sure there are problems with this but i think it would lead to a better dynamic in scenarios. To make world RvR attractive you need to up the drops from looting player corpses, make the xp/rp from killing a player be better than going into a scenario, shove monsters in the RvR areas that give good rewards (like the halloween event).

  11. @Torvik Trouble with individually rewarding people for e.g. carrying the flag is you get an unseemly scrum over who actually picks it up, a wildly inappropriate low-HP cloth-type grabs it, starts running, a healer decides not to bother keeping them alive but waits for them to die then grabs it, etc. As Robert Austin said, “…people have brains and are endlessly creative when it comes to improving their personal well-being at everyone else’s expense”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: