Jump to content


Replay analysis - what does this tell us about the MM?

Lets do science statistics

  • Please log in to reply
228 replies to this topic

Baldrickk #221 Posted 17 July 2018 - 10:27 AM

    Field Marshal

  • Player
  • 29322 battles
  • 13,387
  • [-TAH-] -TAH-
  • Member since:
    03-03-2013

View PostZhul87, on 17 July 2018 - 08:28 AM, said:

 

All matches after 3-5-7 introduction.

Streaks:

 

10.07 - 21 matches played, which had:

 

Five consecutive losses(10.07)

 

Five consecutive wins(10.07)

 

Six consecutive losses(10.07)

 

11.07 - 15 matches played, which had:

 

Nine consecutive wins(11.07)

 

To put those streaks into perspective.

 

Chances for having ONE streak of 9 out of 15 is 0.7%.

Chances for having ONE streak of 5 out of 21 is 26,2%.

Chances for having THREE streaks of at least 5, in just 21 matches is probably close to what a streak of 15 out of 21 would be and thats 0.01%.

Chances for having 5 wins followed by 5 losses or vice versa, out of 10 matches should be a shy 0.2%, if I am not mistaken.

Thanks for that.

You've exposed a couple of issues that need to be fixed (unknown results shouldn't show on the result distribution, and it shouldn't try to draw the graph with no results)

 



Simeon85 #222 Posted 17 July 2018 - 02:30 PM

    Captain

  • Player
  • 1097 battles
  • 2,103
  • Member since:
    04-19-2013

View Postm1x_angelico, on 13 July 2018 - 07:00 PM, said:

Why is SBMM unfair? 

 

Because it fundamentally will treat each player differently, handicapping them or boosting them on their performance. 

 

Your football analogy has no relevance because it's a completely different game and you are using an example of professional team sport to compare it to a computer game full of random players who effectively play on their own and are not paid.

 

Also as Jabster pointed out, many cup competitions can drawn complete mismatched teams together and no one complains about that. To play in the English FA cup you basically have to enter and be FA affiliated, technically the most amateur of sides can end up playing vastly better professional units. But it is still a 'fair' competition because it basically treats all the teams the same, they all play on the same pitches, with the same rules and go into the same draws to face their opponents. Of course better teams generally come in later in the competition but it doesn't stop some successful semi-professional team potentially playing Man Utd. 

 

Fairness is not the same as balanced. 

 

A fair MM system is about giving people the same opportunities to do well and win games, that is what we currently have. 

 

Any SBMM system, even leagues is unfair, and they are generally only tolerated if the rewards for playing the high leagues are huge, which is a concept unlikely to be applied to randoms (it is why we have ranked battles). 



Baldrickk #223 Posted 17 July 2018 - 04:56 PM

    Field Marshal

  • Player
  • 29322 battles
  • 13,387
  • [-TAH-] -TAH-
  • Member since:
    03-03-2013

View PostSimeon85, on 17 July 2018 - 02:30 PM, said:

View Postm1x_angelico, on 13 July 2018 - 07:00 PM, said:

Why is SBMM unfair? 

Because it fundamentally will treat each player differently, handicapping them or boosting them on their performance. 

 

Your football analogy has no relevance because it's a completely different game and you are using an example of professional team sport to compare it to a computer game full of random players who effectively play on their own and are not paid.

 

Also as Jabster pointed out, many cup competitions can drawn complete mismatched teams together and no one complains about that. To play in the English FA cup you basically have to enter and be FA affiliated, technically the most amateur of sides can end up playing vastly better professional units. But it is still a 'fair' competition because it basically treats all the teams the same, they all play on the same pitches, with the same rules and go into the same draws to face their opponents. Of course better teams generally come in later in the competition but it doesn't stop some successful semi-professional team potentially playing Man Utd. 

 

Fairness is not the same as balanced. 

 

A fair MM system is about giving people the same opportunities to do well and win games, that is what we currently have. 

 

Any SBMM system, even leagues is unfair, and they are generally only tolerated if the rewards for playing the high leagues are huge, which is a concept unlikely to be applied to randoms (it is why we have ranked battles). 

 

To add to this:

 

In the graphs seen on the OP, and in all the others people have generated, within a certain amount of error, the MM does, as expected provide teammates for everyone that average out at the same level as the opposition team members.

 

It's then down to each individual player how well they do from there.  The better they are, the more of those battles they will end up winning.

 

A SBMM's role is to provide stronger or weaker teams so that after your skill is taken into account, your team is balanced with the enemy team.

This means that for a good player, they will always end up with weaker team-mates on their team, while poor players get stronger team-mates.

 

Its not included in the OP as it was just an experiment during development, but if we were to include LordMuffin in his team for the ratings, his team comes out roughly 3% stronger.

That's the effect of his skill on the team.

As noted previously, platoons account for the ~2% stronger teams already, so a SBMM would make LordMuffin's team, on average, about 5% weaker than the enemy.

That 2% equates to a stronger team in about 5.5% of battles before taking Muffin into account, and he adds another 4.5% to that (diminishing returns as you get better due to normal distribution)

 

To view it niavely, the MM would stack an equal number of games (10%) against him to even it out (as opposed to being balanced)

More realisticly, every battle he goes into, the enemy team will be ~5% stronger than his team just to cancel him out.

 

Considering that the most common SBMM advocate whines because "I always get matched with the worst teams" (when all evidence so far points towards them getting the same as anyone else - though only one has reported back after running the tool) it is incredibly selfish to want other players to have to shoulder that burden every battle just so they can get 50% WR with no effort.

 



CmdRatScabies #224 Posted 17 July 2018 - 08:06 PM

    Colonel

  • Player
  • 35441 battles
  • 3,662
  • [-MM] -MM
  • Member since:
    10-12-2015
How does the analysis handle incomplete replays?  If it excludes them have you determined that it doesn't bias the results or have you compensated for this?

Baldrickk #225 Posted 17 July 2018 - 09:24 PM

    Field Marshal

  • Player
  • 29322 battles
  • 13,387
  • [-TAH-] -TAH-
  • Member since:
    03-03-2013

View PostCmdRatScabies, on 17 July 2018 - 08:06 PM, said:

How does the analysis handle incomplete replays?  If it excludes them have you determined that it doesn't bias the results or have you compensated for this?

I can't get the results of the battles that were incomplete,  but the players are still listed. 

That means that it can still lookup their stats.

 

On the graphs,  these are shown as the grey dots.


Edited by Baldrickk, 17 July 2018 - 09:25 PM.


CmdRatScabies #226 Posted 17 July 2018 - 10:15 PM

    Colonel

  • Player
  • 35441 battles
  • 3,662
  • [-MM] -MM
  • Member since:
    10-12-2015

View PostBaldrickk, on 17 July 2018 - 09:24 PM, said:

I can't get the results of the battles that were incomplete,  but the players are still listed.

That means that it can still lookup their stats.

 

On the graphs,  these are shown as the grey dots.

 

Okay, I see that the battle result doesn't come into what you are doing.

Baldrickk #227 Posted 17 July 2018 - 11:46 PM

    Field Marshal

  • Player
  • 29322 battles
  • 13,387
  • [-TAH-] -TAH-
  • Member since:
    03-03-2013

New update!  https://github.com/B...releases/latest

 

View Postjabster, on 14 July 2018 - 01:36 PM, said:

 

Have you got the raw output from analysis of Lord Muffins replays as I’m interested in looking at the randomness of teams and also results?

 

Still not yet.  Next update.  I wanted to finish the graph saving and correctness (unfinished replays showing in the results distribution graph (now fixed)

 

View PostOreH75, on 11 July 2018 - 10:47 PM, said:

 

Great, btw suggestion for your analyzer: add option to automatically store all .png files in the replay directory? Saves about 30 mouse clicks :P

 

 

The tool now has the option to automatically save graphs - though due to how the library works, markers in the graphs (e.g. points in the scatter graphs) are larger when automatically saved.

Along with that, it also has the option to not show the graphs as they are generated, if you just want to generate the graphs as images.

I've also fixed the score histogram to not include battles without results, as that was inflating the left-most bar in the histogram above where it should be. (as seen in Zhul87's results)

As an example, here are my results for my last 4070 battles (all the battles with the new MM):

Block Quote

 Total replays:
            4070
Green team average rating:
            4455.65
Red team average rating:
            4401.52
Percentage difference:
            +1.22%
Stronger than enemy:
            2170 battles
Weaker than enemy:
            1900 battles
Percentage Stronger:
            +3.32%

 

Posted Image

Posted Image

Block Quote

 Histogram of team rating differences: μ=1.224779 σ=15.231782

 

Posted Image

Block Quote

 Distribution of results: μ=7.396043 σ=3.293276

 

Posted Image

Posted Image

Posted Image

Posted Image

Block Quote

 All teams rating distribution: μ=4428.586298 σ=948.459155

 

And all the players in my cache:

Posted Image

Block Quote

 Histogram of all players: μ=4458.423901 σ=1829.776614

 

 


Edited by Baldrickk, 17 July 2018 - 11:47 PM.


CmdRatScabies #228 Posted 18 July 2018 - 12:11 PM

    Colonel

  • Player
  • 35441 battles
  • 3,662
  • [-MM] -MM
  • Member since:
    10-12-2015

View PostBaldrickk, on 17 July 2018 - 11:46 PM, said:

Still not yet.  Next update.  I wanted to finish the graph saving and correctness (unfinished replays showing in the results distribution graph (now fixed)

 

 

 

Would that chart not tend to be biased if it excludes a large number unfinished replays?  That is, could you not expect more players to stay to the end in winning steam rolls (cleaning up) than losing ones (killed & quit)?  Do you have access to a data set with very few early battle quits to see if it produces the same distribution?

Baldrickk #229 Posted 18 July 2018 - 01:37 PM

    Field Marshal

  • Player
  • 29322 battles
  • 13,387
  • [-TAH-] -TAH-
  • Member since:
    03-03-2013

View PostCmdRatScabies, on 18 July 2018 - 12:11 PM, said:

 

Would that chart not tend to be biased if it excludes a large number unfinished replays?  That is, could you not expect more players to stay to the end in winning steam rolls (cleaning up) than losing ones (killed & quit)?  Do you have access to a data set with very few early battle quits to see if it produces the same distribution?

Yes. And no.

 

It does miss information.

 

Lets make the niave assumption that a player will only watch battles that they win.

What does the graph look like?

 

Pretty much the same. For every battle won, the opposing team loses.

And vice versa.

The graph doesn't care which team won,  only the score difference. 

Unless the player was good enough that he (for example) took down 5+ enemies in each battle, so never lost worse than 15-5 but won 15-(0-4) more often, then it makes little difference to the ratio of results. 

We can see that a very good player like LordMuffin isn't able to do that, so it isn't really a worry.

 

Which brings me back to LordMuffin's graphs - one of the reasons I chose them for the OP is that he has watched almost every battle in his replay set to the end, so almost every result has a score recorded.


Edited by Baldrickk, 18 July 2018 - 01:37 PM.





2 user(s) are reading this topic

0 members, 2 guests, 0 anonymous users