Jump to content


Matchmaker simulator

matchmaker

  • Please log in to reply
22 replies to this topic

DrP3pp3r #1 Posted 26 May 2018 - 01:57 PM

    Private

  • Player
  • 43502 battles
  • 25
  • [PZBWL] PZBWL
  • Member since:
    02-14-2013

Hi,

 

Preamble

 

This post is about a matchmaker simulator and NOT about the matchmaker itself!

Note to me: Read the whole matchmaker thread!!!

 

Introduction

I'm DrP3pp3r and I'm playing World of Tanks since 2013 and I enjoy the game a lot despite limited success. At least most of the time... :)

In the last days, I watched videos from Quickybaby, DezGamez and the incredible Claus Kellerman about "Perfecting Preferential Premiums". Also playing the game I noticed a lot of complaining about the matchmaker (mm) and the (rather) new template system.
Personally, I do not dislike the template system myself that much. At least as a bottom tier tank, you always have a bunch vehicles of the same tier that you can definitely compete with. Of course, the Tier XIII problem (basically never being top tier as Tier XIII) sux.
The "Russian Refund" (Claus Kellerman) WG offers is, of course, an unfortunate approach. If they did this naively or because they want to make some extra money. I don't know. But I'm digressing. This is already discussed somewhere else.


"Perfecting Preferential Premiums" matchmaker statements, some random thoughts of mine


Claus' rant got me thinking about it and the stuff WG told/wrote. I have the feeling that some statements from "Perfecting Preferential Premiums" can be misinterpreted.
E.g. "the matchmaker starts grabbing an excessive number of Tier IX tanks to fill up team rosters for preferential Prems"
This sounds like the matchmaker deliberately chooses Tier IXs for preferential Prems i.e. that the algorithm is implemented like this and that this is the main strategy. I assume that this could be just a symptom because there are not enough T7s and T6s, so to get the tank into a match, the mm needs tanks from higher tiers. As it cannot pick Tier Xs, it might switch to the 5/10 template which in consequence is "grabbing an excessive number of Tier IX tanks".
To me, this article sounds like some devs gave some information and a non-tech guy or gal wrote this stuff down. While the symptoms are described correctly maybe the whole thing is misleading about the true root causes of the problem.
Despite from all this, I'd imagine that beside the uneven distribution of tanks that join the queue (way more higher tiers than lower tiers), people tend to overlook the fact that Tier Xs are always top tier. Tanks that are always top tier take away top-tier positions from other tanks. So the 1/3 top tier, 1/3 middle tier, 1/3 bottom tier stuff just cannot work out for every tank, can it?
Another side note: Apparently WG thinks short queuing times are super important. It seems most people don't see it that way and would prefer to wait a little longer for "better" matches.

 

Finally to the point: a matchmaker simulator


Currently, I was thinking about a free time programming project. Now I got an idea: a matchmaker simulator, that can be used to test different approaches.
What I would like to do:
- Get data about which tanks join the queue how often in a certain time period (e.g. within 24 hours). This would be the source for the input to the simulated queue.
  Probably different data sets would be needed (a whole day, a peak hour with a lot of users, night time with fewer users). It would be interesting to know if the distribution of tanks over tiers and classes changes a lot during the day or if it is rather constant. I never checked the WG APIs if you can get data like that, but pages like vbaddict make it seem that it is very much possible.
- Get data for each tank which battle tiers it can be placed in.
- Create a simulated queue which is filled based on the data mentioned above.
- An API for a matchmaker
  - to access the queue
  - to place a tank in a match
- Record data for each match in order to be able to get statistics for each tank:
  - how often was is top, middle, and bottom tier
  - what was the average tier of enemy tanks
  - ...
Probably further details need to be considered (maximum and average queuing time, the maximum number of on-going matches, ...). Also, platoons need to be considered somehow. It's hard to be top tier in a three-man platoon in 3/5/7 template...

The whole thing could be implemented in C++ or Java. At least that'd be my preference.
This tool would enable people to implement their own matchmaking algorithm (or to suggest one and let somebody else implement it) and to see how it performs. To compare different matchmakers a standardized set of tank inputs to the queue
could be used to compare them fairly. Another metric could be queuing times.
Possible results:
- We figure out that making a great mm is fudging hard and no surprise WG can't do better
- A great mm algorithm is born and WG can adapt it.

 

Would anybody be interested in a tool like this? Would it be worth to put some more thought into this?

 

Best regards
Dr. P3pp3r



evilchaosmonkey #2 Posted 26 May 2018 - 02:07 PM

    Lieutenant

  • Player
  • 16655 battles
  • 1,756
  • [EIGHT] EIGHT
  • Member since:
    05-04-2013

Funnily enough this isn't a new idea, but has yet to be completed/started by various people who've discussed before.

Indeed I downloaded the data this morning for a quick and dirty look myself.

 

As a first pass, suggest you just stick for the first phase of development with static data based on number of times a tank was played over the last 30 days - then move on to a more detailed look at actual live queues.

However, I am pretty sure WG have blocked (last year) data queries on live queues as it was being exploited in a mod that helped you get top tier by helping you select your tank.

 

Code should be straightforward enough to enable you pick and choose various spreads and templates.

Suspect, the result will show that indeed getting the MM right is a royal pain - but there are fairer options than 3/5/7.

 

Good luck in the project.

 



lgfrbcsgo #3 Posted 26 May 2018 - 02:36 PM

    Second Lieutenant

  • Player
  • 30245 battles
  • 1,015
  • [MOTIV] MOTIV
  • Member since:
    04-04-2012

Sound like a fun project.

 

About the data collection part of the project: Create a mod which sends data to a server for analysis. The data could be: the time when the player entered the queue, the time when he joined the match, the players in the match and maybe platoon mates. The server could then approximate the actual queue data with enough data points, i.e. users of the mod.

I could help with a mod like this, but I am currently short on time. I am also no data scientist to help with analysing the data points.



RamRaid90 #4 Posted 26 May 2018 - 02:38 PM

    Lieutenant General

  • Player
  • 21398 battles
  • 6,457
  • [D0NG] D0NG
  • Member since:
    12-14-2014

View Postlgfrbcsgo, on 26 May 2018 - 01:36 PM, said:

Sound like a fun project.

 

About the data collection part of the project: Create a mod which sends data to a server for analysis. The data could be: the time when the player entered the queue, the time when he joined the match, the players in the match and maybe platoon mates. The server could then approximate the actual queue data with enough data points, i.e. users of the mod.

I could help with a mod like this, but I am currently short on time. I am also no data scientist to help with analysing the data points.

 

Be careful GDPR police arepatrolling these servers :trollface:

lgfrbcsgo #5 Posted 26 May 2018 - 02:42 PM

    Second Lieutenant

  • Player
  • 30245 battles
  • 1,015
  • [MOTIV] MOTIV
  • Member since:
    04-04-2012

View PostRamRaid90, on 26 May 2018 - 02:38 PM, said:

 

Be careful GDPR police arepatrolling these servers :trollface:

 

Please, don't tell them. :hiding:

You should probably consider getting a data protection lawyer on your team...

 

Spoiler

 


Edited by lgfrbcsgo, 26 May 2018 - 02:45 PM.


DrP3pp3r #6 Posted 26 May 2018 - 04:34 PM

    Private

  • Player
  • 43502 battles
  • 25
  • [PZBWL] PZBWL
  • Member since:
    02-14-2013

View Postevilchaosmonkey, on 26 May 2018 - 01:07 PM, said:

As a first pass, suggest you just stick for the first phase of development with static data based on number of times a tank was played over the last 30 days - then move on to a more detailed look at actual live queues.

Yeah, my initial idea was to use static data and create random queue input from that.

Getting data from the live queue would be interesting indeed though.

 

 

View Postevilchaosmonkey, on 26 May 2018 - 01:07 PM, said:

Code should be straightforward enough to enable you pick and choose various spreads and templates.

 I'd envision a clear separation of the framework (filling the queue and access to the queue) and mm "plug-ins".

One "plug-in" could be a WG-like mm based on available info from WG about the current mm.

Other "plug-in"s could be implementations of whatever someone can think of. So if you want to try to write your own mm, you just need to write the plug-in part and don't have to worry about how the queue is formed and how the result data set is created.

 

 

View Postevilchaosmonkey, on 26 May 2018 - 01:07 PM, said:

Suspect, the result will show that indeed getting the MM right is a royal pain - but there are fairer options than 3/5/7.

 That's my feeling. About the fairer options: There probably is, but it is sure not easy to create the algorithm (so it works and is sufficiently efficient).

 

View Postlgfrbcsgo, on 26 May 2018 - 01:36 PM, said:

About the data collection part of the project: Create a mod which sends data to a server for analysis. The data could be: the time when the player entered the queue, the time when he joined the match, the players in the match and maybe platoon mates. The server could then approximate the actual queue data with enough data points, i.e. users of the mod.

I could help with a mod like this, but I am currently short on time. I am also no data scientist to help with analysing the data points.

Like stated above I'd start with static data, but this ofc would be neat.

Dealing with data is no problem as all of this is kinda my profession. :)

 

View Postlgfrbcsgo, on 26 May 2018 - 01:42 PM, said:

View PostRamRaid90, on 26 May 2018 - 02:38 PM, said:

 

Be careful GDPR police arepatrolling these servers :trollface:

 

Please, don't tell them. :hiding:

You should probably consider getting a data protection lawyer on your team...

 

Spoiler

 

Hashing is probably not quite good enough, because the pool of user IDs is public. If you just hash them all you have a reverse look-up table and can figure out who the user was.

But you could just generate a random user id (e.g. create a UUID) in the client and use that. Then there is no connection to the user.



lgfrbcsgo #7 Posted 26 May 2018 - 04:38 PM

    Second Lieutenant

  • Player
  • 30245 battles
  • 1,015
  • [MOTIV] MOTIV
  • Member since:
    04-04-2012

View PostDrP3pp3r, on 26 May 2018 - 04:34 PM, said:

Hashing is probably not quite good enough, because the pool of user IDs is public. If you just hash them all you have a reverse look-up table and can figure out who the user was.

But you could just generate a random user id (e.g. create a UUID) in the client and use that. Then there is no connection to the user.

 

Right, I complete forgot that you can obtain a list of user IDs from Wargaming's API.

UUIDs wouldn't work as clients should ideally also log the players they were in the battle with.



evilchaosmonkey #8 Posted 26 May 2018 - 05:08 PM

    Lieutenant

  • Player
  • 16655 battles
  • 1,756
  • [EIGHT] EIGHT
  • Member since:
    05-04-2013

I really look forward to the result of this, I do think this tool will prove invaluable and very controversial.  Two things that this forum and I personally love.

 

I wish you all the best.


Edited by evilchaosmonkey, 26 May 2018 - 05:08 PM.


DrP3pp3r #9 Posted 10 June 2018 - 03:39 PM

    Private

  • Player
  • 43502 battles
  • 25
  • [PZBWL] PZBWL
  • Member since:
    02-14-2013

Short update to this topic:

I've started the project and got quite some stuff working already. It's basic, but it works.

The project can be found on github: https://github.com/d45w03lfch3n/WotMatchmakerSimulator

Feedback and/or support are appreciated!

 

What exists/works so far

  • Database table with information about existing tanks
  • Database table with information about tank usage
  • Queue that is filled automatically. The ratio of the tanks in the queue reflects the percentage of tanks used by players in the game.
  • API to
    • retrieve tanks from the queue
    • form and store matches (in the database)
    • store unmatched tanks (in the database)
  • (Bad) example matchmaker
  • Analysis of matchmaking result via SQL (via DB Browser for SQLite)
  • Loading (injection), configuration and running of a self-written matchmaker into the simulator (from jar)
          The self written matchmaker has to inherit IMatchmaker or ThreadedMatchmaker

What is still missing (most important stuff)

  • Platoons
    • adding to the queue
    • retrieving form the queue
  • A non-stupid matchmaker example
  • Better analysis facilities
  • Documentation

Edited by DrP3pp3r, 10 June 2018 - 03:39 PM.


HassenderZerhacker #10 Posted 10 June 2018 - 10:34 PM

    Captain

  • Player
  • 27202 battles
  • 2,394
  • [1DPG] 1DPG
  • Member since:
    09-09-2015

i honestly don't see the problem why the mm problem cannot be solved. 

just set for every tier a minimum frequency for top tier mm, except for tier 10. form these matches first, leftover tanks in each tiers go into same tier battles.

 

fixed.

next question please.

 

yes i know, tier 10 will likely see very many same tier battles, but honestly, who is still playing tier 10? it's boring as f*ck



Homer_J #11 Posted 11 June 2018 - 02:46 AM

    Field Marshal

  • Beta Tester
  • 28692 battles
  • 29,980
  • [WJDE] WJDE
  • Member since:
    09-03-2010

View PostDrP3pp3r, on 26 May 2018 - 01:57 PM, said:

 

- Get data about which tanks join the queue how often in a certain time period (e.g. within 24 hours). This would be the source for the input to the simulated queue.
  Probably different data sets would be needed (a whole day, a peak hour with a lot of users, night time with fewer users). It would be interesting to know if the distribution of tanks over tiers and classes changes a lot during the day or if it is rather constant. I never checked the WG APIs if you can get data like that, but pages like vbaddict make it seem that it is very much possible.
 

You can get data of how many battles are played but that tells you nothing about how many join the queue.  Different classes and tiers will queue for longer, especially at certain times of day.  And not everyone who joins the queue gets in a battle.

 

You also need to know about platoons, and how they are made up.



DrP3pp3r #12 Posted 11 June 2018 - 06:04 AM

    Private

  • Player
  • 43502 battles
  • 25
  • [PZBWL] PZBWL
  • Member since:
    02-14-2013

View PostHomer_J, on 11 June 2018 - 01:46 AM, said:

You can get data of how many battles are played but that tells you nothing about how many join the queue.  Different classes and tiers will queue for longer, especially at certain times of day.  And not everyone who joins the queue gets in a battle.

 

You also need to know about platoons, and how they are made up.

 

I don't really think the amount of tanks that join the queue and the tanks that get matched is that different. Do you? And yes different classes queue longer. Maybe I play at the wrong times, but I never have to wait really long. Also for the "never being top tier" problem it should not matter too much, what classes the tanks are - it makes the computations for the mm more complicated (in terms of computation time), but for balancing the tiers it should not matter too much.

But I agree. Like I mentioned above it would we nice to have data sets that represent different times of the day. Where it is not data from a whole month, but maybe form 8 am to 12 am. And of course, real data would be better than an approximation. But I think it is a decent start.

About platoons: I'd didn't check, but isn't that data available through the WG APIs?

In the worst case, one could just generate random data and see if the mm still performs well with that kind of input.



DrP3pp3r #13 Posted 11 June 2018 - 06:09 AM

    Private

  • Player
  • 43502 battles
  • 25
  • [PZBWL] PZBWL
  • Member since:
    02-14-2013

View PostHassenderZerhacker, on 10 June 2018 - 09:34 PM, said:

i honestly don't see the problem why the mm problem cannot be solved. 

just set for every tier a minimum frequency for top tier mm, except for tier 10. form these matches first, leftover tanks in each tiers go into same tier battles.

 

fixed.

next question please.

 

yes i know, tier 10 will likely see very many same tier battles, but honestly, who is still playing tier 10? it's boring as f*ck

That could be true. Wouldn't it be nice to prove this by writing a mm that does this and to be able to back it up by numbers from a simulation?



Homer_J #14 Posted 11 June 2018 - 06:29 AM

    Field Marshal

  • Beta Tester
  • 28692 battles
  • 29,980
  • [WJDE] WJDE
  • Member since:
    09-03-2010

View PostDrP3pp3r, on 11 June 2018 - 06:04 AM, said:

 

I don't really think the amount of tanks that join the queue and the tanks that get matched is that different. Do you? 

I don't think you can say, and if you can't then any conclusions you come to are moot.

 

Go for it, the API is available to anyone but I think you will find it's not as useful as you hope.

 

Vbaddict mostly gets it's stats from replays uploaded by players, so is in reality pretty way off the mark.

 

Other sites track individual players through the API but often only take a snapshot when someone requests info for that player, which is why you often see impossible figures for battles played in the last day.


Edited by Homer_J, 11 June 2018 - 06:30 AM.


HassenderZerhacker #15 Posted 11 June 2018 - 08:13 AM

    Captain

  • Player
  • 27202 battles
  • 2,394
  • [1DPG] 1DPG
  • Member since:
    09-09-2015

View PostDrP3pp3r, on 11 June 2018 - 06:09 AM, said:

That could be true. Wouldn't it be nice to prove this by writing a mm that does this and to be able to back it up by numbers from a simulation?

 

it doesn't need to be proven as it's a flawless mathematical model - it's a question of "just do it"

DrP3pp3r #16 Posted 11 June 2018 - 06:01 PM

    Private

  • Player
  • 43502 battles
  • 25
  • [PZBWL] PZBWL
  • Member since:
    02-14-2013

View PostHassenderZerhacker, on 10 June 2018 - 09:34 PM, said:

 leftover tanks in each tiers go into same tier battles.

 

 

I didn't read that sentence correctly on the first pass. Sorry. So you really got a complete approach and a solid idea.

 

View PostHassenderZerhacker, on 11 June 2018 - 07:13 AM, said:

 

it doesn't need to be proven as it's a flawless mathematical model - it's a question of "just do it"

But your "flawless mathematical model" is a little imprecise when it comes to real-time behavior and some parameters. It is impossible to implement an algorithm with the given information.

- How do you create the matches? "Freeze" the queue, create your mixed-tier matches and the fix single tier matches with the rest, and leave the rest of the tanks in the queue for the next pass? Or is this a continuous process?

- For your mixed-tier matches: How many tanks are top tier? Fixed amount? Dynamic amount? Up to a maximum (e.g. 3) or up to 14? It makes a difference if you are the only top tier tank or 1 of 14... This will also have a big impact on the number of single tier matches. A too big amount of single tier matches is going to make people cry as well IMHO.

- How do you balance different classes of tanks? Random or certain limits?

- How do you deal with platoons?

- ...

 

The big question is: What percentage of single tier matches would be resulting from this approach? This would highly influence acceptance of a matchmaker using this "model". Do you know the answer without a simulation?

Another question: Can this model be implemented in a way that it creates a steady load on the computation capabilities of the machine its running on? Or does it create "waves"?



Homer_J #17 Posted 12 June 2018 - 12:24 AM

    Field Marshal

  • Beta Tester
  • 28692 battles
  • 29,980
  • [WJDE] WJDE
  • Member since:
    09-03-2010

View PostDrP3pp3r, on 11 June 2018 - 06:04 AM, said:

 

About platoons: I'd didn't check, but isn't that data available through the WG APIs?

 

I had a look at the API, as far as I can see it gives data on individual players and on clans.  It doesn't directly give you information about battles or the queue.

 

Also you can only request information about one player at a time.


And there is a ten requests per second limit.

 

The only way you can get what you want from the API is to request information about every player, twice, and then total how many battles are played in each class/tier of tank.  You certainly can't do it on a minute by minute basis (or even hourly) with however many million registered players there are on the EU server.  Even getting data on the commonly accepted 800k active players would take most of a day just to get the first reference set.



DrP3pp3r #18 Posted 12 June 2018 - 06:12 AM

    Private

  • Player
  • 43502 battles
  • 25
  • [PZBWL] PZBWL
  • Member since:
    02-14-2013

View PostHomer_J, on 11 June 2018 - 11:24 PM, said:

Even getting data on the commonly accepted 800k active players would take most of a day just to get the first reference set.

The day is not a problem. Then there would be at least a nice sample data set.

But what is the key for querying the database? Do you need a player's name? Seeing how WoT statistics pages look and work makes it seem that way.

Or is there something like "get all players"?



Homer_J #19 Posted 12 June 2018 - 06:40 AM

    Field Marshal

  • Beta Tester
  • 28692 battles
  • 29,980
  • [WJDE] WJDE
  • Member since:
    09-03-2010

View PostDrP3pp3r, on 12 June 2018 - 06:12 AM, said:

The day is not a problem. Then there would be at least a nice sample data set.

But what is the key for querying the database? Do you need a player's name? Seeing how WoT statistics pages look and work makes it seem that way.

Or is there something like "get all players"?

 

https://developers.w...e=en&r_realm=eu

 

Fill your boots.

 

And I already said, as far as I can see there is no get all players.  You get one player at a time by their ID.  So first you need a list of every player.  Oh, there's problem number 1, how do you get a list of every player?  GL&HF.

 

Clearly you have not got the first clue of how you are going to start this.  But it's your project, any more consultancy will be chargeable.  Does £100 an hour with a minimum 1 day seem reasonable?

 



HassenderZerhacker #20 Posted 12 June 2018 - 08:58 AM

    Captain

  • Player
  • 27202 battles
  • 2,394
  • [1DPG] 1DPG
  • Member since:
    09-09-2015

View PostDrP3pp3r, on 11 June 2018 - 06:01 PM, said:

 

But your "flawless mathematical model" is a little imprecise when it comes to real-time behavior and some parameters. It is impossible to implement an algorithm with the given information.

- How do you create the matches? "Freeze" the queue, create your mixed-tier matches and the fix single tier matches with the rest, and leave the rest of the tanks in the queue for the next pass? Or is this a continuous process?

- For your mixed-tier matches: How many tanks are top tier? Fixed amount? Dynamic amount? Up to a maximum (e.g. 3) or up to 14? It makes a difference if you are the only top tier tank or 1 of 14... This will also have a big impact on the number of single tier matches. A too big amount of single tier matches is going to make people cry as well IMHO.

- How do you balance different classes of tanks? Random or certain limits?

- How do you deal with platoons?

- ...

 

The big question is: What percentage of single tier matches would be resulting from this approach? This would highly influence acceptance of a matchmaker using this "model". Do you know the answer without a simulation?

Another question: Can this model be implemented in a way that it creates a steady load on the computation capabilities of the machine its running on? Or does it create "waves"?

 

err... I'd say it's pretty easy on the contrary.

for each player, the system remembers how many battles the player played as a top tier in each tank tier, and for each tier there is a fixed minimum % of battles that the player should be top tier. so, when the player plays a certain tier, the system knows if he is a candidate for top tier or not (this can be slightly randomized over a few days, so the player doesn't exactly know when a top tier streak is due to happen).

so then the system knows which tanks are due for top tier and picks those for filling top tier matches.

 

some more rules:

if there aren't enough eligible lower tier vehicles for 3/5/7 or 5/10 --> same tier match (which does not count as top tier match and neither as bottom tier)

if players don't get enough top tier matches, they get at worst same tier matches (which does not count as top tier match and neither as bottom tier)

players cannot get more bottom tier matches than they get top tier matches

 

I believe the load on the matchmaker would be equal to what it is now, I can't see what would cause significantly more load, maybe even on the contrary - since I expect a reduction in 3/5/7 and 5/10 matches, the matchmaker should run quicker with an increased number of same tier matches.

 

Regarding the amount of top tier matches, I believe they would decrease overall, but increase for tier 8.

The real pain is the amount of bottom tier matches, especially at tier 8, and these would be replaced by same tier matches.

If they also folllowed my other suggestion, to give some tier 9 and 10 OP tanks +1/-1 MM we could even see more improvement.







Also tagged with matchmaker

1 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users