Introduction - Interest in microRTS as a research tool

[Cody Kearse]

Hi everyone! I wasn't sure what the method of contact was, so I decided to jumpstart a new conversation in this thread. 

Anywho!, I have an strong interest in using microRTS as a tool to test out some novel RL algorithms. I had a question and a request that would most certainly help my initiatives.

Question: On the main competition page, I found published results from the last few years. I am curious if there is a detailed key for all of the metrics, so that I can better understand what is being judged?

Request: On the competition page, it suggests that replays are available upon request. I would definitely be interested in those also.

Okay, no more length to this introductory post! Hello again, thank you for your time in reading and consideration of responses, and be safe out there!

Sinerely,

Cody Kearse

[Santi Ontañón]

Hi Cody, glad you are interested in microRTS!!! Looking forward to your RL experiments, we are currently also experimenting with RL in microRTS (one of my PhD students at Drexel built this, in case it's useful: https://github.com/vwxyzjn/gym-microrts )

As for the metrics used in the competition, You are referring to these types of spreadsheets, right? ( https://docs.google.com/spreadsheets/d/1-jhHGGylVTo7dXGuS88XSNBcs2C7HXHgP6EEHW5T_DQ/edit?usp=sharing ) [note to self: upload the 2020 results]

I don't think I ever wrote a detailed key to the numbers. But I can explain what they are (and I should probably add this explanation to the spreadsheet):

- When 2 agents play a game, if there is a tie, each agent gets a score of 0.5, and if there is a winner, the winner gets 1.0, and the loser 0.0

- For each track of the competition, I just played 5 full round robin rounds on 12 different maps. The "Overall Scores (full matrix):" shows the score that each bot accumulated when playing against each other in all 5 rounds and all 12 maps (row agent playing against column agent).

- Then "Overall Scores (aggregated)" is just aggregated (basically the sum of each row of the matrix above)

- The subsequent tables/matrices are the same but broken by competition iteration (an iteration is 1 map in one of the rounds, so there are 12*5 iterations), and then normalized so that the maximum possible score an agent can hope to attain would be a 1.0

- The remaining of the spreadsheet shows the same, but broken by map

I hope that helps!

As for accessing the competition replays just drop me an email (santi....@gmail.com), so, I remember and I'll send them to you. I don't have them publicly available just because they are large files and I never decided where to host them :)