Invisible lasers in Space Invaders under frame_skip=4

[Bojun Huang]

Hi everyone,

In DeepMind's NIPS Workshop paper in 2013, they mentioned that they used a frame skip of 3 for the game of Space Invaders because the "lasers" become invisible under frame_skip=4. But in their Nature paper they seem to use a uniform frame skip of 4 across all Atari games, including Space Invaders. So I'm wondering what's the latest status about the "invisible laser" problem? 

In fact, I have built and run a 0.4.4 version of ALE and found that, with a frame skip of 4, the lasers are actually only invisible in the GUI window of the game but the image returned by getScreen() indeed contains the lasers. In other words, it appears that having a frame skip of 4 will not make the RL agent blind to enemy's lasers but will only make us (the game observers) blind to the lasers.Is this the consensus about the "invisible lasers" issue at this time? 

I didn't find this issue being mentioned or discussed after DeepMind's Nature paper. The recent OpenAI Gym platform contains Space Invaders, where the frame skip is "uniformly sampled from {2,3,4}". I have also checked the change log of the ALE code, but didn't find it mentioned there either (as of version 5.0) -- it really feels like a bug if the agent can see the lasers but we programmers cannot ...  

Thanks,

Bojun

       

[Tambet Matiisen]

I believe color averaging option in recent ALE versions takes care of this problem. Basically it averages the last two frames, which probably fixes it for Space Invaders. Also DeepMind used the same approach in their code:
https://github.com/deepmind/alewrap/blob/master/alewrap/GameScreen.lua

  Tambet

--
You received this message because you are subscribed to the Google Groups "Deep Q-Learning" group.
To unsubscribe from this group and stop receiving emails from it, send an email to deep-q-learni...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/deep-q-learning/1fc2148e-7965-4496-977d-2dc28176bc1d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alejandro Dubrovsky]

Hi Bojun

The reason the agent is seeing them is because ALE does colour averaging
over the last two frames by default (note that a very recent commit
disabled this) like Tambet mentioned. If you want to replicate
DeepMind's code though, you need to disable the colour averaging and do
a maximisation over the last two frames instead (they mention that this
is what they do in the appendix of one of their papers). There is no
switch to make ALE give you these maximised frames, so you'd have to do
those "manually" ie by disabling frame skipping in ALE and performing a
max over the last two frames every four frame cycle.

ie

Frames 1, 2 are ignored, give agent max of frames 3 and 4, ignore frames
5 and 6, give agent max of frames 7 and 8, etc

Nathan's code does that so you could have a look at that. The relevant
code is a combination of the run_episode and _step methods in
https://github.com/spragunr/deep_q_rl/blob/master/deep_q_rl/ale_experiment.py

On 26/10/16 11:14, Bojun Huang wrote:
> Hi everyone,
>
> In DeepMind's NIPS Workshop paper in 2013, they mentioned that they used
> a frame skip of 3 for the game of Space Invaders because the "lasers"
> become invisible under frame_skip=4. But in their Nature paper they seem
> to use a uniform frame skip of 4 across all Atari games, including Space
> Invaders. So I'm wondering what's the latest status about the "invisible
> laser" problem?
>
> In fact, I have built and run a 0.4.4 version of ALE and found that,
> with a frame skip of 4, the lasers are actually only invisible in the
> GUI window of the game but the image returned by getScreen() indeed

> contains the lasers. In other words,*it appears that having a frame skip

> of 4 will not make the RL agent blind to enemy's lasers but will only

> make us (the game observers) blind to the lasers*.Is this the consensus

> about the "invisible lasers" issue at this time?
>
> I didn't find this issue being mentioned or discussed after DeepMind's
> Nature paper. The recent OpenAI Gym platform contains Space Invaders,
> where the frame skip is "uniformly sampled from {2,3,4}". I have also
> checked the change log of the ALE code, but didn't find it mentioned
> there either (as of version 5.0) -- it really feels like a bug if the
> agent can see the lasers but we programmers cannot ...
>
> Thanks,
> Bojun
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "Deep Q-Learning" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to deep-q-learni...@googlegroups.com

> <mailto:deep-q-learni...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/deep-q-learning/1fc2148e-7965-4496-977d-2dc28176bc1d%40googlegroups.com

> <https://groups.google.com/d/msgid/deep-q-learning/1fc2148e-7965-4496-977d-2dc28176bc1d%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Bojun Huang]

Thanks to Tambet and Alejandro for your comments!

I indeed noticed the Phosphor blend used in the ALE code (which does the color averaging) and have modified it to take the max over two consecutive frames so as to reproduce DeepMind's result. But in both cases of the average-based and max-based blends, the lasers are invisible in the game window, and this has made it harder for me to monitor the agent's behavior (say, to escape from the lasers) during the training time, and so I'm wondering if there is some known solution to this problem...

Best,

Bojun

  

> <mailto:deep-q-learning+unsub...@googlegroups.com>.