Re: [gem5-gpu-dev] Simulating GDDR5 ???

[Jason Power]

Hi Konstantinos,

You'll probably have to modify the DRAM model to simulate GDDR5 precisely. The important files for that are in gem5. Assuming you're using Ruby, you can start by looking at gem5/src/mem/ruby/RubyMemoryControl.cc 

You may be able to get by just modifying the parameters for that class when you create the memory controller in GPUMemConfig.py. You can see all of the parameters available in gem5/src/mem/ruby/RubyMemoryControl.py.

Jason

On Thu, Jan 16, 2014 at 4:51 AM, Konstantinos Koukos <koukos.ko...@gmail.com> wrote:

Hello,

Is it possible to simulate GDDR5 memory? Which files do i need to edit beyond

GPUMemConfig.py? In this file is not very clear how do i set CAS/RAS, etc access

times. Are there any other GPU memory related files?

Best Regards.

[Konstantinos Koukos]

Thanks a lot,

What i actually had in mind was to create an APU like environment with a unified GDDR5 as main memory
shared between CPU and GPU cores. This most likely means that i will need to make most of the changes
in the gem5 part and hint the gpgpu part to use this memory (rodinia no copy as benchmarks). Which files
should i look into in each side GPU/CPU?

Best Regards,
Konstantinos.

[Joel Hestness]

Hi guys,

  What Jason was referring to is that Ruby has it's own memory controller (gem5/src/mem/ruby/system/RubyMemoryControl.*), which models somewhat abstracted DRAM timings in the controller.  It uses a gem5 SimpleMemory as its backing store and ignores the timings in that object rather than using something like gem5's SimpleDRAM, which has timings built in (both of these classes are defined in gem5/src/mem/).  Note that the SimpleDRAM is often used with the gem5 classic memory model (i.e. not Ruby memory model) to get DRAM timings.  The Ruby memory controller can actually be quite accurate in spite of the abstract timings, though it is limited to modeling a closed-page row buffer precharge policy.

  To model something like GDDR5, you can set the parameters of the Ruby memory controller that is instantiated in gem5-gpu configs.  The command line parameters and controller settings for this are defined in gem5-gpu/configs/GPUMemConfig.py, and include the total controller latency (e.g. scheduling delay), memory bus frequency, the number of cycles for a bus transfer (e.g. 64B cache line / 32-bit bus / DDR = 8 cycles), and the bank busy time (CL+tRP+tRCD+CAS).  For GDDR5, you'd want to add the prefetch buffering latency to the total memory controller latency.

  Your other option would require adding a new memory controller in Ruby to wrap gem5's SimpleDRAM (or analogously, the appropriate component in DRAMSim2 if you're using those patches).  This new controller would need to be able to receive Ruby messages from directory controllers (e.g. gem5-gpu/src/mem/protocol/VI_hammer-dir.sm), and convert those to packets to be sent to the SimpleDRAM.  On return, it would have to convert response packets back to Ruby messages for return to the directory.  On brief inspection, I think this would be a fair amount of work.

  Hope this helps,

  Joel

--
  Joel Hestness
  PhD Student, Computer Architecture
  Dept. of Computer Science, University of Wisconsin - Madison
  http://pages.cs.wisc.edu/~hestness/

[Konstantinos Koukos]

Hey Joel and Jason,

Thanks a lot for the help. In my first approach i did the wrong thing to create the memory config in SimpleDRAM.py.
I had a look at gem5-gpu/configs/GPUMemConfig.py. What is not really obvious is how the timing parameters are
given (e.g., in RubyMemoryControl.py timing parameters are explicit). In GPUMemConfig.py the values of the timing
parameters are not explicitly assigned in the same file but i would guess they are read from gem5-gpu/configs/gpu_config/
gpgpusim__arch__.config. Is that correct?

According to what is already suggested i think that the shortest path should be to modify RubyMemoryControl.py similar to
GPUMemConfig.py to include the prefetch latencies using hard coded timing settings. In SimpleDRAM the settings are in
nsec (or usec). What about RubyMemoryControl.py? Does it use wall time or is it in clock cycles?

My last questions is how can we simulate bandwidth using RubyMemoryControl.py? Is it internally calculated using bus
width and latencies or we need to write  code for that ?
Thanks a lot,

Best Regards,
Konstantinos.

On Thursday, January 16, 2014 4:11:49 PM UTC+1, Jason Power wrote:

[Joel Hestness]

Hi Konstantinos,

  I'd recommend reviewing the code in gem5-gpu/configs/GPUMemConfig.py again.  Note that the parameters defined in RubyMemoryControl.py are just parameters of that Python object, and after a RubyMemoryControl is instantiated in a configuration script, these parameters can be set from within other configuration scripts.  The simulation parameters defined in GPUMemConfig.py are used to set the parameters to the RubyMemoryControl components in the setMemoryControlOptions() function that is called from either the se_fusion.py or fs_fusion.py simulation scripts (depending on which you are using).  The RubyMemoryControls are attached to Ruby directories and named system.ruby.dir_cntrl#.memBuffer in the gem5 component hierarchy.  For running a baseline GDDR5 memory, you shouldn't need to modify anything, just pass the appropriate parameters to gem5-gpu on the command line when you start a simulation.  These parameters are for a gem5 component (RubyMemoryControl), so the analogous parameters in the GPGPU-Sim configuration template file are ignored.

  RE: prefetch latency: As I mentioned before, you could begin by simply adding this latency to the overall memory controller latency as that will be close to the full effect of the prefetch buffering.  If you want to model the buffering more accurately, you'd need to add those parameters to the RubyMemoryControl.py script, and modify RubyMemoryControl.cc to model the buffering.

  RE: bandwidth: You can adjust this by setting the memory bus frequency and memory bus transfer latency as appropriate (see my prior response).

  Joel

[Konstantinos Koukos]

Thanks a lot. What you suggest (to give the options at command line) is a more appealing approach. Is there any documentation on the commands.
Should i look at some file for the names to be used? Would it be easy to give an example that would instantiate lets say a 4GB GDDR5 memory
shared between CPU and GPU (without any dedicated GPU memory)?

Best Regards,
Konstantinos.

On Thursday, January 16, 2014 4:11:49 PM UTC+1, Jason Power wrote: