Methodology of capturing google traces

[JINYU DONG]

Hi, I'm doing a final year project on branch prediction and there are several questions. Would be very helpful if could be answered.
1. Are the virtual addresses of branches the same across different Google traces of an application (In other words, is it safe to assume that there are no ASLR effects across different traces of an application)?
2. What do UUIDs correspond to?
3. Can we assume each directory (e.g. charlie) corresponds to a unique application?

[sharma...@google.com]

Hi,

1. All traces within the same workload/trace-X/ directory are for the same run of the application, therefore share the address space.

2. The UUID is simply a unique identifier for the trace.

3. Each top-level directory (charlie, delta, ...) indeed corresponds to a different application, as described in the README file. 

Abhinav

[JINYU DONG]

Hi Abhinav,

Thank you for answering the question so fast! It helps the project a lot.

Regards,

Derek

[Kostya Berestizshevsky]

A follow-up question regarding branch traces:

I would really want to construct an interleaved version of the branch traces that will represent the CPU-centric view of the workload (namely: to construct the exact sequence of instructions that was executed in the cpu core). This of course assumes that all threads were executed on the same cpu core, but nevertheless can be super helpful for a branch predictio research.

For example: 

trace_tid_1 (300 instructions)

Context switch to thread number 2

trace_tid_2 (90000 instructions)

Context switch to thread 1

trace_tid_1 (10000 instructions)

....

Is it possible to know for each context switch instruction - to which thread did it switch?  

Thank you!

[Derek Bruening]

On Fri, Dec 2, 2022 at 5:55 AM Kostya Berestizshevsky <kosty...@gmail.com> wrote:

A follow-up question regarding branch traces:

I would really want to construct an interleaved version of the branch traces that will represent the CPU-centric view of the workload (namely: to construct the exact sequence of instructions that was executed in the cpu core).

You can do this today with custom iteration, but there is no built-in iterator for it: that is filed as https://github.com/DynamoRIO/dynamorio/issues/5694.

 

This of course assumes that all threads were executed on the same cpu core, but nevertheless can be super helpful for a branch predictio research.

For example: 

trace_tid_1 (300 instructions)

Context switch to thread number 2

trace_tid_2 (90000 instructions)

Context switch to thread 1

trace_tid_1 (10000 instructions)

....

Is it possible to know for each context switch instruction - to which thread did it switch?  

I assume by "context switch instruction" you mean the final instruction in a thread-on-cpu segment, whatever it happens to be.

All of the data is there in the timestamp and cpu markers to answer your question.

The new fast skipping introduced a schedule file that condenses the scheduling information into a single file for convenience; but the above-mentioned iterator, which would use this schedule data, is not yet implemented.

Please also note that as a user-mode trace of one process captured on a machine with other processes, there can be gaps where there is nothing on a cpu (because some other process was there).  I.e., trying to convert from the captured software thread trace to a hardware thread trace as exactly recorded will not produce perfect results.  It is also limited to the exact cpu count of the traced machine.  A synthetic schedule of the software threads onto virtual cpus can work better in some ways but has its own tradeoffs.

 

Thank you!

On Saturday, 13 August 2022 at 01:02:22 UTC+2 dere...@connect.hku.hk wrote:

Hi Abhinav,

Thank you for answering the question so fast! It helps the project a lot.

Regards,

Derek

On Friday, August 12, 2022 at 7:41:39 PM UTC+1 sharma...@google.com wrote:

Hi,

1. All traces within the same workload/trace-X/ directory are for the same run of the application, therefore share the address space.

2. The UUID is simply a unique identifier for the trace.

3. Each top-level directory (charlie, delta, ...) indeed corresponds to a different application, as described in the README file. 

Abhinav

On Friday, August 12, 2022 at 1:17:34 PM UTC-4 dere...@connect.hku.hk wrote:

Hi, I'm doing a final year project on branch prediction and there are several questions. Would be very helpful if could be answered.
1. Are the virtual addresses of branches the same across different Google traces of an application (In other words, is it safe to assume that there are no ASLR effects across different traces of an application)?
2. What do UUIDs correspond to?
3. Can we assume each directory (e.g. charlie) corresponds to a unique application?

--
You received this message because you are subscribed to the Google Groups "DynamoRIO Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dynamorio-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/e682ff3d-c9f3-4fe9-9ba4-f6ae5d299194n%40googlegroups.com.

[Kostya Berestizshevsky]

Thank you for your reply, I greatly apprecite it!

I would like to be slightly more specific as my question was only about banch traces that you released. Please refer to my refined question inlined below:

On Friday, 2 December 2022 at 17:42:09 UTC+1 Derek Bruening wrote:

On Fri, Dec 2, 2022 at 5:55 AM Kostya Berestizshevsky <kosty...@gmail.com> wrote:

A follow-up question regarding branch traces:

I would really want to construct an interleaved version of the branch traces that will represent the CPU-centric view of the workload (namely: to construct the exact sequence of instructions that was executed in the cpu core).

You can do this today with custom iteration, but there is no built-in iterator for it: that is filed as https://github.com/DynamoRIO/dynamorio/issues/5694.

 

This of course assumes that all threads were executed on the same cpu core, but nevertheless can be super helpful for a branch predictio research.

For example: 

trace_tid_1 (300 instructions)

Context switch to thread number 2

trace_tid_2 (90000 instructions)

Context switch to thread 1

trace_tid_1 (10000 instructions)

....

Is it possible to know for each context switch instruction - to which thread did it switch?  

I assume by "context switch instruction" you mean the final instruction in a thread-on-cpu segment, whatever it happens to be.

All of the data is there in the timestamp and cpu markers to answer your question.

 In the branch traces there is no timestamp or cpu marker, but there are specific branches of a type context_switch and interrupt. For simplicity I only asked about these context_switch instructions (there are a few thousands of them scattered across each trace). My goal is to break each trace into segments that end with a context_switch instruction, and then to reorder these segments from all the traces into one long branch trace of the entire application run. Now the only missing piece of information that I need is the thread-id to which each context_switch instruction was switching (the branch trace specifies only the target address of the context_switch). I am not intending to use custom iterators, but rather just to process the trace you have provided. Perhaps you have this information somewhere in your release?

The new fast skipping introduced a schedule file that condenses the scheduling information into a single file for convenience; but the above-mentioned iterator, which would use this schedule data, is not yet implemented.

Please also note that as a user-mode trace of one process captured on a machine with other processes, there can be gaps where there is nothing on a cpu (because some other process was there).  I.e., trying to convert from the captured software thread trace to a hardware thread trace as exactly recorded will not produce perfect results.  It is also limited to the exact cpu count of the traced machine.  A synthetic schedule of the software threads onto virtual cpus can work better in some ways but has its own tradeoffs.

Totally agree that I am ignoring multi-tenancy, and the lack of kernel code tracing. But it nevertheless can be a great improvement  for me just to assume that all software threads were indeed hardware threads . This way I can create one long workload that can stress the branch predictor design.

[Derek Bruening]

The terms for those two events in the branch trace can be misleading (and have already been earmarked to change in the next iteration): really "context_switch" is a system call performing a kernel-mediated control transfer (e.g., sigreturn: see https://github.com/DynamoRIO/dynamorio/blob/master/clients/drcachesim/common/trace_entry.h#L259) and "interrupt" is a kernel-initiated disruption of control in user mode (e.g., signal delivery: https://github.com/DynamoRIO/dynamorio/blob/master/clients/drcachesim/common/trace_entry.h#L238).

To view this discussion on the web visit https://groups.google.com/d/msgid/dynamorio-users/8374a65e-61dd-4bb0-a9ab-5d1c5f6ae979n%40googlegroups.com.