Coz, Support for multi-process applications, GSoC 2015

[Alex Lyblinskii]

Hi, i am third year student of St. Petersburg Polytechnic University, CS and Engineering, Lyblinskii Alexander
I am interested in project: "Support for multi-process applications". I have an experience writing C++ and also C applications, using POSIX threads, UNIX IPC, using and writing for Linux(usermode and small kernel modules).
I am already read introducing statement about Coz(https://web.cs.umass.edu/publication/docs/2014/UM-CS-2014-018.pdf), and now i have a general idea about how it works, also i started viewing the code and trying Coz on my Linux machine.
As i can see, causal profiling should be more usable for developers, and i think it is good idea to think about developers and their time, helping them to find place in code, needing their optimization efforts, cause time is the most valuable resource. It is really sounds cool.
Could you say, what exactly means coordination of experiments? How it should be look like?
As i understood, that means functionality similar for multi-threaded applications, but for multi-process applications?
How can i contact with mentor, or i should post to this thread, or irc\mail maybe?
I am appreciate your time, but if you can help me to get started quickly, please help(some statements, hints, links, books or simple advice from which point shall i start). I will be very grateful to you.
Sincerely, Alexander

[Charlie Curtsinger]

Alex,

Yes, by "coordination of experiments" I do mean extending the current functionality to multiple processes. If coz is going to work for multi-process applications, all the running processes will need to select a single section of code to virtually speed up, select some amount of virtual speedup, and begin/end the experiment simultaneously.

It would be up to you to propose a design to support multiple processes. Your design should probably use shared memory because inter-process communication will be very frequent. One source of complexity is with multiple executables. Most processes in the Chromium browser run the same executable (the browser itself) some processes run plugins or background services. During an experiment that virtually speeds up a line of code in the flash plugin executable, Coz should not virtually speed up any code in the browser's main executable.

It's okay if you don't know exactly how to solve all these issues ahead of time. We will work together to come up with a good solution and address issues as you encounter them. The best thing you can do for your proposal is to try to anticipate as many potential issues as possible, and propose a design that is flexible enough to accommodate other issues as we find them.

Don't worry if you aren't comfortable with POSIX processes or parallel programming. Some of the proposed tasks will not require major changes to coz. All of the tasks will require you to run programs with coz and produce profiles, so that's the best way to get started.

Starter Task

Collect a causal profile for pbzip2 (http://compression.ca/pbzip2/). You will need to find an appropriate place to insert a progress point, compile the program from source (be sure to include debug symbols), run the program with coz a few times, and produce a profile graph. You can add a progress point by including "causal.h" from the coz repository and inserting CAUSAL_PROGRESS into the program's code. You should put this point somewhere in the program that you would like to happen more frequently, like just after pbzip2 finishes compressing a block of data.

You should experiment with coz's command line options for profiling scope to control where coz will try virtual speedups (see `coz run --help`). By default, coz will evaluate the effect of optimizations only in pbzip2 itself. You can tell coz to evaluate potential optimizations in libbz2 using profiler scope. Be sure to install debug symbols for libbz2 on your machine (on Ubuntu/Debian the command is `apt-get install libbz2-dbg`).

To complete this task:

1. Build coz on your machine

2. Insert a progress point in pbzip2

3. Build pbzip2 with debug information

4. Collect a causal profile from pbzip2 (may take quite a few runs)

5. Graph the causal profile results using coz

6. (BONUS) Collect and graph another causal profile for pbzip2, this time including potential optimizations in libbz2 instead of just the pbzip2 executable.

Keep me posted on your progress, and let me know if you run into any issues.

--
You received this message because you are subscribed to the Google Groups "PLASMA at UMass: Google Summer of Code" group.
To unsubscribe from this group and stop receiving emails from it, send an email to plasma-umass-g...@googlegroups.com.
To post to this group, send email to plasma-u...@googlegroups.com.
Visit this group at http://groups.google.com/group/plasma-umass-gsoc.
To view this discussion on the web visit https://groups.google.com/d/msgid/plasma-umass-gsoc/d0981e8b-1b81-49c5-89b4-62a15ef064cf%40googlegroups.com.

[Chak Pong Chung]

I looked at your starter task,it said sth like

"git clone git://github.com/ccurtsinger/libelfin", not the "coz"

I thought we are supposed to work on the coz and build it.

Is it really the way you want us to do?

Thanks,

Chak-Pong

[Dan Barowy]

Charlie is traveling internationally at the moment, so his responses will probably be a little delayed.

I believe that you are looking at the section in the coz README about building dependencies for coz.  Yes, you will need to install all of the coz dependencies before you can build coz.  Some of those dependencies are on GitHub, and it looks like Charlie uses his own fork of libelfin, so you’ll need to use the version he points to.  Once you’ve done that, clone the coz repository and then make coz.

For reference, the starter task is here:

https://github.com/plasma-umass/coz/wiki/Coz-Starter-Task-1-%28GSoC-2015%29

And the coz repository is here:

g...@github.com:plasma-umass/coz.git

Dan

To view this discussion on the web visit https://groups.google.com/d/msgid/plasma-umass-gsoc/cd6612e3-20d0-43e6-b50e-dd4d3a278aa8%40googlegroups.com.

[Chak Pong Chung]

Thank you for your reply. I installed it already.That ggplot took more dependencies than I expected.

The new problem is when I add the macro "CAUSAL_PROGRESS" to the file "BZ2StreamScanner.cpp" under pbzip2 at the line 2739(between "

for (;;)

{ " and 

"if (syncGetTerminateFlag() != 0)",

that is :

for (;;)

{

CAUSAL_PROGRESS;

if (syncGetTerminateFlag() != 0)

then I make the project pbzip2 again, then run "coz run --- ./pbzip2 tanGE" ,where "tanGE" is just a big .txt file

it generated a "profile.coz".

Then I run "coz plot", it shows me the following:

cpchung@ubuntu:~/coz/benchmarks/pbzip2$ coz plot 

Traceback (most recent call last):

  File "/home/cpchung/coz/bin/coz", line 5, in <module>

    pycoz.run_command_line()

  File "/home/cpchung/coz/bin/pycoz/__init__.py", line 31, in run_command_line

    args.func(args)

  File "/home/cpchung/coz/bin/pycoz/coz_plot.py", line 35, in run

    data = pandas.DataFrame(p.to_records(), columns=p.get_columns())

  File "/usr/local/lib/python2.7/dist-packages/pandas/core/frame.py", line 264, in __init__

    copy=copy)

  File "/usr/local/lib/python2.7/dist-packages/pandas/core/frame.py", line 399, in _init_ndarray

    return create_block_manager_from_blocks([values.T], [columns, index])

  File "/usr/local/lib/python2.7/dist-packages/pandas/core/internals.py", line 3534, in create_block_manager_from_blocks

    construction_error(tot_items, blocks[0].shape[1:], axes, e)

  File "/usr/local/lib/python2.7/dist-packages/pandas/core/internals.py", line 3515, in construction_error

    passed,implied))

ValueError: Shape of passed values is (0, 0), indices imply (7, 0)

What could be the problem? I also looked into the file "profile.csv", there is no any numerical value but some column names.

Could you provide some help on this?

cpchung

[Charlie Curtsinger]

Chak-Pong,

You may not have enough samples to plot. If you run `coz process` does it produce a csv with only headings? By default `coz plot` will only show data for source lines with a minimum number of different speedups measured. You should run `pbzip` for at least 5-10 minutes to collect enough profile data. One run, even on a large input, is typically not enough.

- Charlie

To view this discussion on the web visit https://groups.google.com/d/msgid/plasma-umass-gsoc/cef47e37-fd64-4a20-b897-a70c162f24f7%40googlegroups.com.

[Chak Pong Chung]

"it produce a csv with only headings?"--------------Yes!

"run `pbzip` for at least 5-10 minutes to collect enough profile data." ----------once the to-be-zipped

 txt file is fixed, how can I control how many minutes the 'pbzip' will run? I dont understand what you meant.

Chak-Pong 

[Charlie Curtsinger]

Chak-Pong,

You can just re-execute `coz run --- pbzip2 <args>` repeatedly. Coz will append new profile results to the end of the existing profile.coz file. This utility may be helpful: http://www.stefanoforenza.com/how-to-repeat-a-shell-command-n-times/

- Charlie

To view this discussion on the web visit https://groups.google.com/d/msgid/plasma-umass-gsoc/c35b214e-6f41-4c52-984d-b84fe1ad44ad%40googlegroups.com.

[Chak Pong Chung]

it has a new problem....since the 'pbzip2' will zip the file to '.bz2' and delete the original one, so there is no easy way to 'repeatedly' run 'pbzip2' since there is no more input file after the first run....

instead, I run the wordcount problem, which is " watch -n 1 coz run --- ./word_count misc.html 

", where 'misc.html' is just a arbitrary '.html' file containing some words.

"watch -n 2" just run it every 2 seconds (1 second could be too short to finish the count, it is working when I just watch a random number generator in c++ and output the result. you can try the 'watch ' command with my attached 'rand.out' by executing "watch -n 2 ./rand.out" )

but when I applied the watch method with "watch -n 2 coz run --- ./word_count misc.html", it seems to have error shown as:

Every 3.0s: coz run --- ./word_count misc.html          Sat Mar 14 10:31:43 2015

^[34m[libcausal.cpp:99] ^[01;34mIncluding MAIN, which is /home/cpchung/coz/bench

marks/word_count/word_count^[0m

^[34m[inspect.cpp:317] ^[01;34m/home/cpchung/coz/lib/libcausal.so is not in scop

e^[0m

^[34m[inspect.cpp:317] ^[01;34m/lib/x86_64-linux-gnu/librt-2.19.so is not in sco

pe^[0m

^[34m[inspect.cpp:317] ^[01;34m/usr/local/lib/libdwarf++.so is not in scope^[0m

^[34m[inspect.cpp:317] ^[01;34m/usr/local/lib/libelf++.so is not in scope^[0m

^[34m[inspect.cpp:317] ^[01;34m/lib/x86_64-linux-gnu/libc-2.19.so is not in scop

e^[0m

^[34m[inspect.cpp:317] ^[01;34m/lib/x86_64-linux-gnu/libgcc_s.so.1 is not in sco

pe^[0m

^[34m[inspect.cpp:317] ^[01;34m/lib/x86_64-linux-gnu/libm-2.19.so is not in scop

e^[0m

^[34m[inspect.cpp:317] ^[01;34m/lib/x86_64-linux-gnu/ld-2.19.so is not in scope^

[0m

^[34m[inspect.cpp:317] ^[01;34m/lib/x86_64-linux-gnu/libdl-2.19.so is not in sco

pe^[0m

^[34m[inspect.cpp:317] ^[01;34m/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19 is

not in scope^[0m

^[34m[inspect.cpp:499] ^[01;34mIncluded source file /home/cpchung/coz/benchmarks

/word_count/sort-pthread.c^[0m

I see the "profile.coz" is updating but "profile.csv" has nothing but some column names, so I cant coz plot.

What could the problem? (Sorry, I dont know why the "repeat" function doesnt work on my computer so I use watch.)

Chak-Pong

[Chak Pong Chung]

Hi Charlie,

Finally I can use the website you suggested to repeatedly run the "repeat 100000000 coz run --- ./word_count misc.html"

I run "watch -n 1 tail profile.coz", and it shows(of course it is changing when I first execute the above  "repeat")

'

startup time=1426381590056086034

shutdown        time=1426381590061876673        samples=0

runtime time=10198384

'

Is it the result you expected? After 20 mins, I try to "coz plot" in the same folder, it still shows the error message I posted before.

Looking forward to your answer.

Chak-Pong

[Chak Pong Chung]

I am surprised that when I try the case of "comsumer producer", it worked and the "coz plot" is working. Repeatedly run the "consumer producer" can generate the "profile.coz" and "coz plot" is also ok~~~

I can also add the "casual" macro after or in between some of the code, it is working....

Why it doesnt work for the case of "word count" and "pbzip2"?  I run them for over an hour~~~

[Charlie Curtsinger]

Chak-Pong,

The word count example doesn't run long enough to complete any performance experiments. The pbzip2 benchmark could have the same issue, or you may have placed the progress point in code that never executes. You should run these programs on larger inputs so a single run will last at least 5-10 seconds. The profile.coz file should have `experiment` lines in addition to the `startup` and `shutdown` lines you were seeing. The log you posted shows that a run did not collect any samples, so you must be running the program for under half a second.

- Charlie

- Charlie

To view this discussion on the web visit https://groups.google.com/d/msgid/plasma-umass-gsoc/165b5511-6702-42e8-a597-2bc7aec2f352%40googlegroups.com.

[Chak Pong Chung]

On Sunday, March 15, 2015 at 11:19:19 AM UTC-4, Charlie Curtsinger wrote:

Hi Charlie,

Thank you so much for pointing this out!! Now I try to zip a larger file(192MB) with pbzip2 and then repeatedly run it with the "repeat" way you suggested, it is working.

You can see the coz plot and the profile.coz result in the link below:

https://www.dropbox.com/sh/066of3phz25hnx2/AAA9DND8c1VY0SHGXebiiJcea?dl=0

For the bonus part, could I pick a project I am more familar with? My background is more on scientific computation and have experience in parallel computation with openMP and MPI.

Should I start writing the proposal now? I am afraid we wont have much time left before March 27th, the deadline to submit the proposa.

[Charlie Curtsinger]

Chak-Pong,

Yes, absolutely! Please try any other applications you would like. Just make sure you try the scope options in coz and that you understand what they are doing. Coz will log to console when it finds debug information and includes source lines from specific files as places for potential optimizations.

Your causal graphs look a little sparse. Maybe you could use the `--fixed-line` argument to coz to get more samples for a few of the lines?

And yes, please do get a start on the proposal. Be sure to explain a plan for how separate processes will select a line to virtually speed up and start and end experiments at the same time. What will you do if the multi-process program has some processes running a different executable? How will you handle inter-process communication via pipes, signals, shared memory, etc. (see the tech report section on thread synchronization for how we deal with explicit thread communication already). Addressing all of these issues is probably not possible in just one summer: just be explicit about what problems you plan to solve. I'm happy to look at the proposal draft when you have something completed.

- Charlie

To view this discussion on the web visit https://groups.google.com/d/msgid/plasma-umass-gsoc/fca2cfe7-2d0b-4085-b14d-1e8cd8a5715e%40googlegroups.com.

[Chak Pong Chung]

Hi Charlie,

I am working on the proposal and understanding your tech report.

I think the key insight is clear by showing one example:

int main()

{

 a();      //take 1 second

b();//take 2 seconds

c();// take 3 seconds

}

So the total run time will be 6 seconds, if "c()" is totally eliminated, the percentage of  run-time decreased is (1+2+0)/6=50%(actual speed-up).If we delay 3 seconds before "c()"(or double the run-time of c()  ), then the percentage of  run-time increased is ( 3 ) / 6=50%(virtual speed-up). So the relative percentage change is the "same".

I dont understand smt about your last question " how separate processes will select a line to virtually speed up and start and end experiments at the same time. What will you do if the multi-process program has some processes running a different executable? How will you handle inter-process communication via pipes, signals, shared memory, etc. "

(1) why you want to "start and end experiments at the same time" and " if the multi-process program has some processes running a different executable". It seems to me that if the first problem is done, how come the second situation will happen? I have ideas how to handle the first one  or do it asynchronously(I think it will be useful to do it asynchronously)

(2) more after you answer the first one

Chak-Pong

[Chak Pong Chung]

sth added:

One possible scenarios is we dont use all the available resources(memory or available cores) to concentrate on evaluating one program.In this case there will be no contradiction mentioned.But is it really good not to concentrate all resources to get one program profiling done first, there will be trade-off.

[Charlie Curtsinger]

Chak-Pong,

Your example speedup is reasonable, but single threaded programs are a simple base case for coz. You should be sure to work through some example parallel programs as well.

All processes in a group need to agree on which line of code is being virtually sped up. Communication mechanisms like pipes can block threads, and blocking requires special handling in coz. You can see my direct email for an example where this would be appropriate, or read about blocking in the coz tech report.

- Charlie

To view this discussion on the web visit https://groups.google.com/d/msgid/plasma-umass-gsoc/6ac57300-674f-4bb5-99a9-3c5964ab8365%40googlegroups.com.