Merge segmented pb data into a whole

[Wanfeng Ma]

I need to obtain Perfetto native heap data by heap profiler script on a low-end machine (Android system is greater than 11), but directly record the raw-trace data, such as 30 minutes of data during the battle, the acquisition of data will fail (maybe the amount of data is too large)。

only use increase --shmem-size was not successful, so I consider using the continuous-dump parameter to stream capture, but the acquired data is a raw-trace file and a lot of pb files.

Open the raw-trace file in Perfetto UI and find that it can only obtain data within a short period of time. For example, during the entire test process, a total of 1000MB of memory is allocated, but the data obtained every 10ms is only about 1MB. 

Is there have a methed that can be used to convert the raw-trace file to the raw-race file that can import Perfetto UI display gen all allocated and released memory from the beginning to the end and remove every small interval in the middle?

The reason why I have this requirement is that, for example, in a 5v5 battle for 18 minutes, I only care about what memory is reserved during the whole process from the beginning to the end, and the current situation may cause the memory allocated in the previours 10ms interval to be released  in later 10ms，Is there a tool or instruction to solve this problem, thank you!

[Daniele Di Proietto]

Once you capture a heap profile (with ./tools/heap_profile, I assume) you get:

1. A raw-trace file (can be opened in the perfetto UI)

2. A bunch of .pb files (that can be visualized with speedscope). One for each continuous-dump

The .pb files are generated on the host from the raw-trace file and don't contain any information that's not in the original raw-trace file.

I'm not aware of any tool to merge different continuous dumps into a single one.

If you only care about the end result after 30 minutes, you should just disable continuous dumps and let the heap profiler only capture everything at the end. Enabling continuous dump will actually generate much more data. If the profiler doesn't work with a single dump for 30 minutes, it will not work with continuous dumps for 30 minutes.

I guess we need to understand why the heap profiler is not able to capture a heap dump after 30 minutes.

* What exactly is the problem?

* Did you find error messages from perfetto/heapprofd while you're capturing the profile?

* Have you tried increasing `size_kb: 63488` to something bigger in `tools/heap_profile`?

* Have you tried on a more recent android version, by chance?

Sorry, I'm not sure this helped. Let me know if you have any other question

--
You received this message because you are subscribed to the Google Groups "Perfetto Development - www.perfetto.dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to perfetto-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/perfetto-dev/59304d22-1078-4725-bf63-a18d3c3067c1n%40googlegroups.com.

[Wanfeng Ma]

Sorry, Maybe massive data is the reason why can't capture native heap memory data，Following is some test case:

一. Test  shmem-size parameter and capture Time:

1. Parameters: Default   Process: Loading to Lobby(Total Process, approximately 60s)--Faild

2. Parameters: Default   Process: Loading to Lobby(Onle Capture 5 second)--Succeed

3. Parameters: --shmem-size 8388608(8MB)   Process: Loading to Lobby(Total Process, approximately 60s)--Faild

4. Parameters: --shmem-size 8388608(8MB)   Process: Loading to Lobby(Onle Capture 5 second)--Succeed

5. Parameters: --shmem-size 33554432(32MB)   Process: Loading to Lobby(Total Process, approximately 60s)--Faild

6. Parameters: --shmem-size 33554432(32MB)   Process: Loading to Lobby(Onle Capture 5 second)--Succeed

7. Parameters: --shmem-size 134217728(128MB)   Process: Loading to Lobby(Total Process, approximately 60s)--Faild

8. Parameters: --shmem-size 134217728(128MB)   Process: Loading to Lobby(Onle Capture 5 second)--Succeed

二. Test sample Interval parameter:

1. Parameters: --shmem-size 134217728(128MB) -i 64K   Process: Loading to Lobby(Total Process, approximately 60s)--Faild

2. Parameters: --shmem-size 134217728(128MB) -i 1MB   Process: Loading to Lobby(Total Process, approximately 60s)--Faild

3. Parameters: --shmem-size 134217728(128MB) -i 16MB   Process: Loading to Lobby(Total Process, approximately 60s)--Faild

三. Test phone Info:  andorid 11

The result can't find error messages from perfetto/heapprofd while capturing the profile, Increase shmem-size and sample interval can't solve the capture problem, only shorten capture time(less capture data) can succeed capture the raw-trace, is version not enough by android 11? can you give me some tips how to solve this problem, thank you! 

[Daniele Di Proietto]

Sorry, I'm not sure what the problem could be.

Let's see:

* If you record an `adb log` while capturing the trace, do you see any error messages related to perfetto/heapprofd? (maybe grep for perfetto or heapprofd).

* Have you tried increasing `size_kb: 63488` to something bigger in `tools/heap_profile`?

* Can you try with a more recent android version? (Not sure it's going to help)

To view this discussion on the web visit https://groups.google.com/d/msgid/perfetto-dev/50fdb40d-4d70-42ed-b3fc-82195690154en%40googlegroups.com.

[Wanfeng Ma]

I record logcat info while capture the trace，Following are the case:

1.  Set shared memory paremeter to 16MB， Record Total Process of Loading to Lobby, can't capture raw-trace .

2.  Set shared memory paremeter to 32MB， Record Total Process of Loading to Lobby, can't capture raw-trace .

3. Set shared memory paremeter to 32MB， But Only Record the Process of Loading to Lobby 10 second, can capture raw-trace successfully.

4. Set shared memory paremeter to 63488KB(62MB)， Command paramter error, can't capture raw-trace .

5. Set shared memory paremeter to 64MB， Record Total Process of Loading to Lobby, can't capture raw-trace .

I compare set shared memory paremeter to 32MB but capture time with 10s or total process, find logcat have the different is faild to write shared ring buffer. 

Base above mention infomation, can you give me some tips how to solve the problem? thank you!

[Daniele Di Proietto]

Thanks for sharing!

We can understand a little bit more about what's happening from the log.

Your app (I assume pid 26010) tries to connect to the central heap profiler, but fails.

This is expected, it's a user (not userdebug) build: before android 12, apps could not connect to the central heap profiler.

07-11 21:25:10.915 I/perfetto(26010): malloc_hooks.cc:231 Constructing client for central daemon.
07-11 21:25:10.915 E/perfetto(26010): client.cc:121 Failed to connect to /dev/socket/heapprofd (errno: 13, Permission denied)
07-11 21:25:10.915 E/perfetto(26010): malloc_hooks.cc:238 Failed to connect to /dev/socket/heapprofd. This is benign on user builds.
07-11 21:25:10.915 I/perfetto(26010): malloc_hooks.cc:247 Setting up fork mode profiling.

So the app resorts to the "fork mode": it forks a process (pid 26036) that acts as its heap profiler:

07-11 21:25:11.022 I/perfetto(26036): approfd_producer.cc:166 Connected to the service, mode [child].
07-11 21:25:11.034 I/perfetto(26010): malloc_hooks.cc:461 com.levelinfinite.sgameGlobal: heapprofd_client initialized.

Now, as the heap profile grows, the forked process consumes more memory to keep track of everything. At some point some system daemon decides to kill the forked process:

07-11 21:25:42.108 I/Athena  ( 5003): MemoryGuardAction: found orphan process [26036][1][10322]heapprofd(0)

07-11 21:25:42.110 I/Athena  ( 5003): OppoClearSystemService : K [26036][10322][heapprofd(0), reason: 71

Immediately after that, your app reports the failure to write to the heap profiling ring buffer:

07-11 21:25:42.140 E/perfetto(26010): client.cc:393 Failed to write to shared ring buffer. Disconnecting. (errno: 104, Connection reset by peer)
07-11 21:25:42.140 E/perfetto(26010): client.cc:393 Failed to write to shared ring buffer. Disconnecting. (errno: 11, Try again)

In order to fix your problem:

1. Perhaps you could find a way to disable OppoClearSystemService? I'm afraid I can't help you much there, I think this is something that's Oppo specific.

2. You can upgrade to Android 12. On Android 12, apps are allowed to connect to the "central" heap profiling daemon. "fork mode" has been removed. This should hopefully allow you to capture longer heap profiles.

--

You received this message because you are subscribed to the Google Groups "Perfetto Development - www.perfetto.dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to perfetto-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/perfetto-dev/c1981ce0-7397-4932-a4c5-684c6690b1b4n%40googlegroups.com.

[Wanfeng Ma]

I bought a new android low-end phone with android version 13, it can work,  thank you very mach!

[Wanfeng Ma]

hello, can you tell me some more detail information about perfetto heapprofd shared memory? 

1. Whether the maximum size of shared memory in defferent mobile phones is fixed?
2. Whether exist a command that can print supported maximum  shared memory size by any android mobile phone?

在2023年7月11日星期二 UTC+8 23:15:47<Daniele Di Proietto> 写道：

[Daniele Di Proietto]

On Wed, Jul 12, 2023 at 7:26 AM Wanfeng Ma <wanf...@gmail.com> wrote:

hello, can you tell me some more detail information about perfetto heapprofd shared memory? 

1. Whether the maximum size of shared memory in defferent mobile phones is fixed?

I don't know. Each phone manufacturer can make arbitrary changes.

 

2. Whether exist a command that can print supported maximum  shared memory size by any android mobile phone?

I'm not aware of any.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/perfetto-dev/cccb3ccb-e063-4f93-876e-1a8affc0d92en%40googlegroups.com.