FuseFileSystem readFileAtPath function issue

[陳維廷]

Hi,

I found a problem of readFileAtPath function in my Fuse filesystem App . When  I play video or read file in my Fuse filesystem mounted path , I found the  used memory increased as file size. I am sure that there isn't any memory leakage  over my App coding. Is the behavior cache ? Do you have any idea about that?  Would you please give me some advice to handle it ? 

thanks.

Henry

 

  

[Sam Moffatt]

Hi Henry,

I'm more familiar with the C API rather than the Obj-C one however it should be requesting data in chunks (size and offset) similar to the C API. If you're not properly free'ing the memory you're using to read the chunks or you're loading the full file in your underlying backing store then you're going to see the situation where your calls use up memory.

Perhaps you can expand on what your file system is doing, where it's sourcing it's data and if possible some sample code?

Cheers,

Sam

Sam Moffatt
http://pasamio.id.au

--
You received this message because you are subscribed to the Google Groups "OSXFUSE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osxfuse-group+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[陳維廷]

Hi Sam Moffatt,

Thank you for your response. I would like to raise an example to introduce my way of implement readFileAtPath function.

Example:copy a mpeg4 video(file size=700MB) from Fuse filesystem mounted path to local folder

Step1. go to readFileAtPath function.

1.do chunk download  data of mpeg4 video from sever.(use NSURL method)

2.store NSData into allocated memory

3.do memcpy:from allocated memory to buffer.

4.return size

Step2: go to releaseFileAtPath function after you have finished downloading

1.remove all NSData  in allocated memory

Problem:

I found  that my Fuse App already used about 700 MB space of memory, but I had removed all NSData in allocated memory.

Therefore, I think those used memory are system cache.

Samuel Moffatt於 2016年12月23日星期五 UTC+8上午2時49分13秒寫道：

Sam Moffatt
http://pasamio.id.au

To unsubscribe from this group and stop receiving emails from it, send an email to osxfuse-grou...@googlegroups.com.

[Sam Moffatt]

My read of what you wrote is that at least during the download, you’re going to have the 700MB in memory until you hit the releaseFileAtPath when you clean it all up. I’d personally aim to stream anything larger than 1MB or so to a temporary directory on disk and then co-ordinate reads off disk. To do that would require some sort of synchronisation between your download thread and any reads that occur in OSXFUSE but at that point you’d do an on disk read and return that avoiding storing too much in memory.

Of course you could download each chunk as necessary but I feel that’s going to be a bit wasteful on your overhead as you’re likely not going to get really large reads from the VFS layer. By shifting the download process to some sort of a stream to a local disk cache (particularly if you’re looking at doing large files) then you’ll likely solve your memory problem.

Cheers,

Sam

Sam Moffatt
http://pasamio.id.au

To unsubscribe from this group and stop receiving emails from it, send an email to osxfuse-group+unsubscribe@googlegroups.com.

[陳維廷]

Hi Sam Moffatt,

Sorry, I did not say my example clearly. I'd like to add some explanation.

1.In the readFileAtPath function:

I only allocated about 60MB memory space for storing  downloaded data, and reuse those 60MB memory space.

Problem:

1.Because  I only allocated  60MB memory for download, the App used memory are more than 700MB.

2.Even if I clear  all object used to store downloaded data in the  releaseFileAtPath function, those used memory didn't decrease.

3.I must close my Fuse APP to clear up used memory.

As mentioned above, I think  system cache caused the situation, not my App itself.

Thanks 

Samuel Moffatt於 2016年12月23日星期五 UTC+8下午1時02分13秒寫道：

[Sam Moffatt]

I’d suggest getting a memory profile/debugger to check out where your memory allocations are going. Without having all of the code you’re using here, it’s hard to exactly tell what’s going on. Part of me wonders if there is a pool somewhere that is holding onto memory for you that you’re not expecting.

Sam Moffatt
http://pasamio.id.au

To unsubscribe from this group and stop receiving emails from it, send an email to osxfuse-group+unsubscribe@googlegroups.com.

[陳維廷]

Sorry about that. Because I can't find any error in profile/leaks instrument, I ask  for help here.
I will provider my Fuse code to you  as soon as possible. Thank you for your help.

Samuel Moffatt於 2016年12月23日星期五 UTC+8下午2時33分45秒寫道：