[Lustre-discuss] Lustre 2.0 client cache size

[DEGREMONT Aurelien]

Hello

I'm trying to control the amount of cache memory used by Lustre clients
with Lustre 2.0
There is a /proc tuning called "max_cache_mb" in llite which is
documented in Lustre manual.
Unfortunately, after testing it and checking the source code it seems
this variable is present but it not used anymore in Lustre code.
This is not the case in 1.8

I did not find if this was removed or this was partially included in
Lustre 2.0.
What's the current status of this and how can I tell to my client to
avoid caching too many data?
Those clients do a lot of read and write in Lustre filesystems but
thoses files will not be re-read soon, so it is useless to fill memory
with it. Moreover, when the client memory is full, Lustre performance
are really impacted.

Thanks in advance

Aurélien Degrémont
_______________________________________________
Lustre-discuss mailing list
Lustre-...@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Andreas Dilger]

On 2011-03-17, at 6:37 AM, DEGREMONT Aurelien wrote:
> I'm trying to control the amount of cache memory used by Lustre clients
> with Lustre 2.0
> There is a /proc tuning called "max_cache_mb" in llite which is
> documented in Lustre manual.
> Unfortunately, after testing it and checking the source code it seems
> this variable is present but it not used anymore in Lustre code.

> This is not the case in 1.8.

>
> I did not find if this was removed or this was partially included in
> Lustre 2.0.
> What's the current status of this and how can I tell to my client to
> avoid caching too many data?

The client VM usage was one of the areas that was completely rewritten by Nikita for 2.0, so it was likely this functionality was lost at that time. I don't have any idea at this time how hard it would be to restore.

> Those clients do a lot of read and write in Lustre filesystems but
> thoses files will not be re-read soon, so it is useless to fill memory
> with it. Moreover, when the client memory is full, Lustre performance
> are really impacted.

It may be possible to use fadvise(FADV_NOREUSE) from the application to cause the VM to discard these pages after the first use.

Cheers, Andreas
--
Andreas Dilger
Principal Engineer
Whamcloud, Inc.

[Oleg Drokin]

Hello!

On Mar 17, 2011, at 5:44 PM, Andreas Dilger wrote:
>> I did not find if this was removed or this was partially included in
>> Lustre 2.0.
>> What's the current status of this and how can I tell to my client to
>> avoid caching too many data?
> The client VM usage was one of the areas that was completely rewritten by Nikita for 2.0, so it was likely this functionality was lost at that time. I don't have any idea at this time how hard it would be to restore.

I wonder if the generic vm tunables could be used instead? I think that was the original plan.

>> Those clients do a lot of read and write in Lustre filesystems but
>> thoses files will not be re-read soon, so it is useless to fill memory
>> with it. Moreover, when the client memory is full, Lustre performance
>> are really impacted.
> It may be possible to use fadvise(FADV_NOREUSE) from the application to cause the VM to discard these pages after the first use.

Yes, that would totally make sense to do regardless of other methods.

Bye,
Oleg

[DEGREMONT Aurelien]

Oleg Drokin a écrit :

> Hello!
>
> On Mar 17, 2011, at 5:44 PM, Andreas Dilger wrote:
>
>> The client VM usage was one of the areas that was completely rewritten by Nikita for 2.0, so it was likely this functionality was lost at that time. I don't have any idea at this time how hard it would be to restore.
>>
>
> I wonder if the generic vm tunables could be used instead? I think that was the original plan.
>

So the issue is that 'max_cached_mb' tunable should be removed if it
makes no sense.

The default with the generic tunable is that you could really control
how they behave per filesystem.

>>> Those clients do a lot of read and write in Lustre filesystems but
>>> thoses files will not be re-read soon, so it is useless to fill memory
>>> with it. Moreover, when the client memory is full, Lustre performance
>>> are really impacted.
>>>
>> It may be possible to use fadvise(FADV_NOREUSE) from the application to cause the VM to discard these pages after the first use.
>>
> Yes, that would totally make sense to do regardless of other methods.
>

Hmm... I do not want to patch 'cp' or 'dd' :)

Aurélien

[Jason Rappleye]

On Mar 18, 2011, at 1:07 AM, DEGREMONT Aurelien wrote:

>>>> Those clients do a lot of read and write in Lustre filesystems but
>>>> thoses files will not be re-read soon, so it is useless to fill memory
>>>> with it. Moreover, when the client memory is full, Lustre performance
>>>> are really impacted.
>>>>
>>> It may be possible to use fadvise(FADV_NOREUSE) from the application to cause the VM to discard these pages after the first use.
>>>
>> Yes, that would totally make sense to do regardless of other methods.
>>
>
> Hmm... I do not want to patch 'cp' or 'dd' :)

You might want to have a look at this:

http://code.google.com/p/pagecache-mangagement/

We use it when copying crash dumps off of our OSSes, as we don't want to disturb cached metadata.

Jason

--
Jason Rappleye
System Administrator
NASA Advanced Supercomputing Division
NASA Ames Research Center
Moffett Field, CA 94035

[DEGREMONT Aurelien]

Jason Rappleye a écrit :

> On Mar 18, 2011, at 1:07 AM, DEGREMONT Aurelien wrote:
>
>>> Yes, that would totally make sense to do regardless of other methods.
>>>
>> Hmm... I do not want to patch 'cp' or 'dd' :)
>>
>
> You might want to have a look at this:
>
> http://code.google.com/p/pagecache-mangagement/
>
> We use it when copying crash dumps off of our OSSes, as we don't want to disturb cached metadata.
>

Interesting project. I will have a look. Thank you.

Aurélien

[Jay]

After checking 2.6.35 kernel source code, POSIX_FADV_NOREUSE actually doesn't do anything. So I don't know how it helps. Probably we should do POSIX_FADV_DONTNEED after reading?

Jay

[Alex Kulyavtsev]

We used POSIX_FADV_DONTNEED in distributed iozone test to clear cache
on slave client after initial write and before client reads back same
data. It helped to see real data rates instead of unrealistically
high read rate due do cacheing. Perhaps it was non-lustre (NFS) file
server.

If you do scripting, small executable like
http://www.citi.umich.edu/projects/asci/benchmarks.html (scroll
down to Clearcache)
can be called after "cp" or "dd."
Alex.

On Mar 19, 2011, at 12:47 AM, Jay wrote:

> After checking 2.6.35 kernel source code, POSIX_FADV_NOREUSE
> actually doesn't do anything. So I don't know how it helps. Probably
> we should do POSIX_FADV_DONTNEED after reading?
>
> Jay

> On Mar 18, 2011, at 1:07 AM, DEGREMONT Aurelien wrote:
>
>>> ... snip...

>> Hmm... I do not want to patch 'cp' or 'dd' :)