High CPU usage when writing large files

[Jimmy Tanzil]

I am having high CPU load issues when writing large files (above 5 GB) into the mounted s3ql filesystem.

Our customers backup data can be as large as 100 GB per file.

I am on m1.large EC2 running Ubuntu 14.04 LTS instance for testing.

I also use the --plain option for mkfs.s3ql to try reduce cpu load, but it's still high. Is this normal?

My mount command: mount.s3ql --compress none --debug all --allow-other s3://bucket /backup15

I got: Using 4 upload threads. Autodetected 4052 file descriptors available for cache entries Using cached metadata. Setting cache size to 42525 MB Mounting filesystem...

The mount.logs for transferring 2 test files: http://54.185.24.96/mount.log http://54.185.24.96/mount.log.1 http://54.185.24.96/mount.log.2http://54.185.24.96/mount.log.3 http://54.185.24.96/mount.log.4 http://54.185.24.96/mount.log.5

Can you please suggest me the proper mount or mksfs command to avoid high CPU usage. The CPU was maxed out just to transfer these 2 test files.

I already use compress none options and --plain option for mkfs, doesn’t seem to help.

I am concern as we are transferring thousands of files on our productions server and many of them are over 5 GB size per file.

[Nikolaus Rath]

Jimmy Tanzil <jimmy....@ichthysmedia.com> writes:
> I am having high CPU load issues when writing large files (above 5 GB) into
> the mounted s3ql filesystem.

Could you quantify that? How do you measure it, and how does it depend
on file size? Is there really a jump at 5 GB, or is it rather a smooth
increase?

> My mount command: mount.s3ql --compress none --debug all --allow-other
> s3://bucket /backup15

Using --debug all is certainly a way to eat up a lot of CPU time. But I
assume the problem also happens with debugging disabled?


Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Jimmy Tanzil]

Hi Nikolaus,

When writing less than 100 MB data, it takes only a few seconds and it finished very quick. But when the file size is larger, the CPU usage keeps climbing and stays there until the file write is complete.

I created a 2 minute video to show you what I mean:

http://youtu.be/-HMac-Hr3bw

Also, this test does not use debug all. The mkfs is using plain option and the mount is using compression none option.

The first file copied in the video is 80 MB in size

The second file copied in the video is 5 GB in size

Is this normal? I am trying to test this using c3.x2large instance today, want to see how it perform transferring many large files using this instance.

Please let me know your insights if there is a work around to reduce the cpu load even more?

Thank you!

--
You received this message because you are subscribed to a topic in the Google Groups "s3ql" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/s3ql/Ts0jgzWRQfI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to s3ql+uns...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Kind Regards,
Jimmy R. Tanzil

[Nikolaus Rath]

Hi Jimmy,

When replying to emails on this list, please do not put your reply
above the quoted text, and do not quote the entire message you're
answering to. This makes it unnecessarily hard for other readers to
understand the context of your email. Instead, please cut quoted parts
that are not relevant to your reply, and insert your responses right
after the points you're replying to (as I have done below). Thanks!

Jimmy Tanzil <jimmy....@ichthysmedia.com> writes:
>>> I am having high CPU load issues when writing large files (above 5
>>> GB) into the mounted s3ql filesystem.
>>
>> Could you quantify that? How do you measure it, and how does it depend
>> on file size? Is there really a jump at 5 GB, or is it rather a smooth
>> increase?
>

> When writing less than 100 MB data, it takes only a few seconds and it
> finished very quick. But when the file size is larger, the CPU usage
> keeps climbing and stays there until the file write is complete.
>
> I created a 2 minute video to show you what I mean:
> http://youtu.be/-HMac-Hr3bw

Sorry, that's not helpful at all. Please provide quantitative data in
text form.

> The first file copied in the video is 80 MB in size
> The second file copied in the video is 5 GB in size

How long does each copy take? Do both files with into the local cache?
Are taking into account the time it takes to flush the cache?

> Is this normal?

Is what normal?

[Jimmy Tanzil]

> Is what normal?

The CPU being utilized more than 100% during writing larger files.

My concern is not the timing, the timing is excellent. 

I am more concerned about the CPU showing above 100% when writing one 5 GB file.

Is there any other tricks that can be done to lower that?

Thank you Nikolaus!

[Jimmy Tanzil]

An update, I just tested using c3.2xlarge and CPU usage is well below 20%. This is very good.

I am in the process of copying 15 TB data from one of the production server to this one. So far looking so good.

Will update you how it goes after switching this server to production and starts writing backups from live data.

Thank you.

[Nikolaus Rath]

I do not understand your concern. Why would you want to artifically slow
down S3QL by forcing it not to use all the available CPU cycles?

You can add a lot of sleep(1) statements to the write() operation in
fs.py. That should bring CPU utilization down to < 1 %.

[Jimmy Tanzil]

My concern that it might crash the file system during busy writes.

I tried it out last night, the server was writing approximately 4,000 files into the file system. 

I got File system appears to have crashed when checking the server this morning.

mount.log is at http://50.112.251.132/mount.log

I am thinking of doing my backups into an EBS drive, then rsync it to the s3ql file system afterwards to avoid file system crashing when doing thousands multiple file writes at the same time.

These 4,000+ files are being written at the same time as we backup hundreds of servers into the same location using s3ql file system.

Do you have any pointers to writing thousands of large files into s3ql simultanesouly over night without crashing? 

--
You received this message because you are subscribed to a topic in the Google Groups "s3ql" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/s3ql/Ts0jgzWRQfI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to s3ql+uns...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nikolaus Rath]

Hi Jimmy,

When replying to emails on this list, please do not put your reply
above the quoted text, and do not quote the entire message you're
answering to. This makes it unnecessarily hard for other readers to
understand the context of your email. Instead, please cut quoted parts
that are not relevant to your reply, and insert your responses right
after the points you're replying to (as I have done below). Thanks!

Jimmy Tanzil <jimmy....@ichthysmedia.com> writes:

>> > I am more concerned about the CPU showing above 100% when writing one 5
>> > GB file.
>> >
>> > Is there any other tricks that can be done to lower that?
>>
>> I do not understand your concern. Why would you want to artifically slow
>> down S3QL by forcing it not to use all the available CPU cycles?
>

> My concern that it might crash the file system during busy writes.

How is crashing related to CPU activity?

> I got *File system appears to have crashed* when checking the server this

> morning.
>
> mount.log is at http://50.112.251.132/mount.log

There is no indication of any crash (nor any other problem) in the log
file at all. Where did you get the "File system appears to have crashed"
message from?


Best,
-Nikolaus

PS: Please remember the first paragraph when replying.

[Jimmy Tanzil]

There is no indication of any crash (nor any other problem) in the log

file at all. Where did you get the "File system appears to have crashed"
message from?

I see it when I try to do umount.s3ql /mountpoint

Then I read on your documentation to just do umount /mountpoint which works

I then did fsck.s3ql which completes under 1 minute, then I can mount it back successfully.

I tried to unmount it in the first place because I cannot access the mounted file system. If I do ls or any command including s3qlstats it just sits there without any response. So then I tried unmounting it and see that message.

Thank you.

[Nikolaus Rath]

Jimmy Tanzil <jimmy....@ichthysmedia.com> writes:
>>
>> There is no indication of any crash (nor any other problem) in the log
>> file at all. Where did you get the "File system appears to have crashed"
>> message from?
>>
> I see it when I try to do umount.s3ql /mountpoint
> Then I read on your documentation to just do umount /mountpoint which works
>
> I then did fsck.s3ql which completes under 1 minute, then I can mount it
> back successfully.
>
> I tried to unmount it in the first place because I cannot access the
> mounted file system. If I do ls or any command including s3qlstats it just
> sits there without any response.

Is the mount.s3ql process still running? (check with ps)

If so, can you obtain a stack trace
(https://bitbucket.org/nikratio/s3ql/wiki/Providing%20Debugging%20Info)?



Best,
-Nikolaus

[Jimmy Tanzil]

Is the mount.s3ql process still running? (check with ps)

No it was not running when I checked with ps. Because I see it’s not running, I tried to unmount it and I saw that “Filesystem appears to have crashed”.

Right now my EC2 instance have 8 cores, but it seems s3ql only utilize a maximum of 50% of the CPU, can’t use more than that.

Is there an option on the mount command to utilize 80% of the available CPU cores?

Thank you.

[Nikolaus Rath]

On 07/20/2014 04:06 PM, Jimmy Tanzil wrote:
> Is there an option on the mount command to utilize 80% of the available
> CPU cores?

No, and I don't think that introducing it would make sense.