Increasing storage space

163 views
Skip to first unread message

James Royal-Lawson

unread,
Sep 16, 2016, 5:20:49 PM9/16/16
to s3backer-devel
Hi everyone,

I set up s3backer on my ubuntu server a few years ago and created a 250gb partition for storing backups on. I've filled that up now and I've been trying to find out how I can make it bigger. I've not found any instructions or information that I feel confident in following.

I wrote a blog post back then detailing what I did (mainly as a way of noting/remembering what I did and why!) which you can see here

I start s3backer with the following...

s3backer --blockSize=1024k --size=250g --listBlocks --vhost --ssl --timeout=250 --blockCacheThreads=3 --maxUploadSpeed=35k --accessFile=/etc/passwd-s3backer --prefix=s3 buggles /s3

I'm running an ext4 file system on top of that mounted as follows

mount -t ext4 -o loop /s3/file /backups

As far as I understand, I need to increase the size of the s3backer file and then increase the size of the ext4 partition. Is that correct? How do I do that without losing data?

/James

Archie Cobbs

unread,
Sep 16, 2016, 5:32:16 PM9/16/16
to s3backe...@googlegroups.com
At the s3backer level, increasing the total size of the s3backer "file" should be easy. Just restart s3backer with a new --size parameter (but the same --blockSize of course).

I've never actually done this. In any case you can easily test that this first step works: just add the --readOnly flag to s3backer, and then mount your filesystem read-only on top of that. Then verify you can still see all your files OK, etc.

If that works, then unmount the top level filesystem, restart s3backer without --readOnly, and then you should be able to run resize2fs to enlarge the ext4 filesystem.

Now this "should" all work but I'm only theorizing. You should test it all out on a throwaway filesystem first.

Report back and let us know how it works.

-Archie


--
You received this message because you are subscribed to the Google Groups "s3backer-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to s3backer-devel+unsubscribe@googlegroups.com.
To post to this group, send email to s3backer-devel@googlegroups.com.
Visit this group at https://groups.google.com/group/s3backer-devel.
For more options, visit https://groups.google.com/d/optout.



--
Archie L. Cobbs

James Royal-Lawson

unread,
Apr 22, 2017, 8:47:50 AM4/22/17
to s3backer-devel

Hi there! I haven't been able to test doing this earlier for a number of reasons - but today I finally got the chance!

I ran s3backer with --size=500g (instead of the usual --size=250g). I also included the --readOnly flag (I tried it later without, but with the same result)
Even though i'm specifying the size, s3backer is autodetecting the size and complaining that the filesystem size and s3backer file size don't match and exits... so I can't run resize2fs as s3backer isn't running.

root@srvr:/var/log# s3backer --readOnly --blockSize=1024k --size=500g --listBlocks --vhost --ssl --timeout=250 --maxUploadSpeed=100k --blockCacheThreads=6 --blockCacheFile=/tmp/s3backerCache --accessFile=/etc/xxx --prefix=s3 srvr /s3
s3backer: auto-detecting block size and total file size...
s3backer: auto-detected block size=1m and total size=250g
s3backer: error: configured file size 500g != filesystem file size 250g

Then I did a bit of checking and spotted in s3b_config.c that s3backer will return a warning rather than an error if I include the --force flag.

So I tried again with 

s3backer --force --blockSize=1024k --size=500g --listBlocks --vhost --ssl --timeout=250 --maxUploadSpeed=100k --blockCacheThreads=6 --blockCacheFile=/tmp/s3backerCache --accessFile=/etc/xxx --prefix=s3 srvr /s3

And I got further!

But before I could extend the filesystem with resize2fs I had to run: 

e2fsck -f /s3/file

It took about 20 mins or so for it to complete recoverig the journal - I was getting worried! But after that it progressed with the passes quite nicely.

Once that was complete I could do 

resize2fs -p /s3/file 

and then remount the enlarged filesystem.

/dev/loop0            493G  226G  242G  49% /backups

/James

On Friday, 16 September 2016 23:32:16 UTC+2, Archie Cobbs wrote:
At the s3backer level, increasing the total size of the s3backer "file" should be easy. Just restart s3backer with a new --size parameter (but the same --blockSize of course).

I've never actually done this. In any case you can easily test that this first step works: just add the --readOnly flag to s3backer, and then mount your filesystem read-only on top of that. Then verify you can still see all your files OK, etc.

If that works, then unmount the top level filesystem, restart s3backer without --readOnly, and then you should be able to run resize2fs to enlarge the ext4 filesystem.

Now this "should" all work but I'm only theorizing. You should test it all out on a throwaway filesystem first.

Report back and let us know how it works.

-Archie

On Wed, Sep 14, 2016 at 2:20 PM, James Royal-Lawson <ja...@ccl4.org> wrote:
Hi everyone,

I set up s3backer on my ubuntu server a few years ago and created a 250gb partition for storing backups on. I've filled that up now and I've been trying to find out how I can make it bigger. I've not found any instructions or information that I feel confident in following.

I wrote a blog post back then detailing what I did (mainly as a way of noting/remembering what I did and why!) which you can see here

I start s3backer with the following...

s3backer --blockSize=1024k --size=250g --listBlocks --vhost --ssl --timeout=250 --blockCacheThreads=3 --maxUploadSpeed=35k --accessFile=/etc/passwd-s3backer --prefix=s3 buggles /s3

I'm running an ext4 file system on top of that mounted as follows

mount -t ext4 -o loop /s3/file /backups

As far as I understand, I need to increase the size of the s3backer file and then increase the size of the ext4 partition. Is that correct? How do I do that without losing data?

/James

--
You received this message because you are subscribed to the Google Groups "s3backer-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to s3backer-deve...@googlegroups.com.
To post to this group, send email to s3backe...@googlegroups.com.



--
Archie L. Cobbs
Reply all
Reply to author
Forward
0 new messages