sdfs installation

[ramu]

Hi all,

Iam new to sdfs dedupfilesystem,please help me to installation steps and example for sdfs on ubuntu-12.04 .

Thanks,

Ramu.

[Pierluigi Ruffo]

http://www.opendedup.org/quickstart

[ramu]

Hi,

I saw that link but fuse is not installing,i downloaded fuse-2.9.1 and configure then make and makeinstall am getting this error

 /usr/bin/install -c -m 644 fusermount.1 ulockmgr_server.1 '/usr/share/man/man1'

test -z "/usr/share/man/man8" || /bin/mkdir -p "/usr/share/man/man8"

 /usr/bin/install -c -m 644 mount.fuse.8 '/usr/share/man/man8'

make[2]: Leaving directory `/home/vu/fuse-2.9.1/doc'

make[1]: Leaving directory `/home/vu/fuse-2.9.1/doc'

make[1]: Entering directory `/home/vu/fuse-2.9.1'

make[2]: Entering directory `/home/vu/fuse-2.9.1'

make[2]: Nothing to be done for `install-exec-am'.

test -z "/usr/lib/pkgconfig" || /bin/mkdir -p "/usr/lib/pkgconfig"

 /usr/bin/install -c -m 644 fuse.pc '/usr/lib/pkgconfig'

make[2]: Leaving directory `/home/vu/fuse-2.9.1'

make[1]: Leaving directory `/home/vu/fuse-2.9.1'

root@vu-AP480-S:/home/vu/fuse-2.9.1# rpm -q fuse

package fuse is not installed

root@vu-AP480-S:/home/vu/fuse-2.9.1# cd ..

root@vu-AP480-S:/home/vu# lsb_release -a 

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 12.04 LTS

Release: 12.04

Codename: precise

Thanks,

Ramu.

[Sam Silverberg]

you can download my fuse packages from here

http://code.google.com/p/opendedup/downloads/detail?name=fuse-2.8.6-deb.tar.gz

[ramu]

Hi Sam,

Iam downloaded installing am getting this error,

root@vu-AP480-S:/home/vu/fuse-2.8.6-deb# sudo dpkg -i libfuse-dev_2.8.6-opendedup_amd64.deb 

(Reading database ... 218150 files and directories currently installed.)

Preparing to replace libfuse-dev 2.8.6-2ubuntu2 (using libfuse-dev_2.8.6-opendedup_amd64.deb) ...

Unpacking replacement libfuse-dev ...

dpkg: dependency problems prevent configuration of libfuse-dev:

 libfuse-dev depends on libfuse2 (= 2.8.6-opendedup); however:

  Version of libfuse2 on system is 2.8.6-2ubuntu2.

dpkg: error processing libfuse-dev (--install):

 dependency problems - leaving unconfigured

Errors were encountered while processing:

 libfuse-dev

[Tom Klein]

This is my installation which seems to work just fine.

 

Install SDFS

sudo apt-get install openjdk-7-jdk
sudo apt-get install attr
wget http://opendedup.googlecode.com/files/sdfs-1.1.6_amd64.deb
wget http://opendedup.googlecode.com/files/fuse-2.8.6-deb.tar.gz
tar zxvf fuse-2.8.6-deb.tar.gz
cd fuse-2.8.6-deb
sudo dpkg --install *.deb
sudo apt-get install -f
cd ..
sudo dpkg -i sdfs-1.1.6_amd64.deb
sudo apt-get install -f


sudo nano /etc/security/limits.conf
### Add the following above # End of file:###


soft nofile 65535
hard nofile 65535
 

[ramu]

Hi Tom,

I followed and imstall above steps after i create file system using this command and am getting error,

root@ramu-TW8-SW8-DW8:/home/ramu# sudo ./mkfs.sdfs --volume-name=sdfs_vol1 --volume-capacity=100GB 

sudo: ./mkfs.sdfs: command not found

On Monday, July 23, 2012 2:54:44 PM UTC+5:30, Tom Klein wrote:

This is my installation which seems to work just fine

 

Install SDFS

[Tom Klein]

Try this and replace/change were needed offcourse.

 

 

 

sudo /sbin/mkfs.sdfs --volume-name=dedup --volume-capacity=3000GB --io-chunk-size=64 --chunk-store-size=1000GB --io-dedup-files true

 

To  let it startup at boot

 

sudo nano /sbin/startdedup

###paste the following in nano####

 

/sbin/mount.sdfs -v dedup -m /mnt/dedup/ &

Save the file

 

Change the permissions for so it can execute

 

sudo chmod 755 /sbin/startdedup

 

Link it to the startup folder

 

sudo ln -s /sbin/startdedup /etc/rc2.d/S20startdedup

 

Create a folder to mount it in

 

sudo mkdir /mnt/dedup

 

After a restart your volume should be ready.

Check it with the mount command to see if its there.

[ramu]

Hi Tom,

Thank u for your help ,it was successfully installed.

Thanks,

Ramu.

[ramu]

Hi Tom,

After sdfs was installed on my system it is very slow sometimes it is going to hang also.I created two volumes the capacity is 3000GB and chunk store size is 1000GB.And how to delete volumes in sdfs.

Thanks,

Ramu.

[Sam Silverberg]

go to /opt/sdfs/volumes and remove the volumes under that directory.

[ramu]

Hi Sam,

What is the steps for  SDFS volume replication ?

Thanks,

Ramu.

[Tom Klein]

Create a master

sudo /sbin/mkfs.sdfs --volume-name=dedup --volume-capacity=3000GB --io-chunk-size=64 --chunk-store-size=1000GB --enable-replication-master

 

Create a slave

sudo /sbin/mkfs.sdfs --volume-name=dedup --volume-capacity=3000GB --io-chunk-size=64 --chunk-store-size=1000GB --enable-replication-slave --replication-master=192.168.10.1

 

Replication setup

sudo cp /etc/sdfs/replication.example.props /etc/sdfs/replication.dedup.props

sudo nano /etc/sdfs/replication.dedup.props

 

###edit####
replication.master=192.168.10.1
schedule.cron=0 0/30 * * * ?               to   schedule.cron=0 0 12 1/1 * ? *       (This will run everyday at 12 afternoon)

 

 

To auto start the slave and replication at startup do the following

 

sudo nano /sbin/startdedup

 

####edit####

/sbin/mount.sdfs -v dedup -m /mnt/dedup/ &

/sbin/sdfsreplicate /etc/sdfs/replication.dedup.props &

 

sudo chmod 755 /sbin/startdedup

sudo ln -s /sbin/startdedup /etc/rc2.d/S20startdedup

sudo mkdir /mnt/dedup

 

[ramu]

Hi Tom and Sam,

How to store data in sdfs volumes.I have 100GB data how to store that data to sdfs volumes.

Thanks,

Ramu.

[Tom Klein]

Well you could share the folder through Samba so you can send files to it from Windows machines.

 

Thats what i'm doing atm.

Op woensdag 1 augustus 2012 11:41:19 UTC+2 schreef ramu het volgende:

[ramu]

Tom,

My sdfs setup on ubuntu-12.04.please send the commands for store data to volumes.

[Tom Klein]

This is more of a linux configuration thing but I have somthing for you I think.

 

Depends when you did the installation of ubuntu if you did check the Samba server option to install.

But if you did skip the first command.

 

sudo apt-get install samba

 

sudo smbpasswd -a (account name where you login with)

sudo nano /etc/samba/smb.conf

 

###edit####

[dedup]
   comment = Dedupe Data
   read only = yes
   locking = no
   path = /mnt/dedup
   guest ok = no
   write list = (account where you login with)

###edit####

 

sudo smbd restart

 

Now you should be able to get to it from a windows machine.

Might want to change the permissions too.

 

sudo chmod -R 777 /mnt/dedup

 

Op woensdag 1 augustus 2012 14:10:51 UTC+2 schreef ramu het volgende:

[ramu]

Tom,

I did the above steps it's fine.But I have some files in ubuntu only,how to store these files or how to write these files to sdfsvolume.

[Tom Klein]

I'm not sure if I understand this right but a simple

 

cp comand will do.

Or just install Midnight Commander and copy it from there to the /mnt/dedup and it will do the deduplication on the fly.

Op woensdag 1 augustus 2012 14:50:08 UTC+2 schreef ramu het volgende:

[Tom Klein]

I think you need to ask Sam.

I'm just a System Admin I don't do programming.

 

Op woensdag 1 augustus 2012 15:15:14 UTC+2 schreef ramu het volgende:

Tom,

Thank u Tom,for your help.I have one more problem,how to write backend for sdfs to talk directly to rados(java-rados).

Am posting total message.

Problem:
Ceph: Provides data distribution and redundancy
opendedup/sdfs: Provides data de-duplication
What we need is a combination of both!!!

Solutions:
1) Create cephfs and make sdfs write data to it.
Issue: The over head in storing the meta-data is high (sdfs has metadata, opendedup has metadata, cephfs has metadata)
2) SDFS supports cloud storage of the data, using amazon s3 style api. Integrate this with the rados s3 gateway.
http://www.opendedup.org/administrators-guide#dse
http://ceph.com/docs/master/radosgw/
3) Write a backend for sdfs to talk directly to rados.


Going forward:
Option 1 not worth trying it.
One of you can try Option 2 and keep it as plan B, with proper steps on how to setup and configure.
Option 3 is the way to go, the other person can use the java rados API and write a backend to sdfs similar to that of the s3 backend.

Thanks,

Ramu.

[Sam Silverberg]

Ramu,

I think I can write a backend for ceph. What is the application? Is it archival storage or primary storage. This might effect the implementation.

Sam

[ramu]

Sam,

    Is there any option for sdfs installation directly from source. 

Ramu.