Seafile Server bulk / site sftp & uploads

[Netflix Boundaries]

we were about to roll-out some test on regards to how to upload files around 4TB of data and looking forward to an alternative instead of using the seafile client to upload/sync

What was proposed and suggested is to SFTP the files to the server and then try uploading the files from the Seafile website to the libraries browsing from the same server. Now on this, we are lacking from:

- The ability to upload multiple files simultaneously

- The ability to upload an entire folder too

Is there a way to enable these features?

The other alternative could be installing the Seafile client on the server. We are running Ubuntu Server 12.04 with the following 

Filesystem      Size  Used Avail Use% Mounted on

/dev/xvda1      7.9G  1.2G  6.4G  15% /

udev            1.9G   12K  1.9G   1% /dev

tmpfs           751M  224K  750M   1% /run

none            5.0M     0  5.0M   0% /run/lock

none            1.9G     0  1.9G   0% /run/shm

/dev/xvdb       394G   20G  355G   6% /mnt1

/dev/md0        4.0T   34M  4.0T   1% /raid0

/dev/md0 is the raid0 array set for uploading the files.  

/dev/xvdb is hosting the /seafile-server-1.4.1 (for now)

Everything is command-based. Where would you recommend installing the client?

Thanks.

[JiaQiang Xu]

The client now requires X and GTK to run. A command line client will
be available in 1.5 release.

It's impractical to upload all the 4TB files at once since Seafile
client has to first write all the data into a cache area before
uploading it. This will take up 4TB cache space. So you won't have
enough free space in /dev/md0.

It's recommended that you divide the 4TB data into multiple libraries
and upload them separately.

Currently the client only cleans up unused cache space on restart, So
you should restart the client after finish uploading a library.

2013/2/13 Netflix Boundaries <netflix.b...@gmail.com>:

> --
> You received this message because you are subscribed to the Google Groups
> "seafile" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to seafile+u...@googlegroups.com.
> To post to this group, send email to sea...@googlegroups.com.
> Visit this group at http://groups.google.com/group/seafile?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Netflix Boundaries]

Thanks for your response.

Can you be more specific regarding X and GTK?

Your methodology for uploading 4TB makes a lot on sense based on the cached functionality. However, in our case, that will require a LOT of manual intervention. Even by doing it separately.

As for example, we have a folder structure containing over 200+ subfolders intrinsically distributed. If we perform manual uploads, that means we need to create the folders through the website manually AND upload each file individually.

However, this can be reduced in a significant manner by adding 'upload a folder' functionality instead of only allowing 'uploading individual files'


If installing Gnome desktop for Ubuntu Server 12.04 on it could do the job, sync would be more faster and maintain same folder's structure?

Again thanks for this incredible project. Looking forward to above mentioned improvements as a lot of implementations over 4 TB are expected, and still need to test MySQL responsiveness and how the server manage large amount of data over RAIDs configs.

[JiaQiang Xu]

Yes, installing Gnome on the server and log in to it should do the job.

If you want to keep the folder structure, you may create one file
library and add your files in a few batches into the library. Because
the client and server are on the same machine, syncing would be very
fast. You just need to restart the client after each batch is synced
with the server. That should clear the cache space.

btw, we're planning to automate the cache cleanup procedure so that
users don't need to restart the client.

2013/2/13 Netflix Boundaries <netflix.b...@gmail.com>:

[Netflix Boundaries]

Thanks. We'll give a try and get back with some results.

What about the features for uploading multiple files and folders from the website?

Automated cache cleanup procedure will be of great help.

[Netflix Boundaries]

Uploading through SFTP worked.

Installing Gnome let us install the Linux client. After changing the Client port on the Server's ccnet.conf file, restarting seafile server & restarting the client (seafile-applet restart) it connected and sync'ed successfully.

Now it's good to make sure that in the original source of the files (Windows on this case), the Seafile client downloads the library with the same path to avoid downloads (of what was uploaded through SFTP) or uploads if there was any new file on the root path. It should only index and sync.

Sync got replicated on both Android & OS X successfully. However some .jpg files doesn't find an application to open the image, even though that this is regularly used without issues. Tried restarting server and the client, but it still persists on Android.

If a thumbnail preview can be added to the mobile client will be great.

Thanks.

[Lingtao Pan]

.jpg files can be opened in my Android phone. Thumbnail is shown when a picture is downloaded.

[Netflix Boundaries]

Yes, you are correct. Thumbnails appears after downloading and being cached. However it would be useful to have it before the download. Just after the files appears listed.

Please see the image attached.

[Lingtao Pan]

I can't find the reason why JPG can not be opened in your phone. The java code is below, first I get mime type by call Android standard class MimeTypeMap  then ask the system to open it.. The mime type only depends on file suffix.

        String mime = MimeTypeMap.getSingleton().getMimeTypeFromExtension(suffix);

        Intent open = new Intent(Intent.ACTION_VIEW, Uri.parse(file.getAbsolutePath()));

        open.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK);

        open.setAction(android.content.Intent.ACTION_VIEW);

        open.setDataAndType((Uri.fromFile(file)), mime);

        try {

            startActivity(open);

            return true;

        } catch (ActivityNotFoundException e) {

            showToast(getString(R.string.activity_not_found));

            return false;

        }

Anyway, in next release, I can directly specify the mime type for JPG files when MimeTypeMap failed to find a proper mime type . This may resolve your problem.

We will also improve thumbnail in future version.

[Daris Tan]

Hi, I really hope seafile can improve the thumbnails before downloading, it doesn't make sense if you have a lot of file in the same folder, I find very hard to do sharing unless you have few photos. Imagine you have 100 photos of 5mb each and each of your friend have to download all. The mobile data would easily exceed. And so much time.

Another thing is can the server be non encrypted while setting up. I am using the pi and I want to share my family albums which already in the pi. Imaging is 100gb, because of eencryption I cannot access the file direct and would have to create another 100gb in the same pi usb Hardisk. Since is for home and nothing really need to keep secret, I only wanted the convenient of sharing and uploading. Although I also use ftp somehow I think cloud would work better when come to easy sharing

[Daris Tan]

[Lingtao Pan]

You can use the FUSE extension to access file directly. See https://github.com/haiwen/seafile/wiki/Seafile-Virtual-Filesystem-via-FUSE 

2014-02-02 Daris Tan <mrd...@gmail.com>:

Hi, I really hope seafile can improve the thumbnails before downloading, it doesn't make sense if you have a lot of file in the same folder, I find very hard to do sharing unless you have few photos. Imagine you have 100 photos of 5mb each and each of your friend have to download all. The mobile data would easily exceed. And so much time.

Another thing is can the server be non encrypted while setting up. I am using the pi and I want to share my family albums which already in the pi. Imaging is 100gb, because of eencryption I cannot access the file direct and would have to create another 100gb in the same pi usb Hardisk. Since is for home and nothing really need to keep secret, I only wanted the convenient of sharing and uploading. Although I also use ftp somehow I think cloud would work better when come to easy sharing

--
You received this message because you are subscribed to a topic in the Google Groups "seafile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/seafile/HiczpRTqekY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to seafile+u...@googlegroups.com.

To post to this group, send email to sea...@googlegroups.com.

Visit this group at http://groups.google.com/group/seafile.

[Daris Tan]

thanks, after getting it done. i don't see any different. how to use it?

[Lingtao Pan]

If you follow the document, you should be able to access files directly in /data/seafile-fuse

2014-02-05 Daris Tan <mrd...@gmail.com>:

[Daris Tan]

Sorry think I never explain clearly, I create a folder call home, then before I activate the fuse, I transfer some files to it .  after that I activate fuse but how do I view it as they do not belong to any library? How do I Create a library that link to it.

[Lingtao Pan]

You have to upload these files to a library first. A library can't be link to an existing folder on the server.

[Daris Tan]

I have thought of that too, however I cannot find the library after I create them, I think they are in blocks too. So when I run fuse, the directory to the library cannot be found.