[rhino-tools-dev] Blob Files

[Rafael Vasconcellos]

Hi,

I have to transfer a blob file and the ETL process is very slow. Has anyone worked with blob files? What´s the best strategy of implementation?

[Nathan Palmer]

I haven't done much with blob files. However in the past with large datasets I've written the data out to text files, compressed and transmitted those files and picked them up at a second location. This was especially useful when running a distributed ETL process between two remote locations.

I would assume the slowness just has to do with the fact that you are dealing with a lot of data transfer. But you can break down the process to find out where the bottleneck is to see if there is a way to optimize. For example.. just export it out (and don't write it) and see how that compares against the entire process. If the destination isn't local you could test writing it out local versus remotely. Break this down a few times and you should be able to identify where in the process it's slow and possibly come up with some ideas for how to speed it up.

Nathan

On Mon, Jan 27, 2014 at 8:19 AM, Rafael Vasconcellos <rfl.vasc...@gmail.com> wrote:

Hi,

I have to transfer a blob file and the ETL process is very slow. Has anyone worked with blob files? What´s the best strategy of implementation?

--
You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhino-tools-d...@googlegroups.com.
To post to this group, send email to rhino-t...@googlegroups.com.
Visit this group at http://groups.google.com/group/rhino-tools-dev.
For more options, visit https://groups.google.com/groups/opt_out.

[Rafael Vasconcellos]

Only to close this thread, it was an application that have to transfer 11k registers. It was spending 30min and doesn´t finalized. So, I decided to do this job in database because of the time spend.

Thanks.