Rhino ETL - loading large pipe-delimited files

[Daventry]

Hi,

 

We've got to load large pipe-delimited files. When loading these into a SQL Server DB by using Rhino ETL (relying upon FileHelpers), is it mandatory to provide a record class?

We have to load files to different tables which have dozens of columns - it might take us a whole day to generate them. I guess we can write a small tool to generate the record classes out of the SQL Server tables.

 

Another approach would be to write an IDataReader wrapper for a FileStream and the pass it on to a SqlBulkCopy.

 

SqlBulkCopy does require column mappings as well but it does allow column ordinals - that's easy.

 

Any ideas/suggestions?

 

Thanks.

[Daventry]

I have already tried, but it's almost 3-4x slower than the bcp command line tool.

 

I think I'm going to use bcp.exe to import the files into staging tables, where all columns are of Varchar type.

Then will run a proc to cast the data and insert into the actual table so that we can get more meaningful logging.

On Wednesday, July 18, 2012 8:28:19 PM UTC+1, Louis Haußknecht wrote:

Why don't you read the file line by line and split each line on the pipe, then yield return the result? 

[Nathan Palmer]

It's going to be hard to beat the speed of bcp. Rhino etl provides flexibility and control and I'm sure we can get close by using SqlBulkCopy.

I'm curious though.. how would you want/expect rhino-etl to work in this instance to make your import easier?

Nathan Palmer

--
You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group.

To view this discussion on the web visit https://groups.google.com/d/msg/rhino-tools-dev/-/jtNCsX_4axAJ.

To post to this group, send email to rhino-t...@googlegroups.com.
To unsubscribe from this group, send email to rhino-tools-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en.