Rhino ETL IS not loading more than 60,000 records to Target

40 views
Skip to first unread message

Vijay Sheganti

unread,
May 3, 2017, 4:27:20 AM5/3/17
to Rhino Tools Dev
I have a package which reads data from the source and insert to target table based on the record existence. Everything works fine when the source record count is up to 60000. Will there be any settings or reasons for such behaviour? BTW, the source contains 700K records. I wanted to load all in a single run.

Oren Eini (Ayende Rahien)

unread,
May 3, 2017, 4:27:53 AM5/3/17
to rhino-tools-dev
Are you doing that as a single transaction?

Hibernating Rhinos Ltd  

Oren Eini l CEO Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 


On Wed, May 3, 2017 at 11:26 AM, Vijay Sheganti <vijayg...@gmail.com> wrote:
I have a package which reads data from the source and insert to target table based on the record existence. Everything works fine when the source record count is up to 60000. Will there be any settings or reasons for such behaviour? BTW, the source contains 700K records. I wanted to load all in a single run.

--
You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhino-tools-dev+unsubscribe@googlegroups.com.
To post to this group, send email to rhino-tools-dev@googlegroups.com.
Visit this group at https://groups.google.com/group/rhino-tools-dev.
For more options, visit https://groups.google.com/d/optout.

Vijay Sheganti

unread,
May 3, 2017, 6:45:16 AM5/3/17
to Rhino Tools Dev
Yes. 
I am reading data from the source and target using InputCommandOperation (2 diff classes) and joining to check for existence in the target table. If the records do not exist in the target, inserting them using SQLBulkInsertOperation.
As I am not seeing any records in target table for all the source data, I have modified my query which pulls data from source to pull a specific number of records.During this experiment, I found it's inserting 60K records and not more than that. 

Verified space in DB and my code. Not sure what else should I do. Thankful for any help on this.

On Wednesday, May 3, 2017 at 1:57:53 PM UTC+5:30, Ayende Rahien wrote:
Are you doing that as a single transaction?

Hibernating Rhinos Ltd  

Oren Eini l CEO Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 


On Wed, May 3, 2017 at 11:26 AM, Vijay Sheganti <vijayg...@gmail.com> wrote:
I have a package which reads data from the source and insert to target table based on the record existence. Everything works fine when the source record count is up to 60000. Will there be any settings or reasons for such behaviour? BTW, the source contains 700K records. I wanted to load all in a single run.

--
You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhino-tools-d...@googlegroups.com.
To post to this group, send email to rhino-t...@googlegroups.com.

Oren Eini (Ayende Rahien)

unread,
May 4, 2017, 4:14:26 AM5/4/17
to rhino-tools-dev
Check for timeout in the bulk insert, I would guess.
To unsubscribe from this group and stop receiving emails from it, send an email to rhino-tools-dev+unsubscribe@googlegroups.com.
To post to this group, send email to rhino-tools-dev@googlegroups.com.

Vijay Sheganti

unread,
May 4, 2017, 7:10:27 AM5/4/17
to Rhino Tools Dev
Actually, I am not setting anywhere the timeout. What would be the default timeout value? 
I verified the source code but could not find the value. Also, the process was not taking more than a minute for my operation. I hope, the default value should be more than a minute. 

Another question I have is on the logging/Exception handling.
Where is the code to handle exceptions raised in the execution? I want to check the reason for termination or not functioning my app as expected.

Vijay Sheganti

unread,
May 9, 2017, 9:06:37 AM5/9/17
to Rhino Tools Dev
The problem was with column value truncation. Nothing with the number of rows or space on the DB. 
Reply all
Reply to author
Forward
0 new messages