SalesForce Bulk API only using 1 batch

[Sharon]

I'm doing some testing to see whether we can switch over to using cozyroc to export data from SalesForce into our on premise database. We currently use a cloud service to do this but are quite frustrated with it. 

I need to export sometimes millions of rows from SalesForce to a SQL database.  I've set up a test SSIS package to export one object into our database using a SalesForce source, with the Bulk Mode.  When I check in the "Monitor Bulk Data Load Jobs" I can see it in there but its only using 1 batch for 2.5 million rows.  I also have a number of retries on there.  

With our current process it processes in smaller batches (we can specify the size).  

Is this something that is also possible to do with the cozyroc salesforce source? 

Also, I keep getting the error "An error occurred with the following error message: "System.IO.IOException: Received an unexpected EOF or 0 bytes from the transport stream." after its loaded around 200,000 rows into the destination table.  I tried increasing the timeout time to 10 minutes but still getting the error.  I don't know if this is related to it only using 1 batch or is a different issue. 

Thanks!

[Ivan Peev]

Hi Sharon,

What do you have specified for the BatchSize parameter ?

[Sharon]

Hi Ivan, 

Sorry about the delay on my reply! I didn't get a notification to say you had replied for some reason. 

When I select to use Bulk mode I am unable to specify a BatchSize - its greyed out.  I've included a screenshot to show what I mean.  

I've tried with a number of different objects and as soon as I select Bulk the BatchSize is greyed out. 

The service URL I am using for the connection is https://test.salesforce.com/services/Soap/u/42.0 just in case that affects things. 

Thanks, 

Sharon

[Sharon]

Here is the screenshot.

[Ivan Peev]

Hi Sharon,

I have checked our code and it appears there is no BatchSize parameter for the bulk-read mode. Therefore the UI you are seeing is correct.

[Sharon]

Thanks Ivan.  So can I just confirm what this means in this case?

When using the bulk UI with cozyroc everything has to come down in 1 batch?  As mentioned, we pull down millions of rows from SalesForce, and using 1 batch is proving problematic.  The system we use currently allows us to set the batch size and we set it to 100,000 which greatly improves things for us.  

If my understanding above is correct - is this something you will be implementing in a future release of cozyroc?  

[Ivan Peev]

Sharon,

You understanding is incorrect. The data is not going to be retrieve as a single batch. Salesforce will send as many batches as needed. You simply don't have an option to configure how big is the block of data you want to retrieve. For further information check the Salesforce Bulk Query API page here. As you can see there is no parameter for batch size available.