Bulk API?

[fmcgeough]

I'm enjoying using what's already developed. Nice work. Any thoughts
on adding a wrapper around the REST based Bulk API in under
databasedotcom?

[Alex Sutherland]

fmcgeough what is your use case for adding support for the Bulk API to the gem?

- Alex

--
You received this message because you are subscribed to the Google Groups "ActiveSalesforce" group.
To post to this group, send email to activesa...@googlegroups.com.
To unsubscribe from this group, send email to activesalesfor...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/activesalesforce?hl=en.

[Frank McGeough]

Sure. Here's my use case :

I am synchronizing data between my company's database (accounts and users) and salesforce.com.

I gather data from my database and update objects in salesforce.com periodically (once an hour in my case) 

In ordinary use I do a diff between what I've already sent to salesforce.com and what is in my own database. Then I send only the changes using databasedotcom gem. This process works fairly quickly (less than 4 minutes in my case). 

Occasionally I get requests to change the business rules that apply to this sync meaning that every row must be updated in sfdc. This could happen in a variety of ways actually. 

Although I can export the data into a csv file and then load it (or use a vendor's tool), it seems easier to have my code recognize this situation and then use the bulk API to load the data. 

The current total amount of data is about 100,000 users and about 18,000 accounts. 

It takes less than 20 minutes to upsert all this data using the Bulk API. (I used a tool that implements a Bulk API call). 

It takes about 24 hours to upsert this data using databasedotcom.gem.

If there is another Ruby based solution that wraps the Bulk API then I'd be happy to use that. I couldn't find one. 

best regards, 

Frank

[Alex Sutherland]

That's a very interesting use case, thanks for sharing it Frank!

Since the Bulk API is such a different animal from the Salesforce REST/JSON data API, I'm thinking that it would probably be better to support it with a separate gem. Also, from the documentation it looks like the "official" method of authenticating to the Bulk API is using the old SOAP API to get a session token (although I'm sure you could use a session token obtained through OAuth just as well). 

Anyone up for the challenge of building a gem for the Salesforce.com/Force.com/Database.com Bulk API (or know of any existing projects around this)?

- Alex

[Frank McGeough]

Alex,

yes it is possible to avoid the SOAP API. http://blogs.developerforce.com/developer-relations/2011/03/oauth-and-the-soap-api.html

Frank

[raygao-gmail]

It might be a good challenge for CloudSpokes. It would be nice to see several versions, each showing its best.

-R

[Jeff Douglas]

It shouldn't be that hard to implement in ruby. Here's a Java version that I did a couple of years ago for the force.com article I wrote.

Essentially, you query for some data, convert the resultset into a CSV (I found an open source Java lib for it), shove the CSV up to the Bulk API and listen for the results.

It actually might be a really cool CloudSpokes challenge! Thanks for the idea Ray!

Thanks!

Jeff Douglas
google voice: (941) 227-4843
mobile: (941) 809-3455
email: je...@appirio.com
blog: http://blog.jeffdouglas.com

Appirio, Inc. 
Accelerating On-Demand in the Enterprise
http://www.appirio.com

Author: Salesforce Handbook & Beginning Java Google App Engine