Bulk API?

87 views
Skip to first unread message

fmcgeough

unread,
Oct 18, 2011, 4:30:43 PM10/18/11
to ActiveSalesforce
I'm enjoying using what's already developed. Nice work. Any thoughts
on adding a wrapper around the REST based Bulk API in under
databasedotcom?

Alex Sutherland

unread,
Oct 24, 2011, 4:10:03 PM10/24/11
to activesa...@googlegroups.com
fmcgeough what is your use case for adding support for the Bulk API to the gem?

- Alex



--
You received this message because you are subscribed to the Google Groups "ActiveSalesforce" group.
To post to this group, send email to activesa...@googlegroups.com.
To unsubscribe from this group, send email to activesalesfor...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/activesalesforce?hl=en.


Frank McGeough

unread,
Oct 24, 2011, 4:21:27 PM10/24/11
to activesa...@googlegroups.com
Sure. Here's my use case :

I am synchronizing data between my company's database (accounts and users) and salesforce.com.
I gather data from my database and update objects in salesforce.com periodically (once an hour in my case) 
In ordinary use I do a diff between what I've already sent to salesforce.com and what is in my own database. Then I send only the changes using databasedotcom gem. This process works fairly quickly (less than 4 minutes in my case). 
Occasionally I get requests to change the business rules that apply to this sync meaning that every row must be updated in sfdc. This could happen in a variety of ways actually. 
Although I can export the data into a csv file and then load it (or use a vendor's tool), it seems easier to have my code recognize this situation and then use the bulk API to load the data. 
The current total amount of data is about 100,000 users and about 18,000 accounts. 
It takes less than 20 minutes to upsert all this data using the Bulk API. (I used a tool that implements a Bulk API call). 
It takes about 24 hours to upsert this data using databasedotcom.gem.

If there is another Ruby based solution that wraps the Bulk API then I'd be happy to use that. I couldn't find one. 

best regards, 
Frank

Alex Sutherland

unread,
Oct 24, 2011, 4:36:06 PM10/24/11
to activesa...@googlegroups.com
That's a very interesting use case, thanks for sharing it Frank!

Since the Bulk API is such a different animal from the Salesforce REST/JSON data API, I'm thinking that it would probably be better to support it with a separate gem. Also, from the documentation it looks like the "official" method of authenticating to the Bulk API is using the old SOAP API to get a session token (although I'm sure you could use a session token obtained through OAuth just as well). 

Anyone up for the challenge of building a gem for the Salesforce.com/Force.com/Database.com Bulk API (or know of any existing projects around this)?

- Alex

Frank McGeough

unread,
Oct 24, 2011, 5:00:30 PM10/24/11
to activesa...@googlegroups.com

raygao-gmail

unread,
Oct 24, 2011, 8:08:50 PM10/24/11
to activesa...@googlegroups.com
It might be a good challenge for CloudSpokes. It would be nice to see several versions, each showing its best.

-R

Jeff Douglas

unread,
Oct 25, 2011, 6:17:04 AM10/25/11
to activesa...@googlegroups.com
It shouldn't be that hard to implement in ruby. Here's a Java version that I did a couple of years ago for the force.com article I wrote.

Essentially, you query for some data, convert the resultset into a CSV (I found an open source Java lib for it), shove the CSV up to the Bulk API and listen for the results.

It actually might be a really cool CloudSpokes challenge! Thanks for the idea Ray!

Thanks!

Jeff Douglas
google voice: (941) 227-4843
mobile: (941) 809-3455
email: je...@appirio.com
blog: http://blog.jeffdouglas.com



Appirio, Inc. 
Accelerating On-Demand in the Enterprise
http://www.appirio.com

Reply all
Reply to author
Forward
0 new messages