Octoparse to BigQuery / Google Sheets

[Charlotte Crowther]

We are looking to bring data from Octoparse (which can automatically export to MySQL or keep the data in Octoparse accessible via APIs).

We want this to be an automated solution, so when Octoparse is set to run on a schedule the data automatically updates in BigQuery and Google Sheets.

Key criteria for our solution is total cost of ownership (cost-effective solution with the ability to easily add new datasets ourselves without needing a developer)
 
Options we’ve been thinking of:

1) Bring the data into Google Sheets using API (and from there to BigQuery)
a.https://www.octoparse.com/faq/how-to-extract-data-to-google-sheet

2) Bring the data into a MySQL Database (default feature of Octoparse) and from there replicate to BigQuery (how?)
a. https://www.octoparse.com/tutorial-7/export-to-database#DatabaseExport2
b. https://www.stitchdata.com/integrations/mysql/google-bigquery/ ($100/month, up to 10 sources, 300M rows/month)

For connecting Google sheets and BigQuery we’ve been looking at utilising: https://codingisforlosers.com/google-sheets-bigquery-connector/ 

Any thoughts, ideas and experience are welcome - thanks!

Charlotte

[Alex]

The main question is 'What is the size of the data?'

Also there is no need connect Google Sheets and BQ. This works manually via BQ project.

[Charlotte Crowther]

Our data is quite small, so there shouldn't be too much problem there.

A couldn't quite find the BQ project they refer to, could you ask for an example/link? Is it this

https://towardsdatascience.com/connect-bigquery-to-google-sheets-e3c4f1377659? Does it work for Connected Sheets but require enterprise Google Suite?

--
You received this message because you are subscribed to a topic in the Google Groups "Google Apps Script Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-apps-script-community/Xt9P1keaP9U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-apps-script-c...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-apps-script-community/bd474e43-d863-4e0f-a02f-a843b12ddc21n%40googlegroups.com.

[Adeel Malik]

I believe you can use Octoparse API and get data directly from Octoparse API to BigQuery. You won't need to involve google sheets into the project unless there is a use case of performing some analysis on the data before syncing to BigQuery. 

Please check Octoparse API doc: http://dataapi.octoparse.com/help

From the Bigquery side, you will use BigQuery Service for apps script to load the data returned from octoparse API -> https://developers.google.com/apps-script/advanced/bigquery

[Ievgen Krasovytskyi]

@Charlotte,

You can use our solution to bring data from Octoparse API to both Sheets and BigQuery.
It's available here on github.

Here is the contributor's guide (but basically you are not obligated to do that - this is also an instruction on how to use our library of apps scripts so you can build your own connector for yourself).

But we would still appreciate your star and contribution.

We are aiming to create the free forever connector library to make the life easier.