Indexing Documents

[thstart]

I would like to use CloudDatastore. With Pricing Calculator I see Indexing Documents.

What Indexing Documents is? Looks like too expensive if I have database of such size.

Indexing Documents: 100 GB

$199.98

[Katayoon (Cloud Platform Support)]

Before you can query a document's contents, you must add the document to an index. Indexing allows the document to be searched with the Search API's query language and query options. Fees for use of the Search API are listed here. As mentioned in the App Engine Pricing, you may take a look at the Java and Python documentation for a detailed description of each type of Search call.

[thstart]

I am trying to calculate how much an index will cost from here:

https://cloud.google.com/datastore/docs/concepts/storage-size

Let say The total size of the index entry is 128 bytes. 1o million records ~1GB

from the calculator:

https://cloud.google.com/products/calculator/#id=9102d5c8-d24f-4b36-a5ad-d4eeb82a7f55

the only price related ti indexes is Indexing Documents. Is this the same index

we are talking about?

[Katayoon (Cloud Platform Support)]

“Indexing Documents” is in App Engine tab of Google Cloud Platform Pricing Calculator. Since you have created this thread in App Engine group, my assumption was you are asking about App Engine Indexing Documents.

For Cloud Datastore you should consider the amount of Stored data including “Index entry size”. For the whole Stored data less than 1GB (as mentioned in this documentation), you would be in free quota limit per day and will be charged USD 0.18 for each extra GB/Month.

[thstart]

App Engine Indexing Documents - do you have a link with 

documentation what that means?

[Katayoon (Cloud Platform Support)]

Sure. I had put the link in my first comment. Here it is. Hope this explanation works for you?

[thstart]

If they are JSON documents where they are stored so to be indexed?

[Katayoon (Cloud Platform Support)]

If you mean using Search API in App Engine to index your structured data (your JSON files), you may take a look at this documentation and the pricing is the same as brought in the App Engine tab of Pricing Calculator (Indexing Documents section). If not, could you provide more clarification about your use case?

[thstart]

Hi Katayoon,

Here is my use case:

1) source : CSV file

2) one master table of size 500 GB and 500 million records. 

3) daily updates of size 10GB and 10 million records.

I need to create 2) one time. Every day I need to update 3).

Right now I import the CSV files and create tables in BigQuery. Then I can do additional work and export as CSV to Cloud Storage.

With BigQuery I can prepare the tables in advance. Let say I have 1 table with 100 columns and 10 indexes on only 10 fields which are fields in the CSV exported from BigQuery in CloudStorage. Let say the master table is 500 GB/500 mil records and the indexes should be  1/10 or 50 GB. Every day I change/delete about 10 GB/10 million records.

BigQuery is not for customer facing apps so I'm looking for the best way to achieve that most effective and cheapest way.

For customer facing app I don't need full SQL -  some limited SQL or if not possible KV store is OK. 

Best,

--Constantine

[Katayoon (Cloud Platform Support)]

Hi Constantine,

I assume your question is more related to the architecture rather than the datastore/ App Engine indexing cost? If so, I recommend that you contact Google Cloud Platform sales team and discuss your project with them. If you have a Premium support package, the architecture advisory service is available for you as well.

I should note that Google Groups are reserved for general product discussions. We cannot provide you any advice on your system’s architecture.