Marcel Manz
Apr 4, 2016, 11:42:22 AM4/4/16
to Google App Engine
Hello
We're glad to see that Google announced a reduced Datastore pricing, starting July 1st 2016:
https://cloudplatform.googleblog.com/2016/03/Google-Cloud-Datastore-simplifies-pricing-cuts-cost-dramatically-for-most-use-cases.html
Finally the costs are entity based, regardless of amount of properties and indexes involved. Big Kudos also on the newly introduced 'Entity deletes' pricing, to finally enable us to delete / offload entities no longer used, an operation which after years of operation was becoming for many use cases too expensive.
Unfortunately Google decided to keep the pricing per Stored GB at $0.18 month, which still means $180 monthly per TB stored. We're having a bit of concern seeing no cost reduction on this part, especially as announcements on BigQuery side are made where 'historical data storage' now costs as less as $0.01 per month, 18x cheaper than DataStore, with no performance reduction:
https://cloud.google.com/blog/big-data/2016/03/google-bigquery-cuts-historical-data-storage-cost-in-half-and-accelerates-many-queries-by-10x
https://cloud.google.com/bigquery/pricing#long-term-storage
Can Google please comment on what justifies a 9 to 18 times higher storage pricing on Datastore compared to BigQuery? I assume both systems are highly distributed and make use of similar storage layers on Google side. How come BigQuery became so cheap, while Datastore Storage is still incredibly expensive? Not to mention that loading operations are even free on BigQuery side.
We would love to store a massive amount of data in our App Engine Datastore, but at a price of $180 per Terabyte we will be forced to offload data from Datastore to some cheaper storage system for long-term retrieval.
A reduction of Datastore Storage Pricing would be highly appreciated.
Best regards,
Marcel
We're glad to see that Google announced a reduced Datastore pricing, starting July 1st 2016:
https://cloudplatform.googleblog.com/2016/03/Google-Cloud-Datastore-simplifies-pricing-cuts-cost-dramatically-for-most-use-cases.html
Finally the costs are entity based, regardless of amount of properties and indexes involved. Big Kudos also on the newly introduced 'Entity deletes' pricing, to finally enable us to delete / offload entities no longer used, an operation which after years of operation was becoming for many use cases too expensive.
Unfortunately Google decided to keep the pricing per Stored GB at $0.18 month, which still means $180 monthly per TB stored. We're having a bit of concern seeing no cost reduction on this part, especially as announcements on BigQuery side are made where 'historical data storage' now costs as less as $0.01 per month, 18x cheaper than DataStore, with no performance reduction:
https://cloud.google.com/blog/big-data/2016/03/google-bigquery-cuts-historical-data-storage-cost-in-half-and-accelerates-many-queries-by-10x
https://cloud.google.com/bigquery/pricing#long-term-storage
Can Google please comment on what justifies a 9 to 18 times higher storage pricing on Datastore compared to BigQuery? I assume both systems are highly distributed and make use of similar storage layers on Google side. How come BigQuery became so cheap, while Datastore Storage is still incredibly expensive? Not to mention that loading operations are even free on BigQuery side.
We would love to store a massive amount of data in our App Engine Datastore, but at a price of $180 per Terabyte we will be forced to offload data from Datastore to some cheaper storage system for long-term retrieval.
A reduction of Datastore Storage Pricing would be highly appreciated.
Best regards,
Marcel
Nick (Cloud Platform Support)
Apr 6, 2016, 7:42:02 PM4/6/16
to Google App Engine
Hey Marcel,
While I can't authoritatively comment on why the price structure is the way it is, as a humble Community Support team member, I can attempt to provide some assistance if you'd like to discuss some issues related to this situation. I'm curious what it is about your app which has it using Datastore on the order of TB's of storage? I'd love to see if we can determine whether there might be a more efficient way to store the data which is currently going into Datastore. Feel free to reply with a description of your data and requirements and perhaps I can help guide a migration to a cheaper service for your needs.
Cheeers,
Nick
Cloud Platform Community Support
While I can't authoritatively comment on why the price structure is the way it is, as a humble Community Support team member, I can attempt to provide some assistance if you'd like to discuss some issues related to this situation. I'm curious what it is about your app which has it using Datastore on the order of TB's of storage? I'd love to see if we can determine whether there might be a more efficient way to store the data which is currently going into Datastore. Feel free to reply with a description of your data and requirements and perhaps I can help guide a migration to a cheaper service for your needs.
Cheeers,
Nick
Cloud Platform Community Support
Marcel Manz
Jul 1, 2016, 9:04:32 AM7/1/16
to Google App Engine
Hey Nick
The new Datastore pricing should go into effect on today (1st July 2016) and I'm wonderning when the documentation will be updated as mentioned (https://cloudplatform.googleblog.com/2016/03/Google-Cloud-Datastore-simplifies-pricing-cuts-cost-dramatically-for-most-use-cases.html):
The new pricing page doesn't yet reveal those new storage calculation details.
Coming back to the topic with the 'Stored data' charge of 0.18 GB (180$ per TB):
Thanks to App Engine we operate a highly scalable backend system that makes heavy use of Datastore as storage system. We designed our backend specificaly to NoSQL, making heavy use of key put/get only operations (among some queries as well).
While Datastore is a perfect fit for us, we were hoping to see some cost reductions on stored data. There hasn't been really any cost reduction since years. With the vast amount of data we like to store for the long term, we're meanwhile at a multi-TB level incurring significant monthly storage costs to us. As the basic idea of Datastore is to use it with such large workloads (as it nearly scales indefinitely), we really would appreciate if Google would reduce the storage pricing, not requiring us to offload Data to some other service (like BigQuery, which is 9 to 18 times cheaper). $0.18 GB may be OK for small apps, but then I don't even see the point why they require Datastore in the first place, as CloudSQL would be far sufficient. With our workload and size, Datastore is the ideal companion, would it be just cheaper to store data for the long term ..
As the delete operations now became cheaper, we're likely forced to start offloading archive data from DS to BigQuery for the sole reason of its more economical storage costs, even though our application doesn't require any of BigQuery's features to operate.
Would be nice if you could discuss the storage costs topic with the PM's, as in my opinion there shouldn't really be a 9 - 18 times premium cost as compared to your other highly scalable BigQuery service.
Cheers
Marcel
The new Datastore pricing should go into effect on today (1st July 2016) and I'm wonderning when the documentation will be updated as mentioned (https://cloudplatform.googleblog.com/2016/03/Google-Cloud-Datastore-simplifies-pricing-cuts-cost-dramatically-for-most-use-cases.html):
New storage usage calculations
To coincide with our pricing changes on July 1st, Cloud Datastore will
also use a new method for calculating bytes stored. This method will be
transparent to developers so you can accurately calculate storage costs
directly from the property values and indexes of the Entity. This new
method will also result in decreased storage costs for the majority of
customers.
Our current method relies heavily on internal implementation details
that can change, so we’re moving to a fixed system calculated directly
from the user data submitted. As the new calculation method gets
finalized, we’ll post the specific details so developers can use it to
estimate storage costs.
The new pricing page doesn't yet reveal those new storage calculation details.
Coming back to the topic with the 'Stored data' charge of 0.18 GB (180$ per TB):
Thanks to App Engine we operate a highly scalable backend system that makes heavy use of Datastore as storage system. We designed our backend specificaly to NoSQL, making heavy use of key put/get only operations (among some queries as well).
While Datastore is a perfect fit for us, we were hoping to see some cost reductions on stored data. There hasn't been really any cost reduction since years. With the vast amount of data we like to store for the long term, we're meanwhile at a multi-TB level incurring significant monthly storage costs to us. As the basic idea of Datastore is to use it with such large workloads (as it nearly scales indefinitely), we really would appreciate if Google would reduce the storage pricing, not requiring us to offload Data to some other service (like BigQuery, which is 9 to 18 times cheaper). $0.18 GB may be OK for small apps, but then I don't even see the point why they require Datastore in the first place, as CloudSQL would be far sufficient. With our workload and size, Datastore is the ideal companion, would it be just cheaper to store data for the long term ..
As the delete operations now became cheaper, we're likely forced to start offloading archive data from DS to BigQuery for the sole reason of its more economical storage costs, even though our application doesn't require any of BigQuery's features to operate.
Would be nice if you could discuss the storage costs topic with the PM's, as in my opinion there shouldn't really be a 9 - 18 times premium cost as compared to your other highly scalable BigQuery service.
Cheers
Marcel
Marcel Manz
Jul 5, 2016, 11:54:34 AM7/5/16
to Google App Engine
It appears that the 'Charges this month' page
(https://console.cloud.google.com/billing/unbilledinvoice) has not yet
been updated according the new pricing scheme.
This page still lists 'Datastore Write Ops' and an estimated $ amount. It should instead show the new billing ops like 'Datastore Entity Writes' and similar.
The dashboard as well as quota usage history are correctly showing the new charges for us. Strangly though that there is an inconsistency between 'Datastore Stored Data' amount from Quota Details (smaller value, matching dashboard value) opposed to Quota History (larger value, according to which charging is done). In our case the difference between these 2 values is around 10%.
Can Google please comment why these values are not the same?
Regards
Marcel
This page still lists 'Datastore Write Ops' and an estimated $ amount. It should instead show the new billing ops like 'Datastore Entity Writes' and similar.
The dashboard as well as quota usage history are correctly showing the new charges for us. Strangly though that there is an inconsistency between 'Datastore Stored Data' amount from Quota Details (smaller value, matching dashboard value) opposed to Quota History (larger value, according to which charging is done). In our case the difference between these 2 values is around 10%.
Can Google please comment why these values are not the same?
Regards
Marcel
Dan McGrath
Jul 6, 2016, 11:37:53 AM7/6/16
to Google App Engine
Hi Marcel,
I'm the PM for Cloud Datastore. Thanks for pointing out the issue on Charges this month, I've sent it on to Engineering to investigate why that page is incorrect and the inconsistencies between values.
As to storage costs, Datastore and BigQuery are very different systems with different performance goals, running in vastly different configurations to meet those goals. This means are underlying costs are different as well.
Regards,
Dan
Reply all
Reply to author
Forward
0 new messages