Feasibility of archival storage

[Upendra Gandhi]

Hi,

Our organisation is looking at storing 5-10TB of data for archival/DR purposes on cloud storage. The initial data shall be around 5-10TB but we will do a weekly sync (upload/deletion) to the archived data.

The data is scattered across solaris, linux, windows systems in in-house data center.

1. Is there a way to get cost estimation?
2. What is the fastest way to export data to google cloud (initial export) to google cloud
3. What different options are there for weekly syncing data from scattered data from different platforms to the storage bucket?

Any help is appreciated.

Thanks,
Upendra

[Vishnu Fafat]

You are probably looking for Google Cloud Storage. (https://cloud.google.com/storage/docs/resources-support#community).

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/4c5aacb1-8d75-4c50-8cfb-249ee910c9e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Vishnu Fafat

[Nicholas (Google Cloud Support)]

Thanks for sharing your questions here!  Cloud Storage may indeed be the appropriate tool for the task you describe.

Cost estimations

The Google Cloud Storage Pricing article describes the cost of storage for the various storage classes as well as the costs of network egress.  With this information, you should be able to estimate the costs your organization would incur for its use.  You can also find in this same article some pricing examples to get a better idea of the end result.

Fastest export to the cloud

Assuming you mean the fastest way to upload data to a cloud storage bucket, gsutil is a command line tool specifically for interacting with cloud storage buckets.  It features many linux-like commands such as gsutil cp that can be used to copy files from local-to-bucket, bucket-to-local or bucket-to-bucket.  One can also use the -r option to perform this copy operation recursively through subdirectories.

Weekly syncing

Sticking with the gsutil tool described above, I would point out the rsync command.  As per the documentation:

The gsutil rsync command makes the contents under dst_url the same as the contents under src_url, by copying any missing files/objects (or those whose data has changed), and (if the -d option is specified) deleting any extra files/objects.

If you cannot install gsutil on each of those systems but CAN read their drive contents remotely (mapped network drive), you could have gsutil from a single machine sync content from the network drive to a storage.  This would increase your internal network traffic though as all data would have to go through the single machine first.  Otherwise, you could simply install gsutil on each of the machines to instead to upload to the bucket directly.

Please note the system requirements for gsutil.  I don't think you'll have any success installing it on a Solaris machine though I've not tested this myself.

Hope this helps!

[Upendra Gandhi]

Thanks Nichols and all for addressing my concerns despite that this group is not meant for answering such type of questions. I appreciate everyone's patience.

Thank you for the information. One last I wanted to know while estimating costs is Class A calls. How do I know how many Class A operations will be made in the Weekly sync(get/post/delete)? If you can point me to the url which has that information, that will be really helpful as well. If not, that's fine. Thanks again!!

~UG

--
You received this message because you are subscribed to a topic in the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-cloud-sql-discuss/dQv2QxhywFE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-cloud-sql-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/4f5fb324-cb1e-4cf7-9fea-4e163d77a479%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Upendra Gandhi
Systems Administrator, OTS
IIT Tower
10 W. 35th Street, 8th Floor, 8E3-2
Chicago, IL 60616
O: (312)567-3283
Alternate Email:
upendra...@iit.edu, upendra...@gmail.com
Keyserver: http://pgp.mit.edu/

 

 

[Nicholas (Google Cloud Support)]

Happy to answer your questions though I would suggest posting such Cloud Storage questions to the App Engine Google Groups in the future.

The operations table shows the different cost of each operation type.  The Cloud Storage APIs article states the following:

By default, gsutil versions starting with 4.0 interact with the JSON API.

As such, you can consider the costs relative to the JSON API when using gsutil for your sync/backups.  The cost of your backups will depend entirely the volume or reads and writes.  Without knowing too much specific detail about your needs, I can only suggest crunching some numbers for yourself with the documentation provided and testing it cautiously.

On Tuesday, June 14, 2016 at 4:11:35 PM UTC-4, Upendra Gandhi wrote: