Migration from Datastore to Firestore in Datastore mode, timeline?

[Khaled Wagdy]

Hello,

I have seen the announcement of the automatic migration of all Datastore projects to Firestore in Datastore mode, I also remember reading somewhere that the entire migration should be done within 2019 (right?) but I can't help but wonder if there is a way we could know when our project Datastore will be migrated? 

We are very much looking forward to having strongly consistent queries, we have a few users now, and we'd like to scale, we've done our data structure using Entity groups and ancestor queries to ensure consistent queries, but there are some corners where this was not implemented and therefore for these entities we are still relying on eventual consistency. In our plan to scale, we'd like to put this behind us, and it would help to know when (even an estimate is fine!) of when the migration would occur.

We don't want to create a new project and manually migrate the datastore entities using the Import Export feature, as currently one can not switch to Firestore in Datastore from within the same project. The move to a new project would require that we re-configure the Domain DNS to the new project, setup the logger export settings, lose access to the previous database exports within Google Storage.. new project means new setup, etc...

Appreciate your feedback,

[Khaled Wagdy]

Hello,

Any notes, advise or general feedback on this?

[Jim Morrison]

HI Khaled,

On Fri, 12 Apr 2019 at 09:45, Khaled Wagdy <khaled....@gmail.com> wrote:

Hello,

I have seen the announcement of the automatic migration of all Datastore projects to Firestore in Datastore mode, I also remember reading somewhere that the entire migration should be done within 2019 (right?) but I can't help but wonder if there is a way we could know when our project Datastore will be migrated? 

Unfortunately, we don't have a timeline to share at the moment, but each project with data will be notified before the migration to Cloud Firestore in Datastore mode for the project.

 

We are very much looking forward to having strongly consistent queries, we have a few users now, and we'd like to scale, we've done our data structure using Entity groups and ancestor queries to ensure consistent queries, but there are some corners where this was not implemented and therefore for these entities we are still relying on eventual consistency. In our plan to scale, we'd like to put this behind us, and it would help to know when (even an estimate is fine!) of when the migration would occur.

We don't want to create a new project and manually migrate the datastore entities using the Import Export feature, as currently one can not switch to Firestore in Datastore from within the same project. The move to a new project would require that we re-configure the Domain DNS to the new project, setup the logger export settings, lose access to the previous database exports within Google Storage.. new project means new setup, etc...

Appreciate your feedback,

--
You received this message because you are subscribed to the Google Groups "Google Cloud Datastore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gcd-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-discuss/259c003b-4d69-47ba-b27b-ac1847fc260b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jim

[Dan Briggs]

Are there any updates about a timeline? I've got essentially the same question as Khaled.

On Tuesday, 16 April 2019 16:23:01 UTC-7, Jim Morrison wrote:

HI Khaled,

On Fri, 12 Apr 2019 at 09:45, Khaled Wagdy <khaled...@gmail.com> wrote:

Hello,

I have seen the announcement of the automatic migration of all Datastore projects to Firestore in Datastore mode, I also remember reading somewhere that the entire migration should be done within 2019 (right?) but I can't help but wonder if there is a way we could know when our project Datastore will be migrated? 

Unfortunately, we don't have a timeline to share at the moment, but each project with data will be notified before the migration to Cloud Firestore in Datastore mode for the project.

 

We are very much looking forward to having strongly consistent queries, we have a few users now, and we'd like to scale, we've done our data structure using Entity groups and ancestor queries to ensure consistent queries, but there are some corners where this was not implemented and therefore for these entities we are still relying on eventual consistency. In our plan to scale, we'd like to put this behind us, and it would help to know when (even an estimate is fine!) of when the migration would occur.

We don't want to create a new project and manually migrate the datastore entities using the Import Export feature, as currently one can not switch to Firestore in Datastore from within the same project. The move to a new project would require that we re-configure the Domain DNS to the new project, setup the logger export settings, lose access to the previous database exports within Google Storage.. new project means new setup, etc...

Appreciate your feedback,

--
You received this message because you are subscribed to the Google Groups "Google Cloud Datastore" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gcd-d...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-discuss/259c003b-4d69-47ba-b27b-ac1847fc260b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jim

[Dan Briggs]

Hello,

Just checking in. Any updates? Has the automatic update process begun? Or any idea when it will begin?

Thanks!

[Khaled Wagdy]

Hello Dan,

I ended up migrating my project and settings the custom domain, and logs sink setup...etc.

Had to pick a time where there is not much traffic to migrate the database, locked the DB so there is no new writes then removed the custom domain from the old project

All the best of luck!

[ThinRhino]

Hello Khaled,

I too need migrate our project, but across geographies. Just curious, how you did the DB & project migration.

For the DB did you do a download & update to new project?

Regards

ThinRhino

[Khaled Wagdy]

Hello ThisRhino,

I created a new project, made sure to choose Firebase in Datastore mode. I then followed this doc "Exporting and Importing Entities" to export the DB to Google Cloud storage, once this is done you need to follow the import steps to import the DB from old project Cloud Storage to new project Firebase in datastore mode. I think there was some Google Cloud storage account access setup so that the new project can have read access to old project Cloud Storage, sorry I don't remember where or how I did this, but I'm sure you will be able to find it with a quick search.

After the DB migration, deploy your app, index.yaml, any other services that you are using (other .yaml files). Make sure to also setup your domain in the new project App Engine -> Settings -> Custom domains. and remove your custom domain from the old project. We also have uploaded a maintenance page in the old project before all that, for users who try to login during the migration process.

I think that was it, we were lucky cuz we don't have many users right now (may also be considered as not being lucky 😅).

Me too am curious, you mentioned across geographies, what do you mean by that? Usually you select the project zone at start and that's it!

Best regards,

Khaled

[Francisco Reis]

I would also like to know, namely if we can ask to be as soon as possible.

My tests with a new project showed that many Cloud Firestore in Datastore Mode operations are considerable quicker (2/3 or 1/2 of the total request time with old datastore).

The advantage still remaining with the old Datastore is that it's easier to detect code/operations that can be optimized!

Francisco

[Francisco Reis]

Hi Jim,

More than one year has passed since your answer.

The migration that was supposed to happen until the end of 2019 did not happen.

Please, tell us something about what is going on for continuing to work with an old

technology when the one that replaced it is much better is quite frustrating.

Thanks!

Francisco

On Wednesday, 17 April 2019 00:23:01 UTC+1, Jim Morrison wrote:

HI Khaled,

On Fri, 12 Apr 2019 at 09:45, Khaled Wagdy <khaled...@gmail.com> wrote:

Hello,

I have seen the announcement of the automatic migration of all Datastore projects to Firestore in Datastore mode, I also remember reading somewhere that the entire migration should be done within 2019 (right?) but I can't help but wonder if there is a way we could know when our project Datastore will be migrated? 

Unfortunately, we don't have a timeline to share at the moment, but each project with data will be notified before the migration to Cloud Firestore in Datastore mode for the project.

 

We are very much looking forward to having strongly consistent queries, we have a few users now, and we'd like to scale, we've done our data structure using Entity groups and ancestor queries to ensure consistent queries, but there are some corners where this was not implemented and therefore for these entities we are still relying on eventual consistency. In our plan to scale, we'd like to put this behind us, and it would help to know when (even an estimate is fine!) of when the migration would occur.

We don't want to create a new project and manually migrate the datastore entities using the Import Export feature, as currently one can not switch to Firestore in Datastore from within the same project. The move to a new project would require that we re-configure the Domain DNS to the new project, setup the logger export settings, lose access to the previous database exports within Google Storage.. new project means new setup, etc...

Appreciate your feedback,

--
You received this message because you are subscribed to the Google Groups "Google Cloud Datastore" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gcd-d...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-discuss/259c003b-4d69-47ba-b27b-ac1847fc260b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jim

[Khaled Wagdy]

If you have the chance to do it manually just do it and get it over with

[Francisco Reis]

Not possible to do it manually. Too many objects with too many connections among them and to Filestore and to Blobstore and to Firebase Real Time database and to Dialog Flow (see scheme at www.umiverse.com).
Of course, a year ago would have been much easier but I relied on the fact that the automatic migration was going to happen during 2019.

[Francisco Reis]

The very high level scheme is at www.umniverse.com . Impossible to migrate manually without stopping the database and investing huge number of hours.

[Volker Braun]

The website https://cloud.google.com/datastore/docs/upgrade-to-firestore now says: "Starting in 2021, Google Cloud will gradually upgrade existing Datastore databases to Firestore in Datastore mode"

[Francisco Reis]

Thanks for the information!
It is bad news but at least we know it is still planned.

[Neerajkumar Dilip Bhatewara]

Hi,
Its almost Aug of 2021, anything concrete on when migration will be done ?
If i have understood the manual migration involves creating a new project then this does not makes sense to have different google project which will just have Firstore in it. But still will re-read and try to understand more on it.

But meanwhile, it will be great if we can get an update on when ? 

[Jim Morrison]

Hi Neerajkumar,

 The migration rate continues to scale up, but it will take us time to get through all of the existing Cloud Datastore databases.   As mentioned in the docs we'll notify before each of your databases is migrated.

Jim

--

You received this message because you are subscribed to the Google Groups "Google Cloud Datastore" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gcd-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gcd-discuss/4a5af6db-8396-43bf-bf14-bbc25690d020n%40googlegroups.com.

--

Jim

[Volker Braun]

By now I've had two projects upgrade and so far it works as advertised ;)