zigzag merge

[Rajesh Gupta]

I have 12-15 composite indexes for few KINDS. They got added 4-5 years ago.  

I am looking to regain indexes

Is there a way to reduce the indexes.  I am seeing if zigzag has any tool or utility to help find out the redundant indexes.

--

Regards,

Rajesh

www.GainERP.com

Accounting/Inventory/Orders/Sales/Purchase/FieldService on Google Cloud Platform and Mobile

[George (Cloud Platform Support)]

When you are sure that old indexes are no longer needed, you can delete them from Cloud Datastore using the datastore cleanup-indexes command. This command deletes all indexes for the production Cloud Datastore instance that are not mentioned in the local version of index.yaml. For more detail, you may read the "Deleting unused indexes" sub-chapter of the "Index Configuration" online document. 

[Rajesh Gupta]

I am not sure whether some indexes are needed,  because, I don't know, when that index is used in the application in which path and options and in which corner case. 

Hence, trying to find any zigzag utility to find that out any equivalent index, so that I am safe, when that corner case gets executed.

On Mon, Jun 25, 2018 at 1:11 AM, 'George (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:

When you are sure that old indexes are no longer needed, you can delete them from Cloud Datastore using the datastore cleanup-indexes command. This command deletes all indexes for the production Cloud Datastore instance that are not mentioned in the local version of index.yaml. For more detail, you may read the "Deleting unused indexes" sub-chapter of the "Index Configuration" online document. 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/b0be29f5-e0e5-442f-a4bf-3416ec23c1e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Regards,

Rajesh

www.VeersoftSolutions.com

www.GainERP.com

Accounting/Inventory/Orders/Sales/Purchase on Google Cloud Platform and Mobile

[George (Cloud Platform Support)]

If you find such a software, how do you plan to instruct it in determining if a set of indexes are to be deleted or not? 

[Kaan Soral]

I might be misunderstanding the question, and interpreting a complex question as a simple question (By "needing an index", if you mean, need an index to increase lookup performance, I do misunderstand the question)

But, to assess whether you need an index or not, run the SDK with the "require indexes"+"don't auto add indexes" (forgot the actual arguments) modes, then run the operations of your app in development, delete indexes one by one, if operations fail with an index error, re-add that index

It's an exhaustive process, but in any case, you need to re-understand what your app does now, so instead of trial error, just studying the app and eliminating the indexes manually could require the same effort

[Kaan Soral]

It would be indeed nice, if the production system kept metrics for index usage, that way, it would be obvious which indexes are needed or not

[Vitaly Bogomolov]

It would be indeed nice, if the production system kept metrics for index usage, that way, it would be obvious which indexes are needed or not

+++

WBR, Vitaly 

[George (Cloud Platform Support)]

Fully agreed here, it would be nice. As they say, usually the problem is in the details. In particular here, how would one determine how often is the index used? What would be the use-case for the new feature? How would you use it, exactly? A good use-case, well-formulated in step-by-step manner might prove a great help for Developers to determine the importance of the new feature and the urgency level for its implementation. 

[Vitaly Bogomolov]

What would be the use-case for the new feature? How would you use it, exactly? A good use-case, well-formulated in step-by-step manner might prove a great help for Developers to determine the importance of the new feature and the urgency level for its implementation. 

There is a large project that uses a datastore with a large number and size of indexes. Serving these indexes consumes a lot of resources when creating/modifying entries in the corresponding tables.

The project is several years old, at different times several development teams worked on it. The project has experienced several cycles of refactoring. There are reasonable suspicions that most of the existing indexes are no longer required for the correct operation of the project.

You need to find and delete indexes that are not used by the current version of the code. Static analysis of the code is laborious and unreliable because the project uses query.filter methods, GQL strings (including dynamically generated ones), indirect calls, etc.

If in the Datastore section of console.cloud.google.com you can see the load (in arbitrary units of measure) for active project indices, then identifying and removing unused indexes will be the simplest task.

In addition, the current implementation of the google.appengine.ext.testbed.init_datastore_v3_stub emulator does not take into account the contents of the project's index.yaml file. All calls to datastore are performed by the emulator successfully, as if all the necessary indexes exist. If testbed will generate errors if there are no necessary indexes in index.yaml, then these problems can be identified at the stage of automatic testing. 

WBR, Vitaly

[Attila-Mihaly Balazs]

- There is an optional parameter to init_datastore_v3_stub which you can use to require indexes: require_indexes=True. In fact we initialize our stub like the following for our unittests:

# this needs to be re-evaluated at every initialization since it's mutable and tests
        # might have overriden it
        consistency_policy = eval(
            '_datastore_stub_util_module.PseudoRandomHRConsistencyPolicy(probability=0)')
        _GAE_TESTBED.init_datastore_v3_stub(
            consistency_policy=consistency_policy, require_indexes=True, use_sqlite=True,
            root_path=_gae_app_path)

(this also makes so that it simulates the worst-case consistency for the datastore - you then need to call _GAE_TESTBED._test_stub_map.GetStub(_testbed_module.DATASTORE_SERVICE_NAME).Flush() explicitly when you want to have your entities visible to queries for example)

- There is also the --require_indexes=True parameter for the dev_appserver

- Finally, you can always have a "staging" project (ie. a separate Google project for testing) where you can try things out.

Attila

[Vitaly Bogomolov]

- There is an optional parameter to init_datastore_v3_stub which you can use to require indexes: require_indexes=True.

Bingo! :) Thanks alot for your advice.

testbed.init_datastore_v3_stub(require_indexes=True, root_path=project_dir)

also works with default in-memory datastore.

But 'require_indexes' and 'root_path' parameters has not referenced in the documentation in context of init_datastore_v3_stub. Appropriate documentation pages must be updated.

https://cloud.google.com/appengine/docs/standard/python/tools/localunittesting#datastore-memcache

WBD, Vitaly

 

[George (Cloud Platform Support)]

In fact the "bingo" info is amply documented on the "Local Development Server Options" documentation page, where usage for the --require_indexes=yes|no options are described in detail. 

[Vitaly Bogomolov]

In fact the "bingo" info is amply documented on the "Local Development Server Options" documentation page, where usage for the --require_indexes=yes|no options are described in detail. 

I was looking for a solution to the problem with testbed in the documentation section that describes automatic testing and could not guess that I need to look for in the description of the development server.

The behavior of the 'require_index' parameter in dev_appserver significantly differs from the behavior of the parameter 'require_indexes' in testbed.

1. If the parameter is not specified, the dev_appserver will change the index.yaml file, according reqired indexes. testbed does not change this file, but simply completes all calls to the datastore successfully.

2. To turn parameter on, it's value must be 'yes' in dev_appserver, but 'True' in testbed.

3. If parameter is on, dev_appserver raises errors for missed indexes in index.yaml. testbed raises errors for all requests, that requires indexes, because need parameter 'root_path'. 'root_path' must point to project directory, where index.yaml is placed.

So that the parameters of the 'require_index' in the testbed and in the dev_appserver can be considered as different parameters. 

WBR, Vitaly.

[George (Cloud Platform Support)]

Hello Vitaly, 

"google.appengine.ext.testbed package" documentation page defines init_datastore_v3_stub(enable=True, datastore_file=None, use_sqlite=False, auto_id_policy='sequential', **stub_kw_args). The signature does not specifically mention the "require_index" parameter. This is passed as one of the  **stub_kw_args – Keyword arguments passed on to the service stub. Would you like to see the require_index parameter defined explicitly in this context? What would you recommend, to improve documentation? 

[Rajesh Gupta]

+1

It would be nice, if the production system keeps this statistics.  It would be give more confidence to remove this, as we know it is not used over time.  

Currently, queries are written few years ago, and refactored by different people over the the years.  There may be a forgetting factor to delete the indexes.

Additionally, the queries, would have been dynamically built using reflection and metadata. 

It is hard to guess and delete the index,with various options and settings existing the software.

Best is, if Google gives an indication on when the index was used and the query used for the index.

On Thu, Jun 28, 2018 at 5:48 PM, Kaan Soral <kaan...@gmail.com> wrote:

It would be indeed nice, if the production system kept metrics for index usage, that way, it would be obvious which indexes are needed or not

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/3931eabd-023b-4650-a5d6-f8473ae26ffd%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Vitaly Bogomolov]

Hello George.

 

"google.appengine.ext.testbed package" documentation page defines init_datastore_v3_stub(enable=True, datastore_file=None, use_sqlite=False, auto_id_policy='sequential', **stub_kw_args). The signature does not specifically mention the "require_index" parameter. This is passed as one of the  **stub_kw_args – Keyword arguments passed on to the service stub. Would you like to see the require_index parameter defined explicitly in this context? What would you recommend, to improve documentation? 

Currently, both 'require_index' and 'root_path' parameters are supported by both testbed stubs: in-memory and sqlite. There are no reasons why the stubs that will be introduced in the future can't support these parameters.

It would be logical to explicitly declare these parameters in init_datastore_v3_stub

init_datastore_v3_stub(enable=True, datastore_file=None, use_sqlite=False, auto_id_policy='sequential', require_index=False, root_path=None, **stub_kw_args)

WBR, Vitaly

 

[George (Cloud Platform Support)]

This is a well-known and popular feature request, and Engineering is already working on its eventual implementation. No estimated time to implementation has been set. Meanwhile, you may follow developments in this thread in the Public Issue Tracker. 

[Vitaly Bogomolov]

This is a well-known and popular feature request, and Engineering is already working on its eventual implementation. No estimated time to implementation has been set. Meanwhile, you may follow developments in this thread in the Public Issue Tracker. 

Liked. But it not testbed-related.

WBR, Vitaly.

[George (Cloud Platform Support)]

Hello Vitaly, 

You are perfectly right, it is not related to testbed, but offers a thorough solution from the ground up, that would finally obviate the need to use testbed to obtain statistics on index usage.