Testing Models

[Lukas Karlsson]

I attempted to create my first model and I am running into trouble. The first time, it responded with status BROKEN, so I tried it with 2 other index IDs and I got the same error.

For example:

$ forseti model create inventory --id 2018-04-18T04:57:38.735183 TEST1

model {

  handle: "e013acfef409c60dead91051c8400e7a"

  name: "TEST1"

  status: "BROKEN"

  description: "{}"

}

$ forseti model list

handle: "32aebc7cbb73b7d1e4f3289302bceee5"

name: "TEST"

status: "BROKEN"

description: "{}"

message: "Traceback (most recent call last):\n  File \"/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/model/importer/importer.py\", line 173, 

in run\n    True) as inventory:\n  File \"/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/inventory/storage.py\", line 906, in __enter__\n    self.open()\n  File \"/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/inventory/storage.py\", line 653, in open\n    self.inventory_index = self._open(existing_id)\n  File \"/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/inventory/storage.py\", line 602, in _open\n    InventoryState.PARTIAL_SUCCESS]))\n  File \"build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py\", line 2854, in one\n    raise orm_exc.NoResultFound(\"No row was found for one()\")\nNoResultFound: No row was found for one()\n"

handle: "e013acfef409c60dead91051c8400e7a"

name: "TEST1"

status: "BROKEN"

description: "{}"

message: "Traceback (most recent call last):\n  File \"/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/model/importer/importer.py\", line 173, 

in run\n    True) as inventory:\n  File \"/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/inventory/storage.py\", line 906, in __enter__\n    self.open()\n  File \"/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/inventory/storage.py\", line 653, in open\n    self.inventory_index = self._open(existing_id)\n  File \"/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/inventory/storage.py\", line 602, in _open\n    InventoryState.PARTIAL_SUCCESS]))\n  File \"build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py\", line 2854, in one\n    raise orm_exc.NoResultFound(\"No row was found for one()\")\nNoResultFound: No row was found for one()\n"

handle: "cf5e7fe0af7dff9a2f04a298d6f77992"

name: "TEST2"

status: "BROKEN"

description: "{}"

message: "Traceback (most recent call last):\n  File \"/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/model/importer/importer.py\", line 173, 

in run\n    True) as inventory:\n  File \"/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/inventory/storage.py\", line 906, in __enter__\n    self.open()\n  File \"/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/inventory/storage.py\", line 653, in open\n    self.inventory_index = self._open(existing_id)\n  File \"/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/inventory/storage.py\", line 602, in _open\n    InventoryState.PARTIAL_SUCCESS]))\n  File \"build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py\", line 2854, in one\n    raise orm_exc.NoResultFound(\"No row was found for one()\")\nNoResultFound: No row was found for one()\n"

Deleting does work properly.  I was able to delete all three BROKEN models and now they are gone.

/l

--

Lukas Karlsson, Cloud Architect / Developer Advocate

Broad Institute of MIT and Harvard
75 Ames Street, 11129, Cambridge, Massachusetts 02142
karl...@broadinstitute.org, +1.6177147142

[Joe Cheuk]

Hi Lukas,

Did you build the data model with inventory index that has status = "SUCCESS"?

Best regards,

Joe

--
You received this message because you are subscribed to the Google Groups "Forseti Security Beta Testers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to beta-testers...@forsetisecurity.org.
To post to this group, send email to beta-t...@forsetisecurity.org.
To view this discussion on the web visit https://groups.google.com/a/forsetisecurity.org/d/msgid/beta-testers/CADT05SRheMzG8G%3Dh35Zu9Ut%3DCwsTLaAz9UMPZureSW6%3Dj-4Xdg%40mail.gmail.com.

[Lukas Karlsson]

Turns out that I have no such index as of yet.  I am hoping that now that I've changed it to 12 hours that it should get one in the next day or so.

I didn't realize that if a new inventory ran that the old one would never complete.  The "most recent one" shows zero objects so that was probably running when I "reset" the server vm just recently.

$ forseti inventory list | grep -v warnings:

id: "2018-04-18T04:57:38.735183"

schema_version: 1

count_objects: 158737

status: "CREATED"

id: "2018-04-18T06:36:17.896636"

schema_version: 1

count_objects: 158999

status: "CREATED"

id: "2018-04-18T08:36:17.636273"

schema_version: 1

count_objects: 158241

status: "CREATED"

id: "2018-04-18T10:36:17.832025"

schema_version: 1

count_objects: 157999

status: "CREATED"

id: "2018-04-18T12:36:18.619962"

schema_version: 1

count_objects: 158885

status: "CREATED"

id: "2018-04-18T13:35:23.344416"

schema_version: 1

count_objects: 156022

status: "CREATED"

id: "2018-04-18T14:36:18.146662"

schema_version: 1

count_objects: 161343

status: "CREATED"

id: "2018-04-18T16:36:18.098157"

schema_version: 1

count_objects: 158883

status: "CREATED"

id: "2018-04-18T18:36:18.683615"

schema_version: 1

count_objects: 159495

status: "CREATED"

id: "2018-04-18T20:36:18.590653"

schema_version: 1

count_objects: 224150

status: "CREATED"

id: "2018-04-18T22:36:18.740627"

schema_version: 1

status: "CREATED"

To unsubscribe from this group and stop receiving emails from it, send an email to beta-testers+unsubscribe@forsetisecurity.org.
To post to this group, send email to beta-testers@forsetisecurity.org.

To view this discussion on the web visit https://groups.google.com/a/forsetisecurity.org/d/msgid/beta-testers/CADT05SRheMzG8G%3Dh35Zu9Ut%3DCwsTLaAz9UMPZureSW6%3Dj-4Xdg%40mail.gmail.com.

[Joe Cheuk]

Hi Lukas,

For the current version (2.0-rc2) of Forseti, since we restart the Forseti server every time we run the cron job so if the inventory process is taking too long, we might ended up being in a state where no inventory is created successfully because the restart will kill all the existing inventory run. The easiest way to resolve this problem for now is to decrease the frequency of the cron job.

Once this issue is resolved, we will no longer have this problem:

https://github.com/GoogleCloudPlatform/forseti-security/issues/1396

Thanks,

Joe

To unsubscribe from this group and stop receiving emails from it, send an email to beta-testers...@forsetisecurity.org.
To post to this group, send email to beta-t...@forsetisecurity.org.

To view this discussion on the web visit https://groups.google.com/a/forsetisecurity.org/d/msgid/beta-testers/CADT05SRheMzG8G%3Dh35Zu9Ut%3DCwsTLaAz9UMPZureSW6%3Dj-4Xdg%40mail.gmail.com.

[Castex, Antoine]

Hello

This feature is to create your personnalized inventory ?

Where is the documentation related to this please ?

Cordialement / Best Regards

CASTEX Antoine

Cloud Developer

Technology & Operations - Digital Factory

Group Information Systems & Technology (IS&T)

VEOLIA 30 rue Madeleine Vionnet / 93300 Aubervilliers / France

Bureau : 0 - Ouest - 137F

Direct: +33 1 85 57 70 91

Mobile: +33 6 10 93 63 49

www.veolia.com

    

--

You received this message because you are subscribed to the Google Groups "Forseti Security Beta Testers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to beta-testers+unsubscribe@forsetisecurity.org.
To post to this group, send email to beta-testers@forsetisecurity.org.

To view this discussion on the web visit https://groups.google.com/a/forsetisecurity.org/d/msgid/beta-testers/CADT05SRheMzG8G%3Dh35Zu9Ut%3DCwsTLaAz9UMPZureSW6%3Dj-4Xdg%40mail.gmail.com.

--------------------------------------------------------------------------------------------

This e-mail transmission (message and any attached files) may contain information that is proprietary, privileged and/or confidential to Veolia Environnement and/or its affiliates and is intended exclusively for the person(s) to whom it is addressed. If you are not the intended recipient, please notify the sender by return e-mail and delete all copies of this e-mail, including all attachments. Unless expressly authorized, any use, disclosure, publication, retransmission or dissemination of this e-mail and/or of its attachments is strictly prohibited. 

Ce message electronique et ses fichiers attaches sont strictement confidentiels et peuvent contenir des elements dont Veolia Environnement et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc destines a l'usage de leurs seuls destinataires. Si vous avez recu ce message par erreur, merci de le retourner a son emetteur et de le detruire ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la publication, la distribution, ou la reproduction non expressement autorisees de ce message et de ses pieces attachees sont interdites.

--------------------------------------------------------------------------------------------

[Lukas Karlsson]

I got my first SUCCESS on a inventory last night, after change it to every 12 hours instead of every 2.

I tried twice to create a model from this successful inventory and it failed with the same issue, seems like MySQL server connection went away:

$ forseti model create inventory --id 2018-04-19T00:36:18.626100 TEST1Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1031, in <module> main(sys.argv[1:], ENV_CONFIG) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1013, in main services[config.service](client, config, output, config_env) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 694, in run_model actions[config.action]() File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 670, in do_create_model config.background) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/client.py", line 219, in new_model background=background)) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 487, in __call__ return _end_unary_response_blocking(state, call, False, deadline) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking raise _Rendezvous(state, None, None, deadline)grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, Exception calling application: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'INSERT INTO `0454555e6f5e870c1966582538c7f321_resources` (full_name, type_name, name, type, policy_update_counter, display_name, email, data, parent_type_name) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5/', 'bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "email": "GROUP_0e0fbda1-daf4-4101-aa78-e712f9aa31f5-REA...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "storageClass": "STANDARD", "timeCreated": "2016-07-08T19:56:33.306Z", "updated": "2016-12-16T19:24:10.582Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd/', 'bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd", "email": "GROUP_0e10e603-5346-4ec0-8ae0-...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e10e603-5346-4ec0-8ae0-f3aa995264fd", "storageClass": "STANDARD", "timeCreated": "2016-05-04T18:51:40.559Z", "updated": "2016-12-16T19:24:12.058Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2/', 'bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e1d0393-9ae4-4d4f-b527-e175573865d2", "email": "806222273987-ksinuuegh...@developer.gserviceaccount.com" ... (1640 characters truncated) ... -0e1d0393-9ae4-4d4f-b527-e175573865d2", "storageClass": "STANDARD", "timeCreated": "2018-01-03T16:44:23.645Z", "updated": "2018-01-03T16:44:23.645Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c/', 'bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "email": "GROUP_I_0e2512e9-ebf4-405...@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "storageClass": "STANDARD", "timeCreated": "2017-05-09T17:40:42.742Z", "updated": "2017-05-09T17:40:42.742Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e30616a-6330-4e32-ab08-87427da34658/', 'bucket/fc-0e30616a-6330-4e32-ab08-87427da34658', 'fc-0e30616a-6330-4e32-ab08-87427da34658', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e30616a-6330-4e32-ab08-87427da34658", "email": "GROUP_0e30616a-6330-4e32...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e30616a-6330-4e32-ab08-87427da34658", "storageClass": "STANDARD", "timeCreated": "2017-11-17T19:23:44.837Z", "updated": "2017-11-17T19:23:44.837Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b/', 'bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b', 'fc-0e33525c-6d13-44c5-afae-c6527312660b', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e33525c-6d13-44c5-afae-c6527312660b", "email": "GROUP_0e33525c-6d13-44c5...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e33525c-6d13-44c5-afae-c6527312660b", "storageClass": "STANDARD", "timeCreated": "2017-10-03T16:46:33.867Z", "updated": "2017-10-03T16:46:33.867Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f/', 'bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "email": "GROUP_0e474340-ba14-42c1...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "storageClass": "STANDARD", "timeCreated": "2018-04-04T15:08:29.946Z", "updated": "2018-04-04T15:08:29.946Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603/', 'bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "email": "GROUP_I_0e4f21f9-ad42-413d-8b6c-df43b5cf...@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "storageClass": "STANDARD", "timeCreated": "2017-02-02T20:23:32.745Z", "updated": "2017-02-02T20:23:32.745Z"}', 'project/broad-dsde-dev') ... displaying 10 of 19378 total bound parameter sets ... ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088/', 'bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'bucket', 0, '', '', '{"acl": [{"bucket": "zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zap-staging-4 ... (4549 characters truncated) ... -41ead8a1-9b62-420e-a57a-dcdfc4a10088", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:25:00.715Z", "updated": "2018-03-20T22:25:07.948Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b/', 'bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'bucket', 0, '', '', '{"acl": [{"bucket": "zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zxcass-sta ... (4612 characters truncated) ... -9888f6f5-01c5-43c7-a52e-521f8f8b808b", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:31:14.668Z", "updated": "2018-03-20T22:31:22.007Z"}', 'project/broad-dsde-dev'))] (Background on this error at: http://sqlalche.me/e/e3q8))>

To unsubscribe from this group and stop receiving emails from it, send an email to beta-testers+unsubscribe@forsetisecurity.org.
To post to this group, send email to beta-testers@forsetisecurity.org.

To view this discussion on the web visit https://groups.google.com/a/forsetisecurity.org/d/msgid/beta-testers/CADT05SRheMzG8G%3Dh35Zu9Ut%3DCwsTLaAz9UMPZureSW6%3Dj-4Xdg%40mail.gmail.com.

--

Lukas Karlsson, Cloud Architect / Developer Advocate

Broad Institute of MIT and Harvard
75 Ames Street, 11129, Cambridge, Massachusetts 02142
karl...@broadinstitute.org, +1.6177147142

[Joe Cheuk]

Hi Antoine,

This is not a feature for creating your personnalized inventory, we build data model from the inventory data so that we can get a better understanding of the relationship between those data.

Once we have the data model ready, you can query the data model directly through Forseti Explainer, the documentation of how to use Forseti Explainer can be found here:

https://docs.google.com/document/d/1e6uJgTZTtTCMFaZ0BMs_EtV9ULcunq-tKsNiSxIMg3M/preview#heading=h.vgovm6yab1n2

Lukas,

Thanks for the updates! I suspect the exception you are getting might be because the default allowed packet size is too small. 

Can you try updating the "max_allowed_packet" flag to a 64000000 (64 mbs) and see if that resolves the problem?

The instructions on how to add a mysql flag on cloud sql instance can be found here:

https://cloud.google.com/sql/docs/mysql/flags

Best regards,

Joe

To unsubscribe from this group and stop receiving emails from it, send an email to beta-testers...@forsetisecurity.org.
To post to this group, send email to beta-t...@forsetisecurity.org.

To view this discussion on the web visit https://groups.google.com/a/forsetisecurity.org/d/msgid/beta-testers/CADT05SRheMzG8G%3Dh35Zu9Ut%3DCwsTLaAz9UMPZureSW6%3Dj-4Xdg%40mail.gmail.com.

--

Lukas Karlsson, Cloud Architect / Developer Advocate

Broad Institute of MIT and Harvard
75 Ames Street, 11129, Cambridge, Massachusetts 02142
karl...@broadinstitute.org, +1.6177147142

[Lukas Karlsson]

That helped, in that I do not get the mysql error anymore.

But based on the mysql activity in the console, it looks like it only ran for around 5 minutes and now the server VM is locked up it seems and I can't ssh into it.

I might have to try making the server VM larger and re-deploying.

/l

On Thu, Apr 19, 2018 at 12:40 PM, Joe Cheuk <joec...@google.com> wrote:

Hi Antoine,

This is not a feature for creating your personnalized inventory, we build data model from the inventory data so that we can get a better understanding of the relationship between those data.

Once we have the data model ready, you can query the data model directly through Forseti Explainer, the documentation of how to use Forseti Explainer can be found here:

https://docs.google.com/document/d/1e6uJgTZTtTCMFaZ0BMs_EtV9ULcunq-tKsNiSxIMg3M/preview#heading=h.vgovm6yab1n2

Lukas,

Thanks for the updates! I suspect the exception you are getting might be because the default allowed packet size is too small. 

Can you try updating the "max_allowed_packet" flag to a 64000000 (64 mbs) and see if that resolves the problem?

The instructions on how to add a mysql flag on cloud sql instance can be found here:

https://cloud.google.com/sql/docs/mysql/flags

Best regards,

Joe

On Thu, Apr 19, 2018 at 7:53 AM Lukas Karlsson <karl...@broadinstitute.org> wrote:

I got my first SUCCESS on a inventory last night, after change it to every 12 hours instead of every 2.

I tried twice to create a model from this successful inventory and it failed with the same issue, seems like MySQL server connection went away:

$ forseti model create inventory --id 2018-04-19T00:36:18.626100 TEST1Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1031, in <module> main(sys.argv[1:], ENV_CONFIG) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1013, in main services[config.service](client, config, output, config_env) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 694, in run_model actions[config.action]() File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 670, in do_create_model config.background) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/client.py", line 219, in new_model background=background)) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 487, in __call__ return _end_unary_response_blocking(state, call, False, deadline) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking raise _Rendezvous(state, None, None, deadline)grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, Exception calling application: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'INSERT INTO `0454555e6f5e870c1966582538c7f321_resources` (full_name, type_name, name, type, policy_update_counter, display_name, email, data, parent_type_name) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5/', 'bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "email": "GROUP_0e0fbda1-daf4-4101-aa78-e712f9aa31f5-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "storageClass": "STANDARD", "timeCreated": "2016-07-08T19:56:33.306Z", "updated": "2016-12-16T19:24:10.582Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd/', 'bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd", "email": "GROUP_0e10e603-5346-4ec0-8ae0-f3aa995264fd-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e10e603-5346-4ec0-8ae0-f3aa995264fd", "storageClass": "STANDARD", "timeCreated": "2016-05-04T18:51:40.559Z", "updated": "2016-12-16T19:24:12.058Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2/', 'bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e1d0393-9ae4-4d4f-b527-e175573865d2", "email": "806222273987-ksinuueghug8u81i36lq8aof266q19h...@developer.gserviceaccount.com" ... (1640 characters truncated) ... -0e1d0393-9ae4-4d4f-b527-e175573865d2", "storageClass": "STANDARD", "timeCreated": "2018-01-03T16:44:23.645Z", "updated": "2018-01-03T16:44:23.645Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c/', 'bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "email": "GROUP_I_0e2512e9-ebf4-4054-9dd8-aa664b171c3c-READER@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "storageClass": "STANDARD", "timeCreated": "2017-05-09T17:40:42.742Z", "updated": "2017-05-09T17:40:42.742Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e30616a-6330-4e32-ab08-87427da34658/', 'bucket/fc-0e30616a-6330-4e32-ab08-87427da34658', 'fc-0e30616a-6330-4e32-ab08-87427da34658', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e30616a-6330-4e32-ab08-87427da34658", "email": "GROUP_0e30616a-6330-4e32-ab08-87427da34658-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e30616a-6330-4e32-ab08-87427da34658", "storageClass": "STANDARD", "timeCreated": "2017-11-17T19:23:44.837Z", "updated": "2017-11-17T19:23:44.837Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b/', 'bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b', 'fc-0e33525c-6d13-44c5-afae-c6527312660b', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e33525c-6d13-44c5-afae-c6527312660b", "email": "GROUP_0e33525c-6d13-44c5-afae-c6527312660b-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e33525c-6d13-44c5-afae-c6527312660b", "storageClass": "STANDARD", "timeCreated": "2017-10-03T16:46:33.867Z", "updated": "2017-10-03T16:46:33.867Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f/', 'bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "email": "GROUP_0e474340-ba14-42c1-ae0e-5cc90e8bfc3f-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "storageClass": "STANDARD", "timeCreated": "2018-04-04T15:08:29.946Z", "updated": "2018-04-04T15:08:29.946Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603/', 'bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "email": "GROUP_I_0e4f21f9-ad42-413d-8b6c-df43b5cf1603-READER@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "storageClass": "STANDARD", "timeCreated": "2017-02-02T20:23:32.745Z", "updated": "2017-02-02T20:23:32.745Z"}', 'project/broad-dsde-dev') ... displaying 10 of 19378 total bound parameter sets ... ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088/', 'bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'bucket', 0, '', '', '{"acl": [{"bucket": "zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zap-staging-4 ... (4549 characters truncated) ... -41ead8a1-9b62-420e-a57a-dcdfc4a10088", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:25:00.715Z", "updated": "2018-03-20T22:25:07.948Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b/', 'bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'bucket', 0, '', '', '{"acl": [{"bucket": "zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zxcass-sta ... (4612 characters truncated) ... -9888f6f5-01c5-43c7-a52e-521f8f8b808b", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:31:14.668Z", "updated": "2018-03-20T22:31:22.007Z"}', 'project/broad-dsde-dev'))] (Background on this error at: http://sqlalche.me/e/e3q8))>

To unsubscribe from this group and stop receiving emails from it, send an email to beta-testers+unsubscribe@forsetisecurity.org.
To post to this group, send email to beta-testers@forsetisecurity.org.

To view this discussion on the web visit https://groups.google.com/a/forsetisecurity.org/d/msgid/beta-testers/CADT05SRheMzG8G%3Dh35Zu9Ut%3DCwsTLaAz9UMPZureSW6%3Dj-4Xdg%40mail.gmail.com.

--

Lukas Karlsson, Cloud Architect / Developer Advocate

Broad Institute of MIT and Harvard
75 Ames Street, 11129, Cambridge, Massachusetts 02142
karl...@broadinstitute.org, +1.6177147142

--

Lukas Karlsson, Cloud Architect / Developer Advocate

Broad Institute of MIT and Harvard
75 Ames Street, 11129, Cambridge, Massachusetts 02142
karl...@broadinstitute.org, +1.6177147142

[yukie sato]

Hi Joe,

Piggy backing on this thread about model Rendezvous error...

I'm getting following errors (removed actual group name and user names from the message).

forseti model create inventory --id 2018-04-19T16:32:18.054460 test_model2

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1031, in <module>

    main(sys.argv[1:], ENV_CONFIG)

  File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1013, in main

    services[config.service](client, config, output, config_env)

  File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 694, in run_model

    actions[config.action]()

  File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 670, in do_create_model

    config.background)

  File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/client.py", line 219, in new_model

    background=background))

  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 487, in __call__

    return _end_unary_response_blocking(state, call, False, deadline)

  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking

    raise _Rendezvous(state, None, None, deadline)

grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, Exception calling application: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.IntegrityError) (1062, "Duplicate entry 'group/ GROUP NAME REMOVED HERE' for key 'PRIMARY'") [SQL: u'INSERT INTO `2cc28387d8da6c6278954677fc2478d0_members` (name, type, member_name) VALUES (%s, %s, %s)'] [parameters: (('user/  HERE LIST OF USERS '))] (Background on this error at: http://sqlalche.me/e/gkpj))>

Thanks.

[Lukas Karlsson]

Now that I'm back to a bigger vm now I'm getting the same error again but I already checked and I have the databaseFlag set for max_allowed_packet=64000000.

$ forseti model create inventory --id 2018-04-19T00:36:18.626100 test2Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1031, in <module> main(sys.argv[1:], ENV_CONFIG) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1013, in main services[config.service](client, config, output, config_env) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 694, in run_model actions[config.action]() File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 670, in do_create_model config.background) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/client.py", line 219, in new_model background=background)) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 487, in __call__ return _end_unary_response_blocking(state, call, False, deadline) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking raise _Rendezvous(state, None, None, deadline)grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, Exception calling application: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'INSERT INTO `1125264ff962ac1469d71bcbd5019e8f_resources` (full_name, type_name, name, type, policy_update_counter, display_name, email, data, parent_type_name) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5/', 'bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "email": "GROUP_0e0fbda1-daf4-4101-aa78-e712f9aa31f5-REA...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "storageClass": "STANDARD", "timeCreated": "2016-07-08T19:56:33.306Z", "updated": "2016-12-16T19:24:10.582Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd/', 'bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd", "email": "GROUP_0e10e603-5346-4ec0-8ae0-...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e10e603-5346-4ec0-8ae0-f3aa995264fd", "storageClass": "STANDARD", "timeCreated": "2016-05-04T18:51:40.559Z", "updated": "2016-12-16T19:24:12.058Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2/', 'bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e1d0393-9ae4-4d4f-b527-e175573865d2", "email": "806222273987-ksinuuegh...@developer.gserviceaccount.com" ... (1640 characters truncated) ... -0e1d0393-9ae4-4d4f-b527-e175573865d2", "storageClass": "STANDARD", "timeCreated": "2018-01-03T16:44:23.645Z", "updated": "2018-01-03T16:44:23.645Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c/', 'bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "email": "GROUP_I_0e2512e9-ebf4-405...@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "storageClass": "STANDARD", "timeCreated": "2017-05-09T17:40:42.742Z", "updated": "2017-05-09T17:40:42.742Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e30616a-6330-4e32-ab08-87427da34658/', 'bucket/fc-0e30616a-6330-4e32-ab08-87427da34658', 'fc-0e30616a-6330-4e32-ab08-87427da34658', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e30616a-6330-4e32-ab08-87427da34658", "email": "GROUP_0e30616a-6330-4e32...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e30616a-6330-4e32-ab08-87427da34658", "storageClass": "STANDARD", "timeCreated": "2017-11-17T19:23:44.837Z", "updated": "2017-11-17T19:23:44.837Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b/', 'bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b', 'fc-0e33525c-6d13-44c5-afae-c6527312660b', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e33525c-6d13-44c5-afae-c6527312660b", "email": "GROUP_0e33525c-6d13-44c5...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e33525c-6d13-44c5-afae-c6527312660b", "storageClass": "STANDARD", "timeCreated": "2017-10-03T16:46:33.867Z", "updated": "2017-10-03T16:46:33.867Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f/', 'bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "email": "GROUP_0e474340-ba14-42c1...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "storageClass": "STANDARD", "timeCreated": "2018-04-04T15:08:29.946Z", "updated": "2018-04-04T15:08:29.946Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603/', 'bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "email": "GROUP_I_0e4f21f9-ad42-413d-8b6c-df43b5cf...@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "storageClass": "STANDARD", "timeCreated": "2017-02-02T20:23:32.745Z", "updated": "2017-02-02T20:23:32.745Z"}', 'project/broad-dsde-dev') ... displaying 10 of 19378 total bound parameter sets ... ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088/', 'bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'bucket', 0, '', '', '{"acl": [{"bucket": "zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zap-staging-4 ... (4549 characters truncated) ... -41ead8a1-9b62-420e-a57a-dcdfc4a10088", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:25:00.715Z", "updated": "2018-03-20T22:25:07.948Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b/', 'bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'bucket', 0, '', '', '{"acl": [{"bucket": "zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zxcass-sta ... (4612 characters truncated) ... -9888f6f5-01c5-43c7-a52e-521f8f8b808b", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:31:14.668Z", "updated": "2018-03-20T22:31:22.007Z"}', 'project/broad-dsde-dev'))] (Background on this error at: http://sqlalche.me/e/e3q8))>

On Thu, Apr 19, 2018 at 2:44 PM, Lukas Karlsson <karl...@broadinstitute.org> wrote:

That helped, in that I do not get the mysql error anymore.

But based on the mysql activity in the console, it looks like it only ran for around 5 minutes and now the server VM is locked up it seems and I can't ssh into it.

I might have to try making the server VM larger and re-deploying.

/l

On Thu, Apr 19, 2018 at 12:40 PM, Joe Cheuk <joec...@google.com> wrote:

Hi Antoine,

This is not a feature for creating your personnalized inventory, we build data model from the inventory data so that we can get a better understanding of the relationship between those data.

Once we have the data model ready, you can query the data model directly through Forseti Explainer, the documentation of how to use Forseti Explainer can be found here:

https://docs.google.com/document/d/1e6uJgTZTtTCMFaZ0BMs_EtV9ULcunq-tKsNiSxIMg3M/preview#heading=h.vgovm6yab1n2

Lukas,

Thanks for the updates! I suspect the exception you are getting might be because the default allowed packet size is too small. 

Can you try updating the "max_allowed_packet" flag to a 64000000 (64 mbs) and see if that resolves the problem?

The instructions on how to add a mysql flag on cloud sql instance can be found here:

https://cloud.google.com/sql/docs/mysql/flags

Best regards,

Joe

On Thu, Apr 19, 2018 at 7:53 AM Lukas Karlsson <karl...@broadinstitute.org> wrote:

I got my first SUCCESS on a inventory last night, after change it to every 12 hours instead of every 2.

I tried twice to create a model from this successful inventory and it failed with the same issue, seems like MySQL server connection went away:

$ forseti model create inventory --id 2018-04-19T00:36:18.626100 TEST1Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1031, in <module> main(sys.argv[1:], ENV_CONFIG) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1013, in main services[config.service](client, config, output, config_env) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 694, in run_model actions[config.action]() File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 670, in do_create_model config.background) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/client.py", line 219, in new_model background=background)) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 487, in __call__ return _end_unary_response_blocking(state, call, False, deadline) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking raise _Rendezvous(state, None, None, deadline)grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, Exception calling application: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'INSERT INTO `0454555e6f5e870c1966582538c7f321_resources` (full_name, type_name, name, type, policy_update_counter, display_name, email, data, parent_type_name) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5/', 'bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "email": "GROUP_0e0fbda1-daf4-4101-aa78-e712f9aa31f5-REA...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "storageClass": "STANDARD", "timeCreated": "2016-07-08T19:56:33.306Z", "updated": "2016-12-16T19:24:10.582Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd/', 'bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd", "email": "GROUP_0e10e603-5346-4ec0-8ae0-f3aa99526...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e10e603-5346-4ec0-8ae0-f3aa995264fd", "storageClass": "STANDARD", "timeCreated": "2016-05-04T18:51:40.559Z", "updated": "2016-12-16T19:24:12.058Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2/', 'bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e1d0393-9ae4-4d4f-b527-e175573865d2", "email": "806222273987-ksinuueghug8u81i36lq8aof266q19hl@developer.gserviceaccount.com" ... (1640 characters truncated) ... -0e1d0393-9ae4-4d4f-b527-e175573865d2", "storageClass": "STANDARD", "timeCreated": "2018-01-03T16:44:23.645Z", "updated": "2018-01-03T16:44:23.645Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c/', 'bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "email": "GROUP_I_0e2512e9-ebf4-4054-9dd8-aa664b17...@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "storageClass": "STANDARD", "timeCreated": "2017-05-09T17:40:42.742Z", "updated": "2017-05-09T17:40:42.742Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e30616a-6330-4e32-ab08-87427da34658/', 'bucket/fc-0e30616a-6330-4e32-ab08-87427da34658', 'fc-0e30616a-6330-4e32-ab08-87427da34658', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e30616a-6330-4e32-ab08-87427da34658", "email": "GROUP_0e30616a-6330-4e32-ab08-87427da34...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e30616a-6330-4e32-ab08-87427da34658", "storageClass": "STANDARD", "timeCreated": "2017-11-17T19:23:44.837Z", "updated": "2017-11-17T19:23:44.837Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b/', 'bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b', 'fc-0e33525c-6d13-44c5-afae-c6527312660b', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e33525c-6d13-44c5-afae-c6527312660b", "email": "GROUP_0e33525c-6d13-44c5-afae-c65273126...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e33525c-6d13-44c5-afae-c6527312660b", "storageClass": "STANDARD", "timeCreated": "2017-10-03T16:46:33.867Z", "updated": "2017-10-03T16:46:33.867Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f/', 'bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "email": "GROUP_0e474340-ba14-42c1-ae0e-5cc90e8bf...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "storageClass": "STANDARD", "timeCreated": "2018-04-04T15:08:29.946Z", "updated": "2018-04-04T15:08:29.946Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603/', 'bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "email": "GROUP_I_0e4f21f9-ad42-413d-8b6c-df43b5cf...@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "storageClass": "STANDARD", "timeCreated": "2017-02-02T20:23:32.745Z", "updated": "2017-02-02T20:23:32.745Z"}', 'project/broad-dsde-dev') ... displaying 10 of 19378 total bound parameter sets ... ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088/', 'bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'bucket', 0, '', '', '{"acl": [{"bucket": "zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zap-staging-4 ... (4549 characters truncated) ... -41ead8a1-9b62-420e-a57a-dcdfc4a10088", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:25:00.715Z", "updated": "2018-03-20T22:25:07.948Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b/', 'bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'bucket', 0, '', '', '{"acl": [{"bucket": "zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zxcass-sta ... (4612 characters truncated) ... -9888f6f5-01c5-43c7-a52e-521f8f8b808b", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:31:14.668Z", "updated": "2018-03-20T22:31:22.007Z"}', 'project/broad-dsde-dev'))] (Background on this error at: http://sqlalche.me/e/e3q8))>

[Joe Cheuk]

Hi Lukas,

We are doing bulk inserts to optimize the performance while building the data model, I think in your case since there are a lot of data so the sql statement becomes very big and unable to send that statement to the server. 

I have opened an issue to not do bulk insert in the future to avoid issues like this:

https://github.com/GoogleCloudPlatform/forseti-security/issues/1442

At this moment I think the only option is to increase the max_allowed_packet again to the maximum 1073741824 (1gb) and see if that works.

Yukie,

Thanks for reporting the issue! This is a known issue on our end and we are working on addressing it.

We have a pull request opened for this issue:

https://github.com/GoogleCloudPlatform/forseti-security/pull/1440

Once it's ready, we will patch the 2.0-rc2 branch with the updates so you can try it out.

Best regards,

Joe

On Thu, Apr 19, 2018 at 1:00 PM Lukas Karlsson <karl...@broadinstitute.org> wrote:

Now that I'm back to a bigger vm now I'm getting the same error again but I already checked and I have the databaseFlag set for max_allowed_packet=64000000.

$ forseti model create inventory --id 2018-04-19T00:36:18.626100 test2Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1031, in <module> main(sys.argv[1:], ENV_CONFIG) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1013, in main services[config.service](client, config, output, config_env) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 694, in run_model actions[config.action]() File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 670, in do_create_model config.background) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/client.py", line 219, in new_model background=background)) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 487, in __call__ return _end_unary_response_blocking(state, call, False, deadline) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking raise _Rendezvous(state, None, None, deadline)grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, Exception calling application: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'INSERT INTO `1125264ff962ac1469d71bcbd5019e8f_resources` (full_name, type_name, name, type, policy_update_counter, display_name, email, data, parent_type_name) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5/', 'bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "email": "GROUP_0e0fbda1-daf4-4101-aa78-e712f9aa31f5-REA...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "storageClass": "STANDARD", "timeCreated": "2016-07-08T19:56:33.306Z", "updated": "2016-12-16T19:24:10.582Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd/', 'bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd", "email": "GROUP_0e10e603-5346-4ec0-8ae0-...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e10e603-5346-4ec0-8ae0-f3aa995264fd", "storageClass": "STANDARD", "timeCreated": "2016-05-04T18:51:40.559Z", "updated": "2016-12-16T19:24:12.058Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2/', 'bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e1d0393-9ae4-4d4f-b527-e175573865d2", "email": "806222273987-ksinuuegh...@developer.gserviceaccount.com" ... (1640 characters truncated) ... -0e1d0393-9ae4-4d4f-b527-e175573865d2", "storageClass": "STANDARD", "timeCreated": "2018-01-03T16:44:23.645Z", "updated": "2018-01-03T16:44:23.645Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c/', 'bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "email": "GROUP_I_0e2512e9-ebf4-405...@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "storageClass": "STANDARD", "timeCreated": "2017-05-09T17:40:42.742Z", "updated": "2017-05-09T17:40:42.742Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e30616a-6330-4e32-ab08-87427da34658/', 'bucket/fc-0e30616a-6330-4e32-ab08-87427da34658', 'fc-0e30616a-6330-4e32-ab08-87427da34658', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e30616a-6330-4e32-ab08-87427da34658", "email": "GROUP_0e30616a-6330-4e32...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e30616a-6330-4e32-ab08-87427da34658", "storageClass": "STANDARD", "timeCreated": "2017-11-17T19:23:44.837Z", "updated": "2017-11-17T19:23:44.837Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b/', 'bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b', 'fc-0e33525c-6d13-44c5-afae-c6527312660b', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e33525c-6d13-44c5-afae-c6527312660b", "email": "GROUP_0e33525c-6d13-44c5...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e33525c-6d13-44c5-afae-c6527312660b", "storageClass": "STANDARD", "timeCreated": "2017-10-03T16:46:33.867Z", "updated": "2017-10-03T16:46:33.867Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f/', 'bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "email": "GROUP_0e474340-ba14-42c1...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "storageClass": "STANDARD", "timeCreated": "2018-04-04T15:08:29.946Z", "updated": "2018-04-04T15:08:29.946Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603/', 'bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "email": "GROUP_I_0e4f21f9-ad42-413d-8b6c-df43b5cf...@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "storageClass": "STANDARD", "timeCreated": "2017-02-02T20:23:32.745Z", "updated": "2017-02-02T20:23:32.745Z"}', 'project/broad-dsde-dev') ... displaying 10 of 19378 total bound parameter sets ... ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088/', 'bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'bucket', 0, '', '', '{"acl": [{"bucket": "zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zap-staging-4 ... (4549 characters truncated) ... -41ead8a1-9b62-420e-a57a-dcdfc4a10088", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:25:00.715Z", "updated": "2018-03-20T22:25:07.948Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b/', 'bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'bucket', 0, '', '', '{"acl": [{"bucket": "zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zxcass-sta ... (4612 characters truncated) ... -9888f6f5-01c5-43c7-a52e-521f8f8b808b", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:31:14.668Z", "updated": "2018-03-20T22:31:22.007Z"}', 'project/broad-dsde-dev'))] (Background on this error at: http://sqlalche.me/e/e3q8))>

On Thu, Apr 19, 2018 at 2:44 PM, Lukas Karlsson <karl...@broadinstitute.org> wrote:

That helped, in that I do not get the mysql error anymore.

But based on the mysql activity in the console, it looks like it only ran for around 5 minutes and now the server VM is locked up it seems and I can't ssh into it.

I might have to try making the server VM larger and re-deploying.

/l

On Thu, Apr 19, 2018 at 12:40 PM, Joe Cheuk <joec...@google.com> wrote:

Hi Antoine,

This is not a feature for creating your personnalized inventory, we build data model from the inventory data so that we can get a better understanding of the relationship between those data.

Once we have the data model ready, you can query the data model directly through Forseti Explainer, the documentation of how to use Forseti Explainer can be found here:

https://docs.google.com/document/d/1e6uJgTZTtTCMFaZ0BMs_EtV9ULcunq-tKsNiSxIMg3M/preview#heading=h.vgovm6yab1n2

Lukas,

Thanks for the updates! I suspect the exception you are getting might be because the default allowed packet size is too small. 

Can you try updating the "max_allowed_packet" flag to a 64000000 (64 mbs) and see if that resolves the problem?

The instructions on how to add a mysql flag on cloud sql instance can be found here:

https://cloud.google.com/sql/docs/mysql/flags

Best regards,

Joe

On Thu, Apr 19, 2018 at 7:53 AM Lukas Karlsson <karl...@broadinstitute.org> wrote:

I got my first SUCCESS on a inventory last night, after change it to every 12 hours instead of every 2.

I tried twice to create a model from this successful inventory and it failed with the same issue, seems like MySQL server connection went away:

$ forseti model create inventory --id 2018-04-19T00:36:18.626100 TEST1Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1031, in <module> main(sys.argv[1:], ENV_CONFIG) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1013, in main services[config.service](client, config, output, config_env) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 694, in run_model actions[config.action]() File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 670, in do_create_model config.background) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/client.py", line 219, in new_model background=background)) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 487, in __call__ return _end_unary_response_blocking(state, call, False, deadline) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking raise _Rendezvous(state, None, None, deadline)grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, Exception calling application: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'INSERT INTO `0454555e6f5e870c1966582538c7f321_resources` (full_name, type_name, name, type, policy_update_counter, display_name, email, data, parent_type_name) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5/', 'bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "email": "GROUP_0e0fbda1-daf4-4101-aa78-e712f9aa31f5-REA...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "storageClass": "STANDARD", "timeCreated": "2016-07-08T19:56:33.306Z", "updated": "2016-12-16T19:24:10.582Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd/', 'bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd", "email": "GROUP_0e10e603-5346-4ec0-8ae0-...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e10e603-5346-4ec0-8ae0-f3aa995264fd", "storageClass": "STANDARD", "timeCreated": "2016-05-04T18:51:40.559Z", "updated": "2016-12-16T19:24:12.058Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2/', 'bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e1d0393-9ae4-4d4f-b527-e175573865d2", "email": "806222273987-ksinuuegh...@developer.gserviceaccount.com" ... (1640 characters truncated) ... -0e1d0393-9ae4-4d4f-b527-e175573865d2", "storageClass": "STANDARD", "timeCreated": "2018-01-03T16:44:23.645Z", "updated": "2018-01-03T16:44:23.645Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c/', 'bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "email": "GROUP_I_0e2512e9-ebf4-405...@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "storageClass": "STANDARD", "timeCreated": "2017-05-09T17:40:42.742Z", "updated": "2017-05-09T17:40:42.742Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e30616a-6330-4e32-ab08-87427da34658/', 'bucket/fc-0e30616a-6330-4e32-ab08-87427da34658', 'fc-0e30616a-6330-4e32-ab08-87427da34658', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e30616a-6330-4e32-ab08-87427da34658", "email": "GROUP_0e30616a-6330-4e32...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e30616a-6330-4e32-ab08-87427da34658", "storageClass": "STANDARD", "timeCreated": "2017-11-17T19:23:44.837Z", "updated": "2017-11-17T19:23:44.837Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b/', 'bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b', 'fc-0e33525c-6d13-44c5-afae-c6527312660b', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e33525c-6d13-44c5-afae-c6527312660b", "email": "GROUP_0e33525c-6d13-44c5...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e33525c-6d13-44c5-afae-c6527312660b", "storageClass": "STANDARD", "timeCreated": "2017-10-03T16:46:33.867Z", "updated": "2017-10-03T16:46:33.867Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f/', 'bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "email": "GROUP_0e474340-ba14-42c1...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "storageClass": "STANDARD", "timeCreated": "2018-04-04T15:08:29.946Z", "updated": "2018-04-04T15:08:29.946Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603/', 'bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "email": "GROUP_I_0e4f21f9-ad42-413d-8b6c-df43b5cf...@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "storageClass": "STANDARD", "timeCreated": "2017-02-02T20:23:32.745Z", "updated": "2017-02-02T20:23:32.745Z"}', 'project/broad-dsde-dev') ... displaying 10 of 19378 total bound parameter sets ... ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088/', 'bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'bucket', 0, '', '', '{"acl": [{"bucket": "zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zap-staging-4 ... (4549 characters truncated) ... -41ead8a1-9b62-420e-a57a-dcdfc4a10088", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:25:00.715Z", "updated": "2018-03-20T22:25:07.948Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b/', 'bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'bucket', 0, '', '', '{"acl": [{"bucket": "zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zxcass-sta ... (4612 characters truncated) ... -9888f6f5-01c5-43c7-a52e-521f8f8b808b", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:31:14.668Z", "updated": "2018-03-20T22:31:22.007Z"}', 'project/broad-dsde-dev'))] (Background on this error at: http://sqlalche.me/e/e3q8))>

To unsubscribe from this group and stop receiving emails from it, send an email to beta-testers...@forsetisecurity.org.
To post to this group, send email to beta-t...@forsetisecurity.org.

--

You received this message because you are subscribed to the Google Groups "Forseti Security Beta Testers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to beta-testers...@forsetisecurity.org.
To post to this group, send email to beta-t...@forsetisecurity.org.
To view this discussion on the web visit https://groups.google.com/a/forsetisecurity.org/d/msgid/beta-testers/CADT05SS-ZaSqrsUZ5Ei%2Br%3DMtFvcVgrXm8vcwvsWqOEvr4VeXOQ%40mail.gmail.com.

[Lukas Karlsson]

Gave 1073741824 a try and same error. 

/l

On Thu, Apr 19, 2018 at 4:56 PM, Joe Cheuk <joec...@google.com> wrote:

Hi Lukas,

We are doing bulk inserts to optimize the performance while building the data model, I think in your case since there are a lot of data so the sql statement becomes very big and unable to send that statement to the server. 

I have opened an issue to not do bulk insert in the future to avoid issues like this:

https://github.com/GoogleCloudPlatform/forseti-security/issues/1442

At this moment I think the only option is to increase the max_allowed_packet again to the maximum 1073741824 (1gb) and see if that works.

Yukie,

Thanks for reporting the issue! This is a known issue on our end and we are working on addressing it.

We have a pull request opened for this issue:

https://github.com/GoogleCloudPlatform/forseti-security/pull/1440

Once it's ready, we will patch the 2.0-rc2 branch with the updates so you can try it out.

Best regards,

Joe

On Thu, Apr 19, 2018 at 1:00 PM Lukas Karlsson <karl...@broadinstitute.org> wrote:

Now that I'm back to a bigger vm now I'm getting the same error again but I already checked and I have the databaseFlag set for max_allowed_packet=64000000.

$ forseti model create inventory --id 2018-04-19T00:36:18.626100 test2Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1031, in <module> main(sys.argv[1:], ENV_CONFIG) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1013, in main services[config.service](client, config, output, config_env) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 694, in run_model actions[config.action]() File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 670, in do_create_model config.background) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/client.py", line 219, in new_model background=background)) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 487, in __call__ return _end_unary_response_blocking(state, call, False, deadline) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking raise _Rendezvous(state, None, None, deadline)grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, Exception calling application: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'INSERT INTO `1125264ff962ac1469d71bcbd5019e8f_resources` (full_name, type_name, name, type, policy_update_counter, display_name, email, data, parent_type_name) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5/', 'bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "email": "GROUP_0e0fbda1-daf4-4101-aa78-e712f9aa31f5-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "storageClass": "STANDARD", "timeCreated": "2016-07-08T19:56:33.306Z", "updated": "2016-12-16T19:24:10.582Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd/', 'bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd", "email": "GROUP_0e10e603-5346-4ec0-8ae0-f3aa995264fd-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e10e603-5346-4ec0-8ae0-f3aa995264fd", "storageClass": "STANDARD", "timeCreated": "2016-05-04T18:51:40.559Z", "updated": "2016-12-16T19:24:12.058Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2/', 'bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e1d0393-9ae4-4d4f-b527-e175573865d2", "email": "806222273987-ksinuueghug8u81i36lq8aof266q19h...@developer.gserviceaccount.com" ... (1640 characters truncated) ... -0e1d0393-9ae4-4d4f-b527-e175573865d2", "storageClass": "STANDARD", "timeCreated": "2018-01-03T16:44:23.645Z", "updated": "2018-01-03T16:44:23.645Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c/', 'bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "email": "GROUP_I_0e2512e9-ebf4-4054-9dd8-aa664b171c3c-READER@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "storageClass": "STANDARD", "timeCreated": "2017-05-09T17:40:42.742Z", "updated": "2017-05-09T17:40:42.742Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e30616a-6330-4e32-ab08-87427da34658/', 'bucket/fc-0e30616a-6330-4e32-ab08-87427da34658', 'fc-0e30616a-6330-4e32-ab08-87427da34658', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e30616a-6330-4e32-ab08-87427da34658", "email": "GROUP_0e30616a-6330-4e32-ab08-87427da34658-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e30616a-6330-4e32-ab08-87427da34658", "storageClass": "STANDARD", "timeCreated": "2017-11-17T19:23:44.837Z", "updated": "2017-11-17T19:23:44.837Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b/', 'bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b', 'fc-0e33525c-6d13-44c5-afae-c6527312660b', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e33525c-6d13-44c5-afae-c6527312660b", "email": "GROUP_0e33525c-6d13-44c5-afae-c6527312660b-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e33525c-6d13-44c5-afae-c6527312660b", "storageClass": "STANDARD", "timeCreated": "2017-10-03T16:46:33.867Z", "updated": "2017-10-03T16:46:33.867Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f/', 'bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "email": "GROUP_0e474340-ba14-42c1-ae0e-5cc90e8bfc3f-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "storageClass": "STANDARD", "timeCreated": "2018-04-04T15:08:29.946Z", "updated": "2018-04-04T15:08:29.946Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603/', 'bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "email": "GROUP_I_0e4f21f9-ad42-413d-8b6c-df43b5cf1603-READER@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "storageClass": "STANDARD", "timeCreated": "2017-02-02T20:23:32.745Z", "updated": "2017-02-02T20:23:32.745Z"}', 'project/broad-dsde-dev') ... displaying 10 of 19378 total bound parameter sets ... ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088/', 'bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'bucket', 0, '', '', '{"acl": [{"bucket": "zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zap-staging-4 ... (4549 characters truncated) ... -41ead8a1-9b62-420e-a57a-dcdfc4a10088", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:25:00.715Z", "updated": "2018-03-20T22:25:07.948Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b/', 'bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'bucket', 0, '', '', '{"acl": [{"bucket": "zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zxcass-sta ... (4612 characters truncated) ... -9888f6f5-01c5-43c7-a52e-521f8f8b808b", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:31:14.668Z", "updated": "2018-03-20T22:31:22.007Z"}', 'project/broad-dsde-dev'))] (Background on this error at: http://sqlalche.me/e/e3q8))>

On Thu, Apr 19, 2018 at 2:44 PM, Lukas Karlsson <karl...@broadinstitute.org> wrote:

That helped, in that I do not get the mysql error anymore.

But based on the mysql activity in the console, it looks like it only ran for around 5 minutes and now the server VM is locked up it seems and I can't ssh into it.

I might have to try making the server VM larger and re-deploying.

/l

On Thu, Apr 19, 2018 at 12:40 PM, Joe Cheuk <joec...@google.com> wrote:

Hi Antoine,

This is not a feature for creating your personnalized inventory, we build data model from the inventory data so that we can get a better understanding of the relationship between those data.

Once we have the data model ready, you can query the data model directly through Forseti Explainer, the documentation of how to use Forseti Explainer can be found here:

https://docs.google.com/document/d/1e6uJgTZTtTCMFaZ0BMs_EtV9ULcunq-tKsNiSxIMg3M/preview#heading=h.vgovm6yab1n2

Lukas,

Thanks for the updates! I suspect the exception you are getting might be because the default allowed packet size is too small. 

Can you try updating the "max_allowed_packet" flag to a 64000000 (64 mbs) and see if that resolves the problem?

The instructions on how to add a mysql flag on cloud sql instance can be found here:

https://cloud.google.com/sql/docs/mysql/flags

Best regards,

Joe

On Thu, Apr 19, 2018 at 7:53 AM Lukas Karlsson <karl...@broadinstitute.org> wrote:

I got my first SUCCESS on a inventory last night, after change it to every 12 hours instead of every 2.

I tried twice to create a model from this successful inventory and it failed with the same issue, seems like MySQL server connection went away:

$ forseti model create inventory --id 2018-04-19T00:36:18.626100 TEST1Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1031, in <module> main(sys.argv[1:], ENV_CONFIG) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1013, in main services[config.service](client, config, output, config_env) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 694, in run_model actions[config.action]() File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 670, in do_create_model config.background) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/client.py", line 219, in new_model background=background)) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 487, in __call__ return _end_unary_response_blocking(state, call, False, deadline) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking raise _Rendezvous(state, None, None, deadline)grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, Exception calling application: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'INSERT INTO `0454555e6f5e870c1966582538c7f321_resources` (full_name, type_name, name, type, policy_update_counter, display_name, email, data, parent_type_name) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5/', 'bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "email": "GROUP_0e0fbda1-daf4-4101-aa78-e712f9aa31f5-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "storageClass": "STANDARD", "timeCreated": "2016-07-08T19:56:33.306Z", "updated": "2016-12-16T19:24:10.582Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd/', 'bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd", "email": "GROUP_0e10e603-5346-4ec0-8ae0-f3aa995264fd-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e10e603-5346-4ec0-8ae0-f3aa995264fd", "storageClass": "STANDARD", "timeCreated": "2016-05-04T18:51:40.559Z", "updated": "2016-12-16T19:24:12.058Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2/', 'bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e1d0393-9ae4-4d4f-b527-e175573865d2", "email": "806222273987-ksinuueghug8u81i36lq8aof266q19h...@developer.gserviceaccount.com" ... (1640 characters truncated) ... -0e1d0393-9ae4-4d4f-b527-e175573865d2", "storageClass": "STANDARD", "timeCreated": "2018-01-03T16:44:23.645Z", "updated": "2018-01-03T16:44:23.645Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c/', 'bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "email": "GROUP_I_0e2512e9-ebf4-4054-9dd8-aa664b171c3c-READER@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "storageClass": "STANDARD", "timeCreated": "2017-05-09T17:40:42.742Z", "updated": "2017-05-09T17:40:42.742Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e30616a-6330-4e32-ab08-87427da34658/', 'bucket/fc-0e30616a-6330-4e32-ab08-87427da34658', 'fc-0e30616a-6330-4e32-ab08-87427da34658', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e30616a-6330-4e32-ab08-87427da34658", "email": "GROUP_0e30616a-6330-4e32-ab08-87427da34658-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e30616a-6330-4e32-ab08-87427da34658", "storageClass": "STANDARD", "timeCreated": "2017-11-17T19:23:44.837Z", "updated": "2017-11-17T19:23:44.837Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b/', 'bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b', 'fc-0e33525c-6d13-44c5-afae-c6527312660b', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e33525c-6d13-44c5-afae-c6527312660b", "email": "GROUP_0e33525c-6d13-44c5-afae-c6527312660b-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e33525c-6d13-44c5-afae-c6527312660b", "storageClass": "STANDARD", "timeCreated": "2017-10-03T16:46:33.867Z", "updated": "2017-10-03T16:46:33.867Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f/', 'bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "email": "GROUP_0e474340-ba14-42c1-ae0e-5cc90e8bfc3f-READER@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "storageClass": "STANDARD", "timeCreated": "2018-04-04T15:08:29.946Z", "updated": "2018-04-04T15:08:29.946Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603/', 'bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "email": "GROUP_I_0e4f21f9-ad42-413d-8b6c-df43b5cf1603-READER@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "storageClass": "STANDARD", "timeCreated": "2017-02-02T20:23:32.745Z", "updated": "2017-02-02T20:23:32.745Z"}', 'project/broad-dsde-dev') ... displaying 10 of 19378 total bound parameter sets ... ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088/', 'bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'bucket', 0, '', '', '{"acl": [{"bucket": "zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zap-staging-4 ... (4549 characters truncated) ... -41ead8a1-9b62-420e-a57a-dcdfc4a10088", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:25:00.715Z", "updated": "2018-03-20T22:25:07.948Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b/', 'bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'bucket', 0, '', '', '{"acl": [{"bucket": "zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zxcass-sta ... (4612 characters truncated) ... -9888f6f5-01c5-43c7-a52e-521f8f8b808b", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:31:14.668Z", "updated": "2018-03-20T22:31:22.007Z"}', 'project/broad-dsde-dev'))] (Background on this error at: http://sqlalche.me/e/e3q8))>

To unsubscribe from this group and stop receiving emails from it, send an email to beta-testers+unsubscribe@forsetisecurity.org.
To post to this group, send email to beta-testers@forsetisecurity.org.

To unsubscribe from this group and stop receiving emails from it, send an email to beta-testers+unsubscribe@forsetisecurity.org.
To post to this group, send email to beta-testers@forsetisecurity.org.

To view this discussion on the web visit https://groups.google.com/a/forsetisecurity.org/d/msgid/beta-testers/CADT05SS-ZaSqrsUZ5Ei%2Br%3DMtFvcVgrXm8vcwvsWqOEvr4VeXOQ%40mail.gmail.com.

[Joe Cheuk]

Hi Lukas,

Thanks for the updates! We are working on this issue now, we will patch branch 2.0-rc2 and let you know once it's ready. I hope to have this fixed early next week.

Thank you for your help in getting Forseti working smoothly!

Best regards,

Joe

On Thu, Apr 19, 2018 at 3:14 PM Lukas Karlsson <karl...@broadinstitute.org> wrote:

Gave 1073741824 a try and same error. 

/l

On Thu, Apr 19, 2018 at 4:56 PM, Joe Cheuk <joec...@google.com> wrote:

Hi Lukas,

We are doing bulk inserts to optimize the performance while building the data model, I think in your case since there are a lot of data so the sql statement becomes very big and unable to send that statement to the server. 

I have opened an issue to not do bulk insert in the future to avoid issues like this:

https://github.com/GoogleCloudPlatform/forseti-security/issues/1442

At this moment I think the only option is to increase the max_allowed_packet again to the maximum 1073741824 (1gb) and see if that works.

Yukie,

Thanks for reporting the issue! This is a known issue on our end and we are working on addressing it.

We have a pull request opened for this issue:

https://github.com/GoogleCloudPlatform/forseti-security/pull/1440

Once it's ready, we will patch the 2.0-rc2 branch with the updates so you can try it out.

Best regards,

Joe

On Thu, Apr 19, 2018 at 1:00 PM Lukas Karlsson <karl...@broadinstitute.org> wrote:

Now that I'm back to a bigger vm now I'm getting the same error again but I already checked and I have the databaseFlag set for max_allowed_packet=64000000.

$ forseti model create inventory --id 2018-04-19T00:36:18.626100 test2Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1031, in <module> main(sys.argv[1:], ENV_CONFIG) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1013, in main services[config.service](client, config, output, config_env) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 694, in run_model actions[config.action]() File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 670, in do_create_model config.background) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/client.py", line 219, in new_model background=background)) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 487, in __call__ return _end_unary_response_blocking(state, call, False, deadline) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking raise _Rendezvous(state, None, None, deadline)grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, Exception calling application: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'INSERT INTO `1125264ff962ac1469d71bcbd5019e8f_resources` (full_name, type_name, name, type, policy_update_counter, display_name, email, data, parent_type_name) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5/', 'bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "email": "GROUP_0e0fbda1-daf4-4101-aa78-e712f9aa31f5-REA...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "storageClass": "STANDARD", "timeCreated": "2016-07-08T19:56:33.306Z", "updated": "2016-12-16T19:24:10.582Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd/', 'bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd", "email": "GROUP_0e10e603-5346-4ec0-8ae0-...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e10e603-5346-4ec0-8ae0-f3aa995264fd", "storageClass": "STANDARD", "timeCreated": "2016-05-04T18:51:40.559Z", "updated": "2016-12-16T19:24:12.058Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2/', 'bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e1d0393-9ae4-4d4f-b527-e175573865d2", "email": "806222273987-ksinuuegh...@developer.gserviceaccount.com" ... (1640 characters truncated) ... -0e1d0393-9ae4-4d4f-b527-e175573865d2", "storageClass": "STANDARD", "timeCreated": "2018-01-03T16:44:23.645Z", "updated": "2018-01-03T16:44:23.645Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c/', 'bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "email": "GROUP_I_0e2512e9-ebf4-405...@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "storageClass": "STANDARD", "timeCreated": "2017-05-09T17:40:42.742Z", "updated": "2017-05-09T17:40:42.742Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e30616a-6330-4e32-ab08-87427da34658/', 'bucket/fc-0e30616a-6330-4e32-ab08-87427da34658', 'fc-0e30616a-6330-4e32-ab08-87427da34658', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e30616a-6330-4e32-ab08-87427da34658", "email": "GROUP_0e30616a-6330-4e32...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e30616a-6330-4e32-ab08-87427da34658", "storageClass": "STANDARD", "timeCreated": "2017-11-17T19:23:44.837Z", "updated": "2017-11-17T19:23:44.837Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b/', 'bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b', 'fc-0e33525c-6d13-44c5-afae-c6527312660b', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e33525c-6d13-44c5-afae-c6527312660b", "email": "GROUP_0e33525c-6d13-44c5...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e33525c-6d13-44c5-afae-c6527312660b", "storageClass": "STANDARD", "timeCreated": "2017-10-03T16:46:33.867Z", "updated": "2017-10-03T16:46:33.867Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f/', 'bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "email": "GROUP_0e474340-ba14-42c1...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "storageClass": "STANDARD", "timeCreated": "2018-04-04T15:08:29.946Z", "updated": "2018-04-04T15:08:29.946Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603/', 'bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "email": "GROUP_I_0e4f21f9-ad42-413d-8b6c-df43b5cf...@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "storageClass": "STANDARD", "timeCreated": "2017-02-02T20:23:32.745Z", "updated": "2017-02-02T20:23:32.745Z"}', 'project/broad-dsde-dev') ... displaying 10 of 19378 total bound parameter sets ... ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088/', 'bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'bucket', 0, '', '', '{"acl": [{"bucket": "zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zap-staging-4 ... (4549 characters truncated) ... -41ead8a1-9b62-420e-a57a-dcdfc4a10088", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:25:00.715Z", "updated": "2018-03-20T22:25:07.948Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b/', 'bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'bucket', 0, '', '', '{"acl": [{"bucket": "zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zxcass-sta ... (4612 characters truncated) ... -9888f6f5-01c5-43c7-a52e-521f8f8b808b", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:31:14.668Z", "updated": "2018-03-20T22:31:22.007Z"}', 'project/broad-dsde-dev'))] (Background on this error at: http://sqlalche.me/e/e3q8))>

On Thu, Apr 19, 2018 at 2:44 PM, Lukas Karlsson <karl...@broadinstitute.org> wrote:

That helped, in that I do not get the mysql error anymore.

But based on the mysql activity in the console, it looks like it only ran for around 5 minutes and now the server VM is locked up it seems and I can't ssh into it.

I might have to try making the server VM larger and re-deploying.

/l

On Thu, Apr 19, 2018 at 12:40 PM, Joe Cheuk <joec...@google.com> wrote:

Hi Antoine,

This is not a feature for creating your personnalized inventory, we build data model from the inventory data so that we can get a better understanding of the relationship between those data.

Once we have the data model ready, you can query the data model directly through Forseti Explainer, the documentation of how to use Forseti Explainer can be found here:

https://docs.google.com/document/d/1e6uJgTZTtTCMFaZ0BMs_EtV9ULcunq-tKsNiSxIMg3M/preview#heading=h.vgovm6yab1n2

Lukas,

Thanks for the updates! I suspect the exception you are getting might be because the default allowed packet size is too small. 

Can you try updating the "max_allowed_packet" flag to a 64000000 (64 mbs) and see if that resolves the problem?

The instructions on how to add a mysql flag on cloud sql instance can be found here:

https://cloud.google.com/sql/docs/mysql/flags

Best regards,

Joe

On Thu, Apr 19, 2018 at 7:53 AM Lukas Karlsson <karl...@broadinstitute.org> wrote:

I got my first SUCCESS on a inventory last night, after change it to every 12 hours instead of every 2.

I tried twice to create a model from this successful inventory and it failed with the same issue, seems like MySQL server connection went away:

$ forseti model create inventory --id 2018-04-19T00:36:18.626100 TEST1Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1031, in <module> main(sys.argv[1:], ENV_CONFIG) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 1013, in main services[config.service](client, config, output, config_env) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 694, in run_model actions[config.action]() File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/cli.py", line 670, in do_create_model config.background) File "/usr/local/lib/python2.7/dist-packages/forseti_security-2.0.0-py2.7.egg/google/cloud/forseti/services/client.py", line 219, in new_model background=background)) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 487, in __call__ return _end_unary_response_blocking(state, call, False, deadline) File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking raise _Rendezvous(state, None, None, deadline)grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, Exception calling application: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'INSERT INTO `0454555e6f5e870c1966582538c7f321_resources` (full_name, type_name, name, type, policy_update_counter, display_name, email, data, parent_type_name) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5/', 'bucket/fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "email": "GROUP_0e0fbda1-daf4-4101-aa78-e712f9aa31f5-REA...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e0fbda1-daf4-4101-aa78-e712f9aa31f5", "storageClass": "STANDARD", "timeCreated": "2016-07-08T19:56:33.306Z", "updated": "2016-12-16T19:24:10.582Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd/', 'bucket/fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e10e603-5346-4ec0-8ae0-f3aa995264fd", "email": "GROUP_0e10e603-5346-4ec0-8ae0-...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e10e603-5346-4ec0-8ae0-f3aa995264fd", "storageClass": "STANDARD", "timeCreated": "2016-05-04T18:51:40.559Z", "updated": "2016-12-16T19:24:12.058Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2/', 'bucket/fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'fc-0e1d0393-9ae4-4d4f-b527-e175573865d2', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e1d0393-9ae4-4d4f-b527-e175573865d2", "email": "806222273987-ksinuuegh...@developer.gserviceaccount.com" ... (1640 characters truncated) ... -0e1d0393-9ae4-4d4f-b527-e175573865d2", "storageClass": "STANDARD", "timeCreated": "2018-01-03T16:44:23.645Z", "updated": "2018-01-03T16:44:23.645Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c/', 'bucket/fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "email": "GROUP_I_0e2512e9-ebf4-405...@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e2512e9-ebf4-4054-9dd8-aa664b171c3c", "storageClass": "STANDARD", "timeCreated": "2017-05-09T17:40:42.742Z", "updated": "2017-05-09T17:40:42.742Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e30616a-6330-4e32-ab08-87427da34658/', 'bucket/fc-0e30616a-6330-4e32-ab08-87427da34658', 'fc-0e30616a-6330-4e32-ab08-87427da34658', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e30616a-6330-4e32-ab08-87427da34658", "email": "GROUP_0e30616a-6330-4e32...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e30616a-6330-4e32-ab08-87427da34658", "storageClass": "STANDARD", "timeCreated": "2017-11-17T19:23:44.837Z", "updated": "2017-11-17T19:23:44.837Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b/', 'bucket/fc-0e33525c-6d13-44c5-afae-c6527312660b', 'fc-0e33525c-6d13-44c5-afae-c6527312660b', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e33525c-6d13-44c5-afae-c6527312660b", "email": "GROUP_0e33525c-6d13-44c5...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e33525c-6d13-44c5-afae-c6527312660b", "storageClass": "STANDARD", "timeCreated": "2017-10-03T16:46:33.867Z", "updated": "2017-10-03T16:46:33.867Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f/', 'bucket/fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "email": "GROUP_0e474340-ba14-42c1...@dev.test.firecloud.org", " ... (4996 characters truncated) ... -0e474340-ba14-42c1-ae0e-5cc90e8bfc3f", "storageClass": "STANDARD", "timeCreated": "2018-04-04T15:08:29.946Z", "updated": "2018-04-04T15:08:29.946Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603/', 'bucket/fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603', 'bucket', 0, '', '', '{"acl": [{"bucket": "fc-0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "email": "GROUP_I_0e4f21f9-ad42-413d-8b6c-df43b5cf...@dev.test.firecloud.org", ... (5176 characters truncated) ... -0e4f21f9-ad42-413d-8b6c-df43b5cf1603", "storageClass": "STANDARD", "timeCreated": "2017-02-02T20:23:32.745Z", "updated": "2017-02-02T20:23:32.745Z"}', 'project/broad-dsde-dev') ... displaying 10 of 19378 total bound parameter sets ... ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088/', 'bucket/zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088', 'bucket', 0, '', '', '{"acl": [{"bucket": "zap-staging-41ead8a1-9b62-420e-a57a-dcdfc4a10088", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zap-staging-4 ... (4549 characters truncated) ... -41ead8a1-9b62-420e-a57a-dcdfc4a10088", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:25:00.715Z", "updated": "2018-03-20T22:25:07.948Z"}', 'project/broad-dsde-dev'), ('organization/548622027621/folder/686906382393/folder/310153630233/project/broad-dsde-dev/bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b/', 'bucket/zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b', 'bucket', 0, '', '', '{"acl": [{"bucket": "zxcass-staging-9888f6f5-01c5-43c7-a52e-521f8f8b808b", "entity": "project-owners-806222273987", "etag": "CAc=", "id": "zxcass-sta ... (4612 characters truncated) ... -9888f6f5-01c5-43c7-a52e-521f8f8b808b", "storageClass": "STANDARD", "timeCreated": "2018-03-20T22:31:14.668Z", "updated": "2018-03-20T22:31:22.007Z"}', 'project/broad-dsde-dev'))] (Background on this error at: http://sqlalche.me/e/e3q8))>

To unsubscribe from this group and stop receiving emails from it, send an email to beta-testers...@forsetisecurity.org.
To post to this group, send email to beta-t...@forsetisecurity.org.

To unsubscribe from this group and stop receiving emails from it, send an email to beta-testers...@forsetisecurity.org.
To post to this group, send email to beta-t...@forsetisecurity.org.

To view this discussion on the web visit https://groups.google.com/a/forsetisecurity.org/d/msgid/beta-testers/CADT05SS-ZaSqrsUZ5Ei%2Br%3DMtFvcVgrXm8vcwvsWqOEvr4VeXOQ%40mail.gmail.com.

[Joe Cheuk]

Hi Lukas,

The 2.0-rc2 branch has been patched with a fix to the issue you reported, can you try resetting the server VM and run the model creation command again to see if you still experience the same error?

Thanks,

Joe