CRITs 3.1.0 to Stable 4 upgrade question

[Steve Castellarin]

Hi all,

I've built a Stable 4 CRITs instance and have done a mongorestore from our current CRITs 3.1.0 system to this new system.  I've read through the Group posting and it looks like I should be focusing on the postings "after" 2014-08-13 in the UPDATING file to begin the data upgrade process.  Doing this I ran the first two "manage.py" commands noted, and both have come up with issues.

The first error was on the "python manage.py create_actors_content" on 2014-09-02.  When I run this I get an error: 

  Unknown command: 'create_actors_content'

The second error came during the Analysis Results migration - noted on 2014-09-25.  I'm not running a sharded Mongo instance, so the three "ensureIndex" commands ran fine within the Mongo shell.  I then ran the "python manage.py upgrade -as" and receive the following error:

Migrating Actors

   No Actors to migrate!

Migrating Backdoors

   No Backdoors to migrate!

Migrating Campaigns

   Magrated 0 of 150

   An error occurred during migration!

   Migrated: 0

   Error: FieldDoesNotExist: The field 'name' does not exist on the document 'EmbeddedObject'

   Document ID: 504f93021d41c823b300000c

I've looked at this Campaign in our current version of CRITs and it does display in 3.1.0.

Any ideas on where I need to go with both of these errors so I can get our CRITs 4 upgrade moving?

Thanks!

[Michael Goffin]

The "create_actors_content" is gone. Back then we didn't have our own vocabulary so we had to import content from STIX and CybOX into the database for Actors to work. Now all of that is managed in code so you can skip that command.

As for the error you are getting with AnalysisResults, without seeing the actual document it's hard to say. Campaigns don't have AnalysisResults since there's no service I'm aware of that creates them for Campaigns. You could try and skip Campaigns and try to migrate everything else first (there are flags to `upgrade` to specify which collection to run on). If you can get through all of those then you'll know there's something up with Campaigns and you can dive in further. Or this might be a weird situation where the Campaign is being flagged even though it's a related TLO that's being migrated.

--
You received this message because you are subscribed to the Google Groups "crits-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crits-users+unsubscribe@googlegroups.com.
To post to this group, send email to crits...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/crits-users/CACKvhwF61oLdZOiOROV%2BSA8dusD2NgqAcXZj55huGeWvordJZw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Steve Castellarin]

Hey Mike,

Here's a sanitized version of the document in our Mongo database that the error noted:

{ "_id" : ObjectId("504f93021d41c823b300000c"), "active" : "on", "aliases" : [ "X" ], "description" : "Bad Guy", "domain_count" : 6, "email_count" : 127, "event_count" : 4, "indicator_count" : 705, "ip_count" : 185, "modified" : ISODate("2015-07-24T10:03:58.674Z"), "name" : "Bad Guy 1", "objects" : [ { "analyst" : "Analyst1", "datatype" : "enum", "date" : ISODate("2015-07-24T10:03:58.673Z"), "name" : "Category", "source" : [ { "instances" : [ { "analyst" : "Analyst1", "date" : ISODate("2015-07-24T10:03:58.674Z"), "method" : "", "reference" : "" } ], "name" : "US" } ], "type" : "Category", "value" : "Bad" } ], "pcap_count" : 0, "relationships" : [ { "relationship" : "Contained_Within", "relationship_date" : ISODate("2015-05-04T12:25:20.203Z"), "value" : ObjectId("554128c35ea09238460dad2a"), "date" : ISODate("2015-05-04T12:25:20.203Z"), "type" : "Campaign", "analyst" : "Analyst2", "rel_reason" : "", "rel_confidence" : "high" } ], "sample_count" : 241, "schema_version" : 2 }

I'll go through and start migrating others to see where I get.

Thanks!

[Steve Castellarin]

Hey Mike,

I've tried another TLO (email), but get a similar error: Error: FieldDoesNotExist: The field 'name' does not exist on the document 'EmbeddedObject' - and gives a document ID.  I also tried the indicator TLO, same error.  Any idea on which way to go next?

[Michael Goffin]

Hmm. It's possible, since you are upgrading after an extremely long time, that the AnalysisResults migration is trying to happen before something else and it's breaking. However, the only change we made to Objects was in an update in June of 2015 and it required no migration on the part of the user. However, I haven't run into this issue in any CRITs instance I maintain so I can't be sure.

You could look into MongoDB's '$unset' (https://docs.mongodb.com/manual/reference/operator/update/unset/) to try and unset that field from each sub-document, but unless you know what you are doing and have a solid backup in place, I wouldn't mess with things too much.

[Steve Castellarin]

The CRITs code we're using is dated April 2015.  Was there something that should have been implemented in my code base to take care of the change you noted in June of 2015?

[Steve Castellarin]

I just tried running an "upgrade" command without the -s switch, and get a totally different error.  I'm not sure if this is part of the data issue I have here or something else:

pymongo.errors.OperationFailure: Index with name: objects.value_1 already exists with different options.