Andrew Brazzatti
Andrew
From: van Schyndel, Jay <jay.van...@jcu.edu.au>
Date: Thu, Jan 10, 2013 at 11:44 AM
Subject: RedBox and The Mint performance Improvements
To: Andrew Brazzatti <and...@redboxresearchdata.com.au>
Cc: Duncan Dickinson <d.dic...@qcif.edu.au>, "Brown, Marianne" <mariann...@jcu.edu.au>
if (curated) {
// Happy ending
if (task.equals("curation-response")) {
log.info("Confirmation of curated object '{}'.", oid);
// Send out upstream responses to objects waiting
JSONArray responses = data.writeArray("responses");
for (Object thisResponse : responses) {
JsonSimple json = new JsonSimple((JsonObject) thisResponse);
String broker = json.getString(brokerUrl, "broker");
String responseOid = json.getString(null, "oid");
String responseTask = json.getString(null, "task");
JsonObject responseObj = createTask(response, broker,
responseOid, responseTask);
// Don't forget to tell them where it came from
String id = json.getString(null, "quoteId");
if (id != null) {
responseObj.put("originId", id);
}
responseObj.put("originOid", oid);
// If NLA Integration is enabled, use the NLA ID instead
if (nlaIntegrationEnabled && metadata.containsKey(nlaIdProperty)) {
responseObj.put("curatedPid", metadata.getProperty(nlaIdProperty));
} else {
responseObj.put("curatedPid", thisPid);
}
}
//JCU: now that the responses have been sent, remove them, so they are not sent again. Otherwise, they just keep getting resent and performance suffers greatly.
responses.clear();
saveObjectData(data, oid);
private void publishRelations(JsonSimple response, String oid) throws TransactionException {
log.debug("Publishing Children of '{}'", oid);
JsonSimple data = getDataFromStorage(oid);
if (data == null) {
log.error("Error accessing item data! '{}'", oid);
emailObjectLink(response, oid,
"An error occured publishing the related objects for this"
+ " record. Please check the system logs.");
return;
}
boolean saveData = false;
JSONArray relations = data.writeArray("relationships");
for (Object relation : relations) {
JsonSimple json = new JsonSimple((JsonObject) relation);
String broker = json.getString(brokerUrl, "broker");
boolean localRecord = broker.equals(brokerUrl);
String relatedId = json.getString(null, "identifier");
// We need to find OIDs to match IDs (only for local records)
String relatedOid = json.getString(null, "oid");
if (relatedOid == null && localRecord) {
String identifier = json.getString(null, "identifier");
if (identifier == null) {
log.error("NULL identifer provided!");
}
relatedOid = idToOid(identifier);
if (relatedOid == null) {
log.error("Cannot resolve identifer: '{}'", identifier);
}
}
boolean authority = json.getBoolean(false, "authority");
if (authority) {
// Is this relationship using a curated ID?
boolean isCurated = json.getBoolean(false, "isCurated");
//JCU: adding check for publishMsgSent
boolean publishMsgSent = json.getBoolean(false, "publishMsgSent");
if (isCurated && !publishMsgSent) {
log.debug(" * Publishing '{}'", relatedId);
// It is a local object
if (localRecord) {
createTask(response, relatedOid, "publish");
// Or remote
} else {
JsonObject task = createTask(response, broker, relatedOid,
"publish");
// We won't know OIDs for remote systems
task.remove("oid") ;
task.put("identifier", relatedId);
}
//JCU: Adding tag to indicate the publish message has been sent.
((JsonObject) relation).put("publishMsgSent", "true");
saveData = true;
} else if (publishMsgSent){
log.debug(" * Ignoring already published relationship '{}'",
relatedId);
}
else {
log.debug(" * Ignoring non-curated relationship '{}'",
relatedId);
}
}
}
if (saveData){
//updating the relations with publishMsgSent
saveObjectData(data, oid);
}
}
Software Engineer - eResearch Centre
Division of Research and Innovation
James Cook University, Townsville QLD 4811 AUSTRALIA
P (07) 4781 3199; I +61 7 4781 3199;
E jay.van...@jcu.edu.au
Location: Faculty of Science & Engineering Room 141 (Building 17.141)
Grant Jackson
Thanks for the contribution Jay. I haven't tested your code, but in addition to the improved performance I suspect it also results in a behavioural change.
Here is an example using v1.5.1 dev-handle.
1. I have an already published collection & related activity (or related person, group or service).
2. I change the (handle) urlTemplate in the activity json file. (I restart Mint & reharvest activities.)
3. I deliberately make a trivial update to the published collection (eg. add a space) & re-save the collection. The intention is to propagate the activity change to the collection's children. (I suspect there is a clever way to do this using messages.)
4. The children of the collection (in this case the activity record) write their new urlTemplate to the corresponding handle record.
If I've understood your description correctly then I suspect the above behaviour will no longer work (because existing child records will not be asked to re-publish).
This example is to encourage discussion. I realise my example might be poor (eg. does not scale well to hundreds or thousands of records) but if there is a behavioural change then I wonder what the implications might be. Perhaps others with more ReDBox-Mint experience than me might have some ideas? Thanks.
Cheers, Grant
--
You received this message because you are subscribed to the Google Groups "ReDBox Development" group.
To post to this group, send an email to redbo...@googlegroups.com.
To unsubscribe from this group, send email to redbox-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Jay van Schyndel
Your comments in the example below are correct. Due to the flag I have added for 'publish' messages, they will not be resent using your scenario below.
I used the flag in this case as the 'relationship' data is used to display the curation information when logged in as admin.
I couldn't just remove the 'relationship' data l like I did for the 'response' due to this.
I haven't used ReDBox in the manner you describe below.
Is it a valid scenario ?
Are other people using ReDBox this way ?
Might need to revisit the fix. Thanks for the feedback.
Cheers,
Jay.
Grant Jackson
It just happened that I was playing with urlTemplates today for all of my data sources so that handles can redirect somewhere other than back to ReDBox-Mint (which is to be put behind a firewall). So initially for me it was an accidental scenario!
I'm not a developer, so if in future I needed to get all *existing* collection, party, activity & service handles to redirect to say a new web server, I might use the example I supplied or update the handle.net records directly (or seek help).
Cheers, Grant
To view this discussion on the web, visit https://groups.google.com/d/msg/redbox-dev/-/8Q9xdPC5NlEJ.
Greg Pendlebury
I wonder if one alternative to avoid this snowball effect is to hold a cache of 'publish' messages and de-dupe them as they are added. Then periodically send the cached messages. By making the cache flush time configurable you could adjust it to match the expected volume of ingest for the data. This would allow you to ensure that publish events are still firing for records that are in a linked network that has undergone change, but you don't have to generate 1500 in a row (or something worse, like the factorial?) if the same guy is related to 1500 newly ingested records.
A similar approach may help with the response messages too, but they carry additional payload data which makes caching de-dupes more complicated. They'd have to merged into the cached message I guess.
Ta,
Greg