Lewis Sick Day

[Lewis John Mcgibbney]

Good Morning Team,

I’m not feeling very well today and will most certainly  miss our meeting this morning.

Please meet as a team and provide an update on current status.

I am available to answer any questions the team may have. I also believe there is a deliverable which has to be submitted, please let me know where we are with this.

Thank you,

Lewis 

--

Lewis

Dr. Lewis J. McGibbney Ph.D, B.Sc

Skype: lewis.john.mcgibbney

[Alexa Huerta]

Hi Dr. McGibbney,

Sorry to hear you are sick, we hope you feel better soon.

Here is our deliverable #3, the Design Document: https://docs.google.com/document/d/19FK1Z75EA08qovRdXk587LTGrvcIRtJFq9hpEuRBrXM/edit?usp=sharing. When possible, please submit your approval to Professor Miller at jeffrey...@usc.edu

Current Status:

Workflow/Pycoal Team: 

• able to view images in QGIS. Will stage images for File Manager to grab. What should our next steps be after this? Begin working on automating the process of staging the products? Or automating the workflow process?
  

File Manager Team:

• working on metadata extraction, managed to run the commands per the instructions on TikaCmdLineMetExtractor documentation. However, the team is unsure on what to do next. No errors was displayed and the metadata information can be seen on localhost:8080/opsui

Finally, assuming you are feeling better, would we be able to schedule another meeting for Thursday morning at 9am?

Thank you for your help.

Best,

Alexa Huerta

--
You received this message because you are subscribed to the Google Groups "COAL - Coal and Open-pit surface mining impacts on American Lands" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coal-capston...@googlegroups.com.
To post to this group, send email to coal-c...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/coal-capstone/CAGaRif2VdixiY3eK%2Br3G%2BOa6%2BU6BR%2BydBsqtrSzjCTu13yUxhg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Alexa Huerta

University of Southern California

B.S. Computer Science, December 2018

alex...@usc.edu

[Lewis John Mcgibbney]

Hi Team,

Back at work today so we are good to meet on a Thursday.

Responses inline...

On Mon, Oct 1, 2018 at 20:19 Alexa Huerta <alex...@usc.edu> wrote:

Hi Dr. McGibbney,

Sorry to hear you are sick, we hope you feel better soon.

Here is our deliverable #3, the Design Document: https://docs.google.com/document/d/19FK1Z75EA08qovRdXk587LTGrvcIRtJFq9hpEuRBrXM/edit?usp=sharing. When possible, please submit your approval to Professor Miller at jeffrey...@usc.edu

I’ll send confirmation once reviewed.

Current Status:

Workflow/Pycoal Team: 

• able to view images in QGIS.

Excellent

• Will stage images for File Manager to grab.

Ok once done please resolve appropriate tickets in Github issue tracker.

• What should our next steps be after this? Begin working on automating the process of staging the products? Or automating the workflow process?

I think moving to workflow is the next step. Start learning about the Workflow Manager. You may want to use WINGS for this as it has an OODT interface which would ease creation and understanding of OODT workflows.

See the following about how to run wings

https://github.com/KnowledgeCaptureAndDiscovery/wings/tree/master/engines/oodt

And the following for wings-OODT integration

https://cwiki.apache.org/confluence/display/OODT/Integrating+OODT+with+Wings

Let’s talk more on Friday and work through what you have uncovered.

File Manager Team:

• working on metadata extraction, managed to run the commands per the instructions on TikaCmdLineMetExtractor documentation. However, the team is unsure on what to do next.

Ok well basically what needs to be done next is that we need to upgrade Apache Tula dependency within OODT which that each field in the hdr document is extracted as an individual field which we can then search over. If you look at the relevant ticket on GitHub you will see that I’ve explained this.

Once this is done then we can further augment the metadata with additional geospatial attributes such as a lat, lon, etc. this will make significant improvements to the search of metadata.

• No errors was displayed and the metadata information can be seen on localhost:8080/opsui

This looks great and is exactly where you should be at. As you can see however, the entire hdr file content is batched into one field called ‘content’. The Tika ENVI parser needs to be improved such that each key is a top-level metadata field and is then searchable.

You can clone Apache Tika and try the extractions yourself this way you will see the short comings.

Finally, assuming you are feeling better, would we be able to schedule another meeting for Thursday morning at 9am?

Yes certainly. Until then, please send me any questions you have about the above.

Also, as previously stated, if it helps you in the administrative portion of your Capstone project, consider writing wiki entries for all major components we are working on e.g. crawler, metadata extraction, workflow, etc.