COAL Updates

[Alexa Huerta]

Hi Dr. McGibbney,

We have been running into some issues and it therefore doesn't look like we will be able to get a coal-sds 0.1 release out today. We wanted to send you updates on our status though and ask if you would be willing to accept documentation for the daemon as the deliverable that our professor wants us to turn in today?

Updates
SDS Team: Got the daemon to continuously run but it keeps ingesting already-ingested products. Posted issue for this here: https://github.com/capstone-coal/coal-sds/issues/25. Also ran into error after trying to manually update Tika to version 1.19, see issue here: https://github.com/capstone-coal/coal-sds/issues/24

Pycoal Team: The newer versions of WINGS that we have access to does not seem to allow for an OODT execution engine. I believe this is related to the warning I received from the people working on WINGS that the oodt-adapter has not been maintained for awhile now.  Should we proceed with not using WINGS for the Workflow Manager? Also, we canceled running example_mineral.py after ~3-4 days of running. The script did not look like it was making any additional progress. Were you able to complete running the file on your machine?

Finally, would be available to meet at 9am tomorrow (Tuesday)?

Thank you,

Alexa Huerta

--

Alexa Huerta

University of Southern California

B.S. Computer Science, December 2018

alex...@usc.edu

[Lewis John Mcgibbney]

Hi Team,

Apologies I've been crammed with work this week. Please see my replies below.

On Mon, Oct 15, 2018 at 1:51 PM Alexa Huerta <alex...@usc.edu> wrote:

Hi Dr. McGibbney,

We have been running into some issues and it therefore doesn't look like we will be able to get a coal-sds 0.1 release out today. We wanted to send you updates on our status though and ask if you would be willing to accept documentation for the daemon as the deliverable that our professor wants us to turn in today?

Yes.

 

Updates
SDS Team: Got the daemon to continuously run but it keeps ingesting already-ingested products. Posted issue for this here: https://github.com/capstone-coal/coal-sds/issues/25. Also ran into error after trying to manually update Tika to version 1.19, see issue here: https://github.com/capstone-coal/coal-sds/issues/24

Let's take both of these conversations over to the issue track thank you.

 

Pycoal Team: The newer versions of WINGS that we have access to does not seem to allow for an OODT execution engine. I believe this is related to the warning I received from the people working on WINGS that the oodt-adapter has not been maintained for awhile now.  Should we proceed with not using WINGS for the Workflow Manager?

Please point me to the code. The code I had committed at https://github.com/capstone-coal/wings appeared to nearly be working. Did you try this and if so what were the results? Did you document what you tried such that I can could reproduce it? Thanks

 

Also, we canceled running example_mineral.py after ~3-4 days of running. The script did not look like it was making any additional progress. Were you able to complete running the file on your machine?

I'll start a run right now and see what I can generate. Thanks for the update.

 

Finally, would be available to meet at 9am tomorrow (Tuesday)?

Seeing as tomorrow is Friday now, lets get something not he calendar for then instead. Please send me an invitation,

Thank you 

[Kristen Cheung]

Hi Dr. McGibbney,

Unfortunately, we are only able to meet for around 45 minutes tomorrow at 9am (I'll create a meeting link shortly).

Please let us know if this time works for you, otherwise we'd be happy to set up a longer time on Monday morning. 

Thank you,

Kristen Cheung

--
You received this message because you are subscribed to the Google Groups "COAL - Coal and Open-pit surface mining impacts on American Lands" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coal-capston...@googlegroups.com.
To post to this group, send email to coal-c...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/coal-capstone/CAGaRif3TFEzoBrcEmG96PegY0SZspjFCFjGM9Sf6ZUY-CB5TVw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Lewis John Mcgibbney]

Let's do 9am tomorrow thanks, see you all then.

Lewis

--

Lewis

Dr. Lewis J. McGibbney Ph.D, B.Sc

Skype: lewis.john.mcgibbney

[Lewis John Mcgibbney]

I’m running a bit behind schedule this morning Folks. I’ll log into the meeting around 9:15.

Please begging reviewing the work I did last night and pull the code and test it out as a team.

I discovered some bugs in the CLI for pycoal as well which was annoying so I will try to fix that tonight.

Speak to you all soon thank you.

Lewis 

[Kristen Cheung]

Hi Dr. McGibbney,

Thank you for meeting with us today--I wanted to summarize the action items since the meeting was rushed.

Please let us know if this looks okay or if you would like to add to the action items. 

Action Items

Dr. McGibbney:

Denim and I started the wiki page for the crawler here. We plan to make this Deliverable #4 instead of the software update.

Would you be able to email Professor Miller (jeffrey...@usc.edu) your approval for this deliverable?

Additionally, Denim and I will update the wiki page with the changes we discussed during the meeting.

Denim and Kristen:

We will continue working on crawler with the changes we discussed during the meeting, ensuring the crawler will delete the files after ingestion. 

Alex and Alexa:

Alex and Alexa will look into automating the start of the pycoal tests and start grabbing the data from the ftp server. 

I set up our next meeting for Wednesday (10/24) at 9am. Please let me know if this time does not work for you.

Thank you,

Kristen Cheung

To view this discussion on the web visit https://groups.google.com/d/msgid/coal-capstone/CAGaRif0ZK1DQ9jhFx%2B7%3D3x%3DfUkOf_mnKMrX1nOBpKitqs5KBxA%40mail.gmail.com.

[Lewis John Mcgibbney]

Hi Team,

Responses inline below

On Fri, Oct 19, 2018 at 1:09 PM Kristen Cheung <cheu...@usc.edu> wrote:

Hi Dr. McGibbney,

Thank you for meeting with us today--I wanted to summarize the action items since the meeting was rushed.

Please let us know if this looks okay or if you would like to add to the action items. 

Action Items

Dr. McGibbney:

Denim and I started the wiki page for the crawler here. We plan to make this Deliverable #4 instead of the software update.

That sounds fine. I think the documentation looks fantastic, thank you for putting it together.

 

Would you be able to email Professor Miller (jeffrey...@usc.edu) your approval for this deliverable?

Yes, I'll CC the team on this.

 

Additionally, Denim and I will update the wiki page with the changes we discussed during the meeting.

I just did that. It's done. Consider it a done deal.

 

Denim and Kristen:

We will continue working on crawler with the changes we discussed during the meeting, ensuring the crawler will delete the files after ingestion. 

That's already done. The code can be seen at https://github.com/capstone-coal/coal-sds/blob/master/crawler/src/main/resources/bin/crawlctl

Note the new flags which determine that the product should be deleted post ingestion.

I've also reflected this within the wiki page so that is done as well.

Please continue to move forward with the Metadata Extraction issue. We need to force Apache Tika to process the .hdr files with the EnviHeaderParser such that the correct metadata is extracted. I'm going to keep working on this as well as I really want to get it working. See the following question I posted over on d...@oodt.apache.org

https://lists.apache.org/api/atom.lua?mid=6dfdc14bfa60ffd9c8446d6e72ffe6ea45a683257359df53bdafb013@%3Cdev.oodt.apache.org%3E

 

Alex and Alexa:

Alex and Alexa will look into automating the start of the pycoal tests and start grabbing the data from the ftp server. 

Yes basically you need to 1) learn about OODT workflows and 2) put in place the correct plumbing to make sure that the workflow(s) kick off after a successful ingestion.

 

I set up our next meeting for Wednesday (10/24) at 9am. Please let me know if this time does not work for you.

Sounds good right now. If anything changes I'll let you know.

Thanks 

[Kristen Cheung]

Great, thank you so much. We look forward to the next meeting. 

Best,

Kristen 

--

You received this message because you are subscribed to the Google Groups "COAL - Coal and Open-pit surface mining impacts on American Lands" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coal-capston...@googlegroups.com.
To post to this group, send email to coal-c...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/coal-capstone/CAGaRif2Gx3HrayLU5ikPVsFvJ_GDcD_iLZ7gF26dBAG2NYsWbQ%40mail.gmail.com.