RE: Customizing lumify

[Charles Greenbacker]

Hi Justin,

Thanks for contacting us, and thanks also for the kind words about Lumify.

> I'm not a java developer but I can read code and was able to find a file
importer. How can I use that to import my own files to test and get a
feel for how Lumify will work with my data set?

The best way to get started trying out Lumify with your own data is by using a public demo instance that we set up on AWS. Once you've logged into Lumify there, you can simply drag & drop files onto your workspace in the browser to import them into the application and test away. More info and a link to the demo instance is available here: http://lumify.io/try.html

> If possible I would also appreciate a bit more information about an
feature mentioned in one of the presentations on the Lumify site. I
think it was called the enterprise importer. If I remember correctly it
would allow for streaming content into Lumify. Is that correct, is that
something that is already in the works? If not is there a format or data
model used to write data to Lumify?

Yes, the enterprise importer for bulk/streaming data ingestion does exist, but we've not yet released that code as open source. One main reason for not releasing it is that our ingest method is only one way of doing it, and we don't want to force people to use our method. Different situations and use cases may require different approaches to ingestion, and our approach might not be best for everybody. In a nutshell, bulk/streaming data is sent to a Storm topology where various open source analytic tools extract & enrich the raw data before it gets indexed in ElasticSearch and stored in SecureGraph/Accumulo.

For more info on storing data in SecureGraph/Accumulo, see Joe's video on our blog: http://lumify.io/blog/2014/04/18/meetup-presentation-securegraph-lumify/ (slides and code examples from Joe's talk are available here: https://github.com/altamiracorp/secure-graph-presentation).

But like I said before, this isn't the only way you can do it. And again, if you're looking for the quickest way to get up & running using Lumify with your own data, we strongly recommend the public demo instance: http://lumify.io/try.html

Cheers,
Charlie

Charlie Greenbacker | Director of Data Science
Altamira Technologies Corporation
charles.g...@altamiracorp.com
8201 Greensboro Dr., Suite 800, McLean, VA 22102
Web: altamiracorp.com | Twitter: @ExploreAltamira

________________________________________
From: Justin Hohner <jus...@scoutswarm.com>
Sent: Thursday, May 22, 2014 12:35 PM
To: info
Subject: Customizing lumify

Hello,

I recently found one of your projects, Lumify, and really like what you
are doing with it. In fact I like it so much I would like to incorporate
it into a product we are developing, ScoutSwarm. However, I can only
find limited documentation and I'm struggling a bit because of it. I
understand that it's a new project so I'm looking for some help to get
me started and not expecting full documentation :).

I'm not a java developer but I can read code and was able to find a file
importer. How can I use that to import my own files to test and get a
feel for how Lumify will work with my data set?

If possible I would also appreciate a bit more information about an
feature mentioned in one of the presentations on the Lumify site. I
think it was called the enterprise importer. If I remember correctly it
would allow for streaming content into Lumify. Is that correct, is that
something that is already in the works? If not is there a format or data
model used to write data to Lumify?

Thanks in advance for your help!

--
Justin Hohner, Co-Founder
www.scoutswarm.com
@scoutswarm
949.636.0499

[Charles Greenbacker]

Hi David,

> I [...] would like to see if I can understand Lumify's schema at that level of detail as well.

I'm afraid the most extensive documentation about how we store data in Accumulo would be Joe's presentation video & slides on SecureGraph (linked above).

> I vote that you open source your ingest process. Use a different project name if you want to separate the ingest process from the Lumify code base because it's opinionated. I don't know of any examples of a sophisticated Accumulo ingest in GitHub. Please lead the way.

We've considered exploring this route for releasing the ingest code, and it may be something that we do eventually.

Thanks,

Charlie

Charlie Greenbacker | Director of Data Science

Altamira Technologies Corporation

charles.g...@altamiracorp.com

8201 Greensboro Dr., Suite 800, McLean, VA 22102

Web: altamiracorp.com | Twitter: @ExploreAltamira

---

From: David Medinets <david.m...@gmail.com>
Sent: Friday, May 23, 2014 8:25 AM
To: Charles Greenbacker
Subject: Re: [lumify:31] RE: Customizing lumify

 

Hi. I am also interested in using Lumify. I recently did a writeup about the D4M schema (https://github.com/medined/D4M_Schema) and would like to see if I can understand Lumify's schema at that level of detail as well.

I vote that you open source your ingest process. Use a different project name if you want to separate the ingest process from the Lumify code base because it's opinionated. I don't know of any examples of a sophisticated Accumulo ingest in GitHub. Please lead the way.

Thanks for Lumify. I look forward to learning more about it.

--
You received this message because you are subscribed to the Google Groups "Lumify" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lumify+un...@googlegroups.com.
Visit this group at http://groups.google.com/group/lumify.
To view this discussion on the web visit https://groups.google.com/d/msgid/lumify/1400845333072.94448%40altamiracorp.com.
For more options, visit https://groups.google.com/d/optout.

[Charles Greenbacker]

Hi Justin,

Yeah... you're not going to want to drag & drop 400GB onto the public-facing demo... nor would we want you to. :)

It sounds like your best bet will be to stand up your own instance of Lumify by building from source. As others have pointed out, building Lumify from source and installing all of its dependencies is a significant undertaking, even for developers. If you run into any problems, please post your questions to the mailing list and we'll do our best to respond.

Good luck!

- Charlie

Charlie Greenbacker | Director of Data Science
Altamira Technologies Corporation
charles.g...@altamiracorp.com
8201 Greensboro Dr., Suite 800, McLean, VA 22102

Office: 703.639.2369 | Mobile: 860.965.8885
Web: altamiracorp.com | Twitter: @ExploreAltamira

________________________________________
From: Justin Hohner <jus...@scoutswarm.com>

Sent: Friday, May 23, 2014 12:38 PM
To: Charles Greenbacker
Subject: Re: Customizing lumify

Charlie

Thanks for the info. Unfortunately I have about 400 GB of data that I
want to use and it's not easily dragable or dropable :). I'm looking to
integrate what you have developed with what we have developed. I guess I
will have to just keep reading the source and learn a bit more about all
the components.

Thanks!

[Hasala M S]

Hi Charles, 

I need some help regarding this issue.
I have a MySQL database with a set of data, some are location data and some are person data. I want to import these data to lumify work space to analyze its patterns. My questions are

1. Is there a method to directly import sql file so that lumify would understand the data model and relationships of the entities I have?
2. If not what method can you suggest me to import MySQL data into lumify?
3. If there's no such method what alternative analysis tools would you suggest?

Your help is much appreciated,

Thanks in advance,
Kind regards,

Hasala 

[Gunjan Kr]

Hi Hasala
Hope you are still working on lumify. Currently I am in process of developing environment for Lumify server and os etc.
My goal is similar as mentioned in your request. Have you done something further i.e. imported the data from mysql tables.
if yes please share the method
thank you
Gunjan Kumar

[issah....@gmail.com]

Got an instance working . has anyone managed data ingestion>?

[ido123]

Can you share how you got an instance running?