Request to Join the project

[venki]

Hi,
I am venkatesh. I am interested in contributing to this project. I am
doing my M.S in software systems
and I am highly interested in data mining. Please let me know what I
need to do to start contributing to this project.

My main skill sets include
Java ( SCJP 1.5),
SQL

Eager to get a reply.

Thanks,
Venkatesh

[David]

Venkatesh,
You might check out this thread on INQLE ideas. Much of the ideas
there are more nuts & bolts programming. There are some listed which
are more interesting
http://groups.google.com/group/inqle-core/browse_thread/thread/435d168d6f4d7af7
David

[David]

Hey Venkatesh,
Great to hear from you! Sounds teriffic. Lots of directions you can
go. I will contact you off the forum to explore directions you are
most interested.
Best,
David Donohue

[venki]

Hi David,
I have downloaded and configured the INQLE, the UI is looking great.
It looks more like a desktop app. I also googled and found some
information on RAP, RCP. Its terrific. I am getting more interested to
work on INQLE. Can you provide me any document reference where I can
try some sample runs of INQLE.

Thanks,
Venki

[David]

Venki,
I am glad you find inqle usable! Still far from a household appliance
but we're making progress.

Any data you can get your hands on in a delimited text format will
do. You can export Excel spreadsheets to delimited text, of course.
I got some standard data sets from here
http://archive.ics.uci.edu/ml/datasets.html
There are some other standard data sets out there but I have not found
a great many that are useful. To be useful, a standard data set
should publish the gold standard results, and those statistical
measures should be the same produced by inqle.

Once you have the data, you can load it as described here
http://code.google.com/p/inqle/wiki/How_To_Load_Data_Into_INQLE

Next you can configure a customized sampler, to use the label of
interest.
http://code.google.com/p/inqle/wiki/How_to_Create_Custom_Sampler
and run the experimenter agent.

Best,
David Donohue

[venki]

David,
1. I Worked on Loading the csv file. I could load the file
successfully. And after loading when i choose the option View RDF
page, The data imported is showing up. But how do I ensure that the
data has got imported successfully? How do i check that?
2. The data set which is used is in the below link,
http://archive.ics.uci.edu/ml/machine-learning-databases/car/car.data
I added the below header line to this data set and imported in inqle
"buying,maint,doors,persons,safety,class"

I followed the steps provided in the below url to get the data
loaded
http://code.google.com/p/inqle/wiki/How_To_Load_Data_Into_INQLE

3. Then I followed the steps in the URL
http://code.google.com/p/inqle/wiki/How_to_Create_Custom_Sampler
to create a customer sampler
The problem that I am facing here is , There is nothing which
shows up in the screen "Select Class of subject", though I have
imported the
data, and defined subjects. Is there anything I am missing or is
this a bug?

Some open questions:
1. How do I find that the data got imported successfully?
2. How is the data internally saved? I guess it is saved as RDF file.
Please provide more information on the structure in which data is
saved.
3. What is a sampler? Why do I need a sampler? What is the job of a
sampler?
4. Though the data is imported successfully, it is not showing up in
"Select class of subject screen" , does this mean that I have not
imported the data properly?
5. What is a URI?

Thanks,
Venki

On Jun 16, 5:40 pm, David <drdono...@gmail.com> wrote:
> Venki,
> I am glad you find inqle usable!  Still far from a household appliance
> but we're making progress.
>
> Any data you can get your hands on in a delimited text format will
> do.  You can export Excel spreadsheets to delimited text, of course.

> I got some standard data sets from herehttp://archive.ics.uci.edu/ml/datasets.html

> There are some other standard data sets out there but I have not found
> a great many that are useful.  To be useful, a standard data set
> should publish the gold standard results, and those statistical
> measures should be the same produced by inqle.
>

> Once you have the data, you can load it as described herehttp://code.google.com/p/inqle/wiki/How_To_Load_Data_Into_INQLE

>
> Next you can configure a customized sampler, to use the label of

> interest.http://code.google.com/p/inqle/wiki/How_to_Create_Custom_Sampler

[David]

Venki,

> 1. I Worked on Loading the csv file. I could load the file
> successfully. And after loading when i  choose the option View RDF
> page, The data imported is showing up. But how do I ensure that the
> data has got imported successfully? How do i check that?

See this wiki page on viewing your data
http://code.google.com/p/inqle/wiki/How_to_Browse_Your_Data

> 3.  Then I followed the steps in the URLhttp://code.google.com/p/inqle/wiki/How_to_Create_Custom_Sampler

>      to create a customer sampler
>      The problem that I am facing here is , There is nothing which
> shows up in the screen "Select Class of subject",

Hmm. You should see this page in the wizard. Could you describe in
more detail what you are doing and seeing in this wizard? Do you see
a Java error in the INQLE system console?

> 2. How is the data internally saved? I guess it is saved as RDF file.

It is saved in a Jena TDB database, which is an embedded semantic
database with great performance.

> Please provide more information on the structure in which data is
> saved.
> 3. What is a sampler? Why do I need a sampler? What is the job of a
> sampler?

A sampler is a piece of code which gathers a chunk of data for the
purposes of data mining by INQLE. See this page, which I am working
on
http://code.google.com/p/inqle/wiki/What_is_a_sampler

> 4. Though the data is imported successfully, it is not showing up in
> "Select class of subject screen"  , does this mean that I have not
> imported the data properly?

See above

> 5. What is a URI?

Uniform Resource Identifier. It is the kind of identifier used by
semantic web/RDF for uniquely identifying things. Looks just like a
URL, ensuring that it is unique and potentially "dereferencable" =
browsable.

[venki]

Hi Dave,
Read on Semantic web and RDF from the below link
http://www.xml.com/pub/a/2001/01/24/rdf.html

Open questions
1. What is caption and what is subject? what is the difference? Please
give some examples.
2. Is an Identifier column mandatory in the data set?
3. Why is the name column null for the Data Table class
4. Though i have imported the data set, I am not seeing any subject
getting listed in the select subject screen while creating custom
sampler.

I have emailed the flash movies of the data import and custom sampler
creation which I have tried out. Have a look on them and guide me if i
am missing something,

Thanks,
Venki

On Jun 16, 10:07 pm, David <drdono...@gmail.com> wrote:
> Venki,> 1. I Worked on Loading the csv file. I could load the file

> > successfully. And after loading when i  choose the option View RDF
> > page, The data imported is showing up. But how do I ensure that the
> > data has got imported successfully? How do i check that?
>

> See this wiki page on viewing your datahttp://code.google.com/p/inqle/wiki/How_to_Browse_Your_Data

>
> > 3.  Then I followed the steps in the URLhttp://code.google.com/p/inqle/wiki/How_to_Create_Custom_Sampler
> >      to create a customer sampler
> >      The problem that I am facing here is , There is nothing which
> > shows up in the screen "Select Class of subject",
>
> Hmm.  You should see this page in the wizard.  Could you describe in
> more detail what you are doing and seeing in this wizard?  Do you see
> a Java error in the INQLE system console?
>
> > 2. How is the data internally saved? I guess it is saved as RDF file.
>
> It is saved in a Jena TDB database, which is an embedded semantic
> database with great performance.
>
> > Please provide more information on the structure in which data is
> > saved.
> > 3. What is a sampler? Why do I need a sampler? What is the job of a
> > sampler?
>
> A sampler is a piece of code which gathers a chunk of data for the
> purposes of data mining by INQLE.  See this page, which I am working

> onhttp://code.google.com/p/inqle/wiki/What_is_a_sampler

[venki]

Hi Dave,
I just added a surrogate key Flower id to the iris flower data set and
made it as the identifier while loading the data.
Now the data I imported is displayed in the "Select Class of
subject", I configured the sampler
and started the Experimenter Agent
Now when i go to window -> Experiments I can see the experiment
results.
Please tell me as to how I can interpret the experiment results.

Thanks,
Venki

On Jun 17, 5:40 pm, venki <venkates...@gmail.com> wrote:
> Hi Dave,

> Read on Semantic web and RDF from the below linkhttp://www.xml.com/pub/a/2001/01/24/rdf.html

[David]

Venki,
Great work! One thing you can do is sort by correlation or accuracy.
I need to improve this aspect of INQLE. Your ideas are valued!

I am keen to find your difficulties and improve on that basis. I am
aware that I do not have enough validation of URIs. As users can
enter URIs in the course of the importation wizard, there is much
potential for error.
David

[venki]

Dave,
In the Iris Flower data set, the attributes are
FlowerId,sepalLength,sepalWidth,petalLength,petalWidth,class

I made the attribute "class" as the Experiment Label.
Let us take an example

I sorted ( desc) on correlation

the result displays Experiment attributes as
IrisFlower/petalLength
irisFlower/petalWidth

correlation is 0.90544

Does this refer to the correlation between the attributes petalLength
and petalWidth? Is this value calculated based on the pearson
correlation coefficient?

and the displayed accuracy is 0.966

Does it mean when I predict the experiment label based on the
attributes petalLength and petalWidth, the resulting accuracy is
0.966?

Did you get chance to look at the flash movies which I sent you? do
you think it will be a good idea to put up the inqle demo over the web
like that so that people can easily get started.

Thanks,
Venki

[David]

Venki,
Have not viewed your stuff yet but will.

> Does this refer to the correlation between the attributes petalLength
> and petalWidth?

No. For the predictive model built using attributes: petalLength and
petalWidth, the predicted class had 0.966 correlation with the actual
class.

> Is this value calculated based on the pearson
> correlation coefficient?

Not sure. Using RapidMiner's measure of correlation coefficient.
Pearson sounds right but would have to check with RapidMiner.

> and the displayed accuracy is 0.966
>
> Does it mean when I predict the experiment label based on the
> attributes petalLength and petalWidth, the resulting accuracy is
> 0.966?

yes!

>
> do you think it will be a good idea to put up the inqle demo over the web
> like that so that people can easily get started.

Brilliant idea! I love it. Go for it! What are you using? Have you
used screentoaster?

Dave

[Venkatesh U]

Dave,
I am signing off for the day. I say the Tim's presentation on SW, read a lot on RDF.
sure this is getting more interesting. I also checked the dev guide you have published to find the possible areas
where I can add some new features. I am interested in making the interface more user friendly and intuitive, we can also
make some flash demos which can serve as documentation. I used a freeware called wink to make those flash videos.
I find wink easy and simple to use.

Have a great day.

Thanks,
Venki

--
Thanks and Best Regards,

Venkatesh Umaashankar
Applications Engineer,
ORACLE,
Mailto    : venka...@gmail.com
Landline : 918040298793
Mobile   :  919902330421
Website: http://www.oracle.com

Arise, awake and stop not till the goal is reached - Swami Vivekanandha

[David Donohue]

testing

[David]

> 1. What is caption and what is subject? what is the difference? Please
> give some examples.

This passage from the wiki page describes captions and subjects.
"Subjects are things that describe a single row in the table. When
importing subjects, you must specify what type of thing each subject
is a member of. Example: we are importing a spreadsheet of companies.
The subject type is "company", and each row represents a different
instance of that type (i.e., each row is a different company).
Captions are things that describe the entire table. Each row does not
represent a new thing of some common type. Rather, the caption
represents a single thing. Example: the above table of companies is
our company's list of suppliers. So we might add a caption of the type
"company", with an identifier of "Our Company". This says that the
entire table is about "Our Company". Another example: we are importing
prices of the S&P 500 Index. We define a single caption, of type
"stock index", and identify it as "the S&P 500 Index"."

I hope that helps. This concept of importing the "caption" is new.
It is our attempt at being able to capture relevant data that is not
represented in the data table being imported. Rather, this data is
about the entire table of data being imported. So if the entire
spreadsheet contains stock pricing and corporate data for a particular
company, the id and name of the company would be captured in the
caption, not in any of the columns.

> 2. Is an Identifier column mandatory in the data set?

No. Subjects can have "blank nodes" as a subject. However you cannot
browse the data if they have blank nodes. It might prevent sampling
from working properly.

> 3. Why is the name column null for the Data Table class

A DataTable is a class containing the subjects and captions within the
table. I need to make a wiki page describing the structure of data
imported by the wizard.

Dave

[Venkatesh U]

Dave,
I already read the passage which you have provided for this question from the wiki page.

> 1. What is caption and what is subject? what is the difference? Please
> give some examples.

I would need more clarifications.

I am importing a list of  companies, each row in the I can describe using the subject "Company"
When it comes to caption, this is a list of our company's supplier, This data is about our company's suppliers.
So I feel a better caption would be "Our company's suppliers",

why should  we use the caption "Our company"

I assume that the caption "Our company", if  already represented by an URI, we can use it to link to this list of  companies
may be something like this in a triple format

<Our company> has < suppliers> who  are <imported companies>

If the intention is to capture the relationship like this when we import the data, the predicate part is missing.

The caption is  "Our company" and the subject is "Company", then how I find the how this list of companies relate to Our company?

Please clarify

Thanks,
Venki

[David Donohue]

Venki,
Glad you are asking these astute questions. You are the first to test
my approach so you are helping revise the strategy.

Probably most imports would not need to use the caption. Let's say
you are importing stock pricing data. If the table has 1 row per
stock, then you simply specify a subject of StockPrice, which has:
* these identifiers: a company, a ticker symbol
* these row data: a stock price, a date

Say however you were importing a table of different structure. The
entire table is about company ABE Dielectrics. Each row is a
different date, and you specify a date column. Each column represents
things like stock price, # of employees, daily income.

Originally I thought I needed the concept of a caption to handle this
but it occurs to me that we could probably get by using subjects
exclusively as follows:
The subject again is Company. This time you use a static URI as the
ID value. You handle the data attributes as in the first case.

So maybe Captions are redundant or not needed! Certainly it will
simplify the process. Great stuff. Do you concur that I should
streamline the importer, removing captions?
Dave

[Venkatesh U]

HI Dave,
   I agree with you, may be we should remove the caption. And I also want to make the user easily understand
that subject is something which refers to the entire row and columns are attributes of the subject. Should we also
add more flexibility to link subjects?

Say I am importing my daily appointments, here the subject  could be appointment and attributes may be
date, time, task description, person to meet.
how does the data get saved. I guess the system generates an URI for each instance of the subject ( i.e each row).

Let us assume, already there is a subject Person, and its instance is Me and I have a URI. How do I link the list of appointments to myself.

Next, I am uploading my contacts, where the rows are instances of the subject  "Person" again,  how do I link
the person to meet in the appointment list to the instances in my contact list?

Does Jena TDB already offer features to create such links? If we start linking data as user uploads the data or
provide a wizard to user through which he can link data. I think we can do more exciting data mining stuff.

I am not sure If  I have put it clearly so that it makes sense. Please let me know your view points

Venki

[David Donohue]

OK great discussion.

> HI Dave,
>    I agree with you, may be we should remove the caption. And I also want to
> make the user easily understand
> that subject is something which refers to the entire row and columns are
> attributes of the subject. Should we also
> add more flexibility to link subjects?

Yes. Maybe the wizard could say something like
"The subject is what each row is about. In this table, what is each
row about? Examples: a company, a person, a country, a tissue
sample."

> Say I am importing my daily appointments, here the subject  could be
> appointment and attributes may be
> date, time, task description, person to meet.
> how does the data get saved. I guess the system generates an URI for each
> instance of the subject ( i.e each row).

Yes, each row has a URI. When the entire table relates to the same
instance (e.g. entire table is various data about company ABC), then
each row has the identical URI.

> Let us assume, already there is a subject Person, and its instance is Me and
> I have a URI. How do I link the list of appointments to myself.

Great question. Yes, and there are standards for personal URIs (e.g.
vcard). Formerly this was handled by the caption. I think we will
replace the caption with a list of static values, which apply to each
row. So you could have the static value "hasPerson" or
"pertainsToPerson" with a value of your URI.

> Next, I am uploading my contacts, where the rows are instances of the
> subject  "Person" again,  how do I link
> the person to meet in the appointment list to the instances in my contact
> list?

Um using the URI of the person. So importing your appointment list,
you might produce the statement
my:appointment_127 event:hasAttendee <http://me.daviddonohue.com>

And importing the contacts list you would produce the statement
<http://venki.name> foaf:knows <http://me.daviddonohue.com>

> Does Jena TDB already offer features to create such links?

Sure. The Jena API and its various back-end model structures (RDB,
SDB, TDB, in-memory models) support doing most anything with RDF.

> If we start
> linking data as user uploads the data or
> provide a wizard to user through which he can link data. I think we can do
> more exciting data mining stuff.

Yes. And we could have other means of importing RDF, e.g. a forms or
survey module.

> I am not sure If  I have put it clearly so that it makes sense. Please let
> me know your view points

Very clear! I hope my responses are clear.

So my to do list:
* Remove captions
* Add a page for subject imports, to import static values
* Make sure we are accurately identifying URI values when the static
or dynamic value of a URI rather than a literal.
* Improve instructions and flow in the wizard to be more intuitive.
* Change documentation on the importation process.
* Capture location info as well in the wizard
* Screencast of the importation process
* Finish a new semantic wiki module, to permit better faceted
browsing of the data. Each semantic subject/object is an editable
wiki page, with links to its associated objects.
* etc.
* etc.

Keep the great questions & ideas coming!
Best,
Dave

[Venkatesh U]

Dave,
 I would also like to get involved in the code changes, so please provide me the details of the software that I would need to download to start coding. I have read about RDF, and I am also looking through documentation of jena TDB. I am interested in exploring lot of data mining through the lens of semantic web. Dave I would love to speak with you over skype,
and we can plan more on future directions. Let me know when you are available on skype

Thanks,
Venki

[Venkatesh U]

I have checked out the bundles as given in the instructions, Have setup RAP in eclipse.
Still I observe some errors in the created packages. attaching the screen shots
please check them and help to get them resolved

On Mon, Jun 22, 2009 at 9:24 PM, David Donohue <d...@daviddonohue.com> wrote:

> do u use Linux or windows for setting up the development environment?

Windows XP

> I think I would need more detailed steps on setting up the environment. I
> run windows vista in my Laptop and I also have a virtual box of fedora which
> use for Linux needs.
>
> Should i first check out all the bundles listed as read only and setup the
> environment?

Sorry!  I need more explicit instructions.
We recommend you use Eclipse 3.4 or above, with the Subclipse plug-in.
 This has problems working behind a firewall.  Check out each of the
required bundles as follows:
In Eclipse, go to File -> New -> Project... ->
Checkout projects from SVN ->
Create a new repository location ->
Enter INQLE's subversion location
https://inqle.googlecode.com/svn/trunk/

From here not sure whether you must check out each project
individually or whether you can do multiple at a time.  I cannot test
from behind firewall.  But you should get 1 project per bundle.

> I am downloading the entire source code by using the command
> svn checkout http://inqle.googlecode.com/svn/trunk/
>
> Does this get me all the source code required for development?

yes.  But need each bundle to be a separate project in Eclipse.

> Also provide the steps to configure eclipse for RAP

See Eclipse RAP getting started:
http://www.eclipse.org/rap/gettingstarted.php
Not that hard really.  They have screencasts.

Dave

[David]

Venki,
Sorry about these difficulties. I got these types of errors. This is
part of the hill to climb that is Eclipse Equinox OSGi development. I
want to document the solution to these difficulties, to remove these
hurdles for future developers. Maven might be the best way to improve
setup of the dev environment, and I am considering mavenizing inqle if
possible.

First I would try this in Eclipse
Project -> Clean... -> Clean all projects
You might have to clean a project or all projects from time to time.

Next consider changing an Eclipse setting
http://lkamal.blogspot.com/2008/09/eclipse-access-restriction-on-library.html

I found this as well
> You can override the access rules on individual system libraries (jars), by
> adding them at the "JRE System Library" level - i.e. the first "access
> rules" entry can be edited.

For the bundles having trouble, you can look at Properties->Build Path
and see what is missing. It seems that 1 problem is you are having
trouble accessing the log4j library, which is exported by bundle
org.inqle.core. I have had this issue before.

Sorry about these hurdles! I hope to resolve these errors without too
much pain. Please let me know how it goes.
Dave

> > > svn checkouthttp://inqle.googlecode.com/svn/trunk/

>
> > > Does this get me all the source code required for development?
> > yes.  But need each bundle to be a separate project in Eclipse.
>
> > > Also provide the steps to configure eclipse for RAP
> > See Eclipse RAP getting started:
> >http://www.eclipse.org/rap/gettingstarted.php
> > Not that hard really.  They have screencasts.
>
> > Dave
>
> --
> Thanks and Best Regards,
>
> Venkatesh Umaashankar
> Applications Engineer,
> ORACLE,

> Mailto    : venkates...@gmail.com

> Landline : 918040298793
> Mobile   :  919902330421
> Website:http://www.oracle.com
>
> Arise, awake and stop not till the goal is reached - Swami Vivekanandha
>

>  devsetup.pdf
> 638KViewDownload

[Venkatesh U]

Hey Dave,
Come on... too many apologies.
   I understand that INQLE is in rudimentary stage and you have lot of ideas to make it a commercially viable tool.
I don't take these hurdles serious, lets get them resolved. If you feel that I can do some help to mavenize inqle, I am ready to do that. I will try my best to contribute what ever I can. After all. these hurdles are part of any software development.
  I feel I am a part of inqle team and Its also my responsiblity to keep the product clean. Let me try out the suggested work arounds and get back to you if I face any further problems. BTW, I started trying out some remo on RAP and RCP, I hope I can make the learning curve as short as possible.

Cheers,
Venki

Mailto    : venka...@gmail.com

[David]

Venki,
Thanks for your patience! I tried a little change to project
org.inqle.experiment.rapidminer
After update from subversion and perhaps cleaning that project, you
might see if the log4j errors resolve for that project.

A new version of Eclipse will be released tomorrow, so if we are still
having trouble i will download that and go thru the setup process
locally and figure out once and for all how to solve these
intermittent errors.
Dave

[Venkatesh U]

Dave, the problems which I am facing are

these imports are not working
import org.inqle.agent.AAgent;
import org.inqle.agent.IAgent;

Bundle 'org.eclipse.rap.core.databinding.beans' cannot be resolved   

The import org.inqle.agent cannot be resolved   

[Venkatesh U]

Dave,
Found a bug with is causing an issue in setting up the dev environment
In the file
org.inqle.data.rdf.jenabean/ src/ main/ org/ inqle/ data/ rdf/ jenabean/ Persister.java

In this file you have replaced all the dataset with Datamodel

But in the latest version of the file
org.inqle.data.rdf.jenabean/ src/ main/ org/ inqle/ data/ rdf/ jenabean/ PersisterInitializer.java

This class still refers to the dataset, is the latest version of this file missing in SVN?

 persister.getInternalDatasets();

[David Donohue]

Good find, Venki. Correct that I renamed all "dataset" with
"datamodel". We have many conflicting naming conventions between data
mining and semantic web.
E.g. a "model" means very distinct things in both domains. As does
"example". A Jena "Dataset" is a particular class. I wanted a
colloquial name for a named graph in INQLE that would not conflict
with Jena and data mining names, so I switched to "datamodel".

PersisterInitializer was removed some time ago. I kept it in the
attic folder, but not in the source folders. Are you seeing it in
your source folder? If so, you have somehow checked out an old
version of the code.

[Venkatesh U]

I think the file is not obsolete yet
Please check the link below and confirm

http://code.google.com/p/inqle/source/browse/trunk/org.inqle.data.rdf.jenabean/src/main/org/inqle/data/rdf/jenabean/PersisterInitializer.java

[Venkatesh U]

And all these imports are not getting resolved though I have created a project for
org.inqle.ui.rap

import org.inqle.ui.rap.IList2Provider;
import org.inqle.ui.rap.IListProvider;
import org.inqle.ui.rap.IValueUpdater;
import org.inqle.ui.rap.pages.NameDescriptionPage;
import org.inqle.ui.rap.pages.SimpleListSelectorPage;
import org.inqle.ui.rap.table.BeanTableSelectorPage;

[Venkatesh U]

Dave,
Find these classes are available in the package org.inqle.ui.rap
but still these imports are failing in the project org.inqle.data.sampling.rap
Am i missing something?

[David Donohue]

this might relate to the eclipse problems we are having. New Eclipse
(Gallileo) is available for me (as a "friend of eclipse") now so i am
downloading.
will set up inqle from scratch there.