Very interesting software

[Ettore Rizza]

Exploratory is a data analytics software with an active community version. It has rich features in data wrangling, data viz and analysis. Its originality is it uses "under the hood" an open source language, R. The operations performed with the interface can also be exported as a perfectly conventional R script.

I mention it because its UI resembles looks in many ways like that of Open Refine. The menus are very similar (see screencast). I think there are inspiring sources in terms of design and UX. For example, see how several dataframes (ie: tables) can be grouped by project, with a Markdown editor to annotate the analysis.The latest version has added as functionality the possibility to reorder the steps of the analysis by drag and drop: it would be great if the OR's operations history could be reordered this way.

[Thad Guidry]

I've played with Exploratory many times.

I'd rather OpenRefine stay fairly close to its heritage of "cleaning messy data".

If during the process of trying to "clean"...there are nicer Facets or Summaries or Grouping, whatever...that you feel would be useful to help "clean" the data prior to importing into Exploratory....then tell us Ettore.  I'd like you to spend some time thinking about some of your messy datasets and how having certain functions, UI, viz, in OpenRefine would be useful to help EXPLORE a bit...but only for Cleaning operations....not pure data analysis, since that's not our domain.

Listening,

-Thad

[Ettore RIZZA]

There is a misunderstanding I think. What I find interesting in this case is its UI, not its features (even if they are). But a large part of them are precisely dedicated to data cleaning, and its drop-down menus are very close to those of Open Refine. I'm not saying they are better or worse, but I think they are an interesting benchmarking point (among others) for the day we'll have to reorganize the menus or the interface, as discussed here: https://github.com/OpenRefine/OpenRefine/issues/1299

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thad Guidry]

Yes, organizing and simplifying our UI to make operations easier is a focus for us longterm.  Yeap, understood, but I still want some specifics from you as a user.  We value your input there !  So take the time to think through it, help us with the redesign by telling us the specific things in Exploratory you like.   Draft it up on a Google Doc and share it, or make a new Wiki page titled "bikeshedding UI ideas from Exploratory" and delight us. :)

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.

[Ettore RIZZA]

I will try to make a proposal that will take into account the good things I've seen in other softwares, not just Exploratory. For example, Talend Data Preparation seems to have been very much inspired by Open Refine, but some of their features could in turn inspire us. 

Here is for example the list of its functions when you click on a column. I put in bold those that seem interesting to me.

SUGGESTIONS

Fill empty cells with text...

Delete the rows with empty cell

Remove trailing and leading characters...

Mask data (obfuscation)

Replace the cells that match...

BOOLEAN

Negate value

COLUMNS

Concatenate with... <- would be very usefull

Delete column

Swap columns...

CONVERSIONS

Convert distance...

Convert duration...

Convert temperature...

DATA CLEANSING

Clear on matching value...

Clear the cells with invalid values

Delete the rows that match...

Delete the rows with empty cell

Delete the rows with invalid cell

Fill cells with value...

Fill empty cells with text...

Fill invalid cells with value...

Remove negative values

DATA MASKING

Mask data (obfuscation)

DATES

Calculate time until...

Calculate timestamp to date...

Change date format...

Compare dates...

Convert date...

Extract date parts...

Modify date...

MATH

Add, multiply, subtract or divide...

Base 10 logarithm

Calculate absolute value

Cosine

Exponential

Max...

Min...

Natural logarithm

Negate

Power...

Sine

Square root

Tangent

NUMBERS

Compare numbers...

Format numbers...

Remove fractional part

Round value using ceil mode...

Round value using down mode...

Round value using floor mode...

Round value using halfup mode...

PHONES

Format phone number...

SPLIT

Extract email parts

Extract number

Extract string parts...

Extract URL parts

Split the text in parts...

STRINGS

Calculate length

Change to lower case

Change to title case

Change to upper case

Contains text...

Extract parts of the text...

Match similar text...

Matches pattern...

Remove consecutive characters...

Remove part of the text...

Remove trailing and leading characters...

Replace the cells that match...

STRINGS ADVANCED

Add extra characters...

Find and group similar text...

Remove all non alpha numeric characters

Remove all non numeric characters

Simplify text (remove case, accent, etc.)

2018-01-04 17:52 GMT+01:00 Thad Guidry <thadg...@gmail.com>:

Yes, organizing and simplifying our UI to make operations easier is a focus for us longterm.  Yeap, understood, but I still want some specifics from you as a user.  We value your input there !  So take the time to think through it, help us with the redesign by telling us the specific things in Exploratory you like.   Draft it up on a Google Doc and share it, or make a new Wiki page titled "bikeshedding UI ideas from Exploratory" and delight us. :)

On Thu, Jan 4, 2018 at 10:47 AM Ettore RIZZA <ettor...@gmail.com> wrote:

There is a misunderstanding I think. What I find interesting in this case is its UI, not its features (even if they are). But a large part of them are precisely dedicated to data cleaning, and its drop-down menus are very close to those of Open Refine. I'm not saying they are better or worse, but I think they are an interesting benchmarking point (among others) for the day we'll have to reorganize the menus or the interface, as discussed here: https://github.com/OpenRefine/OpenRefine/issues/1299

2018-01-04 16:39 GMT+01:00 Thad Guidry <thadg...@gmail.com>:

I've played with Exploratory many times.

I'd rather OpenRefine stay fairly close to its heritage of "cleaning messy data".

If during the process of trying to "clean"...there are nicer Facets or Summaries or Grouping, whatever...that you feel would be useful to help "clean" the data prior to importing into Exploratory....then tell us Ettore.  I'd like you to spend some time thinking about some of your messy datasets and how having certain functions, UI, viz, in OpenRefine would be useful to help EXPLORE a bit...but only for Cleaning operations....not pure data analysis, since that's not our domain.

Listening,

-Thad

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-dev+unsubscribe@googlegroups.com.

[Martin Magdinier]

I started creating issues based on Ettore highlight and I realized that most of those functions are rather shortcut of existing OpenRefine function (or combination of functions and facet). 

Instead of thinking for the user all useful shortcut, I suggest we let the user create a new entry in the a drop down menu and let him link it to a particular GREL function or set of operation (in the same format as the history so it combine facet and function). 

What do you think? 

[Thad Guidry]

Sure. I captured that idea quite a while back in the early days of OpenRefine...

Allow Users and Extensions to customize a Custom Menu (Tools/Other) area
https://github.com/OpenRefine/OpenRefine/issues/109

-Thad

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.

[Ettore RIZZA]

It would be great of course ! But we must not forget the simplicity of use. For many users without any computer skills, for example journalists or librarians, the very word GREL is scary. 

Ideally, the point-and-click menu should make it possible to carry out the majority of the common operations (for example a concatenation of columns).

Remains to list the common operations that should be added by default.

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-dev+unsubscribe@googlegroups.com.

[Flavio Pompermaier]

Why not starting with a minimal UI like the one of Dremio[1] and then target to something better like Dataiku[2]?

They are both very similar to what I expect from the 3.5 release

[1] https://d33wubrfki0l68.cloudfront.net/6ee3200625f18199d8e8a7e8be2ab1c6d954b063/5430e/img/tutorials/connecting-elasticsearch-cluster-to-dremio/image_6.png

[2] https://www.dataiku.com/dss/features/data-wrangling/

[Martin Magdinier]

Ettore, I agree with you. My point was we should make it easy for a super journalist or librarian to custom one those shortcuts and publish them as an extension for OpenRefine.

We won't be able to accommodate each and everyone needs in the core application, your common operations are different than mine ;) 

--

Martin Magdinier

+1 (514) 559 4563