Knight Foundation News Challenge

[Martin Magdinier]

Hello!

We've put a proposal to improve OpenRefine's extensibility with data repository and processing services at the Knight Foundation News Challenge. Through this application we are addressing three challenges facing the OpenRefine community:

• Continue to support and develop OpenRefine core based on the community request
• Improve OpenRefine’s extensibility ; Our plan is to create one of each type of extension (data repository and processing service) and based on our experience, and other community members’ feedback, improve the overall process (documentation, code, hooks …) for other developers.
• Grow OpenRefine user base

Our can read more about it here: https://www.newschallenge.org/challenge/data/entries/improve-openrefine-s-extensibility-with-data-repository-and-processing-services   

Feedback can be made on the news challenge website until September 31.
As always, we're happy to work with anyone interested in this roadmap.

Martin Magdinier

[Thad Guidry]

Haha... its funny... 99% of all of the applications to that Knights Foundation challenge of "How might we make data work for individuals and communities?" ... rely on clean data and making sense of it and perhaps even reconciling it.

I wonder what tool an individual could use to help with that last part ?

So Knights Foundation, you probably want to throw some funds towards the greater goal, rather than small niche uses cases in some of the other applicants like geospatial enhancements for urban planners and individual data contributors. :)

Good Luck Martin and team.  Looks like an easy win for OpenRefine just scrolling through the 100+ applications.

BTW...

The specific technology stack that would help with data repository and processing services...and providing the middleware between OpenRefine and data repos and services would be Apache Gora ... which I have mentioned before...primarily helps towards NoSQL storage but also can take advantage of plain ol SQL storage....and Gora is technology agnostic in both ways...something I think OpenRefine should absorb and use.

No one has had the time to wire it in.

Only Luca of OrientDB has done some work on this, but only towards OrientDB itself.

Thad

+ThadGuidry

--
You received this message because you are subscribed to the Google Groups "OpenRefine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Morris]

Since the goal is to fund RefinePro employees to develop things to benefit their company, wouldn't it make more sense to title the proposal RefinePro instead of OpenRefine? (Is funding commercial companies something the Knight Foundation does?)

Tom

p.s. I'm intentionally ignoring the commentary in the proposal on the relative levels of contribution to the 2.6 release.  It's trivial for anyone who cares to discover the truth.

[Joe Wicentowski]

Tom,

I disagree.  The proposal is clearly focused on enhancing the open source project, with all enhancements being made directly to it - by people with a proven track record of doing exactly that.  I don't see how the proposal narrowly benefits only the RefinePro employees (who are also obviously members of the OpenRefine community) who are being funded to do the work.  The proposal also indicates the funding will help get new developers involved.  Sounds like a win-win to me.

Joe

Sent from my iPhone

[Thad Guidry]

So I see Tom's point.  And his fear which I share.  Let's summarize.

RefinePro is going to try to get funds from Knights Foundation to help make their RefinePro product better, by promoting to make OpenRefine better.

THE FEAR :  RefinePro might not use all the funds directly to benefit OpenRefine.  And the OpenRefine members will have NO VISION into that process, but only a passing trust that Martin and his team will say what they do.  Another fear is that OpenRefine still does not have good documentation on the project management or a foundation letter explaining its organization structure that is agreed to by all contributors.  Which makes this fear compounded.

THE FEAR DISPELLED (a small amount) :  OpenRefine is the project that is being named, not RefinePro, and hence force, Martin damn well better know that any funds accumulated would actually go into OpenRefine's pockets and any funds or allocation would be managed by the 3 of us , Tom, Thad, & Martin https://www.bountysource.com/teams/openrefine/members  (along with my $265 bounty still on BountySource).

Tom, Martin ... thoughts and comments , or anything that I said wrong ?

I want more funds for OpenRefine, yes.  Tom needs to be paid 1st and foremost for his faithful and dedicated help with getting our act together many many times (Thanks Tom!)  He's just to shy to say so.  $1000 USD would be the minimum appropriate amount to start with and it doesn't even cover 1/10 of all his time he has given to this project (emails, code, Github conversion, docs, hundreds of hours of reviewing David and Stephano's code just to make sense of it to begin with)

Thad

+ThadGuidry

[Martin Magdinier]

Thank you all for the feedback. I'd like to clarify few points.

First some quotes from the news challenges FAQ:

• This competition is open to nonprofits, for-profits or individuals anywhere in the world.
• If you win, you’ll own your intellectual property. But under most funding mechanisms, you’ll be required to release what you make as open source (for software)
• Can two organizations/people submit a collaborative project? - Yes. We have funding mechanisms to make partnership projects possible - as said at the beginning of this thread, we're happy to join forces.
• Regarding the funding mechanism: Don’t worry too much about those specifics before applying. If you’re selected to move forward as a semifinalist, we’ll communicate with you about the details.

Regarding the proposal:
The core of the proposal we put through the challenge is to improve the extensibility of OpenRefine, an initial vision I shared here. We want to improve the documentation, hook and API system so other developers can extend OpenRefine core more easily. Our plan is to develop a couple of extension ourself and learn from our experience and feedback from other developers who have been through the process before or planning to develop an extension. Since RefinePro develop extensions will be funded by the grant, they will be released under an open source license.

Yes the funding will goes toward RefinePro employees. Our work will be for OpenRefine as we mention it. I am confident the Knight Foundation have enough experience in funding similar projects to enforce clear and transparent governance rules. I know I can also trust the community here to do friendly reminders if we forgot our engagement.

With freebase being closed now, I think it is time for OpenRefine to get a new direction. The tools proved itself as an excellent data preparation platform and pioneer a new industry. Some of the closed source solutions listed by Gartner here have very similar approach or are build from OpenRefine fork (see OpenRefine discussion here). We just want to help OpenRefine to be an open source alternative to those solution and keep the project where originated the idea growing.

About RefinePro and OpenRefine relationship:
We never try to discard Tom hard work in the 2.6 release. His experience as developer and knowledge of OpenRefine code base is invaluable to the community. However it is not trivial to see that things haven't been moving a lot the last two years. Being not myself a developer, RefinePro is my way to get resources together to help the project to sustain.

RefinePro effort over the summer have been solely focused on helping to ship the 2.6 release. RefinePro has always been working transparently (via OpenRefine mailing list, open chat and github pull request, issue and comment) regarding OpenRefine development. I don't understand why part of the community doesn't want to see business build around OpenRefine. Seeing companies investing resources to an open source project is a sign of good wealth of the project as long as things are done in a transparent fashion and we are working on a consensus base.

I've been working to clarify the governance model (or project management / a foundation letter) last year so we can have a document available for current and future contributors (see blog post here and here and post on the mailing list here ; here or here). Again I am happy to pick up the discussion and reach on an final version.

We consider RefinePro as OpenRefine community member, our goal is not to fork the project to pursue an hidden roadmap. We are here to help existing OpenRefine user to scale (via training, hosted instance and custom development) while working for the project to grow. We are committed to open source principles (transparency, inclusiveness, collaboration). Earlier in August, we offered Tom some kind of sponsoring to take care of OpenRefine core and help with the 2.6 release (see thread here) and we are still waiting for an answer.

We would like to be judged on our contributions and actions, not on assumptions on an hidden agenda that doesn't exist. 

Martin Magdinier
CEO RefinePro

[Thad Guidry]

I'll wait for Tom's response.

Thad

+ThadGuidry