Announcement: From Freebase to Wikidata

[Jason Douglas]

When we publicly launched Freebase back in 2007, we thought of it as a "Wikipedia for structured data." So it shouldn't be surprising that we've been closely watching the Wikimedia Foundation's project Wikidata since it launched about two years ago. We believe strongly in a robust community-driven effort to collect and curate structured knowledge about the world, but we now think we can serve that goal best by supporting Wikidata -- they’re growing fast, have an active community, and are better-suited to lead an open collaborative knowledge base.

So we've decided to help transfer the data in Freebase to Wikidata, and in mid-2015 we’ll wind down the Freebase service as a standalone project. Freebase has also supported developer access to the data, so before we retire it, we’ll launch a new API for entity search powered by Google's Knowledge Graph.

Loading Freebase into Wikidata as-is wouldn't meet the Wikidata community's guidelines for citation and sourcing of facts -- while a significant portion of the facts in Freebase came from Wikipedia itself, those facts were attributed to Wikipedia and not the actual original non-Wikipedia sources. So we’ll be launching a tool for Wikidata community members to match Freebase assertions to potential citations from either Google Search or our Knowledge Vault, so these individual facts can then be properly loaded to Wikidata. 

We believe this is the best first step we can take toward becoming a constructive participant in the Wikidata community, but we’ll look to continually evolve our role to support the goal of a comprehensive open database of common knowledge that anyone can use.

Here are the important dates to know:

• Before the end of March 2015
  
• We’ll launch a Wikidata import review tool
• We’ll announce a transition plan for the Freebase Search API & Suggest Widget to a Knowledge Graph-based solution
• March 31, 2015
  
• Freebase as a service will become read-only
• The website will no longer accept edits 
• We’ll turn down the MQL write API
• June 30, 2015
  
• We’ll retire the Freebase website and APIs
• The last Freebase data dump will remain available, but developers should check out the Wikidata dump

The Knowledge Graph team at Google

[Lydia Pintscher]

Hi everyone,

This is big news for Wikidata and Freebase. I just wanted to say hello
and introduce myself. I am the product manager for Wikidata and
available for any Wikidata related questions you might have. Wikidata
has a great community. I hope you will all join us and feel welcome
and right at home on Wikidata.


Cheers
Lydia

--
Lydia Pintscher - http://about.me/lydia.pintscher
Product Manager for Wikidata

Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de

Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg
unter der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das
Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

[Johannes Athmer]

Disclaimer: I don't really know anything of Wikidata but do have quite some history with Freebase up to now.

Does the Wikidata schema match the Freebase schema? Most importantly we have user created schemas, namespaces, keys and whatnot that would need to be migrated to Wikidata somehow?! Also, the scope of Wikidata looks to be much smaller than the one of Freebase (12 million topics vs. >40 million topics), so there is probably a lot to do on the Wikidata side to transfer all facts from Freebase to Wikidata? Can they even handle that amount of data and can it be migrated from a graph database to whatever Wikidata uses without creating conflicting data or losing data? I suppose these should be core questions that have been reviewed before making this decision.

Unfortunately, this does not really come as a surprise. The latest news on how to add data to Freebase have been long ago. Since the migration to Google hosting, important import tools like the spreadsheet loader etc. have been dead and promised replacements have never been set up. Moreover, performance of the Freebase.com website has been pretty bad regularly. Too bad that all the work people have done on third-party apps and API libraries (since the official Google APIs never got around to support mqlwrite in the first place) will now be void, but I guess we're going to have to do all of that again.

[Johannes Athmer]

Hallo Lydia,

please see my other mail for further questions. But since there seems to be official Wikimedia involvement, I'd like to pose another one:

Ever since I started contributing to Freebase (November 2007, to my shock!), I have been a proponent of the relatively loose notability guidelines on Freebase (especially when voting on delete votes, I do not vote to delete unless the topic is clearly incorrect or fake). Now, Wikipedia is notorious for their (increasingly) weird notability guidelines. How does Wikidata compare to Wikipedia and Freebase? I'd say it would be a bummer to lose millions of topics because of Wikidatas notability guidelines.

Also, is there any general introduction to Wikidata with a focus on a direct comparison with Freebase? Advantages and disadvantages of both platforms should be interesting.

Besten Dank vorab!

Johannes

[Lydia Pintscher]

Hi Johannes,

I don't have answers for your other questions right now.

On Tue, Dec 16, 2014 at 8:45 PM, Johannes Athmer <joha...@athmer.name> wrote:
> Hallo Lydia,
>
> please see my other mail for further questions. But since there seems to be
> official Wikimedia involvement, I'd like to pose another one:
> Ever since I started contributing to Freebase (November 2007, to my shock!),
> I have been a proponent of the relatively loose notability guidelines on
> Freebase (especially when voting on delete votes, I do not vote to delete
> unless the topic is clearly incorrect or fake). Now, Wikipedia is notorious
> for their (increasingly) weird notability guidelines. How does Wikidata
> compare to Wikipedia and Freebase? I'd say it would be a bummer to lose
> millions of topics because of Wikidatas notability guidelines.

In general notability on Wikidata is less strict. The guidelines are
here: https://www.wikidata.org/wiki/Wikidata:Notability That is a
living policy and it can evolve in the future of course.

> Also, is there any general introduction to Wikidata with a focus on a direct
> comparison with Freebase? Advantages and disadvantages of both platforms
> should be interesting.

I am not aware of something like that on the Wikidata side.

[Spencer Kelly]

Thanks for the announcement, 

I agree the sources thing is very awkward. A strategy like this won't realistically markup half a percent of our data, with the best intentions.
Lydia, I hope your team will consider solutions that place more trust in the accuracy of freebase, given its potential to reshape your project.

I don't know much about the promised API for wikidata, but we've been spoiled with the fancy freebase one. It will be sad loosing it.

Looking forward to great things

-spence

--
You received this message because you are subscribed to the Google Groups "Freebase Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to freebase-discu...@googlegroups.com.
To post to this group, send email to freebase...@googlegroups.com.
Visit this group at http://groups.google.com/group/freebase-discuss.
For more options, visit https://groups.google.com/d/optout.

[Jason Douglas]

On Tue Dec 16 2014 at 11:59:12 AM Spencer Kelly <spencer...@gmail.com> wrote:

Thanks for the announcement, 

I agree the sources thing is very awkward. A strategy like this won't realistically markup half a percent of our data, with the best intentions.
Lydia, I hope your team will consider solutions that place more trust in the accuracy of freebase, given its potential to reshape your project.

I don't know much about the promised API for wikidata, but we've been spoiled with the fancy freebase one. It will be sad loosing it.

Which promise?  

In terms of structured querying of wikidata, I think you can try it out (still in development) and track progress here:

http://wdq.wmflabs.org/

https://www.mediawiki.org/wiki/Wikibase/Indexing

-jason

[Thad Guidry]

It's been a fun ride....(at least I got a T-shirt and met a great group of folks who continue to churn out quality software we use in our daily lives)

=)

Welcome Wikidata !  I will be glad to help in the transition next year.

This might make reconciling data for projects like OpenRefine, etc, even easier...if we do things right on the Wikidata side.

Thad

+ThadGuidry

[Paul Houle]

This both a challenge and an opportunity for Wikidata.

Even though Wikidata has thought through the "data wiki" problem further than most,  it's main problem is a lack of instance data,  and the root cause of that is that Wikidata has not reached a "critical" mass of content and community participation.  There's no guarantee that adding Freebase data to Wikidata will cause things to tip,  but it is a good idea.

If you put Freebase in there you are going to expand WD by two orders of magnitude and then you are going to have a very different problem of scalability,  both in terms of WD and of those who would wish to use WD.  It would be a huge mistake to reject this data because WD is unhappy with the attribution.

From the viewpoint of Freebase users this means things will get worse before they get better;  the query interface,  data dumps,  and so forth at Wikidata are in a relatively undeveloped state,  and the tooling to get useful results from WD is not widely available.  On the other hand,  this may give people the incentive to develop the tooling.

I guess I do have some complaints about Wikidata that may be generic to the kind of organization it is and how it is funded.  There are regular uproars on the wikidata mailing list from people who complain that Wikidata is deaf to public input.  That was my first impression and I think the business model that it operates at the pleasure of a few big donors and thus doesn't need to listen to the "voice of the customer."

Freebase provided the MQL API,  Acre and all that so people had a platform to put data to use.  That way Freebase escaped the Cyc trap,  i,e, Cyc is a knowledge management system for building Cyc.)  Had Wikidata worked on a public API that you could do interesting things with from day one,  I think things could have been very different,  or if Wikidata had an "internal customer" building and application.

I guess the gauntlet is thrown down for those who want to run RDF-based "data wikis;"  I've heard a lot of talk,  but now would be the time for action for them because it would be very possible for something based on the RDF-dump to become the dominant fork. 

[Michał Łazowik]

Hi,

W dniu wtorek, 16 grudnia 2014 22:15:51 UTC+1 użytkownik Paul Houle napisał:

There are regular uproars on the wikidata mailing list from people who complain that Wikidata is deaf to public input.  That was my first impression and I think the business model that it operates at the pleasure of a few big donors and thus doesn't need to listen to the "voice of the customer."

You're probably talking about Wikimedia Foundation and the problems with MediaViewer/VisualEditor and the like. Wikidata dev team is based in Wikimedia Deutschland, a german chapter of WMF. I've been on the Wikidata mailing list from before the release of wikidata.org and I have never seen anyone complaining about the contacts with the dev team. I personally think that Lydia is doing a great job at communicating with the community, e.g. there is a planned redesign of the UI and there have been few iterations of proposed mockups. Also, patches welcome. And people do patch :)

Michał

[Denny Vrandečić]

Hi Spence,

I am happy to hear your input on the sources tool. Considering the current size of the community and the activity with tools like the Wikidata Game, I am quite confident that the the primary sources tool has, if done right, a very good chance of becoming a major contributor of data to Wikidata and a good compromise with regards to the requirements of the Wikidata community. And in order to get it right, we should now work together to get a good design and implementation done.

Or do you have specific data that shows that the suggested strategy is not warranted? Even a back-of-the-hand calculation of how you get to this impression would be helpful. Because that's what people told me about Wikidata itself, and it turned out rather nicely. :)

Cheers,

Denny

[Johannes Athmer]

Am Dienstag, 16. Dezember 2014 20:56:03 UTC+1 schrieb Lydia Pintscher:

In general notability on Wikidata is less strict. The guidelines are
here: https://www.wikidata.org/wiki/Wikidata:Notability That is a
living policy and it can evolve in the future of course.

Which seems much more strict than the idea that Freebase has on notability. :)

That probably makes sense for Wikidata, so I do not have a problem with that in general. But I do wonder what would best happen to those topics that do not meet the guidelines and should thus (correctly) not be added to Wikidata at all. It is very well possible that they are better off in specialized databases when Freebase is shut down. In some cases those specialized databases are actually a primary source for the current Freebase data (most prominently sites like MusicBrainz and Open Library). In other cases, it may not be that simple. It really depends on what the schema wizards at Metagoogle make up. (BTW: Thanks for making Freebase in the first place, and for actually trying to get the data published to Wikidata! I know that you are under no obligation to do so.)

-Johannes

[Spencer Kelly]

hi Danny, I'm a big fan. I'm also a philosophy-major turned semantic-web geek. I'm a total believer.

Data-tools like these played a big role in Freebase, which employed oDesk workers for years. Where I work, at state.com, we've done a huge amount of mechanical turking on freebase. A snappy javascript tool and a caffeinated user can get ~3,000 judgements in a good afternoon. Doing this sort of arbitration on each freebase fact would require 898,000 of these afternoons.

What I don't think is necessary is asking users to arbitrate the 5,000 enzymes, for example. To prove, (with more confidence than is offered from freebase) that Caspase 13(WD) is an enzyme... what would that even look like? Having someone click on the first google search for each of these topics seems like a hearty way to dispose the enthusiasm in the wikidata community.

My argument is that freebase is as good a primary source as any. It's certainly more confident than wikipedia as a data reference. These sort of problems shouldn't need to be solved twice.

There's got to be some subset of freebase facts are just too common-sense to need deliberating upon.

-spencer

[Denny Vrandečić]

Hi Spence,

this is a discussion that the Wikidata community has to have and decide upon. Since the source and the data will be put under an open license, they can also decide to directly upload the dataset, or parts of the dataset. Just based on my previous experience and chats with the community, it seems that the tool was the most promising idea to actually get the data integrated.

Cheers,

Denny

[irfan mir]

Will the json that the Knowledge Graph API returns be similar to the one's the current API's return or are we expected to rewrite how are apps process the returned json.

Will there be services like the reconcile, topic, and search APIs? 

[Jason Douglas]

The KG entity search API should be able to substitute for Freebase Suggest-like use cases, but not all current use cases. There will not be deep querying or data retrieval APIs like the current Topic API or MQL. 

We haven't finalized details like response format, but will by the end of March as was mentioned in the announcement.

-jason

--

[irfan mir]

What do you mean by Freebase suggest?

So these APIs will be deprecated and will no longer work on March 31st, correct?

Apologies, but I'm wondering what I can do from here on out.

So I'm going to ask, if you don't mind:

I use the search api and filtres to filtre out titles based on queries. Will I still be able to do this with the KG api?

I use the reconcile api to get MIDs from a textual name. Will I still be able to do this with the KG api?

I then retrieve facts on the topic and based on your previous response the KG api won't be able to do that, right? What I'm looking for is the extract the description of a mid— will the KG api do that?

If the answer is no to any of these, will there be a wikidata replacement? Or just the kg api? I'd greatly appreciate it if you or someone could answer these questions.

Thanks in Advance.

You received this message because you are subscribed to a topic in the Google Groups "Freebase Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/freebase-discuss/s_BPoL92edc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to freebase-discu...@googlegroups.com.

[Jason Douglas]

On Tue Dec 16 2014 at 5:21:21 PM irfan mir <the...@gmail.com> wrote:

What do you mean by Freebase suggest?

https://developers.google.com/freebase/v1/suggest

 

So these APIs will be deprecated and will no longer work on March 31st, correct?

No, Freebase becomes read-only on March 31st.  The APIs don't go away until June 30th.

 

Apologies, but I'm wondering what I can do from here on out.

So I'm going to ask, if you don't mind:

I use the search api and filtres to filtre out titles based on queries. Will I still be able to do this with the KG api?

I use the reconcile api to get MIDs from a textual name. Will I still be able to do this with the KG api?

I then retrieve facts on the topic and based on your previous response the KG api won't be able to do that, right? What I'm looking for is the extract the description of a mid— will the KG api do that?

More details on the new API will be coming later.

[irfan mir]

Okay, thank you.

But the api will be made available before the end of March, right?

Is there a place when I can get updates as more details on the KG api come out? A mailing-list, perhaps?

[Spencer Kelly]

yeah, thanks. 
if the import is done in a onesie-twosie manner, we should be honest about the fate of freebase.

If Wikidata turns its nose at the import, it has 7 years of hard work to reach parity. I'll hit the 'random page' button and buy you dinner when they're matching.

xo

[Michael Luggen]

Hi,

This is exiting news! As researcher, one of the ways to consume this kind of Data was RDF. Freebase offers a way to access the data as RDF.

In Wikidata I see it is also already possible to a certain point through links formed like this: https://www.wikidata.org/wiki/Special:EntityData/Q42.ttl. Although the response is unfortunately not complete at this point. Also I am not able to find more documentation on this topic (Except about this complete dump.)? Do you have any pointers about further development in this regards?

Regards,

Michael

[Lydia Pintscher]

On Wed, Dec 17, 2014 at 9:25 AM, Michael Luggen <atto...@exascale.info> wrote:
> Hi,
>
> This is exiting news! As researcher, one of the ways to consume this kind of
> Data was RDF. Freebase offers a way to access the data as RDF.
>
> In Wikidata I see it is also already possible to a certain point through
> links formed like this:
> https://www.wikidata.org/wiki/Special:EntityData/Q42.ttl. Although the
> response is unfortunately not complete at this point. Also I am not able to
> find more documentation on this topic (Except about this complete dump.)? Do
> you have any pointers about further development in this regards?

Hi Michael,

Currently the best dump to work with for Wikidata is the JSON dump at
http://dumps.wikimedia.org/other/wikidata/. You already found
Special:EntityData. That does however not include the statements yet.
for rdf The task to track this is
https://phabricator.wikimedia.org/T50143 in case you want to
subscribe. There is a 3rd party rdf dump at
https://tools.wmflabs.org/wikidata-exports/rdf/ Though I don't know if
this is up-to-date at the moment. If not I can poke the person
responsible for it.
We will need to spent some time on updating and polishing
https://www.wikidata.org/wiki/Wikidata:Database_download with this
information. (Help welcome.)

[Leeza Rodriguez]

hi Jason--Is there still time to submit a base or type schema for promotion to the commons?  Likewise, what is the future for all user domains? 

thank you,

Leeza

[Denny Vrandečić]

Michael, one important thing that Lydia mentioned at the very end of her Email is: Help is welcome!

As you state, the RDF export of Wikidata is unfortunately incomplete (although there is the full dump). But the code for creating this export is open and welcomes additions! You - and anyone else - can go ahead, fix that issue, and help Wikidata to fulfill its mission even better.

Your help would be truly appreciated.

--

[Peter Patel-Schneider]

The Wikidata notability guidelines are either draconian or nearly vacuous, but I can't figure out which. 

Consider information about movies.  One could argue that entire cast lists can be in Wikidata because they are about a Wikipedia object, and then that all objects that are mentioned in cast lists are notable, and then that all information about these objects are notable, and so on.    However, one might also argue that bit players are not notable by themselves and thus that information about them aside from their presence in cast lists is not notable.

I think that if the Freebase to Wikidata transfer is going to retain much of what makes Freebase more useful than Wikidata then the vacuous reading of notability is going to have to the one used.

Peter F. Patel-Schneider

On Tuesday, December 16, 2014 2:43:26 PM UTC-8, Johannes Athmer wrote:

Am Dienstag, 16. Dezember 2014 20:56:03 UTC+1 schrieb Lydia Pintscher:

In general notability on Wikidata is less strict. The guidelines are
here: https://www.wikidata.org/wiki/Wikidata:Notability That is a
living policy and it can evolve in the future of course.

Which seems much more strict than the idea that Freebase has on notability. :)

[...]

-Johannes

[Thad Guidry]

Notable Topics and Disambiguating Property values are 2 different things.

Freebase's disambiguating marker on properties can help Wikidata know which properties are must have's to work on aligning first.

Freebase's notable type marker is sometimes hit and miss, and I would think less important.

In my opinion, Notability is highly seasonal and irrelevant.

What is more important is having 2 or more disambiguating properties filled in on a given Freebase type, be it a Commons type or a Base type.  That should be the 1st clue to Wikidata that a Freebase topic should warrant importing and merging.

If I were doing the importing...I would start with those topics having the most links in the graph...and then work my way down to nil.

Thad

+ThadGuidry

[Leeza Rodriguez]

Thad--

WRT the 'most links in the graph', would that be the 'relevance score' ?

[Thad Guidry]

'relevance score' is based on a proprietary set of algorithmic ranking, that does factor into it the 'most links in the graph' count,

'most links in the graph' is just a simple total count calculation that anyone can do with the data dump and tools.

Thad

+ThadGuidry

--

[Dan Brickley]

On Wednesday, 17 December 2014 15:48:43 UTC, Peter Patel-Schneider wrote:

The Wikidata notability guidelines are either draconian or nearly vacuous, but I can't figure out which. 

Consider information about movies.  One could argue that entire cast lists can be in Wikidata because they are about a Wikipedia object, and then that all objects that are mentioned in cast lists are notable, and then that all information about these objects are notable, and so on.    However, one might also argue that bit players are not notable by themselves and thus that information about them aside from their presence in cast lists is not notable.

My reading of http://www.wikidata.org/wiki/Wikidata:Notability is that (amongst other things) being notable to any of the different language/region Wikipedias is enough to make something relevant and notable for Wikidata in general. This seems significantly more inclusive e.g. English Wikipedia users might claim non-notability to argue against having a page for a somewhat-locally-famous building in a non-English-speaking country. But if it was notable in its "home country" 's Wiki it would qualify for Wikidata without debate. Having wasted a chunk of my life arguing (in en.wikipedia.org) about whether DBpedia was notable, this feels like progress.

Dan

[Thad Guidry]

Notability is a defense protection that guards against .... what ?  Majority agreement....and good luck getting it.

In my opinion, Notability does not guard against SPAM, there are other policies that take care of SPAM and malicious users.

So, what does Notability protect a system from ?  From growing larger and more useful for the long tail domains....and space is cheap....you'll have a harder time finding volunteers to fill that cheap space anyways.  Just as Freebase has seen.

Forgot about Notability...everything is relevant in someone's context, country, or mind of the week. :)  Majority agreement does not work in the long tail domains... because you won't find a majority who even cares in the first place.

Thad

+ThadGuidry

[Peter Patel-Schneider]

Would that include help in trying to figure out a way to get unnotable or unsourced information into Wikidata or an extension of Wikidata?

If so, count me in.

peter

[Jason Douglas]

Could you be more specific? I'm not sure I understand the question as it seems like the future of user domains would be the same as any other domain in Freebase.

On Wed Dec 17 2014 at 5:45:08 AM Leeza Rodriguez <leezaro...@gmail.com> wrote:

--

[Leeza Rodriguez]

The specific example is my user domain which was created to help classify the /medicine domain.   My end goal was to submit this base or some of the data types for admission to the commons. Most of western medicine is connected to 37 ABMS Medical Specialties and 84 ABMS Subspecialties. I have modeled data types to incorporate these relationships, as well as other practical components of medical practice.  Therefore, is there still time for this submittal process to take place?  Thank you for answering.

[Brian Garnick]

Hi Lydia,

I'm excited to get started with Wikidata. I have spent some time poking around, and can't seem to figure out how to map relationships between a business and its branch offices as I do using Freebase: https://www.freebase.com/business/business_location

Can you point me in the right direction?

Many Thanks,

Brian

On Tuesday, December 16, 2014 2:26:03 PM UTC-5, Lydia Pintscher wrote:

Hi everyone,

This is big news for Wikidata and Freebase. I just wanted to say hello
and introduce myself. I am the product manager for Wikidata and
available for any Wikidata related questions you might have. Wikidata
has a great community. I hope you will all join us and feel welcome
and right at home on Wikidata.

[Lydia Pintscher]

Hi Brian,

On Thu, Dec 18, 2014 at 6:22 PM, Brian Garnick <bgar...@gmail.com> wrote:
> Hi Lydia,
> I'm excited to get started with Wikidata. I have spent some time poking
> around, and can't seem to figure out how to map relationships between a
> business and its branch offices as I do using Freebase:
> https://www.freebase.com/business/business_location
>
> Can you point me in the right direction?

I am not sure if the right property for that has already been decided
on and created by the Wikidata community. Best ask on the Project chat
at https://www.wikidata.org/wiki/Wikidata:Project_chat

[ben]

Wikidata seems like a completely unsuitable home for this data due to the notability requirement that an entry must have a Wikipedia article for it. Wikidata will only ever be able to host a fraction of the data that Freebase did and provide a fraction of the value that Freebase did. This is extremely disappointing.