Help with dates not uploading from OpenRefine to Wikidata

29 views
Skip to first unread message

Jeannette Ho

unread,
May 6, 2021, 10:23:46 PM5/6/21
to openr...@googlegroups.com
I am learning to use the tool in OpenRefine to reconcile data and upload it to Wikidata.

A month ago, I was able to create a schema and successfully upload data to automatically generate items in Wikidata.  This month, I tried it again and everything uploaded fine EXCEPT for any statements involving dates.

in the spreadsheet , the date values are recorded in YYYY format as single years (e.g., 2020).  I get an error message saying they are incorrectly formatted:

Invalid date formats.

Some dates are incorrectly formatted, such as 2016.0. See the allowed formats.


I think I got the same error message last month but it let me upload the dates that appear in new Wikidata items that were created anyway.  So I don't know why it would be affecting it now.  


I tried cleaning up the dates in the spreadsheet prior to putting it in OpenRefine (getting rid of decimal places) as well as cleaning whitespaces afterwards, but it didn't help.


I also tried reconciling them against the years existing on Wikidata.  But when I ran it again,l the dates still don't appear in the  Wikidata items. They are also skipped in the "Preview" tab prior to uploading to Wikidata.


Can someone help with this?  Again, the dates uploaded successfully last month, so I'm not sure what changed.  


Jeannette 

Antonin Delpeuch (lists)

unread,
May 7, 2021, 3:25:28 AM5/7/21
to openr...@googlegroups.com
Hi Jeannette,

Could it be that your years are input as numbers (displayed in green)
rather than as strings (displayed in black)? I suspect this could be the
cause. You could try converting your column to strings first (and
possibly removing the `.0`) that is appended at the end with
value.replace('.0', '')

Best,
Antonin

On 07/05/2021 04:21, Jeannette Ho wrote:
> I am learning to use the tool in OpenRefine to reconcile data and upload
> it to Wikidata.
>
> A month ago, I was able to create a schema and successfully upload data
> to automatically generate items in Wikidata.  This month, I tried it
> again and everything uploaded fine EXCEPT for any statements involving
> dates.
>
> in the spreadsheet , the date values are recorded in YYYY format as
> single years (e.g., 2020).  I get an error message saying they are
> incorrectly formatted:
>
>
> Invalid date formats.
>
> Some dates are incorrectly formatted, such as 2016.0. See the allowed
> formats
> <https://www.wikidata.org/wiki/Wikidata:Tools/OpenRefine/Editing/Schema_alignment#Dates>.
>
>
> I think I got the same error message last month but it let me upload the
> dates that appear in new Wikidata items that were created anyway.  So I
> don't know why it would be affecting it now.  
>
>
> I tried cleaning up the dates in the spreadsheet prior to putting it in
> OpenRefine (getting rid of decimal places) as well as cleaning
> whitespaces afterwards, but it didn't help.
>
>
> I also tried reconciling them against the years existing on Wikidata. 
> But when I ran it again,l the dates still don't appear in the  Wikidata
> items. They are also skipped in the "Preview" tab prior to uploading to
> Wikidata.
>
>
> Can someone help with this?  Again, the dates uploaded successfully last
> month, so I'm not sure what changed.  
>
>
> Jeannette 
>
> --
> You received this message because you are subscribed to the Google
> Groups "OpenRefine" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to openrefine+...@googlegroups.com
> <mailto:openrefine+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/openrefine/CAM7SnMKO970d%3DkcRP1Bu6N5zm0DJKmrKV0uxuP0ccLsroSBRVQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/openrefine/CAM7SnMKO970d%3DkcRP1Bu6N5zm0DJKmrKV0uxuP0ccLsroSBRVQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

Jevon, Graham

unread,
May 7, 2021, 4:25:39 AM5/7/21
to openr...@googlegroups.com
Hi Antonin

Is the appended .0 a bug or is there a logic to it? I regularly have to remove it and I keep meaning to find out why.

Thanks

Graham

-----Original Message-----
From: openr...@googlegroups.com <openr...@googlegroups.com> On Behalf Of Antonin Delpeuch (lists)
Sent: 07 May 2021 08:25
To: openr...@googlegroups.com
Subject: Re: [OpenRefine] Help with dates not uploading from OpenRefine to Wikidata

Hi Jeannette,

Could it be that your years are input as numbers (displayed in green) rather than as strings (displayed in black)? I suspect this could be the cause. You could try converting your column to strings first (and possibly removing the `.0`) that is appended at the end with value.replace('.0', '')

Best,
Antonin

On 07/05/2021 04:21, Jeannette Ho wrote:
> I am learning to use the tool in OpenRefine to reconcile data and
> upload it to Wikidata.
>
> A month ago, I was able to create a schema and successfully upload
> data to automatically generate items in Wikidata. This month, I tried
> it again and everything uploaded fine EXCEPT for any statements
> involving dates.
>
> in the spreadsheet , the date values are recorded in YYYY format as
> single years (e.g., 2020). I get an error message saying they are
> incorrectly formatted:
>
>
> Invalid date formats.
>
> Some dates are incorrectly formatted, such as 2016.0. See the allowed
> formats
> <https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.wikidata.org%2Fwiki%2FWikidata%3ATools%2FOpenRefine%2FEditing%2FSchema_alignment%23Dates&amp;data=04%7C01%7C%7C44031a9c1cbf4c31e72a08d911294b7a%7C21a44cb7f9c34f009afabd1e8e88bcd9%7C0%7C0%7C637559691319202364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=CQ0Hi629Gmzy2ACWn6fzxgchFQSG1lAgbYyD3FLblqs%3D&amp;reserved=0>.
>
>
> I think I got the same error message last month but it let me upload
> the dates that appear in new Wikidata items that were created anyway.
> So I don't know why it would be affecting it now.
>
>
> I tried cleaning up the dates in the spreadsheet prior to putting it
> in OpenRefine (getting rid of decimal places) as well as cleaning
> whitespaces afterwards, but it didn't help.
>
>
> I also tried reconciling them against the years existing on Wikidata.
> But when I ran it again,l the dates still don't appear in the
> Wikidata items. They are also skipped in the "Preview" tab prior to
> uploading to Wikidata.
>
>
> Can someone help with this? Again, the dates uploaded successfully
> last month, so I'm not sure what changed.
>
>
> Jeannette
>
> --
> You received this message because you are subscribed to the Google
> Groups "OpenRefine" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to openrefine+...@googlegroups.com
> <mailto:openrefine+...@googlegroups.com>.
> To view this discussion on the web visit
> https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgrou
> ps.google.com%2Fd%2Fmsgid%2Fopenrefine%2FCAM7SnMKO970d%253DkcRP1Bu6N5z
> m0DJKmrKV0uxuP0ccLsroSBRVQ%2540mail.gmail.com&amp;data=04%7C01%7C%7C44
> 031a9c1cbf4c31e72a08d911294b7a%7C21a44cb7f9c34f009afabd1e8e88bcd9%7C0%
> 7C0%7C637559691319202364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=fe8b9e2
> fM%2FoQbtxygwfaTwi7f3LpRvZlGNeN52%2Bu%2Brs%3D&amp;reserved=0
> <https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fopenrefine%2FCAM7SnMKO970d%253DkcRP1Bu6N5zm0DJKmrKV0uxuP0ccLsroSBRVQ%2540mail.gmail.com%3Futm_medium%3Demail%26utm_source%3Dfooter&amp;data=04%7C01%7C%7C44031a9c1cbf4c31e72a08d911294b7a%7C21a44cb7f9c34f009afabd1e8e88bcd9%7C0%7C0%7C637559691319202364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=4hTP5Q955zXKxgJJ0JBR2%2FF6rMFgChRf8Ri4ZbDbFpc%3D&amp;reserved=0>.

--
You received this message because you are subscribed to the Google Groups "OpenRefine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine+...@googlegroups.com.
To view this discussion on the web visit https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fopenrefine%2Ff523ab72-f1f0-00e2-fc2b-eb27b9122acc%2540antonin.delpeuch.eu&amp;data=04%7C01%7C%7C44031a9c1cbf4c31e72a08d911294b7a%7C21a44cb7f9c34f009afabd1e8e88bcd9%7C0%7C0%7C637559691319212319%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=v4trfF7nSwz3W2%2B93RMPaVJ1OcFtR8DGXeZiiiJ2C%2Fs%3D&amp;reserved=0.


******************************************************************************************************************
Experience the British Library online at www.bl.uk<http://www.bl.uk/>
The British Library’s latest Annual Report and Accounts : www.bl.uk/aboutus/annrep/index.html<http://www.bl.uk/aboutus/annrep/index.html>
Help the British Library conserve the world's knowledge. Adopt a Book. www.bl.uk/adoptabook<http://www.bl.uk/adoptabook>
The Library's St Pancras site is WiFi - enabled
*****************************************************************************************************************
The information contained in this e-mail is confidential and may be legally privileged. It is intended for the addressee(s) only. If you are not the intended recipient, please delete this e-mail and notify the postm...@bl.uk<mailto:postm...@bl.uk> : The contents of this e-mail must not be disclosed or copied without the sender's consent.
The statements and opinions expressed in this message are those of the author and do not necessarily reflect those of the British Library. The British Library does not take any responsibility for the views of the author.
*****************************************************************************************************************
Think before you print

Baer,Helen

unread,
May 7, 2021, 5:47:51 PM5/7/21
to openr...@googlegroups.com
Hi all,

I had this happen to me for the first time this week. We're in the process of a platform migration, and I've literallly seen dozens and dozens of spreadsheets in the past 2 months. This one had the .0 on the dates; it came from the same data source (DSpace) as all the others.

Best,

Helen
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.wikidata.org%2Fwiki%2FWikidata%3ATools%2FOpenRefine%2FEditing%2FSchema_alignment%23Dates&amp;data=04%7C01%7Chelen.baer%40colostate.edu%7C0f5b198cfd7d4389324f08d91131b325%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637559727448289577%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=gysl%2Bhn3Q0uDxogq%2B0SHIaZT0tiha7L57LH%2FcKH%2BAdA%3D&amp;reserved=0>.
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fopenrefine%2FCAM7SnMKO970d%253DkcRP1Bu6N5zm0DJKmrKV0uxuP0ccLsroSBRVQ%2540mail.gmail.com%3Futm_medium%3Demail%26utm_source%3Dfooter&amp;data=04%7C01%7Chelen.baer%40colostate.edu%7C0f5b198cfd7d4389324f08d91131b325%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637559727448289577%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=cXdMd1627pP5O%2BtHDchlf2XAkrtnHDIXGCrnkVmNQjs%3D&amp;reserved=0>.

--
You received this message because you are subscribed to the Google Groups "OpenRefine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine+...@googlegroups.com.
To view this discussion on the web visit https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fopenrefine%2Ff523ab72-f1f0-00e2-fc2b-eb27b9122acc%2540antonin.delpeuch.eu&amp;data=04%7C01%7Chelen.baer%40colostate.edu%7C0f5b198cfd7d4389324f08d91131b325%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637559727448289577%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=B44uI7IDvL4l9b63ZaPV2TF%2B8SHlh1qE%2FLNa8SyU5Tg%3D&amp;reserved=0.


******************************************************************************************************************
Experience the British Library online at https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.bl.uk%2F&amp;data=04%7C01%7Chelen.baer%40colostate.edu%7C0f5b198cfd7d4389324f08d91131b325%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637559727448289577%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=ATgK79FqaKZaICaxmGA9DGLErhk8NgOMyfCaTz8mjes%3D&amp;reserved=0<https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.bl.uk%2F&amp;data=04%7C01%7Chelen.baer%40colostate.edu%7C0f5b198cfd7d4389324f08d91131b325%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637559727448289577%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=ATgK79FqaKZaICaxmGA9DGLErhk8NgOMyfCaTz8mjes%3D&amp;reserved=0>
The British Library's latest Annual Report and Accounts : https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.bl.uk%2Faboutus%2Fannrep%2Findex.html&amp;data=04%7C01%7Chelen.baer%40colostate.edu%7C0f5b198cfd7d4389324f08d91131b325%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637559727448289577%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=YfA1LNzQ8C6ViluNiN98ZWcD71ZOCJ%2BQ0fTAkvrisLw%3D&amp;reserved=0<https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.bl.uk%2Faboutus%2Fannrep%2Findex.html&amp;data=04%7C01%7Chelen.baer%40colostate.edu%7C0f5b198cfd7d4389324f08d91131b325%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637559727448299581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=ZmfSOg%2F0%2Fh8RvuAMezQ8BkUbg%2FPE62DNProig%2Bg%2FVhQ%3D&amp;reserved=0>
Help the British Library conserve the world's knowledge. Adopt a Book. https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.bl.uk%2Fadoptabook&amp;data=04%7C01%7Chelen.baer%40colostate.edu%7C0f5b198cfd7d4389324f08d91131b325%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637559727448299581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=MJBh5lWIwNmYyaigHfqw0wpSrkcPdAMmPHHVCKzKppY%3D&amp;reserved=0<https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.bl.uk%2Fadoptabook&amp;data=04%7C01%7Chelen.baer%40colostate.edu%7C0f5b198cfd7d4389324f08d91131b325%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637559727448299581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=MJBh5lWIwNmYyaigHfqw0wpSrkcPdAMmPHHVCKzKppY%3D&amp;reserved=0>
The Library's St Pancras site is WiFi - enabled
*****************************************************************************************************************
The information contained in this e-mail is confidential and may be legally privileged. It is intended for the addressee(s) only. If you are not the intended recipient, please delete this e-mail and notify the postm...@bl.uk<mailto:postm...@bl.uk> : The contents of this e-mail must not be disclosed or copied without the sender's consent.
The statements and opinions expressed in this message are those of the author and do not necessarily reflect those of the British Library. The British Library does not take any responsibility for the views of the author.
*****************************************************************************************************************
Think before you print

--
You received this message because you are subscribed to the Google Groups "OpenRefine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine+...@googlegroups.com.
To view this discussion on the web visit https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fopenrefine%2FCWXP265MB2951BD94DCF32B26DF7BD26682579%2540CWXP265MB2951.GBRP265.PROD.OUTLOOK.COM&amp;data=04%7C01%7Chelen.baer%40colostate.edu%7C0f5b198cfd7d4389324f08d91131b325%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637559727448299581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=UOOyyoQrufaNOJrVTad2XORMaTxED0n3aYV0iRckvHw%3D&amp;reserved=0.

jahot...@gmail.com

unread,
May 7, 2021, 6:04:41 PM5/7/21
to OpenRefine
I tried what Antonin suggested and he was right--I had to change the formatting of the dates to text. Once I did this in OpenRefine the extra decimal and zero appeared (such as "2019.0") so I went to each individual cell and got rid of them.

Then I ran it through the Wikidata schema I had created again and the dates got uploaded.

Antonin, thanks for letting me know how to resolve this problem.  It seems to be working now.

Jeannette
Reply all
Reply to author
Forward
0 new messages