Publishing large DDI-Codebook metadata via OAI-PMH
7 views
Skip to first unread message
Knut Wenzig
Jun 7, 2024, 1:27:12 PMJun 7
to OAI-PMH
Hello everyone,
In a small project, we are investigating the possibilities of publishing DDI-Codebook Metadata (at the variable level) via OAI-PMH. In a prior project, we accessed more than 150,000 records of this kind, harvested and analyzed them, but we found that only information at the study level was available (see our presentation https://doi.org/10.5281/zenodo.10107875).
Often, studies in the social sciences involve data from surveys, which are stored in data tables. In these tables, objects (such as persons, households, etc.) are organized in rows, and the characteristics of these objects are organized in columns. These columns are often called variables, and they are where the responses are stored.
The metadata standard DDI Codebook (https://ddialliance.org/Specification/DDI-Codebook/2.5/) allows for the description of these variables in datasets within the studies using XML files. One of our use cases is a study (the German Socio-Economic Panel Study, SOEP) with more than 100,000 variables, and we expect the complete XML file to be at least 400 MB.
What do you think? We are unsure if it would be a better approach to split the record at the level of datasets or variables. I would be very happy to hear any pros and cons of these approaches.
A similar question was sent to the DDI community. I will compile an overview of all the feedback I received and inform this list of the results.
Thank you and best regards,
Knut Wenzig
(he/him/his #happytobeinformal)
--
German Institute for Economic Research (DIW Berlin)
SOEP
Mohrenstr. 58 | 10117 Berlin | Germany
https://www.diw.de/cv/en/kwenzig
Board:
Prof. Marcel Fratzscher, Ph.D. (President)
Dr. Denise Rüttinger (Managing Director)
Prof. Dr. Alexander Kritikos (Board Member)
Chairwoman of the Board of Trustees: Dr. Sigrid Nikutta
Registered Office: Berlin
Association Registry: Berlin-Charlottenburg
Association Registry Number: 95 VR 136 NZ
You can also find DIW Berlin on Facebook (facebook.com/diw.de) and Twitter (twitter.com/DIW_Berlin).
In a small project, we are investigating the possibilities of publishing DDI-Codebook Metadata (at the variable level) via OAI-PMH. In a prior project, we accessed more than 150,000 records of this kind, harvested and analyzed them, but we found that only information at the study level was available (see our presentation https://doi.org/10.5281/zenodo.10107875).
Often, studies in the social sciences involve data from surveys, which are stored in data tables. In these tables, objects (such as persons, households, etc.) are organized in rows, and the characteristics of these objects are organized in columns. These columns are often called variables, and they are where the responses are stored.
The metadata standard DDI Codebook (https://ddialliance.org/Specification/DDI-Codebook/2.5/) allows for the description of these variables in datasets within the studies using XML files. One of our use cases is a study (the German Socio-Economic Panel Study, SOEP) with more than 100,000 variables, and we expect the complete XML file to be at least 400 MB.
What do you think? We are unsure if it would be a better approach to split the record at the level of datasets or variables. I would be very happy to hear any pros and cons of these approaches.
A similar question was sent to the DDI community. I will compile an overview of all the feedback I received and inform this list of the results.
Thank you and best regards,
Knut Wenzig
(he/him/his #happytobeinformal)
--
German Institute for Economic Research (DIW Berlin)
SOEP
Mohrenstr. 58 | 10117 Berlin | Germany
https://www.diw.de/cv/en/kwenzig
Board:
Prof. Marcel Fratzscher, Ph.D. (President)
Dr. Denise Rüttinger (Managing Director)
Prof. Dr. Alexander Kritikos (Board Member)
Chairwoman of the Board of Trustees: Dr. Sigrid Nikutta
Registered Office: Berlin
Association Registry: Berlin-Charlottenburg
Association Registry Number: 95 VR 136 NZ
You can also find DIW Berlin on Facebook (facebook.com/diw.de) and Twitter (twitter.com/DIW_Berlin).
Reply all
Reply to author
Forward
0 new messages