Open data can also be linked data; when it is, it is linked open data. One of the most important forms of open data is open government data (OGD), which is a form of open data created by ruling government institutions. Open government data's importance is borne from it being a part of citizens' everyday lives, down to the most routine/mundane tasks that are seemingly far removed from government.
The abbreviation FAIR/O data is sometimes used to indicate that the dataset or database in question complies with the principles of FAIR data and also carries an explicit data‑capable open license.
The concept of open data is not new, but a formalized definition is relatively new. Conceptually, open data as a phenomenon denotes that governmental data should be available to anyone with a possibility of redistribution in any form without any copyright restriction. One more definition is the Open Definition which can be summarized in the statement that "A piece of data is open if anyone is free to use, reuse, and redistribute it – subject only, at most, to the requirement to attribute and/or share-alike." Other definitions, including the Open Data Institute's "Open data is data that anyone can access, use or share", have an accessible short version of the definition but refer to the formal definition.
Open data may include non-textual material such as maps, genomes, connectomes, chemical compounds, mathematical and scientific formulae, medical data, and practice, bioscience and biodiversity. Problems often arise because these are commercially valuable or can be aggregated into works of value. Access to, or re-use of, the data is controlled by organisations, both public and private. Control may be through access restrictions, licenses, copyright, patents and charges for access or re-use. Advocates of open data argue that these restrictions are against the common good and that these data should be made available without restriction or fee. In addition, it is important that the data are re-usable without requiring further permission, though the types of re-use (such as the creation of derivative works) may be controlled by a license.
A typical depiction of the need for open data:
Numerous scientists have pointed out the irony that right at the historical moment when we have the technologies to permit worldwide availability and distributed process of scientific data, broadening collaboration and accelerating the pace and depth of discovery... we are busy locking up that data and preventing the use of correspondingly advanced technologies on knowledge.
Creators of data often do not consider the need to state the conditions of ownership, licensing and re-use; instead presuming that not asserting copyright puts the data into the public domain. For example, many scientists do not regard the published data arising from their work to be theirs to control and consider the act of publication in a journal to be an implicit release of data into the commons. However, the lack of a license makes it difficult to determine the status of a data set and may restrict the use of data offered in an "Open" spirit. Because of this uncertainty it is also possible for public or private organizations to aggregate said data, claim that it is protected by copyright and then resell it.
The issue of indigenous knowledge (IK) poses a great challenge in terms of capturing, storage and distribution. Many societies in third-world countries lack the technicality processes of managing the IK.
At his presentation at the XML 2005 conference, Connolly displayed these two quotations regarding open data:
"I want my data back." (Jon Bosak circa 1997)
"I've long believed that customers of any application own the data they enter into it." (This quote refers to Veen's own heart-rate data.)
While the open-science-data movement long predates the Internet, the availability of fast, ubiquitous networking has significantly changed the context of Open science data, since publishing or obtaining data has become much less expensive and time-consuming.
The Human Genome Project was a major initiative that exemplified the power of open data. It was built upon the so-called Bermuda Principles, stipulating that: "All human genomic sequence information ... should be freely available and in the public domain in order to encourage research and development and to maximize its benefit to society'. More recent initiatives such as the Structural Genomics Consortium have illustrated that the open data approach can also be used productively within the context of industrial R&D.
In 2004, the Science Ministers of all nations of the Organisation for Economic Co-operation and Development (OECD), which includes most developed countries of the world, signed a declaration which essentially states that all publicly funded archive data should be made publicly available. Following a request and an intense discussion with data-producing institutions in member states, the OECD published in 2007 the OECD Principles and Guidelines for Access to Research Data from Public Funding as a soft-law recommendation.
Examples of open data in science:
The Dataverse Network Project – archival repository software promoting data sharing, persistent data citation, and reproducible research
data.uni-muenster.de – Open data about scientific artifacts from the University of Muenster, Germany. Launched in 2011.
linkedscience.org/data – Open scientific datasets encoded as Linked Data. Launched in 2011.
systemanaturae.org – Open scientific datasets related to wildlife classified by animal species. Launched in 2015.
There are a range of different arguments for government open data. For example, some advocates contend that making government information available to the public as machine readable open data can facilitate government transparency, accountability and public participation. "Open data can be a powerful force for public accountability—it can make existing information easier to analyze, process, and combine than ever before, allowing a new level of public scrutiny." Governments that enable public viewing of data can help citizens engage within the governmental sectors and "add value to that data."
Some make the case that opening up official information can support technological innovation and economic growth by enabling third parties to develop new kinds of digital applications and services.
Several national governments have created websites to distribute a portion of the data they collect. It is a concept for a collaborative project in the municipal Government to create and organize culture for Open Data or Open government data.
At the international level, the United Nations has an open data website that publishes statistical data from member states and UN agencies, and the World Bank published a range of statistical data relating to developing countries. The European Commission has created two portals for the European Union: the EU Open Data Portal which gives access to open data from the EU institutions, agencies and other bodies and the PublicData portal that provides datasets from local, regional and national public bodies across Europe.
The debate on open data is still evolving. The best open government applications seek to empower citizens, to help small businesses, or to create value in some other positive, constructive way. Opening government data is only a way-point on the road to improving education, improving government, and building tools to solve other real world problems. While many arguments have been made categorically, the following discussion of arguments for and against open data highlights that these arguments often depend highly on the type of data and its potential uses.
Arguments made on behalf of open data include the following:
In scientific research, the rate of discovery is accelerated by better access to data.
Making data open helps combat "data rot" and ensure that scientific research data are preserved over time.
Statistical literacy benefits from open data. Instructors can use locally relevant data sets to teach statistical concepts to their students.
It is generally held that factual data cannot be copyrighted. However, publishers frequently add copyright statements (often forbidding re-use) to scientific data accompanying publications. It may be unclear whether the factual data embedded in full text are part of the copyright.
While the human abstraction of facts from paper publications is normally accepted as legal there is often an implied restriction on the machine extraction by robots.
Unlike open access, where groups of publishers have stated their concerns, open data is normally challenged by individual institutions. Their arguments have been discussed less in public discourse and there are fewer quotes to rely on at this time.
Arguments against making all data available as open data include the following:
Government funding may not be used to duplicate or challenge the activities of the private sector (e.g. PubChem).
Governments have to be accountable for the efficient use of taxpayer's money: If public funds are used to aggregate the data and if the data will bring commercial (private) benefits to only a small number of users, the users should reimburse governments for the cost of providing the data.
Open data may lead to exploitation of, and rapid publication of results based on, data pertaining to developing countries by rich and well-equipped research institutes, without any further involvement and/or benefit to local communities (helicopter research); similarly to the historical open access to tropical forests that has led to the disappropriation ("Global Pillage") of plant genetic resources from developing countries.
The revenue earned by publishing data can be used to cover the costs of generating and/or disseminating the data, so that the dissemination can continue indefinitely.
The revenue earned by publishing data permits non-profit organisations to fund other activities (e.g. learned society publishing supports the society).
The government gives specific legitimacy for certain organisations to recover costs (NIST in US, Ordnance Survey in UK).
Privacy concerns may require that access to data is limited to specific users or to sub-sets of the data.
Collecting, 'cleaning', managing and disseminating data are typically labour- and/or cost-intensive processes – whoever provides these services should receive fair remuneration for providing those services.
Sponsors do not get full value unless their data is used appropriately – sometimes this requires quality management, dissemination and branding efforts that can best be achieved by charging fees to users.
Often, targeted end-users cannot use the data without additional processing (analysis, apps etc.) – if anyone has access to the data, none may have an incentive to invest in the processing required to make data useful (typical examples include biological, medical, and environmental data).
There is no control to the secondary use (aggregation) of open data.
Relation to other open activities
The goals of the Open Data movement are similar to those of other "Open" movements.
Open access is concerned with making scholarly publications freely available on the internet. In some cases, these articles include open datasets as well.
Open specifications are documents describing file types or protocols, where the documents are openly licensed. Usually these specifications are primarily meant to improve different software handling the same file types or protocols, but monopolists forced by law into open specifications might make it more difficult.
Open content is concerned with making resources aimed at a human audience (such as prose, photos, or videos) freely available.
Open knowledge. Open Knowledge International argues for openness in a range of issues including, but not limited to, those of open data. It covers (a) scientific, historical, geographic or otherwise (b) Content such as music, films, books (c) Government and other administrative information. Open data is included within the scope of the Open Knowledge Definition, which is alluded to in Science Commons' Protocol for Implementing Open Access Data.
Open notebook science refers to the application of the Open Data concept to as much of the scientific process as possible, including failed experiments and raw experimental data.
to deposit bioinformatics, atomic and molecular coordinate data, experimental data into the appropriate public database immediately upon publication of research results.
to retain original data sets for a minimum of five years after the grant. This applies to all data, whether published or not.
Other bodies active in promoting the deposition of data as well as full text include the Wellcome Trust. An academic paper published in 2013 advocated that Horizon 2020 (the science funding mechanism of the EU) should mandate that funded projects hand in their databases as "deliverables" at the end of the project, so that they can be checked for third party usability then shared.
Several mechanisms restrict access to or reuse of data (and several reasons for doing this are given above). They include:
making data available for a charge.
compilation in databases or websites to which only registered members or customers can have access.
use of a proprietary or closed technology or encryption which creates a barrier for access.
copyright statements claiming to forbid (or obfuscating) re-use of the data, including the use of "no derivatives" requirements.
patent forbidding re-use of the data (for example the 3-dimensional coordinates of some experimental protein structures have been patented).
restriction of robots to websites, with preference to certain search engines.
^Auer, S. R.; Bizer, C.; Kobilarov, G.; Lehmann, J.; Cyganiak, R.; Ives, Z. (2007). "DBpedia: A Nucleus for a Web of Open Data". The Semantic Web. Lecture Notes in Computer Science. 4825. p. 722. doi:10.1007/978-3-540-76298-0_52. ISBN 978-3-540-76297-3.
^Kitchin, Rob (2014). The Data Revolution. London: Sage. p. 49. ISBN 978-1-4462-8748-4.
^Kassen, Maxat (1 October 2013). "A promising phenomenon of open data: A case study of the Chicago open data project". Government Information Quarterly. 30 (4): 508–513. doi:10.1016/j.giq.2013.05.012. ISSN 0740-624X.
^See Open Definition home page and the full Open Definition
^Connolly, Dan (16 November 2005). "Semantic Web Data Integration with hCalendar and GRDDL". W3C Talks and Presentations. XML Conference & Exposition 2005, Atlanta, Georgia, USA: W3C. p. 2. Retrieved 2 May 2015.CS1 maint: location (link)
^Veen, Jeffrey (2 November 2005). "Polar Heart Rate Monitors: Gimme my data!". A website by Jeffrey Veen.
^Committee on Scientific Accomplishments of Earth Observations from Space, National Research Council (2008). Earth Observations from Space: The First 50 Years of Scientific Achievements. The National Academies Press. p. 6. doi:10.17226/11991. ISBN 978-0-309-11095-2. Retrieved 24 November 2010.
^World Data System (27 September 2017). "Data Sharing Principles". www.icsu-wds.org. ICSU-WDS (International Council for Science - World Data Service). Retrieved 27 September 2017.
^Vuong, Quan-Hoang (12 December 2017). "Open data, open review and open dialogue in making social sciences plausible". arXiv:1712.04801. Bibcode:2017arXiv171204801V. Retrieved 30 June 2018. Cite journal requires |journal= (help)
^Human Genome Project, 1996. Summary of Principles Agreed Upon at the First International
Strategy Meeting on Human Genome Sequencing (Bermuda, 25–28 February 1996)
^Perkmann, Markus; Schildt, Henri (2015). "Open Data Partnerships between Firms and Universities: The Role of Boundary Organizations". Research Policy. 44 (5): 1133–1143. doi:10.1016/j.respol.2014.12.006.
^OECD Declaration on Open Access to publicly funded data Archived 20 April 2010 at the Wayback Machine
^OECD Principles and Guidelines for Access to Research Data from Public Funding
^Gray, Jonathan (2014). "Towards a Genealogy of Open Data". Social Science Research Network (SSRN). doi:10.2139/ssrn.2605828. Cite journal requires |journal= (help)
^Brito, Jerry. "Hack, Mash, & Peer: Crowdsourcing Government Transparency". Colum. Sci. & Tech. L. Rev. 119 (2008).
^Yu, Harlan; Robinson, David G. (28 February 2012). "The New Ambiguity of 'Open Government'". Rochester, NY: Social Science Research Network. SSRN2012489. Cite journal requires |journal= (help)
^Robinson, David G.; Yu, Harlan; Zeller, William P.; Felten, Edward W. (1 January 2009). "Government Data and the Invisible Hand". Rochester, NY: Social Science Research Network. SSRN1138083. Cite journal requires |journal= (help)
^"data.ca.gov". data.ca.gov. Retrieved 7 May 2019.
^Data, City of New York, NYC Open. "NYC Open Data". NYC OpenData. Retrieved 7 May 2019.
^"Linee Guida per la Gestione Open Data - Città di Reggio Calabria".
^"Linee guida programmatiche della Città Metropolitana di Genova" (PDF).
^"The Open Data Charter: A Roadmap for Using a Global Resource". The Huffington Post. 27 October 2015. Retrieved 29 October 2015.
^Green, Arthur C. "OpenNWT announces launch of new election information website". My Yellowknife Now.
^Oyuela, Andrea; Walmsley, Thea; Walla, Katherine (30 December 2019). "120 Organizations Creating a New Decade for Food". Food Tank. Retrieved 21 January 2020.
^"dblp: How can I download the whole dblp dataset?". dblp.uni-trier.de. Dagstuhl. Retrieved 21 January 2020.
^Victor, Patricia; Cornelis, Chris; De Cock, Martine; Herrera-Viedma, Enrique (2010). "Bilattice-based aggregation operators for gradual trust and distrust". World Scientific Proceedings Series on Computer Engineering and Information Science. World Scientific: 505–510. doi:10.1142/9789814324700_0075. ISBN 978-981-4324-69-4.
^Dandekar, Pranav. "Analysis & Generative Model for Trust Networks" (PDF). Retrieved 21 January 2020. Cite journal requires |journal= (help)
^Overgoor, Jan; Wulczyn, Ellery; Potts, Christopher (20 May 2012). "Trust Propagation with Mixed-Effects Models". Sixth International AAAI Conference on Weblogs and Social Media.
^Lauterbach, Debra; Truong, Hung; Shah, Tanuj; Adamic, Lada (August 2009). "Surfing a Web of Trust: Reputation and Reciprocity on CouchSurfing.com". 2009 International Conference on Computational Science and Engineering. 4: 346–353. doi:10.1109/CSE.2009.345. ISBN 978-1-4244-5334-4.
^Rustam Tagiew; Dmitry I. Ignatov; Radhakrishnan Delhibabu (2015). Hospitality Exchange Services as a Source of Spatial and Social Data?. (IEEE) International Conference on Data Mining Workshop (ICDMW). Atlantic City. pp. 1125–1130. doi:10.1109/ICDMW.2015.239.
^Rustam Tagiew; Dmitry I. Ignatov; Radhakrishnan Delhibabu (2015). Economics of Internet-Based Hospitality Exchange. (IEEE/WIC/ACM) International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT). Singapore. pp. 493–498. arXiv:1501.06941. doi:10.1109/WI-IAT.2015.89.
^"On the road to open data, by Ian Manocha". Archived from the original on 29 March 2012. Retrieved 12 August 2011.
^"Big Data for Development: From Information- to Knowledge Societies", Martin Hilbert (2013), SSRN Scholarly Paper No. ID 2205145. Rochester, NY: Social Science Research Network; https://ssrn.com/abstract=2205145
^How to Make the Dream Come True[permanent dead link] argues in one research area (Astronomy) that access to open data increases the rate of scientific discovery.
^Khodiyar, Varsha (19 May 2014). "Stopping the rot: ensuring continued access to scientific data, irrespective of age". F1000 Research. F1000. Retrieved 11 March 2015.
^Magee AF, May MR, Moore BR (24 October 2014). "The dawn of open access to phylogenetic data". PLOS One. 9 (10): e110268. arXiv:1405.6623. Bibcode:2014PLoSO...9k0268M. doi:10.1371/journal.pone.0110268. PMC4208793. PMID25343725.
^Rivera, Roberto; Marazzi, Mario; Torres, Pedro (19 June 2019). "Incorporating Open Data Into Introductory Courses in Statistics". Taylor and Francis. Retrieved 7 May 2020.
^Rivera, Roberto. "Principles of Managerial Statistics and Data Science". Wiley. Retrieved 15 February 2020.
^Towards a Science Commons Archived 14 July 2014 at the Wayback Machine includes an overview of the basis of openness in science data.
^Low, A., 2001. The Third Revolution: Plant Genetic Resources in Developing Countries and China: Global Village or Global Pillage. Int'l. Trade & Bus. L. Ann. 323
^Sharif, Naubahar; Ritter, Waltraut; Davidson, Robert L; Edmunds, Scott C (31 December 2018). "An Open Science 'State of the Art' for Hong Kong: Making Open Research Data Available to Support Hong Kong Innovation Policy". Journal of Contemporary Eastern Asia. 17 (2): 200–221. doi:10.17477/JCEA.2018.17.2.200.
^"Protocol for Implementing Open Access Data". Archived from the original on 30 January 2017. Retrieved 17 April 2009.
^http://drexel-coas-elearning.blogspot.com/2006/09/open-notebook-science.html creation of term
^Kauppinen, T.; Espindola, G. M. D. (2011). "Linked Open Science-Communicating, Sharing and Evaluating Data, Methods and Results for Executable Papers". Procedia Computer Science. 4: 726–731. doi:10.1016/j.procs.2011.04.076.