Metadata Quality Assessment & OLAC

Metadata Quality is a fascinating topic that I’m interested in more professional engagement around. I am specifically interested in user-contributed metadata platforms and quality control in those digital library contexts. I have long noticed the need for spell checking and for the ability to move folksonomy terms to controlled vocabulary terms at certain archives stewarding linguistic and language resources. 

I read the following three articles for UNT work:

• Hider, P., & Tan, K.-C. (2008). Constructing Record Quality Measures Based on Catalog Use. Cataloging & Classification Quarterly, 46(4), 338–361. https://doi.org/10.1080/01639370802322515
• Kumar, V., Maurya, S., & Ji, G. (2022). An Assessment of Spelling and Typographical Errors in WebOPAC of Academic Libraries. Journal of Library Metadata, 22(1–2), 63–73. https://doi.org/10.1080/19386389.2022.2051978
• Petrucciani, A. (2015). Quality of Library Catalogs and Value of (Good) Catalogs. Cataloging & Classification Quarterly, 53(3–4), 303–313. https://doi.org/10.1080/01639374.2014.1003669 

Across these articles, I really appreciate the internationally diverse perspectives brought by these authors. Kumar et al. (2022) has some really clear methodologies (I feel like I could replicate the experiment in my location) and looks at spelling mistakes in a catalog where the language of instruction (at the institution) may be English, but the first language of many (maybe most) of the people there is not English. It seems natural that there would be spelling mistakes in these environments. In the archive that I worked at (in Dallas, Tx. no less) I offered to do a spell-check analysis on titles and folksonomy terms. The staff were less than ecstatic to receive any feedback on the records as that would mean that they would need to do more work to correct them. (In that situation the archive’s catalog has central staff supporting some collections and core infrastructure, and remote staff around the world supporting location specific collections.) Depending on which collection the misspelling was found in would determine which remote staff would need to approve the changes. So I appreciate the management at the instution where Kumar and collegues did their work to have a willingness to allow this kind of research. I hope that they then engaged with the results. In the case of the archive at which I worked, they also have a remote accession tool for digital objects. I’ve suggested they add a spell check system to that tool to catch errors in the accession process, but it has been argued that those features are not mission critical. This leads back to an analysis of what are the financial priorities of the management. If an archive has no specific materials re-use objective then this could have a direct impact on the quality of the records in interesting ways. For example, an archive without a specific requirement to engage the public in innovative ways (or at all) or where this process of engagement is disjunct from cataloging, curation, and preservation activities may not seek to maintain their records in a way where potential targest would benefit.

Since I am interested in linguistics and language resource records and their quality I found the description of the types of errors in Italian libraries interesting. Petrucciani (2015), an article which was derived from a speech given at a conference, says the following of the Italian situation (p. 304 & 309):

In Italy, for example, the main factors may be the widespread outsourcing of cataloging activities (especially retrospective conversion and cataloging of backlogs, bequests, etc.) with very low remuneration and lack of quality control, and the participation in library networks (mainly the national network SBN, Servizio Bibliotecario Nazionale) of small libraries—public, religious, or special—run by unqualified people or volunteers, without skilled staff. In many countries, an apparent factor is the lack of competence in foreign languages among cataloging staff, resulting in wrong language coding (e.g., Spanish for Italian or vice versa, Russian for books in any language written in Cyrillic script) and the abundance of misprinted words.

These (and many more) examples are not errors. Or better, they are not simply errors. They are the result of cataloging rules and policies, of the design of OPAC interfaces, or of bibliographic formats. The low quality of large catalogs is not simply due to the vast amount of duplicated records, unprofessional work, errors of interpretation, miskeyings, and so on, but in the first place to the misunderstanding (and undervaluing) of the functions and aims of bibliographic information and to unwise cataloging standards, rules, and policies.

Again he attributes the errors to management practices established at institutions. In other parts of his paper Petrucciani points out that tools like Google Search and full text search options available exploit the problems that less-than-perfect catalogs have. That is, end-users for a variety of reasons turn to Google Search instead of catalog searches. I have long found interesting that Google doesn’t return results for books found in library catalogs. Its like library catalogs are part of the dark web.

I found the approach by Hider and Tan (2008) inspiring. I often look at records found in the Open Language Archives Community (OLAC). One of the open questions there has been how to assess data providers and the quality of the data provided. OLAC records are often sourced from digital repositories where accessions are described by the creators/depositors. This means that variation in management practices at the data provider to OLAC impact the quality of record which OLAC has to aggregate. The current method (Hughes, 2005)  is to count the DC elements on a record, average the count on each record and assign that average to the archive to rank the data providers. In this framework quality is only measured on the basis of the syntax present in the record rather than any sort of syntax-semantics relationship. e.g., Does a date element really contain valid date data? This method does not attempt to measure the validity or conformity of the data values to Dublin Core or OLAC recommendations. 

With regards to OLAC metadata quality, the current status quo follows the methods laid out by B. Hughes which is to count the DC elements on a record, average the count on each record and assign that average to the data provider and then to rank the data providers. In this framework Quality is only measured on the basis of the syntax (number of elements) present in the record rather than any sort of syntax-semantics relationship. E.g., does a date really appear in a date field and are they really valid dates? This method does not attempt to measure the validity or conformity of the data values to Dublin Core or OALC recommendations in terms of the semantics or the well-formedness of the values. In a sense the OLAC method of quality is similar to what MARC record generators call Encoding Level.

In contrast to B. Hughes’ approach I have suggested that the DCMIType vocabulary in conjunction with the dc:format , dc:relation , dc:medium & tableOfContents elements could be used to provide continuity checks to the type of artifact being described. The purpose here would be to check for the completeness of a description if there is evidence in the record that specific type is present. This would would be a step towards enforcing the Dublin Core 1-to-1 principle. In a way the suggested process uses a fingerprinting process for each DCMIType. Common linguistic artifact types could be crafted (fingerprinted) and also added to the checks. These crafted fingerprints would appear in the documentation where recommendations existed for describing materials.

A third method to look at quality is to look at the records that people are visiting and clicking through. People found these records and they must be interesting for some reason. An investigation of the most successful records in providing click-throughs to target websites is one way to assess the quality of a record. That is, is the record valuable to an audience? This third method of establishing quality is discussed in Hider & Tan (2008). They look at MARC records, but we have Dublin Core records, so some generalities and abstractions to their arguments may be needed before application.

I find the idea of measuring record quality based on its use interesting. With nearly 450K records in OLAC but only some 8,000 views per month (Bird & Simons, 2021) a natural question arises. Which records are being viewed and are these the same ones month-over-month? Is there something “higher quality” about these records that they are being found and used? Recent work by Husic (2014) and Osborn (2019) show that adding Library of Congress Subject Headings (LCSH) to records does make them more visible to search engines. So maybe the highly used OLAC records have LCSH terms attached? This is something worth looking into and reporting on.

References

Bird, Steven, and Gary Simons. 2021. “Towards an Agenda for Open Language Archiving.” In Proceedings of the International Workshop on Digital Language Archives: LangArc 2021, edited by Oksana Zavalina and Shobhana Lakshmi Chelliah, 25–28. Denton, Texas: University of North Texas. doi:10.12794/langarc1851171.

Hider, Philip, and Kah-Ching Tan. 2008. “Constructing Record Quality Measures Based on Catalog Use.” Cataloging & Classification Quarterly 46 (4). Routledge: 338–61. doi:10.1080/01639370802322515.

Hughes, Baden. 2005. “Metadata Quality Evaluation: Experience from the Open Language Archives Community.” In Digital Libraries: International Collaboration and Cross-Fertilization, edited by Zhaoneng Chen, Hsinchun Chen, Qihao Miao, Yuxi Fu, Edward Fox, and Ee-peng Lim, 320–29. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer. doi:10.1007/978-3-540-30544-6_34.

Husic, Geoff. 2014. “Enhancing an Open-Access Linguistics Journal Archive with Library of Congress-like Metadata: A Case Study of the Effectiveness for Improving Discovery.” Kansas Working Papers in Linguistics 35 (December): 1–8. doi:10.17161/KWPL.1808.15945.

Kumar, Vinit, Sharad Maurya, and Gopal Ji. 2022. “An Assessment of Spelling and Typographical Errors in WebOPAC of Academic Libraries.” Journal of Library Metadata 22 (1–2). Routledge: 63–73. doi:10.1080/19386389.2022.2051978.

Osborn, Carlyn. 2019. “How Do Levels of Description Affect Discoverability of the Web Archives at the Library of Congress?” Poster presented at the SAA Research Forum, Austin, Texas. https://www2.archivists.org/sites/all/files/flattened_Osborn_SAA_Research_Forum_2019.pdf.

Petrucciani, Alberto. 2015. “Quality of Library Catalogs and Value of (Good) Catalogs.” Cataloging & Classification Quarterly 53 (3–4). Routledge: 303–13. doi:10.1080/01639374.2014.1003669.

User-Generated Metadata

The following is a quick extract of academic papers from my files on the topics of user-generated metadata and metadata quality.