As Dieter recently announced, the Language Archive Cologne (LAC) is the first service provider connected to the new CLARIN single sign-on (SSO) proxy. This post describes the rebuilt repository and the two ways it connects to CLARIN: metadata harvesting and federated login.
The LAC is a research data repository for linguistics and for the humanities disciplines that work with audiovisual material. Three institutions at the University of Cologne run it: the Department of Linguistics, the Data Center for the Humanities, and the IT Center, which develops the software and operates the infrastructure.
From two applications to one
For years LAC ran as two separate applications: KA3, a backend that managed the data and exposed it through a REST API, and a frontend that consumed the API to provide the user interface. This made even small changes expensive. Most changes had to be made twice, once in the backend and once in the frontend, and the two applications had to be released in step. So we merged them into one application, where the pages are rendered on the server by the same code that defines the data models and enforces the access rules.
We built the new application with Django, a Python web framework. The same data models now serve three interfaces: the website, a REST API for programmatic access, and an OAI-PMH endpoint for metadata harvesting. Everything is defined once, in the data model, so the three interfaces cannot contradict one another.
Structured metadata lives in PostgreSQL. The deposited files, mostly audio and video recordings and often large, live in S3-compatible object storage. Object storage has two virtues for an archive. Unlike a filesystem, it has no volume to size in advance and extend when it fills: the store grows behind its interface. And that interface is a standard, so the repository is not tied to one storage system or vendor.
A good part of the material is available only to logged-in academic users, so the application checks access before it plays or hands out a file. Access rules work per object and are inherited down the deposit hierarchy. A depositor can open a whole collection to the public and still restrict a single bundle inside it: a bundle follows its collection’s policy unless a rule of its own overrides it. So a project can share most of a corpus openly while protecting the few bundles that contain sensitive material.
Long-term storage and versioning
The archival objects follow the Oxford Common File Layout (OCFL), an open specification that keeps stored digital objects readable independently of the software on top of them. The specification is meant to protect the holdings for decades, but it already paid for itself during the rewrite. Switching to the new system meant moving the holdings from a filesystem into object storage. OCFL prescribes the same layout for both, so the move preserved the structure of every object exactly, and the new application reads the migrated objects without knowing anything about its predecessor. We replaced the storage and the application at the same time, and the data in the archive came through unchanged.
The OCFL specification defines each object as an explicit sequence of versions. Versions are necessary because an archived object is not static: files change over the years, and an archive must keep the history of those changes. Versioning the archived material is what we are working on now. It is the most important part of the current phase of the project, funded by the DFG (KA3OS), and the same work will also bring the archive to full conformance with OCFL 1.1.
Metadata and findability
Our metadata model, BLAM (Basic Language Archive Metadata), spells out the structure of a deposit. It has three levels. A collection is a corpus or a project. A collection contains bundles, and a bundle is typically one recording together with its transcription, annotations, and accompanying documentation. Within a bundle, BLAM distinguishes three kinds of resources: media resources for recordings, written resources for transcriptions and annotation files, and other resources for other type of documents.
Every level carries a persistent identifier, so a citation can target a whole corpus, a single bundle, or one individual file. The identifiers, issued by the Data Center for the Humanities, are Handles, the system on which digital object identifiers (DOIs) are built.
LAC has been a CLARIN center for years, and its records have been findable in the Virtual Language Observatory (VLO) under the data provider name Language Archive Cologne. The new system did not change that: the VLO still harvests the complete holdings, every collection and every bundle. A standard OAI-PMH endpoint serves the records in several metadata formats, including Dublin Core, OLAC, and BLAM itself, defined as a profile within CLARIN’s CMDI framework.
The focus of the holdings is the documentation of endangered languages and endangered oral traditions. Two examples from the VLO: recitations of the Rigveda in Vedic Sanskrit, one of the oldest oral traditions still in active transmission, and the largest documentation corpus assembled for Totoli, an endangered Austronesian language of Sulawesi, Indonesia. A researcher who finds either can follow the result straight to the recordings, transcriptions, and annotations in LAC.
Federated login through the CLARIN SSO proxy
Most of the bundles in LAC are public. But four in ten are not, and for them the archive must know who is asking, so authentication is a central part of the service. LAC is the first repository connected to the proxy, which the CLARIN authentication and authorization infrastructure (AAI) team introduced earlier this year.
The traditional federated model demands a great deal from a service provider. It has to join a national identity federation. It has to collect and trust the metadata of every identity provider whose users it wants to admit. And it has to offer institution discovery itself, across the whole of eduGAIN, the interfederation that links national research and education identity federations worldwide. The proxy replaces all of this with a single trust relationship. LAC trusts one instance, the proxy, and the proxy deals with the thousands of identity providers in eduGAIN. A researcher who wants to sign in picks their home institution in the CLARIN discovery service and logs in with their usual institutional credentials. LAC never sees a password.
Our experience with the proxy
We tested the whole flow together with the CLARIN AAI team: the discovery feed, the attribute release (the user information an institution passes on), and the metadata exchange between the repository and the proxy. Every LAC login now goes through the proxy, and we are glad it does.
We thank the CLARIN AAI team for their help and patience. We encourage other centers to contact them and adopt the SSO: it has simplified our work. The source code of the application is publicly available at github.com/language-archive-cologne/lac-app. The archive itself is online at lac.uni-koeln.de.