Categories
access archives legal reuse

Endangered language archives: what do their terms allow you to do?

In the last post, I discussed technical possibilities to query and download data from endangered language archives. But once you got the data, what can you do with them? Obviously, you can read, view and listen to them, but what are the possibilities for enrichment, aggregation, reuse, and sharing of your work?

There are two aspects to this. The first one is technical: some formats like PDF are notoriously difficult for information extraction, while others, like CSV, JSON or XML allow for a much easier access to the information contained. But today I want to focus on the second aspect, namely legal interoperability. When downloading data from an endangered language archive, you typically have to register and agree to some terms and conditions, which determine what you are allowed to do with the data, and what is forbidden. It is important that the legal condition be clear so that other users do not run into legal risks when reusing the data, and also for the wishes of the original depositors and the people recorded to be respected.

Some parts of this blogpost will be somewhat lame, and discussion of legal terms tends to be rather dull, so you have been warned, but there are only 2000 words left 😉

There are two basic approaches to regulating access: allow everything and restrict certain usages, or forbid everything and grant particular usages.

image via Peakpx

I will survey four endangered language archives, namely AILLA (US), ELAR (UK), TLA (NL), and PARADISEC (AUS). I will first outline some quality measures for legal language. Then, I will present four toy reuse scenarios and see whether these scenarios would be allowed under the present terms. The full texts of the terms are given at the very end for reference.

Quality criteria

Usage terms can be worded very clearly, or they can be extremely opaque. I identify three main points to check:

  • clarity: is the language used specific, explicit, and unambiguous?
    Good: People younger than 18 years of age may not consume beverages containing alcohol on the premises.
    Bad: Young people are not allowed to drink here.
  • relevance: the terms make a legal difference. They do not simply restate the law
    Good: You are allowed to carry one bag of 20 kg.
    Bad: You are allowed to read a book.
  • coherence: There are no contradictions
    Good: Cars are not allowed on the premises. Really not.
    Bad: Cars are not allowed on the premises. The speed limit is 30.

Legal domains

The domains concerned are copyright (who is allowed to derive commercial value from a resource) and privacy law (who is allowed to produce and distribute content where people can be identified).

Legal spheres and civil/common law

One word of caution: what I write here is mainly based on my understanding of continental European law. In the anglophone world, the organization of intellectual property law has completely different foundations. I try to take them into account, but there are good chances I miss the mark for an Anglo context.

By Maximilian Dörrbecker (Chumwa) – Own work, usingWorld map by Canuckguy and othersUNESCO World Heritage map by NNWThe data sources are:University of Ottawa: JuriGlobe – World Legal Systems Research GroupWikipedia: List of national legal systemsEuropean legal systems map by Ain92 and others (which seems to be based on this map by C.Löser and others)World legal systems map by Robinkissac, CC BY-SA 2.5, https://commons.wikimedia.org/w/index.php?curid=40154967

For instance, the concept of Fair Use is well-established in the US, but completely unknown in Germany (and the EU). Terms and conditions from the US are written with the idea of Fair Use in mind, while terms and conditions in the EU are different, as we will see in the discussion below.

The four freedoms

The Open X movement was pioneered by Open Source software and later extended to Open Access, Open Data, Open Science, Open GLAM (Galleries, archives, libraries, museums) and so on.
In the context of Free Software, there are four well established freedoms which determine whether a software is indeed Free Software: the rights to use, study, share, and modify it without legal barriers. We can test our four language archives with respect to these criteria.

Freedom 0: Use

This refers to the functionality of the software. In the context of language data, we are not dealing with executable programs, so this point is moot

Freedom 1: Study

The AILLA terms do not say anything about the permission or restriction of study.

ELAR explicitly allows research and education: I will use the materials only for non-commercial private research or educational activity. Note, however, that it is unclear whether non-commercial private refers only to research or to both research and educational activity. And it is unclear what non-private research might be. If there is a difference between private and non-private research, it might be the case that professional research is actually not covered by the terms. This would be very strange and is probably not intended, but the wording here lacks clarity.

PARADISEC states: To use the data for purposes of private scholarly research and/or for purposes of educational use. The scope of private is better here, since educational use is not “private” here, but it is still unclear what private research is to begin with. In any case, the use of data for at least some research is permitted.

TLA does not state anything about study, but since the general legal situation allows that you read whatever you like, this also applies here.

Freedom 2: Share

With respect to sharing of downloaded material, AILLA states: I will respect the intellectual property rights and copyrights that pertain to AILLA’s holdings. This is thus just a reaffirmation of the legal situation. Obviously, you should respect the law, just like you respect the traffic rules when you rent a car. Under European continental terms, this basically means that you are not allowed to share anything since you are not the copyright holder. Under US Fair Use, sharing is less of a legal problem. We will see further down that AILLA’s terms are very close to Fair Use and trust that the users will not do Bad Things.

Vassia Atanassova – Spiritia / CC BY-SA (https://creativecommons.org/licenses/by-sa/3.0)

ELAR states: I will not transfer the materials to any other person or equipment. If the materials are used in any way in the preparation of any publication, broadcast or dissemination, I will acknowledge ELAR
This is a very clear statement that you are not allowed to share the data, be it with friends, colleagues, or the general public (“any other person”). But we have a contradiction here: you are not allowed to share the data, but if you broadcast it, sharing it with the population at large, you should acknowledge ELAR. ELAR thus wants to be acknowledged if you breach their terms. ELAR is aware of the value of the archive data for informing the public about language endangerment, and would like to regulate the use for broadcasting, but their very own terms are worded in a way which make broadcasting impossible to begin with

PARADISEC states: Not to copy the data in whole or in part except insofar as this may be necessary for security purposes or for my own personal use. So you are allowed to make backups and transfer the files between your own devices, but the data may never go to a third person’s device. This condition is somewhat loosened thereafter: To give access to the data only to persons directly associated with me or working under my control, so you are allowed to have student assistants, but you have to require of such persons signed undertakings neither to use the data except in connection with my academic or educational purposes, nor to give access to the datas to others; these signed undertakings to be made available to PARADISEC on request.
This says that if you have a student assistant and have them work on PARADISEC data, PARADISEC could write an email requesting that you hand over the paperwork. I doubt anybody has ever respected that clause. It would also be interesting to see what happens if a researcher does not comply with the request to hand over “these signed undertakings”.

TLA does not state anything about sharing, so it falls back on standard European law, which forbids sharing without explicit consent.

Freedom 3: improve/modify

AILLA states: I will not alter or modify any resource in the AILLA database without a legal authorization to do so. Again, the terms simply reaffirm what is stated in the law anyway. If the law allows some modifications, this is fine, if the laws bans some modifications, the terms will not grant you extra rights.

ELAR terms do not say anything about modifications

PARADISEC states: Where substantial revisions or reformatting of the data is permitted by the conditions of deposit and carried out as a part of my research, to inform PARADISEC, and also the original depositor if this is a condition of deposit, of the nature of such revisions and to deposit any such revised version at PARADISEC […]
PARADISEC thus assumes that revisions and reformatting are a possible scenario for the data it provides. There is also reference to “the conditions of deposit”, but I have not been able to find out what this refers to.

TLA does not say anything about modifications.

Commercial use

According to the opendefinition.org, resources are only open if they require at most attribution to the original creator and nothing else. In particular, commercial reuse is allowed (See here for an extended discussion about what (non-)commercial use means and what often unintended consequences are).
The archives are very clear that they have no interest in complying with this particular definition of openness.
AILLA: I will not use any resource in the AILLA database for commercial purposes.
ELAR: I will use the materials only for non-commercial private research or educational activity.
PARADISEC: To use the data for purposes of private scholarly research and/or for purposes of educational use, and not for profit

TLA does not say anything about commercial use.

Summary

TLA (1254 words)

You may have noticed that the TLA terms do not really say anything about the data. The terms contain detailed instructions about your credentials (username/password), what you are allowed to do with credentials, what TLA is allowed to do with the credentials, but TLA is surprisingly silent about what you are allowed to do with the content. There are 15 subcategories (ID, AFFIL=x, PERM, FF, PLAN, BY, NC, INF, LOC, LRT, PRIV, NORED, DEP, SA, ND), but it is unclear what legal consequences they have. As far as I can tell, you can accept the terms and conditions and no further measures are undertaken to inform you about which resource falls under which subcategory.

ELAR (519 words)

The ELAR terms are contradictory and unclear. You are not allow to share the data with anybody, but if you broadcast them, you should inform ELAR.

PARADISEC (405 words)

The PARADISEC terms are clearer than the ELAR terms, but they point to “conditions of deposit”, which I have not been able to identify.

AILLA (366 words)

I personally like the AILLA terms. They do not pretend to be “real” legal terms. They just say “listen, there is a legal context, and you should respect that. But fear not, it might very well be the case that Fair Use will help you do what you want”. In a sense, this is not a legally binding contract, but it is a performative act of admonition. The other archives try to put some legally binding language in their terms, but as far as I can see that extra language will either have no effect or unintended effects as compared to the fallback legal situation. This unclear legal language creates legal uncertainty, which is fatal for uptake, mobilization, and reuse.

Use cases

To test the effect of these terms and conditions, I made up four different use cases of people who might want to interact with endangered language archives

Use case 1: laughter around the world.

Kim wants to extract laughter from videos from many different cultures and post a compilation on YouTube

Use case 2: Installation: “ceci n’est pas un [pa]”.

Sam wants to extract all audio snippets transcribed as “pa” and have a spatial audio installation in a room where [pa] is being uttered by many different speakers. In order to cover the rent for the room, a small entry fee is required.

The syllable [pa] produced by a German male speaker, aged 43.

Use case 3: Volkshochschule course on language diversity

The German Volkshochschule (public university for adults) offers a course on language diversity. Videos of different languages and cultures are to be shown once a week at Wednesday evening, accompanied by some cultural and linguistic explanation. The course fee for 3 months is 65 EUR.

Use case 4: Big Data

The LinguTech company develops The Definite Translation Software™ which can translate any human language into any other human language, based on some neural network fed with endangered language data. That software is sold on GooglePlay and AppStore.

Evaluation

Use case 1: laughter around the world.

The extraction and compilation of video is a modification. Posting on YouTube is sharing. YouTube is a commercial platform. This usage is not allowed. Furthermore, people can be identified, raising privacy issues.

Use case 2 ceci n’est pas un [pa].

Extraction of [pa] is a modification. No privacy issues, though, since the sequences are very short. The content is shared with the visitors. The entry fee makes this project a commercial one. The terms do not allow this project, but it might be possible under Fair Use in the US.

Use case 3: Volkshochschule course.

The data are used for educational purposes, but the education is not “private”, whatever that might mean, and there is a moderate fee, making this usage commercial. Using the data in class means sharing the data with the students. This project is not allowed.

Use case 4: translate everythin’ into everythin’.

The data are not really modified, nor are they shared. The trained network is used to generate revenue, but it does not contain any data. The terms of AILLA, ELAR, and PARADISEC seem to ban this usage nevertheless.

Conclusion

ALl of the toy use cases are banned by current terms. There are certainly more use cases for mobilization. If you can think of some additional one which would be allowed by the present terms and conditions, please leave a comment.

The legal language of archives is not satisfactory. TLA focuses on credentials and ignores reuse as a category requiring legal clarity. ELAR and PARADISEC have terms which are contradictory (broadcasting) or out of touch with reality and unenforceable (paperwork). The AILLA terms are modest and differ from the other archives’ terms in that their main goal seems to be to enable users to do things, whereas the other archives’ terms give the impression that their main goal is to protect resources from the users.

Way forward

From this short overview, I venture the following recommendations:

  • TLA should provide meaningful terms.
  • ELAR should get rid of contradictions.
  • TLA, ELAR, and PARADISEC should get rid of legalese signalling erudition and get in touch with linguists studying readability. “Hereinafter”, “you undertake not to”, “any part of any material however modified”, “legally capable of entering into this agreement”, “this shall not preclude the publication …, unless such publication is precluded” only add to the length of the terms, but not to their purpose. The terms can be worded in a much more readable way, allowing a larger share of the population to interact with the archive.
  • ELAR and PARADISEC should review their conditions with a focus on attracting and enabling users. This does not mean that the level of protection of resources has to be lowered; it is mainly about attitude and wording. Users are a boon, not a threat.
  • AILLA should keep its kind spirit and user-centered language ;): If you produce transcriptions or translations or other derivative materials from AILLA resources that our user community would find interesting or useful, we would appreciate it if you would deposit those materials with us as well. As a user, I feel very welcome.

Feel free to skip to comments now. What follows is the text of the terms and conditions of the four archives for reference.

Appendix

AILLA terms

https://ailla.utexas.org/site/rights/use_conditions:
Resources deposited in the AILLA database are protected by the AILLA License Agreement or by more specific agreements signed by the depositors and AILLA’s directors. While AILLA can not guarantee that no violations of these agreements will occur, if we learn of a violation, AILLA will notify the AILLA community of the violation.

“I agree to the following conditions on the use of AILLA resources:

  1. I will not use any resource in the AILLA database for commercial purposes.
  2. I will respect the intellectual property rights and copyrights that pertain to AILLA’s holdings.
  3. I will not alter or modify any resource in the AILLA database without a legal authorization to do so, for example, explicit permission from the rights holder or Fair Use of Copyrighted Materials (US Code Title 17, Sec. 107).
  4. I will not create works derived from any resource in the AILLA database, without a legal authorization to do so, for example, explicit permission from the rights holder or Fair Use of Copyrighted Materials (US Code Title 17, Sec. 107).
  5. I will act in good faith in interactions with AILLA’s User Registration and Graded Access Systems. This means that I will not attempt to disguise my electronic identity or impersonate an AILLA depositor or resource creator, and I will abide by the decisions made by resource controllers concerning access to AILLA resources.
  6. If the metadata for a resource states that names of creators and participants must be kept anonymous, I will respect their anonymity in any spoken or written representation of that resource that I produce.
  7. I will cite any AILLA resource that I make use of in any published work according to the AILLA Citation Guidelines.”

If you produce transcriptions or translations or other derivative materials from AILLA resources that our user community would find interesting or useful, we would appreciate it if you would deposit those materials with us as well. AILLA has many recordings that are untranscribed, untranslated, and unanalyzed, and we hope that members of our user community will contribute to AILLA by helping to make these oral works more accessible with transcriptions, translation and analyses.

Thank you for your interest in our collection.

ELAR terms

https://www.soas.ac.uk/elar/using-elar/terms-and-conditions
The Endangered Languages Archive (ELAR) provides access to materials which remain the property of their depositors and information providers. By agreeing to the Terms and Conditions and Data Protection statement below, you as an ELAR user undertake to use any materials for private study purposes only, and you undertake not to make any further copies, nor to distribute, publish, broadcast or disseminate, any part of any material however modified without explicit written permission. ELAR will hold data about you and may provide it to the owners of materials and others in accordance with UK and EU laws and only for the purposes of management and protection of materials.

Terms and Conditions and Data Protection statement

  1. I declare that I am the person named as the LAT user, and that I am legally capable of entering into this agreement with ELAR.
  2. ‘Materials’ refers to all or any part of data received, or played on any equipment, or any further material derived from or originating in the data received.
  3. I will use the materials only for non-commercial private research or educational activity. Any other usage needs explicit written agreement from the depositor and any other rights holders.
  4. I will not transfer the materials to any other person or equipment, or do anything to allow such transfer to take place, unless permitted by an explicit written agreement from the depositor and any other rights holders.
  5. The materials may include documents that state further conditions of use or instructions for acknowledgement. I will make reasonable efforts to identify such documents and I will observe any stated restrictions or instructions.
  6. If the materials are used in any way in the preparation of any publication, broadcast or dissemination, I will acknowledge ELAR and, if required, the depositor and any other rights holders in the materials. I will, wherever possible, send a copy of such publication to ELAR.
  7. I will not knowingly use materials for any purpose which causes disparagement, disrespect, damage to reputation, or harm to any individual or group.
  8. I understand that the materials are supplied ‘as is’, and are not guaranteed to fulfil any purpose or function, or to be compatible with any particular computing system or software. I release and indemnify ELAR from any damages or consequences resulting from my receipt or use of the materials. I understand that ELAR may take legal action on behalf of owners of materials in the case of serious infringement of this agreement.
  9. ELAR collects data to create User Accounts, to facilitate users’ access to ELAR’s resources, and to manage, protect, and report usage of materials. ELAR will store data about users and their access to materials. The data will be stored securely in accordance with the laws of the United Kingdom and the European Union. I understand that the data may identify me and may be shared with the owners of the materials and others only for the purposes of managing the materials. ELAR may contact me in the future about my Account. I consent to such uses of information about me.
  10. I have read this agreement and agree to be bound by it.

PARADISEC terms

https://www.paradisec.org.au/deposit/access-conditions/
The following are general conditions for obtaining material from the Pacific And Regional Archive for Digital Sources in Endangered Cultures (hereinafter called PARADISEC). They assume that the depositor’s permission has been obtained for copying. Specific restrictions accompany each item in PARADISEC.

In consideration of PARADISEC agreeing to supply me with certain data in machine-readable form together with supporting documentation, I hereby undertake: –

Purpose:
To use the data for purposes of private scholarly research and/or for purposes of educational use, and not for profit (this shall not preclude the publication in a scholarly context of analyses or interpretations derived from the data, unless such publication is precluded by the conditions of deposit of the data in the PARADISEC).

Acknowledgement:
To acknowledge in any work, published or unpublished, based in whole or in part on analyses of the data, both the original depositors of the data and PARADISEC (by use of the PARADISEC persistent identifier and citation form).

Copyright:
Not to copy the data in whole or in part except insofar as this may be necessary for security purposes or for my own personal use. Not to distribute the data to third parties, nor to publish or reproduce it in any way. Copyright of all machine-readable data issued by PARADISEC is reserved.

Access:
To give access to the data only to persons directly associated with me or working under my control and to require of such persons signed undertakings neither to use the data except in connection with my academic or educational purposes, nor to give access to the datas to others; these signed undertakings to be made available to PARADISEC on request. Not to give access to any third party if the conditions of deposit preclude this.

Errors:
Not to hold PARADISEC liable for any errors of transcription or marking-up which may be found in the data, but to notify PARADISEC of such errors wherever possible.

Revisions:
Where substantial revisions or reformatting of the data is permitted by the conditions of deposit and carried out as a part of my research, to inform PARADISEC, and also the original depositor if this is a condition of deposit, of the nature of such revisions and to deposit any such revised version at PARADISEC subject to the conditions of deposit of the original.

Charges:
To pay such charges as PARADISEC may require to cover the cost of supplying the data (not applicable unless an invoice accompanies the data).

TLA terms

https://archive.mpi.nl/tla/legal/document/terms-of-use
Preamble

Welcome to The Language Archive. This repository service is provided to you by Max Planck Institute for Pycholinguistics, located at Wundtlaan 1 in Nijmegen, The Netherlands, hereinafter referred to as CLARIN Centre. The mission of CLARIN is to provide as wide access as possible to language research materials and tools around Europe. To archive this goal, we set some ground rules in this document. By accessing or using, and in consideration of the CLARIN Services provided to you, you agree to abide by the Terms of Use below, which can be updated or modified according to article 9.

Terms of Use

1. Governing Terms

1.1 The access to and use of CLARIN Services are governed by this Agreement and by its terms and conditions, which are hereby incorporated into its End-User License Agreements. The User’s attention is drawn to the provisions limiting access to certain classes of resources.

1.2 Any use of the CLARIN Services by means of the User Identity granted by the local identity provider also signifies the User’s acceptance of the Terms of Use and the User’s agreement to be bound by them.

2. Definitions

2.1 All terms used in this Agreement, unless specifically defined herein, will have the meanings attributed to them in this Agreement. The following terms shall have the following meaning:

“Authorised Use” means use by a User who is accessing the CLARIN Services via the identity provided by a national identity provider.

“Academic Use” means use by a User who fulfils the criteria set by the national identity providers for Academic users and defined by CLARIN ERIC.

“Non-Commercial Use” refers to any use that does not generate income or is not used to promote the generation of income.

“User Identity” refers to the identity granted by a local identity provider or a CLARIN Centre of type A or B.

3. Access to CLARIN Services

3.1 CLARIN hereby grants the User a limited non-exclusive non-transferable licence to access and use CLARIN resources under the terms of this Agreement and via the chosen User Identity.

3.2 Access to certain Features and Databases may be restricted by CLARIN.

3.3 The User will be responsible for all access to CLARIN by means of the chosen User Identity whether or not the User has knowledge of such access and use. CLARIN reserves the right to cancel any User Identity without notice.

3.4 The User understands and acknowledges that the User Identity is for his or her use only and that the User is prohibited from permitting any third party from accessing CLARIN by means of the User Identity.

3.5 Content Categories

CLARIN offers content in three different main categories:

  • • Public Content (PUB)
  • • Academic Content (ACA)
  • • Restricted Content (RES)

The User understands and acknowledges that depending on the content the access may be limited based on this categorization and the User Identity. CLARIN may also require acceptance of additional licensing or usage terms for Academic and Restricted Content. The User agrees to use the content according to the conditions.

3.6 Sub-categories

The offered content may belong to certain sub-categories indicated by the following category labels:

  • •Identification and Access conditions
    • ID: The user needs to be authenticated or identified
    • AFFIL=x: The user needs to be affiliated with some community, e.g. a community of academic researchers (x=EDU) or a community of language research and technology researchers more generally (x=META)
    • PERM: The user can only be given permission to use the resource on a case-by-case basis, such as a mandatory fee or a research plan
    • FF: A fee is required to get access to the resource
    • PLAN: The right holder requires a research plan for granting access
    • • General Use conditions
    • BY: Attribution, i.e. acknowledgement of authorship, is required
    • NC: The content is available only for non-commercial purposes
    • INF: Informing the rights holder about the use of the resource is required
    • LOC: The content is available only at a single location, center, or site?
    • LRT: The content is available only for language research and technology development
    • PRIV: There are personal data in the resource
  • • Distribution conditions
    • NORED: The user is not permitted to redistribute the resource
    • DEP: The user is not permitted to redistribute the resource but as an exception to this rule, the user may still distribute modified versions via CLARIN
    • SA: The resource can be redistributed under similar conditions, i.e. is the license reciprocal
    • ND: The user is not permitted to make derivate works, i.e. works containing copyrighted parts of the original
  • • Other conditions
    • *: There are other non-standard conditions in the license that the user should pay attention to
    • The User agrees to adhere to these requirements.

3.7 Content-Specific Licenses

In addition to the aforementioned categories, some content has its own license (for example a Creative Commons-license), which may set additional limitations and requirements. The User agrees to follow these limitations and requirements.

4. Research Ethics

The User agrees to observe best practices regarding research ethics. This includes treating colleagues, stakeholders, customers, suppliers and the public respectfully and professionally, taking into account confidentiality when appropriate, respecting cultural differences and having an open and explicit relationship with government, the public, the private sector and other funders.

5. Availability of Services

CLARIN disclaims all responsibility and liability for the availability, timeliness, security or reliability of the services, software or other content provided through the CLARIN Services. CLARIN reserves the right to modify, suspend, or discontinue the services or access to the services without any notice at any time and without any liability to you.

6. Governing Law and Entire Agreement

These Terms are governed by the laws of The Netherlands without regard to the rules of conflict of law that may cause the laws of another jurisdiction to apply. The User agrees to the sole and exclusive jurisdiction and venue of the Gelderland District Court of Law in the event of any dispute of any kind arising from or relating to the CLARIN Services, or the User’s use or review of it. However, CLARIN has the right to use the laws of another jurisdiction to get injunctive measures against the misuse of the CLARIN Service.

The Terms of Service constitute the entire agreement between the parties with respect to the subject matter hereof and supersedes and replaces all prior or contemporaneous understandings or agreements, written or oral, regarding such subject matter. If for any reason a court of competent jurisdiction finds any provision or portion of these Terms to be unenforceable, the remainder of the Terms will continue in full force and effect.

7. Data Protection and Privacy

The User agrees to follow the data protection policy of the CLARIN Services.

8. Usage Statistics and Automated Querying

CLARIN maintains usage statistics as a measure of readership and other use of the CLARIN Services by authors and researchers. It is a violation of CLARIN policy for a party to directly or indirectly use CLARIN with a view to affecting download and other usage statistics, or to encourage others to do so. As part of its general right to refuse or terminate service and remove or edit the content of the CLARIN Services, CLARIN reserves the right in its sole discretion to limit access, remove content, and adjust usage statistics to respond to any activity that appears likely to have such an effect.

9. Amendments

9.1 If these Terms of Service are modified for legal, administrative or any other reasons, the CLARIN Centre shall notify the users about these amendments by publishing the relevant information on the CLARIN Centre website without unreasonable delay.

9.2 If the User continues to use the services after being properly notified about the amendment of the Terms of Service, it implies his consent to the new Terms of Service.

10. Termination

If the User violates the letter or spirit of this agreement, or otherwise creates a risk or possible legal exposure for CLARIN, CLARIN can stop providing all or part of CLARIN Services. CLARIN will notify the User at the next time he or she attempts to access the CLARIN Services.

Categories
access Allgemein

How to get things out of a language archive

There are about 7000 languages in the world. Many of those languages are severely endangered and many will lose their last speakers in the years to come. For about 25 years now, several language documentation programs have funded projects to collect data from those languages (texts, audio, video). The collected resources are deposited in specialized archives. Of those archives, TLA (Nijmegen), ELAR (London), PARADISEC (Australia) and AILLA (Texas) are the largest ones. Each archive has its own procedures to upload stuff and to access stuff for download.

The fields of Linguistic Typology and Diversity Linguistics are interested in the variation we find in the languages of the world. For instance, some languages use different words for ‘hand’ and ‘arm’ (red dots in map below), others use the same words (yellow dots). Why is this the case, and can we predict which languages in which areas are more likely to choose one value over the other?

Languages of the world differ on whether the words for ‘hand’ and ‘arm’ are different or identical https://wals.info/feature/129A

For questions like this one, information about a maximum of very different languages is crucial. As such, the language archives should be a prime resource for research in these areas. This is, however, not really the case. Very few studies make use of the primary resources deposited in the archives. This has to do with hurdles to access related to 1) the sensitivity of the data, 2) lacking querying facilities, and 3) no uniform access protocols between archives. I will discuss those issues in turn

Sensitivity of data

Given that recordings found in language archives are often from marginalized communities, the archives have to be very cautious to not cause any harm to these communities. Generally, archives require registration, which entails the signing of a code of conduct. A discussion of the different codes of conduct will follow in a future blog post, but they generally say “do respect the community and their values. Use the data for research, but do not share them”.

The requirement of registration means that a link to a resource in an archive will only work when credentials are provided, and automated access in a software library will have to deal with this.

Archives generally have different levels of access, typically “accessible to anybody”, “accessible to registered users”, “available on request”, “closed”. The first two levels can be used for automated queries, while this is not possible for the latter two.

Access levels for ELAR https://lat1.lis.soas.ac.uk/ds/RRS_V1/RrsRegistration

Lack of querying facilities

For a phonetician, primary data involving audio are a great resource. Word lists and dictionaries, on the other hand, are less useful. Sign language researchers need video, while sociolinguists might be interested in sessions which involve at least two speakers of different genders. Areal specialists might only be interested in languages of the Himalayas and so on. Given the wealth of data contained in the archives, good querying facilities are paramount. Furthermore, the querying facilities should take into account access rights. A list of audio recordings which are set to “no access” is of less value for a phonetician than accessible ones. All language archives provide different querying facilities. For instance, this link provides a list of all ELAR collections for Africa.

Query result for all ELAR collections from Africa.

Unfortunately, it is not possible to restrict these 87 collections to the ones which contain at least some video. One has to access each and every collection to see whether the required data type is found in there. Within a collection, say the Tabaq collection, browsing by media type is possible (all sessions with video from Tabaq), so that one can arrive at a session with video, which can then be downloaded.

All sessions with video from the Tabaq collection
A particular session, from where video can be downloaded

This process is of course rather cumbersome, and it would be nice to be able to tell the computer “get me all videos from Africa and report back when you are done”.

Several archives expose their metadata via the Open Archives Initiative Protocol for Metadata Harvesting. The OLAC project harvests these metadata and provides a uniform querying interface.

This allows to formulate nice queries, like this one for videos from Africa. Following some hops, one can end up at a session page like this one on playing the Bao game in the Hai//om language, where one can then download the video. Or, alternatively, one can end up at a page like this one about melons in West !Xoon, which does list video files, but they are of the type “restricted access” so not available for inspection. Since there are over 1200 hits for this query on OLAC, a manual inspection of which files are actually available for inspection is not practically feasible.

A faceted query for videos from Africa in OLAC.

No uniform access protocols

Following up on our OLAC query above, once you have identified the relevant sessions from the different archives (of course you are already a registered user everywhere), you have to get hold of your own copy of the media files. In order to retrieve the download links, a variety of different strategies have to be followed. As far as I can see, there is no way to get JSON data. Instead, one has to crawl the web page and look for the elements where the links to media files are listed. For TLA, you have to find the links with the CSS class flat-compound-caption-link. For ELAR and AILLA, you have to find the elements matching the XPATH .//tbody/tr/td/a. For PARADISEC, the XPATH is .//div/div/div/fieldset/table/tr/td/a. The best approach then is to try and download the files and see whether your permissions are sufficient.

Identifying the CSS class for the TLA download links.

It would obviously be much nicer if some JSON were offered next to the HTML so that machine crawlers can easily retrieve the URLs they are interested in. The server would not have to assemble the HTML and the client would not have to parse the HTML, so it is a win-win. TLA and AILLA use Islandora, and some documentation suggests that the underlying server should be able to deliver JSON without too much trouble, so this would definitely be a desideratum for the future. Even better would be possibilities for bulk download of all (accessible) data.

eldpy, a library for interacting with endangered language data

To recapitulate, for quantitative research in linguistic typology, a programmatic access to endangered language data via a clearly specified interface is a desideratum. In the context of the QUEST project, I have worked on a python library to provide such programmatic access. This library should be able to interact with the various endangered language archives and retrieve metadata and media files according to the user’s specifications. As of today, the library can download files of a given type from AILLA, ELAR, PARADISEC and TLA. Futher processing and analysis functions are also available within the library, which will be covered in future blog posts.

Categories
Allgemein

mobELD

There are about 7.000 languages in the world, but we have computational resources for only a dozen of them. On the other hand, there are endangered language archives, which host data from very many other languages. These languages are small, remote, or simply from poorer countries. This blog will present efforts to make these archive collections accessible for computational research.

Search OpenEdition Search

You will be redirected to OpenEdition Search