Datasets

English–Punjabi (Shahmukhi) Parallel Sentences Corpus (Mediamen Archives)

This parallel sentences corpus containing 30,405 aligned sentence pairs with a total of approximately 0.62 million tokens, curated from the archival materials of Mediamen (Advertising Agency). The sentences were professionally translated from English into Punjabi (Shahmukhi) and are intended to support machine translation, linguistic research, and Punjabi language technology development, particularly for real-world and contemporary language use.

License: CC-BY-NC-4.0

Locale: en-PK, pnb

Task: MT

Format: CSV

Size: 1.08 MB

Fundación Vía Libre

HESEIA Sentence Bias Dataset

This repository contains a dataset collected during the teacher training course HESEIA Sentence Bias (Tools for Exploring Biases and Artificial Intelligence). organized by Vía Libre, the Ministry of Education, and FAMAF-UNC. The course had an initial enrollment of 370 participating teachers, who also involved over 5,000 students in building a dataset that reflects stereotypes present in Argentina.

License: CC-BY-SA-4.0

Locale: es-AR

Task: OTH

Format: CSV

Size: 235.43 KB

RFERL

RFE/RL Tatar-Bashkir News Text Corpus

This dataset is longitudinal news corpus for the Tatar and Bashkir languages, sourced from Azatliq Radiosi, from 2001 to 2025, it contains over 105,000 articles (30M tokens).

License: CC-BY-NC-SA-4.0

Locale: tt,ba,ru

Task: NLP

Format: TXT

Size: 102.44 MB

Digital Divide Data

Luhya ASR data subset 70 hours

A 70-hour subset of Luhya speech data collected by Digital Divide Data in Kenya. The dataset includes recorded sentences from native speakers and is intended to support research and development in Automatic Speech Recognition for low-resource African languages.

License: CC-BY-4.0

Locale: luy

Task: ASR

Format: WAV, XLSX

Size: 13.90 GB

Effect AI

Effect AI Scripted Speech 1.0 - English

A collection of scripted spoken phrases in English.

License: CC0-1.0

Locale: en

Task: TTS

Format: CSV, MP3

Size: 663.45 MB

Amara Hub

DataTrust Africa: Speech Corpus of Public Radio Recordings from Northern Uganda

This is an open-access corpus of short clips of public radio content from Mega 100 FM, Q FM, Radio Pacis and Radio Rupiny in Northern Uganda. As of now, the online corpus has over 350 clips of recordings in English. We also hope to add finely-annotated transcripts to them. The dataset is for use in NLP research and non-commercial use. Upcoming datasets to look out for from Amara Hub are public radio recordings in other languages spoken in the region like Acholi, Lango, Lugbara and Akaramajong.

License: NOODL-1.0

Locale: en-US

Task: NLP

Format: MP3

Size: 179.82 MB

Digital Divide Data

Khmer ASR Cultural Dataset

37.62 hours manually curated speech-text pairs by native speakers in Khmer language about Cambodian cultural topics. On average, each recording is 8.29 seconds with the standard deviation of 3.87. Speaker metadata (gender, age group, and origin city) is provided.

License: CC-BY-SA-4.0

Locale: khm

Task: ASR

Format: WAV

Size: 12.59 GB

PT Pancaran Semangat Jaya

Corpus of Panjebar Semangat Javanese-Language Magazine

This dataset is a TXT-format collection compiled from three years of popular articles published in the Javanese-language weekly magazine Panjebar Semangat. It compiles widely read, non-academic Javanese texts reflecting contemporary themes and language use.

License: CC-BY-SA-4.0

Locale: Jav

Task: OTH

Format: TXT

Size: 4.31 MB

Center za jezikovne vire in tehnologije Univerze v Ljubljani

SI-NLI

SI-NLI (Slovene Natural Language Inference Dataset) contains 5,937 human-created Slovene sentence pairs (premise and hypothesis) that are manually labeled with the labels "entailment", "contradiction", and "neutral". We created the dataset using sentences that appear in the Slovenian reference corpus ccKres (http://hdl.handle.net/11356/1034). Annotators were tasked to modify the hypothesis in a candidate pair in a way that reflects one of the labels. The dataset is balanced since the annotators created three modifications (entailment, contradiction, neutral) for each candidate sentence pair. The dataset is split into train, validation, and test sets, with sizes of 4,392, 547, and 998. We used Slovenian pre-trained language models to create splits, thereby ensuring that difficult and easy instances are evenly distributed in all three subsets. The dataset is released in a tabular TSV format. The README.txt file contains a description of the attributes. Only the hypothesis and premise are given in the test set (i.e. no annotations) since SI-NLI is integrated into the Slovene evaluation framework SloBENCH (https://slobench.cjvt.si/). If you use the dataset to train your models, please consider submitting the test set predictions to SloBENCH to get the evaluation score and see how it compares to others.

License: CC-BY-NC-SA-4.0

Locale: sl

Task: NLU

Format: TSV

Size: 392.44 KB

Pro Svizra Rumantscha

Vallader Newspaper Corpus

6.2 million tokens in the Vallader variety of Romansh from the daily newspaper ”La Quotidiana”.

License: CC0-1.0

Locale: rm-vallader

Task: OTH

Format: TSV

Size: 18.71 MB

Kaleem Art Press

Multilingual Religious Parallel Corpus (Kaleem Art Press)

This dataset is a multilingual parallel sentences corpus containing 6,465 aligned sentence units with approximately 0.98 million words, curated from Kaleem Art Press archives. It includes parallel religious text data in Arabic, Urdu, Saraiki (standard and dialectal), Punjabi (Shahmukhi), and English, supporting research in machine translation, comparative linguistics, digital humanities, and low-resource language studies.

License: CC-BY-SA-4.0

Locale: mul

Task: MT

Format: CSV

Size: 2.27 MB

Sindh Line Publishers

The corpus contains 1.029 million tokens from the Sindh Line a Sindhi Newspaper published from the year 2024-2025. The text consists of the complete newspaper content including headlines, editorials, finance news and advertisements. The newspaper published in Karachi, Pakistan on daily basis

License: CC-BY-SA-4.0

Locale: snd

Task: NLP

Format: TXT

Size: 2.22 MB

Institute of African Digital Humanities

Spoken-Congolese-French-Dataset

The dataset consists of paired audio and text resources on spoken French from the Republic of the Congo. The audio files were extracted from longer recordings of semi-guided interviews conducted in Brazzaville, and orthographic transcriptions were added. The long audio recordings and their corresponding TRJS transcription files were automatically clipped alongside their respective transcriptions. The dataset comprises ten folders containing audio files and ten audio/text mapping files.

License: NOODL-1.0

Locale: fr-CG

Task: NLP

Format: MP3, WAV, TSV

Size: 3.44 GB

Institute of African Digital Humanities

Ewondo_Mbida-Mbani_ALCAM-MultimodalDataset

This dataset comprises a datasheet of Ewondo (Ewo) lexical entries collected in the speech area known as Mbida Mbani. Each entry is accompanied by illustrative sentences, word-by-word glosses and French translations. The resource is enriched with aligned audio recordings, making it suitable for linguistic analysis and speech technology development.

License: NOODL-1.0

Locale: ewo

Task: NLP

Format: MP3, TSV

Size: 19.25 MB

Balochi Academy

Balochi Academy Text Corpus

This corpus contains approximately 500k tokens of text from novels, poetry, articles, riddles, and proverbs, covering both literary and traditional genres. It is intended for linguistic research, NLP tasks (e.g., language modeling and text analysis), and cultural documentation.

License: CC-BY-NC-SA-4.0

Locale: bgn

Task: NLP

Format: TXT

Size: 1.88 MB

Institute of African Digital Humanities

Mada Narratives

This dataset contains 17 transcribed oral narratives in Mada (mxu), a language belonging to the Afro-Asiatic family that is spoken in Cameroon. The texts, derived from audio recordings of oral literature, reflect natural spoken discourse. This dataset can be used for language modelling, text analysis and other natural language processing (NLP) tasks.

License: NOODL-1.0

Locale: mxu

Task: NLP

Format: TXT

Size: 65.04 KB

Institute of African Digital Humanities

Bamun-French Parallel Corpus

This dataset is a parallel corpus of Bamun (Shupamem) to French texts. Text were obtained by transcription of raw audio files. Translation were added to enrich the original corpus. Alignment of Bamun and French texts were made in the process of creating this dataset.

License: NOODL-1.0

Locale: bax

Task: MT

Format: TSV

Size: 99.24 KB

Pro Svizra Rumantscha

Surmiran Newspaper Corpus

2.9 million tokens in the Surmiran variety of Romansh from the daily newspaper “La Quotidiana”.

License: CC0-1.0

Locale: rm-surmiran

Task: OTH

Format: TSV

Size: 11.89 MB

Maseno Centre for Applied Artificial Intelligence (MCAAI)

DhoNam: Dholuo Speech dataset

DhoNam: Dholuo Speech dataset is a speech corpus designed to supercharge Automatic Speech Recognition (ASR) and other speech technologies for Dholuo, one of Kenya’s major indigenous languages. This dataset contains native-speaker audio recordings collected through a platform where users read aloud a displayed sentence. The dataset includes the audio recordings and the corresponding prompt/sentence that was read.

License: NOODL-1.0

Locale: Luo

Task: ASR

Format: WEBM

Size: 2.49 GB

Amnesia

Archivo de la Comisionada María de los Ángeles Guzmán García (COTAI Nuevo León / InfoNL)

Este archivo preserva la memoria institucional y académica de la gestión de la Dra. María de los Ángeles Guzmán García como Comisionada de la Comisión de Transparencia y Acceso a la Información del Estado de Nuevo León (COTAI / INFONL) durante el periodo 2018-2025. El dataset consolida el legado documental de una de las perfiles más técnicos y académicos del Sistema Nacional de Transparencia. Doctora en Derecho Constitucional por la Universidad Complutense de Madrid, la Comisionada Guzmán García

License: CC-BY-4.0

Locale: es-MX

Task: NLP

Format: ZIP, PDF, CSV, XLSX

Size: 866.15 MB

Common Voice

Common Voice Spontaneous Speech 2.0 - Kenyah

A collection of spontaneous spoken phrases in Kenyah.

License: CC0-1.0

Locale: xkl

Task: ASR

Format: MP3

Size: 212.06 MB

Common Voice

Common Voice Spontaneous Speech 2.0 - Ushojo

A collection of spontaneous spoken phrases in Ushojo.

License: CC0-1.0

Locale: ush

Task: ASR

Format: MP3

Size: 102.83 MB

Common Voice

Common Voice Spontaneous Speech 2.0 - Kuku

A collection of spontaneous spoken phrases in Kuku.

License: CC0-1.0

Locale: ukv

Task: ASR

Format: MP3

Size: 233.85 MB

Common Voice

Common Voice Spontaneous Speech 2.0 - Rutoro

A collection of spontaneous spoken phrases in Rutoro.

License: CC0-1.0

Locale: ttj

Task: ASR

Format: MP3

Size: 272.63 MB