Authors
My Chiffon NguyenTianyi HuVinod AnbalaganHenry GagnierPouya SadeghiFarah AbdouTidor-Vlad PricopeDiego MMS OliveiraJocelyn K D'ArcyMadeline SmithOluwaseun A AjayiYusser Al GhussinZena Al-KhaliliMicol AltomareShayan AmirAhanaf AzizBryan ChenRuchira DharAbdellah El MekkiRawan Mohammed ElframawyAkhil EppaSilvia FernandezAlex GimeiArya HariharanChristian HoangRamMohanRao KadiyalaDiana KozachekPeerat LimkonchotiwatAnkita MaityDi MoAyush MunotAylin NaebzadehAnastasiia NikitinaDina PisarevskayaDivas PoudelJebish PurbeyMd Rabius Sany ApuAhmad Mufli RamadhanRalitza Nikolova RizovaDiogo M. Serrano-MirandaVivek Gopalkrishna SilimkhanLuis Fernando Solis NavarroKaiser SunArya SuneeshManaswitha SunkaraTejaswi SunkaraJonathan E SwindellHanna YukhymenkoJingyi ZhengEsther AdenugaIlia AfanasevWafa AissaLisa AlazrakiLauren C AltomarePriscilla Amondi AmuokEmile AnandGabriel AssisJosé Matías AvalosSilvia Lorena AvilaZakaria BaannouNiyati BafnaYash BaglaSarah BahramiBontu F BalchaVanya Bannihatti KumarIrene BaucellsAbdumutalib BegmuratovNaman BhatiaSouvik BhattacharyyaIsabel Blanco FerreiroFelipe BochiniSayuru BopitiyaAngana BorahTulika BoseOlena BugaiovaJuliana CarvalhoNancy F ChenStephy ChenRochelle ChoenniJayanth Krishna ChundruLeandro A CossioNebojša DamnjanovićOna de GibertMarco De SantisPedro DelgadoKaustubh DholéAdnan El AssadiWesam El-SayedNadine M ElframawyOumayma EssarhiNaome A EtoriReuben Chagas FernandesSaanidhi Pravin GadeJavier García GilabertÁlvaro González-EscapaBin HuKaiyu HuangAbderrahmane IssamDanka JokićUlugbek KhalilovDominik KrzemińskiRoy Ka-Wei LeeHaichang LiChen Cecilia LiuZhengyuan LiuCedric LothritzHouman MehrafarinAntonio Miranda-EscaladaFengran MoMehrnaz MofakhamiWafaa MohammedFelipe Moreno-VeraPrahitha MovvaJannatul NayemVan NgoJoy OlusanyaMarc PàmiesAvinash Kuney PandeyYurii PanivFred PhilippyVladislav PoritskiDikshya PoudelJoanna RadołaSara RajaeeSwati RajwalPrerana RaneMuhammad Ravi Shulthan HabibiLucas ResckShubhashis Roy DiptaAnanya SahuEduardo SánchezBorja Sánchez ClementeAleix SantNobin SarwarIstiak ShihabSanthoshini Pranathi SingarajuJane SmithDavid StapEmily Z SuAnanth K SureshSajag SwamiRuiyang TanNazia TasnimKhalid TounoussiPatteera TriamamornwoothMuhammad Saad UmarShankar VenkitachalamOksana VolchekLingbo WangDebela Desalegn YadetaRushikesh ZawarJinghan ZhangMike ZhangWeihua ZhengMarzieh FadaeeAkhil AroraKároly BoczkaJulia Kreutzer
Abstract
As LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important. However, many benchmarks remain English-centric, with limited cultural coverage and a focus on factual knowledge over its application in real-world cultural contexts. We propose CulturalRiddles, a benchmark of original, culturally grounded riddles created entirely by humans through a participatory community effort, spanning 62 cultures (45 countries, 51 languages). These riddles are deeply rooted in their respective cultures and require abductive and compositional reasoning, alongside detailed factual and social knowledge, language skills, and an understanding of culturally specific associations. We benchmark 28 LLM configurations using both automatic and large-scale human evaluation. Even frontier models achieve at most 22% accuracy in some communities. We publicly release all artifacts alongside an open contribution process to expand the benchmark's linguistic and cultural coverage.