{"id":10323,"date":"2025-02-27T10:36:36","date_gmt":"2025-02-27T10:36:36","guid":{"rendered":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/?p=10323"},"modified":"2025-12-12T16:00:10","modified_gmt":"2025-12-12T16:00:10","slug":"gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms","status":"publish","type":"post","link":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/","title":{"rendered":"GSMA Open-Telco LLM Benchmarks Launches to Advance AI in Telecoms"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\" id=\"h-new-community-provides-an-open-source-framework-to-assess-large-language-models-for-capability-energy-efficiency-and-safety\">New community provides an open-source framework to assess large language models for capability, energy efficiency and safety<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/www.mwcbarcelona.com\/articles\/gsma-open-telco-llm-benchmarks-launches-advance-ai-telecoms?utm_campaign=gsma_blog&amp;utm_medium=social&amp;utm_source=linkedin\"><strong>Read the full press release here<\/strong><\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>25 February 2025, London<\/strong>&nbsp;\u2013 The&nbsp;<a href=\"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/\" target=\"_blank\" rel=\"noreferrer noopener\">GSMA Foundry<\/a>, the GSMA\u2019s innovation hub, today announced the launch of&nbsp;<a href=\"https:\/\/huggingface.co\/blog\/otellm\/gsma-benchmarks\" target=\"_blank\" rel=\"noreferrer noopener\">GSMA Open-Telco LLM Benchmarks<\/a>, an open-source community aimed at improving the performance of large language models (LLMs) for telecom-specific applications. The community provides an industry-first framework for evaluating AI models in real-world telecom use cases and is supported at launch by&nbsp;<a href=\"https:\/\/huggingface.co\/\" target=\"_blank\" rel=\"noreferrer noopener\">Hugging Face<\/a>,&nbsp;<a href=\"https:\/\/www.ku.ac.ae\/\" target=\"_blank\" rel=\"noreferrer noopener\">Khalifa University<\/a>,&nbsp;<a href=\"https:\/\/www.linuxfoundation.org\/\" target=\"_blank\" rel=\"noreferrer noopener\">The Linux Foundation<\/a>&nbsp;and a host of leading mobile network operators and vendors.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As AI adoption in telecoms accelerates, LLMs have demonstrated significant shortcomings in handling technical telecom knowledge, regulatory compliance and network troubleshooting. In recent tests, GPT4\u00b9 scored less than 75% on TeleQnA\u00b2 \u00b3, a comprehensive dataset tailored to assess the knowledge of LLMs in the field of telecommunications, and less than 40% on 3GPPTdocs Classification\u2074, a dataset based on 3GPP standards documentation. Microsoft&#8217;s Phi2\u2075, a much smaller model, scored only 10% on MATH500\u2076 \u2077, a benchmark of 500 general maths questions.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">These results highlight the current limitations of AI models in addressing telecom-specific queries. GSMA Open-Telco LLM Benchmarks will address these gaps by providing transparent, open evaluations of AI models across capabilities, energy efficiency and safety.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u201c<em>Today\u2019s AI models struggle with telecom-specific queries, often producing inaccurate, misleading or impractical recommendations<\/em>,\u201d said Louis Powell, Head of AI Initiatives, GSMA. \u201c<em>By creating an industry-wide set of benchmarks, we\u2019re not only improving model performance but also ensuring AI in telecoms is safe, reliable and aligned with real-world operational needs<\/em>.\u201d<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The mobile network operators supporting the launch of GSMA Open-Telco LLM Benchmarks include&nbsp;Deutsche Telekom,&nbsp;LG Uplus, SK Telecom and Turkcell and technology vendor, Huawei.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The GSMA Open-Telco LLM Benchmarks community enables mobile network operators, AI researchers and developers to submit use cases, datasets and models for evaluation.&nbsp;A standardised benchmarking framework&nbsp;ensures that all AI models are evaluated against real-world challenges in areas such as telecoms domain knowledge, mathematical reasoning, energy consumption and safety.&nbsp;The resulting benchmarks will be hosted on Hugging Face to ensure transparency and encourage community engagement.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Mobile network operators, vendors, startups and researchers are now encouraged to contribute, by submitting interest and LLM telcos use cases, to\u00a0<a href=\"mailto:aiusecase@gsma.com\" target=\"_blank\" rel=\"noreferrer noopener\">aiusecase@gsma.com<\/a>\u00a0and for more information visit\u00a0<a href=\"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/gsma-open-telco-llm-benchmarks\/\" target=\"_blank\" rel=\"noreferrer noopener\">www.gsma.com\/get-involved\/gsma-foundry\/gsma-open-telco-llm-benchmarks<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/www.mwcbarcelona.com\/articles\/gsma-open-telco-llm-benchmarks-launches-advance-ai-telecoms?utm_campaign=gsma_blog&amp;utm_medium=social&amp;utm_source=linkedin\"><strong>Read the full press release here<\/strong><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>New community provides an open-source framework to assess large language models for capability, energy efficiency and safety Read the full press release here 25 February 2025, London&nbsp;\u2013 The&nbsp;GSMA Foundry, the GSMA\u2019s innovation hub, today announced the launch of&nbsp;GSMA Open-Telco LLM Benchmarks, an open-source community aimed at improving the performance of large language models (LLMs) for [&hellip;]<\/p>\n","protected":false},"author":9,"featured_media":10324,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_oasis_is_in_workflow":0,"_oasis_original":0,"_oasis_task_priority":"","footnotes":""},"categories":[125],"tags":[],"algolia_discover_type":[],"class_list":["post-10323","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-intelligence"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v24.4 (Yoast SEO v27.9) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>GSMA Open-Telco LLM Benchmarks Launches to Advance AI in Telecoms - GSMA Foundry<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"GSMA Open-Telco LLM Benchmarks Launches to Advance AI in Telecoms\" \/>\n<meta property=\"og:description\" content=\"New community provides an open-source framework to assess large language models for capability, energy efficiency and safety Read the full press release here 25 February 2025, London&nbsp;\u2013 The&nbsp;GSMA Foundry, the GSMA\u2019s innovation hub, today announced the launch of&nbsp;GSMA Open-Telco LLM Benchmarks, an open-source community aimed at improving the performance of large language models (LLMs) for [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/\" \/>\n<meta property=\"og:site_name\" content=\"GSMA Foundry\" \/>\n<meta property=\"article:published_time\" content=\"2025-02-27T10:36:36+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-12-12T16:00:10+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-content\/uploads\/2025\/02\/LLM-press-release.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Nicola\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@gsma\" \/>\n<meta name=\"twitter:site\" content=\"@gsma\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nicola\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"GSMA Open-Telco LLM Benchmarks Launches to Advance AI in Telecoms - GSMA Foundry","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/","og_locale":"en_US","og_type":"article","og_title":"GSMA Open-Telco LLM Benchmarks Launches to Advance AI in Telecoms","og_description":"New community provides an open-source framework to assess large language models for capability, energy efficiency and safety Read the full press release here 25 February 2025, London&nbsp;\u2013 The&nbsp;GSMA Foundry, the GSMA\u2019s innovation hub, today announced the launch of&nbsp;GSMA Open-Telco LLM Benchmarks, an open-source community aimed at improving the performance of large language models (LLMs) for [&hellip;]","og_url":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/","og_site_name":"GSMA Foundry","article_published_time":"2025-02-27T10:36:36+00:00","article_modified_time":"2025-12-12T16:00:10+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-content\/uploads\/2025\/02\/LLM-press-release.png","type":"image\/png"}],"author":"Nicola","twitter_card":"summary_large_image","twitter_creator":"@gsma","twitter_site":"@gsma","twitter_misc":{"Written by":"Nicola","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/#article","isPartOf":{"@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/"},"author":{"name":"Nicola","@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/#\/schema\/person\/26f709086056ec8c41f00429223058fc"},"headline":"GSMA Open-Telco LLM Benchmarks Launches to Advance AI in Telecoms","datePublished":"2025-02-27T10:36:36+00:00","dateModified":"2025-12-12T16:00:10+00:00","mainEntityOfPage":{"@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/"},"wordCount":414,"commentCount":0,"publisher":{"@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/#organization"},"image":{"@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/#primaryimage"},"thumbnailUrl":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-content\/uploads\/2025\/02\/LLM-press-release.png","articleSection":["Artificial Intelligence"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/","url":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/","name":"GSMA Open-Telco LLM Benchmarks Launches to Advance AI in Telecoms - GSMA Foundry","isPartOf":{"@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/#primaryimage"},"image":{"@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/#primaryimage"},"thumbnailUrl":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-content\/uploads\/2025\/02\/LLM-press-release.png","datePublished":"2025-02-27T10:36:36+00:00","dateModified":"2025-12-12T16:00:10+00:00","breadcrumb":{"@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/#primaryimage","url":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-content\/uploads\/2025\/02\/LLM-press-release.png","contentUrl":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-content\/uploads\/2025\/02\/LLM-press-release.png","width":1200,"height":628,"caption":"Digital binary code overlays a cityscape at dusk. The flowing ones and zeros create the illusion of data streaming through city streets, symbolizing digital communication. Blurred city lights and skyscrapers form the background, highlighting the blend of technology and urban life."},{"@type":"BreadcrumbList","@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/artificial-intelligence\/gsma-open-telco-llm-benchmarks-launches-to-advance-ai-in-telecoms\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/"},{"@type":"ListItem","position":2,"name":"GSMA Open-Telco LLM Benchmarks Launches to Advance AI in Telecoms"}]},{"@type":"WebSite","@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/#website","url":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/","name":"GSMA Foundry","description":"","publisher":{"@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/#organization","name":"GSMA Foundry","url":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/#\/schema\/logo\/image\/","url":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-content\/uploads\/2024\/06\/GSMA-Logo-Red-RGB_square.jpg","contentUrl":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-content\/uploads\/2024\/06\/GSMA-Logo-Red-RGB_square.jpg","width":600,"height":600,"caption":"GSMA Foundry"},"image":{"@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/gsma"]},{"@type":"Person","@id":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/#\/schema\/person\/26f709086056ec8c41f00429223058fc","name":"Nicola","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/cb9419751dd1f000eb04eef857447d4cfb304dd39358b796378bd657b1fa4272?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/cb9419751dd1f000eb04eef857447d4cfb304dd39358b796378bd657b1fa4272?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/cb9419751dd1f000eb04eef857447d4cfb304dd39358b796378bd657b1fa4272?s=96&d=mm&r=g","caption":"Nicola"}}]}},"featured_image_url":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-content\/uploads\/2025\/02\/LLM-press-release.png","_links":{"self":[{"href":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-json\/wp\/v2\/posts\/10323","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-json\/wp\/v2\/comments?post=10323"}],"version-history":[{"count":3,"href":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-json\/wp\/v2\/posts\/10323\/revisions"}],"predecessor-version":[{"id":11442,"href":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-json\/wp\/v2\/posts\/10323\/revisions\/11442"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-json\/wp\/v2\/media\/10324"}],"wp:attachment":[{"href":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-json\/wp\/v2\/media?parent=10323"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-json\/wp\/v2\/categories?post=10323"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-json\/wp\/v2\/tags?post=10323"},{"taxonomy":"algolia_discover_type","embeddable":true,"href":"https:\/\/www.gsma.com\/get-involved\/gsma-foundry\/wp-json\/wp\/v2\/algolia_discover_type?post=10323"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}