A European HPC-centric Benchmark Framework

The HORIZON-EUROHPC-JU-2024-BENCHMARK-05 call, titled "A European HPC-centric Benchmark Framework," is an upcoming funding opportunity as part of the Horizon Europe program. This initiative is focused on developing a comprehensive benchmarking framework and suite for high-performance computing (HPC) and artificial intelligence (AI) systems. The goal is to enhance decision-making in the procurement and operation of exascale and post-exascale supercomputers.

Eligible applicants include national HPC centers, research and academic institutions focused on HPC, standardization bodies, and public or private entities involved in significant HPC resource procurement and operation. However, for-profit entities with business models focused on HPC hardware and software are not eligible due to potential conflicts of interest.

The funding type is specifically a grant under the Horizon Europe Research and Innovation Actions (HORIZON-RIA) framework, with budgets ranging from €500,000 to €1 million per project. The total budget for the initiative is set at €2 million, with only one project expected to receive funding. Co-funding is a requirement, with an EU-funding rate of up to 50% of eligible costs.

The call is framed as a single-stage application process, scheduled to open for submissions on December 11, 2025, with a deadline of March 24, 2026. Proposals must be comprehensive, covering a range of performance metrics for various benchmarks, including microbenchmarks and application benchmarks, while also ensuring energy efficiency metrics are integrated.

Key outcomes sought include the establishment of a production-quality benchmarking framework and a well-maintained benchmarking suite that can effectively assess the performance of HPC and HPC-AI systems. The opportunity emphasizes collaboration and community-driven development to ensure sustainability and relevance going forward. All participating consortia are required to show complementary expertise in both HPC and AI domains while fostering strategic partnerships aligned with European technological interests.

Overall, this funding opportunity is strategically significant for establishing a unified benchmarking standard that can enhance procurement decision-making and operational optimization in European supercomputing, thereby bolstering the region's competitive standing in the global HPC landscape.

The HORIZON-EUROHPC-JU-2024-BENCHMARK-05 call, titled "A European HPC-centric Benchmark Framework," is part of the Horizon Europe (HORIZON) program. It aims to fund Research and Innovation Actions (RIA) to enhance decision-making in the procurement and operation of exascale, post-exascale supercomputers, and supercomputers with dedicated AI capabilities. The call emphasizes the development of a unified, extensible, and well-documented benchmarking framework and suite.

The expected outcomes of the project include: Enhanced decision-making through comprehensive system comparisons, aiding in the procurement of advanced supercomputers. Competent HPC application developers and end-users capable of selecting systems that best meet their needs, considering accuracy and cost. Improved operation and fine-tuning of HPC and HPC-AI systems, leading to better performance, throughput, energy optimization, and user experience. A unified, extensible, and well-documented benchmarking framework with common standards, versioning, and control. A well-maintained and continuously updated benchmarking suite for exascale and post-exascale HPC, including applications and AI models.

The scope of the call is divided into two main objectives:

A. Deployment of a Benchmarking Framework: This involves creating a framework for designing, developing, and executing exascale HPC and HPC-AI benchmarks. The framework should: Offer a fine-grained and fair comparison methodology among different HPC systems, ensuring reproducibility, repeatability, and replicability of metrics. Define precise performance metrics with a clear focus on energy-related performance indicators. Standardize all benchmarking input and output formats. Collect and report all benchmarking results while offering statistically sound result analyses. Ensure that all benchmarks are executable on the respective target environment(s). Offer a standardized structured workflow capturing and streamlining the entire benchmarking process. Offer a standardized repository with transparent version control. Provide a reference implementation for each benchmark. Use a EuroHPC reference system, where applicable, to normalize the performance metrics produced by the benchmarking suite. Be of production quality and ready to assess all EuroHPC supercomputers and supercomputers with AI capabilities. Provide all required templates with relevant input data to properly execute the benchmarking suite on every EuroHPC system. The benchmarking framework and its workflows should be realized in a software implementation providing a dynamic workspace for the end-user.

B. Establishing a Comprehensive Benchmarking Suite: This involves creating a suite with performance metrics to measure and assess the performance of HPC and HPC-AI systems at various levels of granularity, including: Microbenchmarks: Focusing on small building blocks of real programs for component developers and system integrators. Examples include dense and sparse linear algebra operations including tensor operations, spectral methods, n-body methods, (un)structured grid methods. Application and workflow benchmarks: Measuring system performance under typical user workloads for end-users, system administrators, and procurement authorities. Examples include CFD, molecular dynamics simulation, numerical weather prediction, atomic scale materials modelling, AI model training, service/inference. Workflow benchmarks account for system-performance effects of integrated scientific workflows. System benchmarks: Offering a comprehensive system performance assessment under conditions where multiple, diverse workloads are concurrently executed, reflecting a realistic, multi-user production environment. An example is running an ensemble of large AI multimodal model training simultaneously with large scale lattice Boltzmann simulations. The benchmarking suite is expected to: Be generally hardware agnostic. Provide documentation for developers and end-users. Catalogue well-established benchmarks of both technical areas. Continuously update the portfolio with novel benchmarks of both technical areas. Ensure that each benchmark produces at least one metric, such as time-to-solution, throughput, or utilization. Define reliable and appropriate common metrics to compare different architectures based on pre-defined criteria (e.g., efficiency). Ensure comprehensive coverage of relevant workloads and performance aspects to meet the diverse needs of the European HPC-AI community. Offer comprehensive coverage of contemporary and upcoming architectures, utilizing current representative and upcoming workloads from the HPC and HPC-AI domains. Be application-oriented, reflecting actual use-patterns, use-cases, and diverse workloads in all three technical areas (exascale HPC and HPC-AI). Ensure the scalability of each benchmark by identifying relevant scale parameters.

Proposals should thoroughly justify the selection of each benchmark and performance metric, aligning them with the requirements and priorities of the European HPC-AI landscape. Integration of existing benchmarks is encouraged, provided there are agreements with benchmark owners and compatibility with licensing conditions. A strategy for ensuring the sustainability and availability of the benchmarking suite beyond the project's duration is required, focusing on transforming it into a community-driven effort. An IP plan targeting industry needs is also necessary. The consortium will coordinate with international collaborators to establish common benchmarking standards and propose a detailed strategic development roadmap. The consortium will actively engage with industry and research communities through workshops and feedback loops.

Specific requirements for the proposals include: Building on existing European benchmarking efforts and initiatives in HPC, such as the Unified European Application Benchmark Suite (UEABS). Specifying workloads in an implementation-independent way. Defining datasets and quality criteria. Detailing benchmark specifications to test supercomputing systems against state-of-the-art metrics. Addressing relevant metrics of the target system per technical area (Performance, Scalability, Resource utilization, Energy efficiency). Contributing to relevant standardization efforts, including security standards such as ISO 27001 and AI standards like ISO 22989 and 23053. Defining a methodology to deal with legacy applications and systems. Defining HPC utilization metrics, including breakdown by benchmarking area and corresponding KPIs. Collecting and analyzing user feedback to evaluate the benchmark suite's efficiency and fairness. Defining a mechanism to monitor the benchmarking framework and pool existing benchmark suites in a common data repository. Enabling continuous improvement through automated integration and testing workflows. Defining a mechanism for extending the benchmark suite. Preparing extensive user documentation. Demonstrating complementary expertise regarding the two main technical areas. Making the benchmarking framework and suite available under the European Union Public Licence (EUPL). Defining the benchmarking framework through a consensus among stakeholders. Addressing all technical and legal aspects at the proposal stage.

The admissibility conditions include a proposal page limit of 70 pages, as described in General Annex A of the Horizon Europe Work Programme 2023-2025. Eligible countries are detailed in Annex B of the Work Programme General Annexes, with specific provisions for non-EU/non-Associated Countries outlined in the Horizon Europe Programme Guide. Other eligible conditions are described in General Annex B of the Horizon Europe Work Programme 2023-2025. Eligible participants include National HPC centers, Research and academic institutions focused on HPC, Standardisation bodies, and Other public and private entities regularly procuring, operating or using significant HPC resources, if clearly explained and duly justified in the proposal, and provided no conflict of interest exists. For-profit entities with business models around hardware and software for HPC are generally not eligible.

Financial and operational capacity and exclusion criteria are described in Annex C of the Work Programme General Annexes. The evaluation and award process is described in General Annex F of the Horizon Europe Work Programme 2023-2025, with the granting authority funding a maximum of one project. Award criteria are described in Annex D of the Work Programme General Annexes. The submission and evaluation processes are described in General Annex D of the Horizon Europe Work Programme 2023-2025. The indicative timeline for evaluation and grant agreement is described in Annex F of the Work Programme General Annexes. The EU-funding rate for eligible costs will be up to 50%, as an exception from General Annex G of the Horizon Europe Work Programme. The rules are described in General Annex G of the Horizon Europe Work Programme 2023-2025.

The documents are described in General Annex E of the Horizon Europe Work Programme 2023-2025. Application form templates are available in the Submission System, including standard forms for HE RIA, IA, CSA, RI, PCP, PPI, COFUND, FPA, MSCA DN/PF/SE/COFUND, ERC STG/COG/ADG/POC/SYG, and EIC PATHFINDER CHALLENGES/OPEN/TRANSITION. Evaluation form templates will be used with necessary adaptations, including standard forms for HE RIA/IA, CSA, PCP/PPI, COFUND, FPA, MSCA, and EIC PATHFINDER CHALLENGES/OPEN/TRANSITION/Accelerator. Guidance documents include the HE Programme Guide, Model Grant Agreements (MGA), and call-specific instructions. Additional documents include the HE Main Work Programme 2023–2025, HE Programme Guide, HE Framework Programme 2021/695, HE Specific Programme Decision 2021/764, EU Financial Regulation 2018/1046, Rules for Legal Entity Validation, LEAR Appointment and Financial Capacity Assessment, EU Grants AGA, Funding & Tenders Portal Online Manual, and Funding & Tenders Portal Terms and Conditions.

The total budget for the HORIZON-EUROHPC-JU-2024-BENCHMARK-05 call is EUR 2,000,000. The opening date for submissions is planned for 11 December 2025, with a deadline of 24 March 2026. The expected contribution per project ranges from EUR 500,000 to EUR 1,000,000, and the indicative number of grants is 1.

This funding opportunity aims to establish a comprehensive and sustainable benchmarking framework for European HPC and HPC-AI systems. It seeks to create tools and methodologies that will enable better procurement decisions, system optimization, and application development, ultimately enhancing the European HPC landscape and ensuring its competitiveness in the global arena. The project emphasizes collaboration, standardization, and community involvement to create a lasting impact on the field.

Eligible Applicant Types: The eligible applicant types for this opportunity are national HPC centers, research and academic institutions focused on HPC, standardisation bodies, and other public and private entities regularly procuring, operating, or using significant HPC resources, if clearly explained and duly justified in the proposal, and provided no conflict of interest exists. For-profit entities with business models around hardware and software for HPC are generally not eligible for participation due to potential conflicts of interest.

Funding Type: The funding type is a HORIZON Research and Innovation Action (HORIZON-RIA), which is a grant. The type of Model Grant Agreement (MGA) is a HORIZON Action Grant Budget-Based [HORIZON-AG].

Consortium Requirement: A consortium is required for this opportunity.

Beneficiary Scope (Geographic Eligibility): The geographic eligibility includes EU and non-EU/non-Associated Countries that have made specific provisions for funding their participants in Horizon Europe projects. Details can be found in the Horizon Europe Programme Guide.

Target Sector: This program targets the digital, industry, and space sectors, specifically focusing on high-performance computing (HPC) and artificial intelligence (AI). It aims to enhance the European HPC-AI landscape.

Mentioned Countries: The opportunity is open to EU member states and associated countries, as well as non-EU/non-Associated Countries that have specific provisions for funding their participants in Horizon Europe projects.

Project Stage: The project stage is focused on development and deployment, aiming to create a production-quality benchmarking framework and suite ready for assessing EuroHPC supercomputers and supercomputers with AI capabilities.

Funding Amount: The total budget for the topic is EUR 2,000,000. The indicative contributions range from EUR 500,000 to EUR 1,000,000 per project. The granting authority can fund a maximum of one project.

Application Type: The application type is a single-stage call for proposals.

Nature of Support: Beneficiaries will receive money in the form of a grant to cover eligible costs, with an EU-funding rate of up to 50% of the eligible costs.

Application Stages: The application process is a single-stage process.

Success Rates: The indicative number of grants is 1.

Co-funding Requirement: Co-funding is required, as the EU-funding rate is up to 50% of the eligible costs.

Summary:

This Horizon Europe call, titled "A European HPC-centric Benchmark Framework," aims to establish a comprehensive and sustainable benchmarking framework and suite for exascale and post-exascale high-performance computing (HPC) and HPC-AI systems. The project seeks to enhance decision-making in the procurement of supercomputers, aid application developers in system selection, and improve the operation and optimization of HPC systems. The framework should offer a fair comparison methodology, standardized workflows, and a transparent repository. The benchmarking suite should cover microbenchmarks, application and workflow benchmarks, and system benchmarks, reflecting diverse workloads and architectures. The successful project will receive between EUR 500,000 and EUR 1,000,000, with a total budget of EUR 2,000,000 available, and will need to ensure the sustainability and community-driven development of the benchmarking suite. Eligible applicants include national HPC centers, research institutions, standardization bodies, and other entities involved in HPC resource procurement and operation, excluding for-profit entities with hardware and software business models due to potential conflicts of interest. The call follows a single-stage application process, with the submission deadline on March 24, 2026. The EU-funding rate is up to 50% of the eligible costs, implying a co-funding requirement.