The University of Oldenburg connects its data centre and HPC cluster to OpenWebSearch.eu, demonstrating how academic infrastructure can add to a distributed European web index
A European open web search infrastructure should not depend on a single data centre or a single organisation. Instead, the idea is to connect different institutions, different countries, different kinds of computing resources, who all contribute to the same shared infrastructure. The NordLink project, carried out by the University of Oldenburg following the OpenWebSearch.EU third-party funding call, is a concrete step in that direction: it connects a university’s high-performance computing resources to the already existing OpenWebSearch network.
What NordLink brings to the table
The University of Oldenburg’s contribution is not trivial. The resources committed to the project include 50 terabytes of S3-compatible cloud storage, two dedicated physical servers with 200 terabytes of combined storage, three virtual machines for testing and deployment, and – most notably – access to the university’s HPC cluster. This is a serious piece of computing infrastructure: 161 nodes, over 20,000 CPU cores, 145 terabytes of RAM, and 36 high-end NVIDIA GPUs (including A100 and H100 models) with a combined peak GPU performance exceeding 2 TFlop/s. The storage subsystem provides more than 4 petabytes of capacity.
For context, this is the kind of computing power typically used for large-scale scientific simulations, machine learning training runs, and data-intensive research. Making it available to the OpenWebSearch.eu project beautifully demonstrates that European academic HPC centres can play a meaningful role in search infrastructure – a domain traditionally dominated by commercial tech companies.
The Integration Challenge
The primary technical challenge for NordLink was integrating the university’s resources with the OWS infrastructure through HEAppE, a middleware system designed to provide HPC-as-a-Service capabilities. This middleware allows remote users and automated systems to submit computing jobs to the university’s cluster without needing direct access to the local systems.
The NordLink team deployed HEAppE on both virtual machines and physical servers, set up comprehensive monitoring using Prometheus and Grafana, configured the university’s S3 storage as a data staging area for the project, and provided the IP addresses of both physical servers and virtual machines for whitelisting to enable web crawling. A functional account was created to link the physical infrastructure to the HPC cluster, enabling job submission from the OWS network.
Challenges to consider
The team reported that the documentation for the HEAppE middleware was incomplete and difficult to follow, making deployment more laborious. Notably, the infrastructure provider EXAION reported the same issue in their final report as well.
Why university infrastructure matters for Open Search
European universities collectively operate enormous computing resources. HPC clusters, large-scale storage systems, high-bandwidth network connections, and skilled technical teams exist across hundreds of institutions. Most of this capacity is used for scientific research – climate modelling, genomics, particle physics, engineering simulations. But much of it also has periods of underutilisation, and the skills required to operate it overlap significantly with those needed for a web search infrastructure.
NordLink demonstrates that these resources can be connected to a shared infrastructure with reasonable effort.
What’s Next
Beyond the formal project period, the University of Oldenburg plans to maintain all committed infrastructure for a while – the S3 storage, physical servers, and virtual machines – and to complete the HEAppE integration with the HPC cluster. The team is also considering provisioning additional VMs running search index software such as OpenSearch or Vespa.ai, which would allow the university to host a searchable subset of the Open Web Index locally.
In conjunction with EXAION’s contribution from France, NordLink underpins the kind of infrastructure network that OpenWebSearch.eu is building: a distributed system where European organisations of different types – universities, data centre operators, research institutions – contribute to a shared, sovereign search infrastructure.
To read the full technical report, go here: https://zenodo.org/records/18259771
The project was funded under the OpenWebSearch.EU initiative (Horizon Europe, Grant Agreement 101070014, Call #3).