In September 2022 the project OpenWebSearch.eu has officially kicked-off: OpenWebSearch.EU is the first project the EU has funded to get tomorrow’s web search up and running.
The project has its origins in concerns over the imbalance of the search engine market. Despite being a backbone of our digital economy, web search is dominated and limited by a few gatekeepers like Google, Microsoft, Baidu or Yandex. Thus, information as public good, with free, unbiased and transparent access is not under public control anymore. This imbalance endangers democracy and limits the innovative potential of Europe’s research landscape and economy.
Over the next three years the researchers will develop the core of a European Open Web Index (OWI) as a basis for a new Internet Search in Europe. In addition, the project will set the foundation for an open and extensible European open Web Search and Analysis Infrastructure (OWSAI), based on Europe’s values, principles, legislation and standards.
Project Partners: 14
Time-frame: 3 years
Kick-off: September 2022
Funding: 8,5 million euros
Grant agreement no.: 101070014
Through the Open Search Foundation and its wider community we are already engaged in making OpenWebSearch.EU a reality. Also pay attention to the Open Calls of the project. You can also read more about the current status of the project here.
Chat Noir and Args – Argument Search
ChatNoir enables retrieval research by providing researchers with easy and ready-to-use access to billions of indexed web pages, allowing for reproducible experiments on them. Unlike traditional search engines such as Google or DuckDuckGo, ChatNoir does not index the live web but fixed snapshots of it while being transparent about its retrieval decisions.
… for web masters
The “Open Console” infrastructure attempts to combine factual knowledge about websites into a network of cooperation. It exchanges detailed information about websites and domain-names, on a huge scale. Website owners can see what third-parties distribute about their site, and add, correct or annotate that data. Open Console is the open collaborative answer to Google’s Search Console, building one single interface for many data collectors.
The “Crawl Pipeline” gives access to data which has been collected by Web Crawlers (the automated processes which collect the content of websites). These crawlers produce huge WARC- files. Instead of everyone processing these huge files with their own crawlers, the filtering pipeline will provide a multitude of participants simultaneously with customized filtered extracts of these files as needed.
Web search has become an essential technology and commodity, driving not only future innovations but forming a backbone for our digital economy.
Currently few non-European gatekeepers control Web search. This situation creates a biased, one-sided information access centred around economic success rather than the needs of citizens or European values and jurisdiction. This one-sided ecosystem puts pressure on many small Web contributors from science, economy, art, culture, media and society requiring them to optimize their content for a few gatekeepers. A system greatly benefiting the gatekeepers thereby creates a vicious cycle, which leads to locked-in effects and a closed search engine market.
To promote an open human-centred search engine market and provide a true choice to users when selecting search engines based on their own preferences, OpenWebSearch.EU proposes to develop and pilot the core of a European Open Web Index (OWI) and the foundation for an open and extensible European open Web Search and Analysis Infrastructure (OWSAI).
Our approach is based on four objectives, namely
The envisioned infrastructure will not only contribute to Europe’s sovereignty for navigating and searching the web, it will also empower Europe’s researchers, innovators and business to systematically tap into the Web as business and innovation resource, without paying huge upfront costs. This will be particularly crucial for future AI innovations and relevant for other European infrastructure like e.g. the European Open Science Cloud, GAIA-X.
The open web index and its creation is the core of the project. The web index contains data and metadata in a structured form derived from the analysis of web documents (6).
To create such an index, web documents are systematically retrieved from the web (1) and stored on a server (2). A list of initial web documents serves as a starting point, then all links found in the source web documents and in additional web documents are used to traverse search and store documents. Blacklists guide the capture process by excluding websites according to certain characteristics.
The content analysis of the web documents consists of several steps. The text content is extracted (3), the metadata is processed (4), and an analysis is performed to obtain certain features, including qualitative, ethical and legal aspects (5). All this derived information is stored in the index together with the respective web addresses of the analysed web documents (6).
In order to enable the search for information, the index is deployed in two different ways to make it accessible (7). The entire index is deployed in a computing centre, and a part of the index can be downloaded and provided on a personal server.
Data products or search applications are created using and demonstrating the open web index.
Search applications enable a user to search for general information or domain-specific information for a particular purpose, e.g. for restaurants in the vicinity. For this purpose, the index is queried and a selection of matching web documents is retrieved (a). The documents found are then ranked based on purpose-specific characteristics, user preferences and usage data (b).
The search application provides the user with the search functionality via a user interface where the result is presented according to the application purpose (c). The user is taken into account by specifically addressing cognitive aspects to better perform and understand the search process (d).
The data products are knowledge representation models derived from the index, e.g. knowledge graphs containing structured conceptual knowledge or AI language models capable of text generation. Applications can use these components for a variety of purposes.
Openness is one of the key concepts of this technology. Not only can the web index be used by external applications, but any of the modules described above can be enriched. In particular, new data products or search applications can be created by other parties using the open web index.
Methodologically OWS follows a mixture of conceptual considerations, algorithmic innovations and technology developments. We structure our research along the data processing chain for search and discov-ery systems while considering different stakeholders and services. The following figure gives an overview of the differ-ent components, the envisioned chain, corresponding stakeholders as well as the resulting data products.
