MANTELET Gregory

Software engineer at CDS (Strasbourg, France) working mostly on Simbad and contributing to the IVOA.


Sessions

11-02
10:00
9min
VizieR versus increasing bot scraping
MANTELET Gregory, Gilles Landais, Fernique

The CDS VizieR service, like many Data Centers, is facing a dramatic increase in bot crawling. These are millions of queries each day, which pollute and impact the VizieR service. This massive and blind harvesting targets static and dynamic resources and is generally controlled by AI to populate knowledge bases. Log analysis shows many queries, executed by a wide range of IPs, all of which look legitimate without any obvious AI marker: it becomes harder to detect their origin and to manage the flow.
Maintaining good availability for users requires redefining data access management. This is often done urgently, but in the background, the CDS is investigating access management for the long term. In this poster, we present the solutions used, the solution implemented today, and those planned.

Usability, accessibility and security in astronomy software
Seminar Room 2
11-04
17:30
60min
How to cope with the rapidly increasing number of aggressive incoming requests?
Simon O'Toole, MANTELET Gregory, Christophe Arviset, Gilles Landais

For a few years now, some astronomical data centers have observed an increasing number of incoming requests. This usually indicates that a service is becoming more popular and useful. However, the number and rate of these requests is so high that the services are becoming increasingly overwhelmed, ending up by a saturation and complete unavailability. Worryingly, this trend is speeding up. Consequently, services are becoming unavailable more often and more quickly than before.

The recently observed growth in activity actually hides a different user profile. Until now, humans and indexing crawlers were the only users of our services. Now, there is a new category of users: AI bots. Unlike the other users, they are much more aggressive. There are two main reasons for this. First, they rely on cloud infrastructures. This allows them to massively parallelize the web scrapping operations and use an enormous number of different IP addresses, making them impossible to block with the usual methods. Second, they do not adhere to any best practices, such as the well-known robots.txt, which indexing crawlers (search engines) read and follow.

Through this BoF, CDS (France), ESAC (Spain), and Data Central (Australia) aim to discuss beyond the negative impact of this recent practice on their infrastructures and explore strategies to maintain the high availability and quality of their services while allowing AI to reasonably interact with them. Everyone is welcome to join the discussion and share experiences, solutions (e.g. mirrors, proxy, authentication, ban with well-identified and shared IP ranges), and, more generally, ideas.

AI as a tool for data discovery and data management
Seminar Room 3