Why www.clarin.eu is moving behind Anubis

General
,
1

In recent times, www.clarin.eu has been experiencing a dramatic increase in unwanted automated traffic. These aren’t search engine crawlers or legitimate research tools, they’re high‑volume scrapers that:

  • Ignore robots.txt
  • Bypass rate limits
  • Rotate through residential proxy networks
  • Generate traffic spikes that degrade performance for real users
  • Because we cannot scale our infrastructure indefinitely to feed pathological bots, we’re introducing a protective layer: Anubis.

What Anubis Does
Anubis performs a lightweight verification step to distinguish legitimate human traffic from abusive automated traffic. It does not track users, does not require solving captchas, and does not block normal browsing.

What Users Will See
When accessing www.clarin.eu, users will encounter a short verification page:

  • It appears once per browser session
  • It completes automatically in most cases
  • It adds only a brief delay
  • It ensures the site remains responsive under load

After that, users continue to the website as usual.

This Shouldn’t Worry You

  • No personal data is collected beyond what’s needed for traffic validation
  • No change to your permissions or access
  • No impact on API usage or research workflows
  • No persistent tracking or fingerprinting
  • No change to the content or structure of the website

If Something Goes Wrong
If the verification page loops, blocks you, or behaves unexpectedly, please contact CLARIN support via sysops@clarin.eu. We can whitelist legitimate use cases or investigate misclassifications.

Thank You!
This step helps us keep the CLARIN website stable, fast, and accessible for the community, especially during peak usage periods. We appreciate your understanding as we take measures to protect our shared infrastructure.

1 Like
2

For the most part, the ERCC has been relatively unaffected by “unwanted automated traffic” and rather simplistic ‘rate limit’ rules did the job; however, since last Saturday, the ERCC has been affected by a large influx of automated requests. The volume of requests could not be handled properly by the service and, as a result, denied requests led to database timeouts and errors.

The requests were not easy to block with our existing measures. They came from many different IP addresses, locations, user-agent signatures, and request patterns. At the same time, they were all targeting the discover endpoint. Since this endpoint is central to the repository interface, simply disabling it would effectively make large parts of the site unusable.

As a more robust mitigation, we have taken a page out of CLARIN-ERIC’s book :folded_hands: :wink: We now also have additional protection in front of the (old) CLARIN-DSpace service in place. Technically, the site is now routed through SWAG + CrowdSec + nginx-crowdsec-bouncer, and the Anubis Web AI Firewall Utility. These tools help detect and slow down automated or suspicious traffic before it reaches the repository application and database - as already described in the post above (for Anubis)…

Whether CrowdSec is really necessary, we will see - we will continue to monitor the situation and adjust the configuration as needed. For interested people, our configuration is available as a docker-compose stack (that needs to go in front of - in our case - another nginx+shibboleth and the actual dspace service):

1 Like
3

Thanks a lot for this report @egon. The details on your setup are definitly useful for all of us. Please keep us posted on your results.

On our side, we are observing a drop on DB heavy queries of about 60%, so we could again relax all rate limiting and geoblock systems we have in place. We also observe the same behavior of IP rotation you reported, so we believe these come residential proxy IP pools.

1 Like