About our crawler.
Quagga runs a crawler that reads law firms' own public websites for firm size.
It identifies itself as QuaggaFirmBot.
How to keep it off your site.
Exclude QuaggaFirmBot in your robots.txt, or write to us.
Our address is matthew.maus@mqsale.com.
How it behaves.
- It reads public pages only.
- It follows each site's robots.txt. If it cannot read robots.txt, it does not crawl.
- It makes at most one request to a site every two seconds, one at a time, and waits longer if robots.txt asks it to.
- It stops at a login, a paywall, a block or a challenge. It does not retry or change identity, and it keeps no cookies.
- It may visit again on a later day.
Write to us and it stays away.
We would rather hear from you than be unwelcome.