MJ12Bot

Majestic is a UK-based specialist search engine company serving hundreds of thousands of businesses across more than 60 countries and in 13 languages. It develops tools and data services that help people understand the structure and changing relationships of the web, independently of consumer-focused search engines. Its distributed crawler, MJ12bot, gathers information about links between websites rather than caching web pages or personal data.

This web graph supports Majestic’s backlink index and Site Explorer, a dedicated tool for investigating links pointing to websites. Website owners can use Majestic. com to explore data about their own sites, while the company also makes its index available for keyword and website searches and to other technologies.

Majestic’s work includes research into building a full-text search engine, with research partly funded through commercialisation. MJ12bot is presented as a transparent crawler that identifies itself and respects robots. txt and crawl-delay instructions.

The broader community project behind the crawler is associated with Majestic-12, and its distributed approach allows people with broadband connections to contribute to web crawling