Search
⌘K

Design a Web Crawler with Limited Communication

Design a distributed web crawler using 10,000 machines to crawl 1 billion URLs without any centralized components (no master nodes, databases, queues, or cloud services). Focus on decentralized coordination, load distribution, and handling communication constraints between nodes.

Asked at:

Meta

Bloomberg

Bloomberg