30% off Lifetime Premium Access expires December 2nd!

30% off Lifetime Premium Access expires Dec 2nd!

Your Dashboard

Interview Coaching

Mock Interviews

1:1 Mentorship

Salary Negotiation

Practice

Learn

System Design

ML System Design

Code

Behavioral

Salary Negotiation

Interview Guides

Blog

Community

Interview Questions

Interview Experiences

Peer System Design Library

Ask The Community

Discord

Refer a Friend

Pricing

Become a Coach

Sign in / Sign up

Search

⌘K

Pricing Become a Coach

Search

⌘K

Get Premium

Design a Web Crawler with Limited Communication

Design a distributed web crawler using 10,000 machines to crawl 1 billion URLs without any centralized components (no master nodes, databases, queues, or cloud services). Focus on decentralized coordination, load distribution, and handling communication constraints between nodes.

Asked at:

Meta

Bloomberg