Design Wikipedia Crawler

Design a distributed web crawler system that can crawl and store Wikipedia pages at scale using multiple machines or edge devices, with focus on coordination, deduplication, and efficient resource utilization.