Design a Web Crawler

Design a web crawler that takes seed URLs as input, discovers and crawls linked pages, extracts image URLs from web pages, and handles duplicate URL filtering while scaling to enterprise-level requirements.

Asked at:

Atlassian