1. Seed URLs (Root URLs)

URLs represent links to servers across the internet. The root URL typically has the fewest slashes, indicating that all associated directives fall under it. This URL is crucial for feeding into the web crawler, as it serves as the starting point to extract all its sub-URLs. A strategy for organizing seed URLs into various groups or validating them is often necessary. For instance, one might segment the entire URL space into smaller categories.

2. URL Frontier

The URL frontier's primary role is to track each URL's status. This can be achieved by maintaining a boolean flag for each URL that indicates whether it has been downloaded. Furthermore, the frontier will hold the downloaded content of each URL. This can be effectively managed using a First In First Out (FIFO) queue, where URLs are queued for download and subsequently removed after processing. Queues are a fundamental data structure in computer science; if you are unfamiliar with them, I recommend exploring further.

3. HTML Downloader

This component is straightforward as it handles the downloading of HTML content.

4. DNS Resolver

This function converts a URL into its corresponding IP address.

5. Content Parser

The content parser is essential for validating web pages. Recall from part 1 that avoiding web traps was a key requirement. Each URL must be classified as either valid or invalid. Implementing this function on a separate server from the crawler is advisable for enhanced efficiency.

6. Content Seen Gate

To prevent the storage of duplicate content, this component checks for redundancies. It serves as a data structure that reduces duplication and accelerates processing times. A basic method for verifying if an HTML page is already stored involves a character-by-character comparison with another HTML page. However, this approach can be slow. A more efficient method is to hash each HTML page and compare the hashes. Hashing generates a unique value for any input. If the same input is processed through a hashing algorithm twice, it yields identical results. For those unfamiliar with hashing, I suggest researching this topic.

Stay tuned for part 3, where we will explore additional components and optimization strategies for the system. This tutorial is based on the Systems Design Interview book, which you can find here. Please note that this is an Amazon affiliate link, and I will earn a commission if you purchase the book. I highly recommend it as it has greatly contributed to my understanding and helped me secure a new job.