Explore the intricate components of a distributed web crawler and optimize its design for efficiency and effectiveness.