The Capabilities of Scrapy

Scrapy is an open-source framework designed for web crawling that streamlines the process of extracting data from various websites. With its user-friendly API and built-in tools, it serves as an excellent option for both novices and seasoned developers.

Getting Started: Setup and Initial Use

Before we delve into examples, let’s ensure that Scrapy is installed on your system. You can install it by executing the following command in your terminal:

pip install scrapy

Next, let’s create a basic Scrapy project. In your terminal, run:

scrapy startproject myproject

This command sets up a new Scrapy project named "myproject." Now, we can create a spider, which is a script that defines the rules for scraping a specific website.

# File: myproject/spiders/example_spider.py

import scrapy

class ExampleSpider(scrapy.Spider):

name = 'example'

def parse(self, response):

# Extract data here

title = response.css('title::text').get()

paragraphs = response.css('p::text').getall()

# Print the results

print(f"Title: {title}")

print(f"Paragraphs: {paragraphs}")

scrapy crawl example

Executing this command will run the spider, displaying the extracted data in your terminal.

Navigating and Extracting Information

Scrapy greatly simplifies the navigation of HTML and the extraction of content. Let’s examine a slightly more complex example where we scrape quotes from http://quotes.toscrape.com.

# File: myproject/spiders/quotes_spider.py

import scrapy

class QuotesSpider(scrapy.Spider):

name = 'quotes'

start_urls = ['http://quotes.toscrape.com']

def parse(self, response):

# Extract quotes and authors

quotes = response.css('div.quote span.text::text').getall()

authors = response.css('div.quote small::text').getall()

# Print the results

for quote, author in zip(quotes, authors):

print(f"Quote: {quote}")

print(f"Author: {author}")

# Follow pagination links

next_page = response.css('li.next a::attr(href)').get()

if next_page:

yield scrapy.Request(url=next_page, callback=self.parse)

In this example, the spider extracts both quotes and their corresponding authors from the specified URL. The yield scrapy.Request statement allows the spider to follow pagination links, continuing to scrape additional pages.

Storing Extracted Data

Scrapy has built-in capabilities for storing the data it collects. Let’s enhance our quotes spider to save the scraped information in a JSON file.

# File: myproject/spiders/quotes_spider.py

import scrapy

import json

class QuotesSpider(scrapy.Spider):

name = 'quotes'

start_urls = ['http://quotes.toscrape.com']

def parse(self, response):

# Extract quotes and authors

quotes = response.css('div.quote span.text::text').getall()

authors = response.css('div.quote small::text').getall()

# Store data in a JSON file

data = [{'quote': quote, 'author': author} for quote, author in zip(quotes, authors)]

with open('quotes.json', 'a') as file:

json.dump(data, file)

# Follow pagination links

next_page = response.css('li.next a::attr(href)').get()

if next_page:

yield scrapy.Request(url=next_page, callback=self.parse)

In this version, the spider generates a list of dictionaries where each dictionary contains a quote and its author. This data is appended to a JSON file named 'quotes.json.'

Handling Forms and Authentication

Scrapy makes it easy to manage forms and log into websites. Below is an example of a spider that logs into a hypothetical site and retrieves data from a secured area.

# File: myproject/spiders/login_spider.py

import scrapy

from scrapy.http import FormRequest

class LoginSpider(scrapy.Spider):

name = 'login'

def parse(self, response):

# Extract login token

token = response.css('input[name="csrf_token"]::attr(value)').get()

# Log in

yield FormRequest.from_response(

response,

formdata={'username': 'your_username', 'password': 'your_password', 'csrf_token': token},

callback=self.after_login

)

def after_login(self, response):

# Scraping logic for the authenticated area

authenticated_data = response.css('div.authenticated-data::text').get()

print(f"Authenticated Data: {authenticated_data}")

In this example, the spider first retrieves the login token and then uses it to log in through the FormRequest.from_response method. The after_login method contains the logic for scraping data from the authenticated section.