Python Web Scraping Tools Compared: Which to Use

Beautiful Soup, Scrapy, Playwright and Selenium solve different scraping problems. See what each does, when it fits, and how to pick.

For most scraping jobs, start with the simplest tool that works: fetch the page with Requests or HTTPX and parse it with Beautiful Soup. Move to Scrapy when you are crawling lots of pages, and to Playwright or Selenium only when the data needs a real browser to appear.

At a glance

Beautiful Soup: parse a page you already have

Beautiful Soup turns HTML into a tree you can search. It does not fetch anything, so you combine it with an HTTP client:

import httpx
from bs4 import BeautifulSoup

html = httpx.get("https://example.com").text
soup = BeautifulSoup(html, "html.parser")

for link in soup.select("a[href]"):
    print(link.get_text(strip=True), link["href"])

It uses the best parser you have installed (lxml, then html5lib, then Python’s built-in html.parser), and you can choose one explicitly as above. If you are unsure which HTTP client to use, see HTTPX vs Requests.

Choose it for one-off scripts and pages whose content is already in the HTML. Its limit is that it only reads markup: no JavaScript, no crawling, no retries, no scheduling.

Scrapy: when you are crawling, not just scraping

Scrapy is a framework for crawling websites and extracting structured data. You write a spider, and Scrapy handles the requests, concurrency, retries and exports. This is the example spider from its own documentation:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/tag/humor/"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "author": quote.xpath("span/small/text()").get(),
                "text": quote.css("span.text::text").get(),
            }
        next_page = response.css('li.next a::attr("href")').get()
        if next_page is not None:
            yield response.follow(next_page, self.parse)

Run it with scrapy runspider quotes_spider.py -o quotes.jsonl and the results are written to a file. It follows pagination and links for you, and it supports asyncio coroutines.

Choose it when the job is “get everything from this site” and you want throttling, retries and exports built in. The cost is a framework to learn, which is more than a ten-line script needs.

Playwright: a real browser, with less friction

When a page builds its content with JavaScript, the HTML you download is nearly empty. Playwright launches an actual browser (Chromium, Firefox or WebKit), lets the scripts run, and gives you the result:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://example.com")
    print(page.title())
    browser.close()

It has both a sync and an async API, runs headless by default, and its documentation lists auto-waiting among its core guides, so you wait for an element rather than guessing a delay. It is also a popular end-to-end testing tool.

Choose it for pages that need JavaScript, or when you need to click, type or log in. The cost is speed and resources: a browser is far heavier than an HTTP request, so don’t use it for pages that don’t need one.

Selenium: the established option

Selenium’s WebDriver controls a real browser through the automation APIs the browser vendors provide. It has been the standard for browser automation for years, so there are a great many tutorials and Stack Overflow answers.

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://example.com")
print(driver.title)
driver.quit()

You don’t need to download browser drivers yourself: Selenium Manager, which ships with every Selenium release, handles it. Choose Selenium if you already know it or have existing Selenium code. For a new scraping project, Playwright is usually the smoother start, but both can do the job.

How to choose

1. View the page’s source, or fetch it with HTTPX. If the data you want is in the HTML, use Requests or HTTPX plus Beautiful Soup. You’re done.
2. Is it a big crawl? Many pages, pagination, retries, exports: use Scrapy.
3. Is the data missing from the downloaded HTML? Check your browser’s network tab first: the page may call a JSON API you can use directly, which is faster than any browser. If not, use Playwright or Selenium.
4. Mixing is normal. A common setup renders a page in Playwright and passes the HTML to Beautiful Soup, or uses Scrapy for the crawl and a browser only for the awkward pages.

Scrape politely

Check the site’s robots.txt and terms of service before you start. Slow your requests down so you don’t overload the server, identify your scraper honestly, and avoid personal data. Scrapy has built-in throttling you can turn on. If a site offers an API or a data download, use that instead of scraping it.