Scrapy

Scrapy is an application framework for crawling websites and extracting structured data. Instead of a single script, you write spiders: classes that say where to start, how to follow links and how to pull data out of each page. The framework handles the request scheduling, concurrency, retries, throttling and exporting results (JSON lines, CSV and more) that you would otherwise have to build yourself.

It is aimed at crawling many pages reliably, and it can also be used for extracting data from APIs. Scrapy has its own CSS and XPath selectors and supports coroutines and asyncio, and its documentation includes guidance on selecting dynamically-loaded content.

See Python Web Scraping Tools Compared for how it compares with Beautiful Soup, Playwright and Selenium.