Playwright for Python

Playwright for Python controls real browsers (Chromium, Firefox and WebKit) from Python code. Because it runs an actual browser, it can load pages that build their content with JavaScript, click and type, and take screenshots, which makes it a common choice for scraping sites that a plain HTTP request can’t read. It is also widely used for end-to-end testing, with a pytest plugin.

It offers both a synchronous and an asynchronous API, so it fits asyncio projects, and it runs headless by default. Its documentation lists auto-waiting among its core guides, so scripts wait for elements to be ready instead of relying on fixed sleeps.

See Python Web Scraping Tools Compared for when a browser is worth the extra weight.