Skip to content
@scrapy

Scrapy project

An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

Pinned Loading

  1. scrapy scrapy Public

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    Python 65.4k 12.1k

  2. scrapyd scrapyd Public

    A service daemon to run Scrapy spiders

    Python 3.1k 581

  3. parsel parsel Public

    Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

    Python 1.4k 188

  4. w3lib w3lib Public

    Python library of web-related functions

    HTML 422 148

  5. protego protego Public

    A pure-Python robots.txt parser with support for modern conventions.

    DIGITAL Command Language 97 41

  6. itemadapter itemadapter Public

    Common interface for data container classes

    Python 71 27

Repositories

Showing 10 of 35 repositories
  • itemadapter Public

    Common interface for data container classes

    scrapy/itemadapter's past year of commit activity
    Python 71 BSD-3-Clause 27 3 3 Updated Nov 13, 2026
  • scrapy Public

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    scrapy/scrapy's past year of commit activity
    Python 65,365 BSD-3-Clause 12,130 99 (4 issues need help) 72 Updated Nov 13, 2026
  • scrapy-lint Public

    A linter for Scrapy projects.

    scrapy/scrapy-lint's past year of commit activity
    Python 22 MIT 10 19 21 Updated Nov 13, 2026
  • frostwork Public

    Fast HTML extraction for Python and Rust, without a DOM

    scrapy/frostwork's past year of commit activity
    Python 17 Apache-2.0 3 0 0 Updated Nov 13, 2026
  • cssselect Public

    CSS Selectors for Python

    scrapy/cssselect's past year of commit activity
    Python 313 76 5 5 Updated Nov 12, 2026
  • w3lib Public

    Python library of web-related functions

    scrapy/w3lib's past year of commit activity
    HTML 422 BSD-3-Clause 147 4 10 Updated Nov 12, 2026
  • queuelib Public

    Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python

    scrapy/queuelib's past year of commit activity
    Python 301 BSD-3-Clause 60 0 1 Updated Nov 12, 2026
  • parsel Public

    Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

    scrapy/parsel's past year of commit activity
    Python 1,366 BSD-3-Clause 188 7 6 Updated Nov 12, 2026
  • protego Public

    A pure-Python robots.txt parser with support for modern conventions.

    scrapy/protego's past year of commit activity
    DIGITAL Command Language 97 BSD-3-Clause 39 0 2 Updated Nov 1, 2026
  • unattended-pr-guard Public

    A GitHub action to close pull requests that an LLM is opening unattended.

    scrapy/unattended-pr-guard's past year of commit activity
    JavaScript 2 Apache-2.0 1 0 0 Updated Nov 1, 2026
View all repositories