soup-strainer is a Python implementation of the Readability "decruft" algorithm used to remove navigation, advertising and boilerplate from web pages. It uses BeautifulSoup and html5lib to parse malformed HTML5, then returns the article body from a raw HTML string. The project predates most current Python Readability implementations.
The algorithm scores candidate block elements by analysing their text density, link density, class names, and tag types — high text density with low link density signals article content. The top-scoring candidate and its siblings are kept; everything else is stripped. html5lib parses even malformed HTML correctly, so it handles the full range of real-world pages that trip up lxml-based parsers.
Strips nav, ads, headers, and footers — returns the main readable content block as clean HTML.
Uses html5lib for spec-compliant HTML5 parsing — handles broken markup, missing closing tags, and encoding issues gracefully.
Scores candidate elements by text-to-link ratio — correctly identifies navigation blocks that look like content.
No native extensions, no headless browser — runs anywhere Python and BeautifulSoup are available.
The entire algorithm fits in one importable Python file — drop it into any scraping or content pipeline.