When feeding web data into a RAG pipeline, it is convenient to have a tool that accepts a URL and converts the page to Markdown. defuddle does exactly that. The problem is that results vary dramatically depending on the site’s structure. Whether Semantic HTML Exists or Not Tech blogs and official documentation work fine. When heading hierarchy is intact and body tags are clear, the extraction output is usable. The trouble starts with e-commerce sites and layout-driven pages. Without semantic structure, body text and ads come out mixed together. Feeding this into RAG indexing contaminates search quality. In dynamic environments where JavaScript renders the page, the content itself often drops out entirely. Static wiki-level sources work well enough, but complex commercial site structures hit clear limits. Automation Tools Do Not Replace Preprocessing The expectation was less effort than writing parsing logic from scratch, but inspecting extraction results ended up taking even more time. Metadata and body boundaries blurring was a recurring issue. Maintaining separate preprocessing scripts per source domain is the realistic approach. No matter how good the model, low-quality source data corrupts the index itself, and that corruption spreads across all search results. Key Takeaways defuddle extraction performance depends heavily on the target site’s adherence to semantic HTML On sites with poor SEO optimization, distinguishing body from noise (ads, menus) is difficult When building RAG, a separate preprocessing stage to validate auto-extracted results must be designed Related Posts Silver Layer — Lifting Bronze to an Analyzable State — Data cleansing principles and quality gates What Is GEO — AI Citation Strategy Beyond SEO — The importance of semantic HTML and structured data RAG Quality and Markdown Conversion Tools for Data Preprocessing — Document preprocessing with MarkItDown Source https://share.google/8V29VWarTG9YMxXI7