Repository navigation
Conversation
The Extract API now centers on Markdown: include_markdown defaults to true and the content is returned in a top-level `markdown` field, while text_content, headings, breadcrumbs, lists, tables, forms, price_offers, and include_inline_assets are gone. - Rewrite the extract doc comment around Markdown, list the current response sections and parameters, and document the String return for output: 'md' / 'html', crawl credit usage, and the API timeout (ms) vs. the client HTTP timeout (s). - Update the README Extract section with Markdown examples, crawl credit usage, the two timeouts, and the current redirect wording. - Fix the demo for the new response shape (markdown, page[:url]) and drop max_pages from the map call, as it only applies to crawl. - Check markdown in the live example spec and cover output: 'md'. Co-authored-by: Claude Opus 5.5 (xhigh effort) <[email protected]>
A crawl without max_pages extracts up to 10 pages, so it can consume up to 10 credits. Co-authored-by: Claude Opus 5.5 (xhigh effort) <[email protected]>
Describe include_sitemap and include_subdomains, and note that include_subdomains also applies to crawl mode. Co-authored-by: Claude Opus 5.5 (xhigh effort) <[email protected]>
adarshdigievo
approved these changes
Nov 13, 2026
adarshdigievo
left a comment
adarshdigievo
left a comment
Member
There was a problem hiding this comment.
Looks all good.
Double-checked everything with our live docs using Claude Fable 5.2 extra.
Unrelated, but we need to update the CI tests, as they are failing on the ubuntu-latest image.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a follow-up to #37. The Extract API changed after #37 was written and it now centers on Markdown, and several response sections and parameters are gone. The
extractmethod itself still works, since parameters pass through to the backend, but our docs and demo describe the old response.Below what changed. Made with Opus 5.5 High Effort.
What changed in the API
include_markdownnow defaults totrue, and the Markdown is returned in a top-levelmarkdownfield (previouslytext_content.markdown).text_content,headings,breadcrumbs,lists,tables,forms,sections,price_offers,scripts, andstylesheetswere removed from the response, and theinclude_inline_assetsparameter is gone.pagescarry the page URL inurl.page_informationno longer has one.Changes
extractdoc comment: lists the current sections and parameters (addsno_cache,async,zero_trace,output,json_restrictor), documents theStringreturn foroutput: 'md'/'html', the crawl credit usage, and that a call-leveltimeoutis the API fetch timeout in milliseconds, while the clienttimeoutis the HTTP timeout in seconds.results[:markdown],output: 'md'), crawl credit usage, the two timeouts, and the current redirect wording. The map example usesbooks.toscrape.com, matching the live spec.markdowninstead of the removedtext_content, usespage[:url]for crawled pages, and dropsmax_pagesfrom the map call, as it only applies to crawl.markdownand coversoutput: 'md'.There's no behavior change in the client.