Can these 100 apps become agent toolkits?

Where a human was needed

• Rebrands and moved docs. fanbasis.com now 301s to commas.com; the agent saw an 'Enterprise & APIs' marketing page and called it partner-gated. The real API reference is at commasdocs.com with self-serve scoped keys. Coda's docs moved to docs.superhuman.com and Binance's to developers.binance.com - the agent handled both of those on its own.
• Bot-blocked sites. pitchbook.com returns 403 to anything that is not a browser. The agent still found the public OpenAPI spec at api.pitchbook.com/api-docs (130 operations), but only a real Chrome session could read the help page that says API access 'requires a standalone contract agreement'.
• Pricing tables are JavaScript. The 'access' field lives on pricing/signup pages, and those are mostly JS-rendered comparison grids that WebFetch flattens. Attio (API on Free), Devin (API from Pro only) and iPayX (5 free audits, API on paid) were all resolved by opening the page in Chrome and reading the tick marks.
• Quote grounding fails on JS docs. Mintlify/ReadMe-style docs (Intercom, Clay, Coda, SE Ranking, Datadog, MongoDB) return near-empty HTML to a plain fetch, so correct quotes could not be grounded and confidence dropped. 12 of the 18 queued apps were queued for this reason and turned out to be right on every field checked.
• Ambiguous identity. 'Paygent Connect' matches a Japanese payment gateway, an unrelated crypto wallet and an NMI reseller brand; no primary doc could be found. Left as partner-gated with low confidence. 'Consensus' has an MCP in the official registry - but it belongs to goconsensus.com (sales demos), not consensus.app; the agent caught that itself.
• CLIs are not APIs. Sherlock and Mermaid CLI have no hosted API. They are buildable as local agent skills, not as Composio toolkits; the verdict rule marks them red with blocker 'CLI, not an API' and breadth was reset to unknown by hand.

Corrections log

• Attio · access: paid_plan → self_serve_freePricing comparison table (JS-rendered) shows 'API and webhook access' ticked on the Free $0 plan. Pass 2 read the marketing copy, not the table. Confirmed in Chrome. evidence
• Attio · blocker: paid_only → noneFollows from the access correction. evidence
• fanbasis · access: partner_gated → self_serve_freefanbasis.com 301-redirects to commas.com (rebranded July 2026). The agent saw a marketing site with 'Enterprise & APIs' and concluded gated. The public API reference lives at commasdocs.com: x-api-key header, scoped keys, sandbox environment, test cards. evidence
• fanbasis · blocker: partner_gate → noneFollows from the access correction (requires a seller account, which is self-serve). evidence
• fanbasis · api_breadth: unknown → smallcommasdocs.com documents ~6 resource groups (checkout sessions, products, discount codes, transactions, webhooks, customers). evidence
• Slack · mcp: community → officialSlack shipped an official remote MCP server (mcp.slack.com) on 17 Feb 2026. Pass 1 actually had this right; pass 2 anchored on Smithery/GitHub community listings. evidence
• Amazon Selling Partner · mcp: community → officialAmazon publishes @amazon-sp-api-release/sp-api-dev-mcp in its official samples repo. Pass 2 found it; the judge could not ground the quote in GitHub's HTML, so the merge demoted it. evidence
• iPayX · mcp: community → officialVendor site is JS-rendered so no quote grounded. In Chrome: 'Native Model Context Protocol server', docs at /docs/mcp-server, MCP-native positioning on the homepage. evidence
• iPayX · docs_url: https://www.ipayx.ai/docs/api → https://www.ipayx.ai/docs/apiNo change - recorded because the assignment hint (ipayx.ai/docs) is a 404; the agent found the real path itself. evidence
• Datadog · mcp: community → officialdocs.datadoghq.com/mcp_server/ is Datadog's own MCP server documentation (200 OK). Quote grounding failed on the JS-rendered docs, so the merge fell back to registry 'community'. evidence
• Devin · access: self_serve_free → paid_planPlan matrix on devin.ai/pricing: 'Devin API' is crossed out on Free and ticked from Pro ($20/mo). Confirmed in Chrome. evidence
• Devin · blocker: none → paid_onlyFollows from the access correction. evidence
• Paygent Connect · access: admin_approval → partner_gatedIdentity could not be resolved: 'Paygent' matches a Japanese gateway (pay-gent.com, no public dev docs), an unrelated crypto wallet (api.paygent.tech) and the hint says 'NMI-powered'. NMI gateway accounts are provisioned through resellers/ISOs. Marked partner-gated with low confidence; see failures. evidence
• Paygent Connect · mcp: official → noneThe 'official' MCP vote came from an unrelated registry match on the word 'paygent'. No vendor MCP exists for this product. evidence
• Sherlock · api_breadth: large → unknownA CLI has no endpoints; pass 2 counted supported sites as breadth. evidence
• Mermaid CLI · api_breadth: large → unknownA CLI has no endpoints. evidence

Per field

Is confidence calibrated?

Share of gold apps where auth, access and verdict were all right, by the agent's own confidence bucket. Errors should concentrate in the low bucket — that is what makes the human queue worth working.

Composio's own catalog covers 67 of the 100. Pass 2, working independently of the catalog, agreed with Composio's declared auth on 60/67 (90%); the differences are explained in the failures section.

Source: github.com/NikhilGaur1022/composio-app-research. Needs Python 3.11+ and either a logged-in Claude Code CLI or ANTHROPIC_API_KEY. A Composio key is optional (falls back to the public catalog).

pip install -r requirements.txt
python -m agent.run all --app "Attio"        # one app, every stage
python -m agent.run all                       # all 100, resumable
python -m verification.scoring                # score against gold
python -m page.build                          # rebuild this page

Runnable trigger: the repo's Research one app GitHub Action (workflow_dispatch) runs the full pipeline for any app name and commits the JSON. Add a new app by appending to apps.json.

For agents: the page embeds the dataset (window.RESEARCH) and links raw JSON above. Enums are documented in schema/result.schema.json.