Agent experience (AX): Comparing four social media API providers

We gave AI agents 17 complex research workflows using four social media API providers. The best result was 32 passes out of 51 tests. Each workflow involved finding posts, reading responses, checking dates and writing a report backed by evidence. Here is what helped them finish, and what got in the way.

An AI agent is software that uses tools to do work for you. A social media API lets it find public posts, comments and video transcripts. A transcript is a written record of what someone said in a video. You might ask an agent to compare coding tools or check what people said about a company.

Agent experience (AX) is how easily an AI agent can use tools to complete a workflow. We tested ScrapeCreators, SociaVault, Social Fetch and SocialCrawl using Intermesh, our platform for testing how agents use software. Each provider got the same workflows.

How often agents completed the workflow

We tested 17 workflows in three agent apps. That gave each provider 51 tests, or 204 tests in total. Each test paired one workflow, one app and one provider.

ScrapeCreators passed 32 tests. SociaVault and SocialCrawl each passed 30. Social Fetch passed 27. Just five tests separated the highest and lowest scores.

Loading comparison chart…

A pass means every required part of the workflow was marked complete. A failed test could still produce useful research while missing a requirement. Scores combine automated checks with the agent’s own review of its work.

How we tested

Each workflow came with a question, dates to follow and a list of requirements. The agent had to write a short report, save its evidence and keep a search log. It had to do the work itself, with no help from other agents, and stay within 40 tool calls.

We used Codex, Claude Code and OpenCode with Luna, Haiku 4.5 and DeepSeek models. The app handles tools and files; the model decides what to do and writes the answer. All four providers used MCP, a standard way for agents to call tools such as search or comment retrieval.

Workflow briefs

Choose a coding subscriptionInvestigate tour backlashCheck Ronaldo and Portugal claimsAssess Bali tourist-trap claimsCheck Paris scam claimsInvestigate Spotify and AI musicInvestigate Meta glasses privacyCompare views on MicronInvestigate Nebius growth and financingInvestigate SpaceX investor accessCheck Marvell AI growth claimsCompare Cerebras and Nvidia argumentsInvestigate layoffs and financial stressAssess Meta’s investment in Alexandr WangCompare AI video and motion modelsCompare built-in and personal agentsCompare Jev, System One and Decisions

Agent client

Social provider

Codex

Working through the interface

Choose a coding subscription

Compare Codex and Claude Code subscriptions using firsthand user reports.

Report and supporting sources

ScrapeCreators

Search

Find public posts, replies and video transcripts

Posts, source links and dates

evaluation run

Review

Checks + self-review

Requirement outcomes

Interface friction

Helpful behavior

Review saved

The agent receives the question and the rules for the workflow.

Social providersScrapeCreatorsSociaVaultSocial FetchSocialCrawl

Agent clientsCodexClaude CodeOpenCode

We test the same 17 research workflows with four providers and three agent apps. Each agent collects sources and saves its report, evidence and search log. Reviews include the agent’s own check of its work.

Restart0.5×1×2×

The 17 research workflows

The workflows covered coding tools, travel claims, company news and AI video tools. Each needed different evidence. A travel claim might need firsthand visitor accounts. A company question might need both supporting and opposing views.

Browse the cards to see each workflow and what it required.

Each workflow uses a selected set of public posts. Its findings describe those sources, with gaps and conflicting views recorded in the report.

Why pass rates were around 50–60%

Many agents found useful posts but missed a required part of the workflow:

• Some claims had no matching evidence.
• Some reports left part of the question unanswered.
• Some source or comment counts were wrong.

A workflow had to meet every requirement to pass.

The saved interactions also showed problems with the tools themselves. The cases below explain what got in the agent’s way, how it responded and what still went wrong in the final report.

What the agent interface made harder

Some problems were bugs in how the tools were described. Others came from limits or confusing responses. Following the agent through a real workflow showed where each problem added work or blocked access to evidence.

The instructions should describe what the tool actually accepts

• Scenario: A developer wanted to compare Codex and Claude Code, two AI coding tools. The agent needed user reviews about prices, usage limits and coding work.
• What happened: Codex searched YouTube through SociaVault. It used date and month, which appeared in the tool’s instructions. The API rejected both and gave different options. The agent corrected its requests.
• Why it matters: Incorrect instructions caused avoidable failed calls. The errors helped the agent recover, but accurate instructions would have prevented the problem.
• Outcome: The agent collected evidence. The workflow later failed because some quotes did not support the report’s claims, and a result count was wrong.

See the example.

Show limits before the agent chooses a tool, and explain what it can try when a source exceeds them

• Scenario: The agent needed to check what artists had said about a concert controversy before comparing public reactions.
• What happened: SociaVault could not transcribe Ed Sheeran’s 3-minute-19-second Instagram video. Its two-minute limit was missing from the tool’s instructions. Social Fetch also rejected the video as too long.
• Why it matters: The agent found the original video but could not read its full spoken explanation. Stating limits upfront would help it choose a suitable tool or source.
• Outcome: Both agents continued with other sources. Both workflows failed because some report details lacked matching evidence. The Social Fetch workflow also recorded the wrong comment count.

SociaVault example · Social Fetch example.

A misleading result count made the next step unclear

• Scenario: The agent used SocialCrawl to find public discussions that could help it check claims about AI tools.
• What happened: The tool displayed “0 rows,” then explained that 37 results were saved separately. The agent noticed that explanation and made another call to read them.
• Why it matters: “0 rows” could make an agent think the search found nothing. “37 results found. Use this call to read them” would make the next step clear.
• Outcome: The agent read the results and continued. The workflow later failed because a claim pointed to the wrong evidence. The confusing response added a step; the citation mistake caused the failed requirement.

See the example.

Cost per workflow

There are two costs: the AI model’s work and the provider’s charge for getting data.

Model usage cost averaged about 19–23 cents per test. For runs using a subscription, we priced recorded model usage at pay-per-use rates. For DeepSeek, we used the recorded charges. Subscription fees are separate.

Loading comparison chart…

We estimated provider API costs from recorded credit usage and the prices of the purchased credit packs.

Loading comparison chart…

The chart shows average costs for workflows with complete billing records. Cost coverage differs by provider. These usage estimates cover the research calls and exclude the review.

Time per workflow

Time covers the full research workflow: searching, reading results, writing the report and waiting for responses or access. It measures how long the agent took to finish.

ScrapeCreators averaged 8.2 minutes, SocialCrawl 10.1, Social Fetch 11.0 and SociaVault 12.1.

Loading comparison chart…

The next chart puts pass rate and time together. Higher means more tests passed. Further left means less time per workflow.

Loading comparison chart…

Read the full results

All 204 tests finished. 119 passed, 82 failed and three could not be scored.

The full comparison links to the results, provider reports and saved interaction logs. The reports explain the scoring, cost calculations and evidence checks. Public logs are shortened to fit size limits.

Test the agent interface from start to finish

Software teams use end-to-end tests to check whether a person can complete a workflow through a user interface. Agent interfaces need the same kind of testing: give an agent a real goal and follow it from choosing a tool to checking the result.

These tests can catch software bugs, such as instructions that disagree with the API. They can also reveal AX problems, such as an unclear result count or a limit the agent discovers too late. Watching the full workflow shows whether the agent can recover and finish the work.

A useful test checks both the tool interaction and the final answer. That helps builders see whether to fix the software, clarify the interface or improve how the agent uses the evidence.