Scrape arXiv with AI

# Scrape arXiv — SoloStack Skill File
> Drop this file into your SoloStack repo and Claude Code will use it as
> context to run this workflow automatically.
---
## Overview
Extract arXiv research papers, abstracts, author data, and citation info using Apify and Claude Code.
**Apify Actor:** `epctex/arxiv-scraper`
**Pricing:** ~$0.10 per 1,000 papers
## Capabilities
1. **Data Extraction** — Pull key data points from arXiv including profiles, content, and metadata.
2. **Search & Filter** — Search by keywords, categories, or specific URLs to target exactly what you need.
3. **Engagement Metrics** — Capture engagement signals — views, likes, shares, and comments for every item.
4. **Bulk Processing** — Process hundreds or thousands of records in a single run with automatic pagination.
5. **Export & Integration** — Output clean JSON ready for CRM import, analysis, or integration with other tools.
## Pipeline
1. **Define targets** — Specify arXiv URLs, search terms, or categories to scrape
2. **Run scraper** — Apify actor extracts data, handles pagination and rate limits automatically
3. **Process data** — Clean, deduplicate, and enrich the raw output for analysis
4. **Load into CRM** — Push structured data into Neon database with source attribution
## Use Cases
- **Research trend monitoring:** Track new paper volume by topic to identify emerging research areas before they hit mainstream.
- **Competitive R&D tracking:** Monitor papers from competitor company researchers to understand their R&D direction.
- **AI/ML trend analysis:** arXiv is ground zero for AI research. Track model announcements, benchmark results, and new techniques.
- **Content creation:** Summarize trending research for non-technical audiences in blog posts and newsletters.
## Warnings
- arXiv has a public API and bulk data access. Use official channels when possible.
- Paper quality varies — arXiv is a preprint server with no peer review.
- Citation counts lag publication by months. Use for trend detection, not impact measurement.
## Sample Output
```json
{
  "id": "12345",
  "name": "Sample Record",
  "platform": "arXiv",
  "url": "https://arxiv.com/sample",
  "dataField1": "value1",
  "dataField2": 42,
  "scrapedAt": "2026-03-25T12:00:00Z"
}
```
Download the complete skill file with full implementation details,
agent prompts, and ready-to-run scripts.