Scrape Academia with AI

# Scrape Academia — SoloStack Skill File
> Drop this file into your SoloStack repo and Claude Code will use it as
> context to run this workflow automatically.
---
## Overview
Extract academic papers, citations, author profiles, and research data from Google Scholar using Apify and Claude Code.
**Apify Actor:** `apify/google-scholar-scraper`
**Pricing:** ~$0.50 per 1,000 papers
## Capabilities
1. **Data Extraction** — Pull key data points from Google Scholar including profiles, content, and metadata.
2. **Search & Filter** — Search by keywords, categories, or specific URLs to target exactly what you need.
3. **Engagement Metrics** — Capture engagement signals — views, likes, shares, and comments for every item.
4. **Bulk Processing** — Process hundreds or thousands of records in a single run with automatic pagination.
5. **Export & Integration** — Output clean JSON ready for CRM import, analysis, or integration with other tools.
## Pipeline
1. **Define targets** — Specify Google Scholar URLs, search terms, or categories to scrape
2. **Run scraper** — Apify actor extracts data, handles pagination and rate limits automatically
3. **Process data** — Clean, deduplicate, and enrich the raw output for analysis
4. **Load into CRM** — Push structured data into Neon database with source attribution
## Use Cases
- **Research landscape mapping:** Track research output in your domain to understand who is publishing, where, and on what topics.
- **Author network analysis:** Map co-authorship networks to identify key researchers and institutions in your field.
- **Patent and IP research:** Track academic publications that may lead to patents or commercial applications.
- **Evidence-based content:** Find peer-reviewed research to back up marketing claims and product positioning.
## Warnings
- Google Scholar aggressively blocks scrapers. Success rates are lower than most platforms.
- Citation counts differ from other databases (Scopus, Web of Science).
- Google Scholar includes non-peer-reviewed content (preprints, theses). Filter for quality.
## Sample Output
```json
{
  "id": "12345",
  "name": "Sample Record",
  "platform": "Google Scholar",
  "url": "https://academia.com/sample",
  "dataField1": "value1",
  "dataField2": 42,
  "scrapedAt": "2026-03-25T12:00:00Z"
}
```
Download the complete skill file with full implementation details,
agent prompts, and ready-to-run scripts.