---
name: website-data-agent
description: >-
  Use this skill whenever the user has a list of company domains, websites, or
  work emails and wants a specific piece of information pulled from the actual
  websites at scale. Natural requests include 'check which of these companies
  are hiring sales roles', 'scrape the careers pages', 'pull pricing from these
  sites', 'check partner directories', 'find case studies', or 'get this custom
  website signal for every company'. Prefer it when the needed data is not a
  normal database field. Infer which pages to find, scrape them free-first over
  the user's own connection, return the requested website data, and ask before
  any paid fallback. The user never needs to mention the skill name.
compatibility: >-
  Needs Claude Code on macOS, Linux, or Windows through WSL, with Python 3 and
  internet access. Uses the user's own internet connection for the normal run.
allowed-tools: Bash, Read
---

# Website Data Agent

This skill is for **custom website data that is not already sitting in a normal company database**.

The user already has the companies. The job is to go to the right pages on those companies' websites and bring back the specific information they asked for.

Good examples:

- careers / jobs pages
- pricing / plans pages
- partner directories
- case studies
- certifications
- office / location pages
- product or service pages
- any other clearly described website signal

The normal workflow is:

**company list already exists -> user says what they want to know -> infer the pages and signals -> scrape those pages free-first -> return the website data -> optionally summarize the requested signal -> ask before any paid fallback.**

Do not turn this into generic homepage enrichment when the user asked for a specific page or signal.

---

# Semantic routing

The user does **not** need to say "Website Data Agent" or "scraper".

Prefer this skill for natural requests such as:

- "Go through these companies and find which are hiring sales, RevOps or GTM roles."
- "Scrape the careers pages for this list."
- "Pull pricing from these company websites."
- "Check which of these sites have a partner program."
- "Find the case studies on these domains."
- "Get the office locations from these websites."
- "I already have the domains. Find this one thing on each website."

If the user already has the company list, do not ask them to rebuild it or explain where it came from.

If they mention MoltSets, treat MoltSets as the source of the existing company/contact list, not as a replacement for the custom website signal. This skill should fetch the part that the database does not already provide.

---

# The spine

0. Check the local scraper quietly and install it once if needed.
1. Take the existing company/domain list.
2. Understand the exact website information the user wants.
3. Infer the relevant page type, page-discovery terms, likely paths, and signal terms.
4. Show one compact `Website data plan` so the user can see what will be collected.
5. Scrape the relevant pages across the list over the user's own connection.
6. Return the raw website data and a concise summary of the requested signal.
7. If some sites remain unresolved, ask once before using any paid fallback.

The user should not have to tell Claude which URL path to use. They describe the information they want; the agent finds the relevant pages.

---

# Global rules

- Start from the list the user already has.
- Never modify the user's source file.
- The default run is free and uses the user's own internet connection.
- Do not use a paid scraping API, proxy service, or data provider without explicit approval.
- Do not recreate generic company descriptions if the user asked for a specific website signal.
- Do not silently replace missing website evidence with database data.
- Do not classify a company as having a signal unless the scraped page text supports it.
- If a page is blocked or unavailable, report that state honestly.
- Keep the main conversation clean. Do not dump raw HTML or debug output.
- Important results must appear in chat and in a CSV.
- One row per unique company/domain in the primary results file.
- If the source has duplicate rows for the same company, preserve the original rows in the joined file.
- Ask at most one paid-fallback question, and only after the free run is complete.

---

# Step 0 - Local scraper availability

Do this silently before the first visible response.

The local engine lives at:

`~/.website-data-agent/`

Check:

```bash
WDA="$HOME/.website-data-agent"
if [ -x "$WDA/venv/bin/python" ] && [ -f "$WDA/targeted_web_data.py" ] && "$WDA/venv/bin/python" -c "import curl_cffi, bs4" >/dev/null 2>&1; then
  echo READY
else
  echo INSTALL
fi
```

If `READY`, continue without setup narration.

If `INSTALL`, say one short line:

> Setting up the free Website Data Agent on this machine. This is a one-time local install.

Then create the environment and extract the embedded script from this skill file.

Locate this `SKILL.md` using the skill base directory provided by Claude Code. If needed, search for it under the normal Claude skill folders.

Create the environment:

```bash
WDA="$HOME/.website-data-agent"
mkdir -p "$WDA"
python3 -m venv "$WDA/venv"
"$WDA/venv/bin/python" -m pip install --quiet --upgrade pip
"$WDA/venv/bin/python" -m pip install --quiet curl_cffi beautifulsoup4
```

Extract the embedded script exactly from the markers at the bottom of this file into:

`~/.website-data-agent/targeted_web_data.py`

Use the actual path to this `SKILL.md` in `SKILL_MD`:

```bash
SKILL_MD="/absolute/path/to/this/SKILL.md"
WDA="$HOME/.website-data-agent"
awk '/^# === WEBSITE DATA AGENT SCRIPT BEGIN ===$/{f=1;next} /^# === WEBSITE DATA AGENT SCRIPT END ===$/{f=0} f' "$SKILL_MD" > "$WDA/targeted_web_data.py"
```

Then run:

```bash
"$WDA/venv/bin/python" "$WDA/targeted_web_data.py" --selftest
```

The final line must be:

`WEBSITE_DATA_AGENT_READY`

If setup fails, explain the concrete failure in one sentence and stop. Do not switch to a paid scraper automatically.

---

# Step 1 - Take the existing list

Accepted inputs:

- CSV
- pasted domains / websites
- work-email list

If the user already supplied a file, use it.

If they pasted domains, create a working CSV in a new output folder.

## Pick the company/domain column yourself

Prefer headers containing:

- website
- domain
- url
- site

If none exist, use a work-email column and derive the domain after `@`.

Ask only if there is genuinely no plausible company/domain field.

Do not ask where the list came from.

Create a new output folder beside the source file:

`<source-name> - Website Data Agent/`

Never overwrite the source.

---

# Step 2 - Understand what the user wants from the websites

The user should describe the outcome in normal language.

Example:

> Go through these companies and tell me which are hiring sales, RevOps or GTM roles right now.

Turn that into four internal inputs:

1. **page_type** - the kind of page likely to contain the answer
2. **page_terms** - words that identify the right page/link
3. **candidate_paths** - common paths worth trying when the homepage does not expose a usable link
4. **signal_terms** - words/phrases that support the specific thing the user wants to know

### Example: careers hiring signal

`page_type = careers/jobs`

`page_terms = careers, career, jobs, job openings, join us, join our team, work with us, vacancies, open positions, hiring`

`candidate_paths = /careers, /jobs, /join-us, /work-with-us, /open-positions, /vacancies`

`signal_terms` should reflect the user's exact interest. For:

> sales, RevOps or GTM roles

use strong terms such as:

`sales, account executive, sales development, business development, revenue operations, revops, sales operations, go-to-market, gtm`

Do not add unrelated signals just to increase hit rate.

### Other page types

For pricing, partner directories, case studies, certifications, locations, or another request, derive the equivalent page terms, paths, and signal terms from the user's language.

If the request is clear, do not ask the user to approve technical keywords one by one.

---

# Step 3 - Show the website data plan

Before the run, render exactly:

### Website data plan

| Item | What I'm using |
|---|---|
| Companies | [N rows / N unique domains] |
| What you want to know | [plain-language request] |
| Pages I'll look for | [page type] |
| Signals I'll look for | [short signal list, or `page content only`] |
| Cost | $0 on the normal run |

Then say one short line:

> I'll find the relevant page on each domain, read it, and return the source URL plus the page text and the signals found.

Do **not** add another approval gate unless the request itself is ambiguous.

Proceed to the run.

---

# Step 4 - Run the free targeted scrape

Tell the user one short progress line:

> Reading the relevant pages across [N] companies now.

Run the local script.

Conceptual command:

```bash
WDA="$HOME/.website-data-agent"
"$WDA/venv/bin/python" "$WDA/targeted_web_data.py" \
  --input "INPUT.csv" \
  --column "DOMAIN_COLUMN" \
  --out "OUTPUT_FOLDER/targeted-website-data.csv" \
  --join-out "OUTPUT_FOLDER/source-rows-plus-website-data.csv" \
  --page-type "careers/jobs" \
  --page-terms "careers|career|jobs|job openings|join us|join our team|work with us|vacancies|open positions|hiring" \
  --paths "/careers|/jobs|/join-us|/work-with-us|/open-positions|/vacancies" \
  --signal-terms "sales|account executive|sales development|business development|revenue operations|revops|sales operations|go-to-market|gtm" \
  --workers 20
```

Build the real values from the user's request. Do not copy the careers example when the user asked for a different page type.

## How the free discovery works

For each domain the script:

1. fetches the homepage when reachable
2. reads homepage links and looks for anchors/URLs matching the page terms
3. adds the relevant common paths as fallback candidates
4. fetches the strongest candidate pages
5. if signal terms were supplied, follows one additional relevant hop from the best target page when needed
6. chooses the strongest page evidence
7. records the real source URL, title, description, visible text, matched terms, and status

External ATS or partner URLs are allowed when they were linked from the company's own site and match the requested page type.

Do not expand into an unrestricted crawl.

---

# Step 5 - Return the data

Primary file:

`targeted-website-data.csv`

One row per unique domain.

Fields:

- `domain`
- `page_type`
- `status`
- `source_url`
- `page_title`
- `meta_description`
- `matched_page_terms`
- `matched_signal_terms`
- `signal_hit_count`
- `pages_checked`
- `text`
- `text_chars`
- `reason`
- `checked_at`

Joined file:

`source-rows-plus-website-data.csv`

This preserves the user's original rows and appends the website-data fields.

## Show in chat

Render a concise summary that directly answers the user's request.

For the careers demo, show:

### Careers-page results

| Company/domain | Careers page | Relevant roles/signals found |
|---|---|---|

Show up to 10 useful rows, then say:

> **Companies checked:** [N]  
> **Relevant signal found:** [N]  
> **No matching signal on the page:** [N]  
> **Unresolved/blocked:** [N]  
> **File:** `targeted-website-data.csv`

Do not invent a positive signal when the page text does not contain supporting evidence.

For another page type, adapt the table label and final count to the user's actual request.

The output should be **the scraped website data plus a concise evidence-backed summary**, not an unrelated company qualification workflow.

---

# Step 6 - Paid leftovers gate

If every domain was resolved well enough, stop.

If some domains remain blocked/unresolved, ask once:

> [N] sites are still unresolved after the free run. Want me to leave them as unresolved, or use a connected paid web-fetch/proxy option for just those leftovers?

Use interactive options when available:

- Leave them unresolved
- Use a paid fallback

Stop.

If the user chooses a paid fallback:

- use only a provider/tool that is actually connected in the current environment
- show the provider and estimated cost before spending
- require one explicit approval for that cost
- run only the unresolved URLs
- merge the successful results back into the same output files

Never silently use MoltSets, Apify, or another data provider as a substitute for the requested page scrape.

---

# Failure states

### Source file has no usable domain/email field
Ask for the correct column or a CSV with a website/domain field.

### Homepage is blocked but a common target path works
Use the target path and continue. The homepage itself is not the deliverable.

### Target page is on an external ATS / partner host
Use it only when the company's own site links to it and the link clearly matches the requested page type.

### No relevant page exists
Return `status=no_target_page` and do not manufacture data.

### Target page exists but the requested signal is absent
Return the page data with an empty `matched_signal_terms`. This means no supporting signal was found in the scraped page, not that the company can never have that signal elsewhere.

### Page is blocked/unreachable
Return `status=unresolved`, preserve the reason, and include it in the paid-leftovers gate.

### User asks for another website signal after the run
Reuse the same source list. Build a new page plan and create a new output file. Do not overwrite the earlier dataset.

---

# Recording rules

For the careers-page demo, the ideal visible sequence is:

1. the company list already exists from the prior workflow
2. user types a plain-English request to find companies hiring sales, RevOps or GTM roles
3. `Website data plan` appears
4. one short visible line says the agent is reading the relevant pages
5. scraper runs without raw debug output taking over the screen
6. `Careers-page results` appears
7. final CSV path appears
8. if there are unresolved sites, the paid-leftovers gate appears only after the main result

Do not make the demo about uploading a random CSV.
Do not make the demo about generic homepage descriptions.
Do not add a second "Claude qualifies the companies" stage after the scrape. The requested signal and evidence should already be visible in the returned data.

---

# Final rules

- The user already has the companies.
- The user describes the information they want, not the URL paths.
- Infer the page type and discovery logic yourself.
- Fetch the actual website pages where the data lives.
- Free-first over the user's own connection.
- Ask before anything paid.
- Return the source URL and evidence.
- Careers is one example, not the only use case.
- Do not duplicate normal company-database enrichment.
- Do not turn the result into an unrelated qualification workflow.
- Keep the raw website evidence available in the CSV.

---

# Embedded local scraper

Extract everything between the markers into:

`~/.website-data-agent/targeted_web_data.py`

```python
# === WEBSITE DATA AGENT SCRIPT BEGIN ===
from __future__ import annotations

import argparse
import csv
import re
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
from pathlib import Path
from typing import Iterable
from urllib.parse import urljoin, urlsplit, urlunsplit

from bs4 import BeautifulSoup
from curl_cffi import requests

FREE_MAIL = {
    "gmail.com", "googlemail.com", "yahoo.com", "outlook.com", "hotmail.com",
    "icloud.com", "aol.com", "proton.me", "protonmail.com", "live.com",
}
USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/124 Safari/537.36"
TIMEOUT = 18
MAX_TEXT = 12000
MAX_TARGET_CANDIDATES = 6
MAX_SECOND_HOP = 5
DEFAULT_WORKERS = 20


def split_pipe(value: str) -> list[str]:
    return [x.strip() for x in (value or "").split("|") if x.strip()]


def normalize_domain(value: str) -> str:
    v = (value or "").strip()
    if not v:
        return ""
    if "@" in v and "://" not in v:
        v = v.rsplit("@", 1)[-1].strip()
        if v.lower() in FREE_MAIL:
            return ""
    if "://" not in v:
        v = "https://" + v
    try:
        host = (urlsplit(v).hostname or "").lower().strip(".")
    except Exception:
        return ""
    if host.startswith("www."):
        host = host[4:]
    return host


def clean_url(url: str) -> str:
    try:
        p = urlsplit(url)
        scheme = p.scheme if p.scheme in {"http", "https"} else "https"
        return urlunsplit((scheme, p.netloc, p.path or "/", p.query, ""))
    except Exception:
        return url


def same_companyish(base_domain: str, url: str) -> bool:
    try:
        host = (urlsplit(url).hostname or "").lower()
    except Exception:
        return False
    if host == base_domain or host.endswith("." + base_domain):
        return True
    known_external = (
        "greenhouse.io", "lever.co", "ashbyhq.com", "workable.com", "smartrecruiters.com",
        "jobvite.com", "myworkdayjobs.com", "workdayjobs.com", "bamboohr.com", "icims.com",
        "recruitee.com", "teamtailor.com", "applytojob.com", "personio.com",
    )
    return any(host == d or host.endswith("." + d) for d in known_external)


def fetch(url: str):
    last = None
    for impersonate in ("chrome124", "chrome120", "chrome"):
        try:
            r = requests.get(
                url,
                timeout=TIMEOUT,
                allow_redirects=True,
                impersonate=impersonate,
                headers={"User-Agent": USER_AGENT, "Accept-Language": "en-US,en;q=0.9"},
            )
            return r
        except Exception as exc:
            last = exc
    raise RuntimeError(str(last or "fetch failed"))


def page_data(url: str) -> dict:
    try:
        r = fetch(url)
    except Exception as exc:
        return {"ok": False, "url": url, "reason": f"fetch_error: {exc}"}
    status = int(getattr(r, "status_code", 0) or 0)
    final_url = str(getattr(r, "url", url) or url)
    ctype = str(r.headers.get("content-type", "")).lower()
    if status >= 400:
        return {"ok": False, "url": final_url, "reason": f"http_{status}"}
    if "html" not in ctype and ctype:
        return {"ok": False, "url": final_url, "reason": f"non_html:{ctype.split(';')[0]}"}
    html = r.text or ""
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "noscript", "svg", "template"]):
        tag.decompose()
    title = soup.title.get_text(" ", strip=True) if soup.title else ""
    meta = ""
    md = soup.find("meta", attrs={"name": re.compile("^description$", re.I)})
    if md and md.get("content"):
        meta = str(md.get("content")).strip()
    text = " ".join(soup.stripped_strings)
    text = re.sub(r"\s+", " ", text).strip()
    links = []
    for a in soup.find_all("a", href=True):
        href = str(a.get("href") or "").strip()
        if not href or href.startswith(("mailto:", "tel:", "javascript:", "#")):
            continue
        abs_url = clean_url(urljoin(final_url, href))
        anchor = " ".join(a.stripped_strings)
        links.append((abs_url, anchor))
    low = text.lower()
    obvious_block = any(x in low[:2000] for x in (
        "checking your browser", "access denied", "verify you are human", "captcha",
        "just a moment", "enable javascript and cookies",
    ))
    if obvious_block and len(text) < 2500:
        return {"ok": False, "url": final_url, "reason": "blocked_or_challenge"}
    return {
        "ok": True,
        "url": final_url,
        "title": title,
        "meta": meta,
        "text": text[:MAX_TEXT],
        "full_text": text,
        "links": links,
        "status": status,
    }


def term_hits(text: str, terms: Iterable[str]) -> list[str]:
    low = (text or "").lower()
    return [t for t in terms if t.lower() in low]


def link_score(url: str, anchor: str, page_terms: list[str]) -> int:
    hay = f"{url} {anchor}".lower()
    score = 0
    for term in page_terms:
        t = term.lower()
        if t in anchor.lower():
            score += 6
        if t.replace(" ", "-") in url.lower() or t.replace(" ", "") in url.lower():
            score += 4
        if t in hay:
            score += 2
    return score


def unique_keep_order(items: Iterable[str]) -> list[str]:
    seen = set()
    out = []
    for x in items:
        if x and x not in seen:
            seen.add(x)
            out.append(x)
    return out


def build_candidates(domain: str, homepage: dict, page_terms: list[str], paths: list[str]) -> list[str]:
    scored = []
    if homepage.get("ok"):
        for url, anchor in homepage.get("links", []):
            if not same_companyish(domain, url):
                continue
            s = link_score(url, anchor, page_terms)
            if s > 0:
                scored.append((s, url))
    scored.sort(key=lambda x: (-x[0], x[1]))
    linked = [u for _, u in scored]
    base = f"https://{domain}"
    path_urls = [urljoin(base + "/", p.lstrip("/")) for p in paths]
    return unique_keep_order(linked + path_urls)[:MAX_TARGET_CANDIDATES]


def second_hop_candidates(domain: str, page: dict, signal_terms: list[str], page_terms: list[str]) -> list[str]:
    scored = []
    terms = signal_terms + page_terms
    for url, anchor in page.get("links", []):
        if not same_companyish(domain, url):
            continue
        hay = f"{url} {anchor}".lower()
        score = sum(4 for t in signal_terms if t.lower() in hay) + sum(1 for t in page_terms if t.lower() in hay)
        if score > 0:
            scored.append((score, url))
    scored.sort(key=lambda x: (-x[0], x[1]))
    return unique_keep_order(u for _, u in scored)[:MAX_SECOND_HOP]


def choose_best(pages: list[dict], page_terms: list[str], signal_terms: list[str]) -> dict | None:
    good = [p for p in pages if p.get("ok")]
    if not good:
        return None
    def score(p):
        text = f"{p.get('url','')} {p.get('title','')} {p.get('full_text','')}"
        page_hit = len(term_hits(text, page_terms))
        signal_hit = len(term_hits(text, signal_terms))
        return (signal_hit * 12) + (page_hit * 4) + min(len(p.get("full_text", "")) // 1000, 5)
    return max(good, key=score)


def process_domain(domain: str, page_type: str, page_terms: list[str], paths: list[str], signal_terms: list[str]) -> dict:
    checked = []
    homepage = page_data(f"https://{domain}/")
    if not homepage.get("ok"):
        alt = page_data(f"https://www.{domain}/")
        if alt.get("ok"):
            homepage = alt
    if homepage.get("url"):
        checked.append(homepage.get("url"))

    candidates = build_candidates(domain, homepage, page_terms, paths)
    pages = []
    for url in candidates:
        p = page_data(url)
        checked.append(p.get("url", url))
        pages.append(p)

    best = choose_best(pages, page_terms, signal_terms)

    if best and signal_terms and not term_hits(best.get("full_text", ""), signal_terms):
        for url in second_hop_candidates(domain, best, signal_terms, page_terms):
            p = page_data(url)
            checked.append(p.get("url", url))
            pages.append(p)
        best = choose_best(pages, page_terms, signal_terms)

    if best:
        full = best.get("full_text", "")
        page_hits = term_hits(f"{best.get('url','')} {best.get('title','')} {full}", page_terms)
        signal_hits = term_hits(full, signal_terms)
        return {
            "domain": domain,
            "page_type": page_type,
            "status": "resolved",
            "source_url": best.get("url", ""),
            "page_title": best.get("title", ""),
            "meta_description": best.get("meta", ""),
            "matched_page_terms": "; ".join(page_hits),
            "matched_signal_terms": "; ".join(signal_hits),
            "signal_hit_count": str(len(signal_hits)),
            "pages_checked": str(len(unique_keep_order(checked))),
            "text": best.get("text", ""),
            "text_chars": str(len(full)),
            "reason": "",
            "checked_at": datetime.now(timezone.utc).isoformat(),
        }

    reasons = [p.get("reason", "") for p in pages if p.get("reason")]
    if homepage.get("reason"):
        reasons.insert(0, homepage.get("reason"))
    status = "unresolved" if reasons else "no_target_page"
    return {
        "domain": domain,
        "page_type": page_type,
        "status": status,
        "source_url": "",
        "page_title": "",
        "meta_description": "",
        "matched_page_terms": "",
        "matched_signal_terms": "",
        "signal_hit_count": "0",
        "pages_checked": str(len(unique_keep_order(checked))),
        "text": "",
        "text_chars": "0",
        "reason": "; ".join(unique_keep_order(reasons))[:500],
        "checked_at": datetime.now(timezone.utc).isoformat(),
    }


def load_rows(path: Path) -> tuple[list[dict], list[str]]:
    with path.open("r", encoding="utf-8-sig", errors="replace", newline="") as fh:
        reader = csv.DictReader(fh)
        rows = list(reader)
        return rows, list(reader.fieldnames or [])


def write_csv(path: Path, rows: list[dict], fields: list[str]):
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("w", encoding="utf-8", newline="") as fh:
        w = csv.DictWriter(fh, fieldnames=fields, extrasaction="ignore")
        w.writeheader()
        w.writerows(rows)


def run(args):
    inp = Path(args.input).expanduser()
    out = Path(args.out).expanduser()
    join_out = Path(args.join_out).expanduser()
    page_terms = split_pipe(args.page_terms)
    paths = split_pipe(args.paths)
    signal_terms = split_pipe(args.signal_terms)
    rows, original_fields = load_rows(inp)

    domains = []
    row_domains = []
    for row in rows:
        d = normalize_domain(row.get(args.column, ""))
        row_domains.append(d)
        if d and d not in domains:
            domains.append(d)

    by_domain = {}
    done = 0
    workers = max(1, min(int(args.workers or DEFAULT_WORKERS), 40))
    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = {
            pool.submit(process_domain, domain, args.page_type, page_terms, paths, signal_terms): domain
            for domain in domains
        }
        for fut in as_completed(futures):
            domain = futures[fut]
            try:
                result = fut.result()
            except Exception as exc:
                result = {
                    "domain": domain, "page_type": args.page_type, "status": "unresolved",
                    "source_url": "", "page_title": "", "meta_description": "",
                    "matched_page_terms": "", "matched_signal_terms": "", "signal_hit_count": "0",
                    "pages_checked": "0", "text": "", "text_chars": "0",
                    "reason": f"worker_error: {exc}",
                    "checked_at": datetime.now(timezone.utc).isoformat(),
                }
            by_domain[domain] = result
            done += 1
            print(f"[{done}/{len(domains)}] {domain}", flush=True)
    results = [by_domain[d] for d in domains]

    result_fields = [
        "domain", "page_type", "status", "source_url", "page_title", "meta_description",
        "matched_page_terms", "matched_signal_terms", "signal_hit_count", "pages_checked",
        "text", "text_chars", "reason", "checked_at",
    ]
    write_csv(out, results, result_fields)

    joined = []
    append_fields = [f for f in result_fields if f != "domain"]
    for row, domain in zip(rows, row_domains):
        new = dict(row)
        r = by_domain.get(domain, {})
        new["website_data_domain"] = domain
        for f in append_fields:
            new[f"website_data_{f}"] = r.get(f, "")
        joined.append(new)
    joined_fields = original_fields + ["website_data_domain"] + [f"website_data_{f}" for f in append_fields]
    write_csv(join_out, joined, joined_fields)

    resolved = sum(1 for r in results if r["status"] == "resolved")
    signal = sum(1 for r in results if int(r.get("signal_hit_count") or 0) > 0)
    unresolved = sum(1 for r in results if r["status"] == "unresolved")
    no_target = sum(1 for r in results if r["status"] == "no_target_page")
    print(f"DONE domains={len(results)} resolved={resolved} signal={signal} unresolved={unresolved} no_target_page={no_target}")
    print(f"OUT={out}")
    print(f"JOIN_OUT={join_out}")


def selftest():
    assert normalize_domain("https://www.Example.com/careers") == "example.com"
    assert normalize_domain("person@example.com") == "example.com"
    assert normalize_domain("person@gmail.com") == ""
    assert split_pipe("a|b| c ") == ["a", "b", "c"]
    assert term_hits("Revenue Operations Manager and Sales", ["revops", "revenue operations", "sales"]) == ["revenue operations", "sales"]
    print("WEBSITE_DATA_AGENT_READY")


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--selftest", action="store_true")
    ap.add_argument("--input")
    ap.add_argument("--column")
    ap.add_argument("--out")
    ap.add_argument("--join-out")
    ap.add_argument("--page-type", default="target page")
    ap.add_argument("--page-terms", default="")
    ap.add_argument("--paths", default="")
    ap.add_argument("--signal-terms", default="")
    ap.add_argument("--workers", type=int, default=DEFAULT_WORKERS)
    args = ap.parse_args()
    if args.selftest:
        selftest()
        return
    required = [args.input, args.column, args.out, args.join_out]
    if not all(required):
        ap.error("--input, --column, --out, and --join-out are required")
    run(args)


if __name__ == "__main__":
    main()
# === WEBSITE DATA AGENT SCRIPT END ===
```
