The Problem with Single-Page Lead Research
Most lead-generation tools do the same thing: they visit a company's homepage, scrape whatever text and contact info they find, and call it "enrichment." That gives you a description and maybe an email address. It does not give you the context you need to write a personalized email that gets a reply.
The homepage is rarely where the useful information lives. Contact details are on a dedicated contact page. Team members and leadership names live on an about or team page. Social media links, pricing, case studies, and legal pages each sit on their own URL. A tool that only reads the homepage is leaving 80% of the available intelligence on the table.
AutoReach's refine stage used to work this way too. It scanned a single page per lead. That changed with the deep website crawl, which now navigates each prospect's site the way a human researcher would, page by page, following the links that matter and skipping the ones that do not.
How the Deep Crawl Works
When AutoReach reaches the refine stage of a workflow, it takes each lead's website and runs a depth-first crawl with the following characteristics:
Priority-Driven Page Selection
Not every page on a website is worth visiting. The crawler maintains a priority queue and ranks internal links by how likely they are to contain useful sales intelligence:
- Contact pages (/contact, /contact-us, /get-in-touch) highest priority
- About and team pages (/about, /our-team, /staff, /leadership) high priority
- Legal pages (/privacy, /terms, /legal) lower priority
- Everything else only followed in the first few pages, then pruned
Depth and Page Limits
Each crawl is bounded by two safety rails:
- Max depth: 2 the crawler follows links up to two clicks from the homepage. This covers homepage, contact, about, team, and one level deeper without going down rabbit holes.
- Max pages: 8 no crawl visits more than eight pages per lead. This keeps credit costs predictable and prevents the crawler from spending minutes on a single prospect.
Invalid Links and Backtracking
Websites are messy. Links break, redirect to external domains, point to PDFs, or lead nowhere useful. The crawler handles all of this:
- Internal-only following links to other domains are ignored. The crawl stays on the prospect's own site.
- Already-visited skipping a visited set prevents re-crawling the same page twice, even if multiple pages link to it.
- Malformed URL handling any link that fails URL parsing is silently skipped.
- Transient error retries HTTP 429 (rate limited), 502, 503, and similar transient statuses trigger a single retry with backoff before the page is marked as failed.
- Total timeout if a crawl exceeds 30 seconds for any reason, it stops and returns whatever intelligence it gathered so far.
robots.txt Awareness
Before fetching any page, the crawler checks the site's robots.txt rules. If a path is disallowed, it is logged as "disallowed" and skipped. This respects site owner preferences and avoids fetching pages the site does not want automated visitors to access.
SSRF Protection
Every URL is validated against an SSRF guard before any HTTP request is made. Internal network addresses, localhost, and non-public IPs are rejected. Redirect targets are re-validated before they are followed. This prevents a malicious or misconfigured lead website from triggering requests to internal infrastructure.
What Gets Extracted
Each successfully crawled page contributes to a consolidated intelligence profile for the lead:
- Description the best meta description or first paragraph found across all pages
- Contact email the first real email address found (filtered to exclude image filenames, tracking pixels, and common false positives)
- Contact phone extracted from tel: links or phone-number patterns in page text
- Social links Facebook, Twitter/X, Instagram, LinkedIn, YouTube, and TikTok profile URLs
- Page title the site's title tag
- Meta keywords from the homepage meta tags
- Team members names and titles parsed from team, staff, and leadership page layouts, with heuristic title detection (founder, CEO, director, partner, etc.)
- LinkedIn person URLs any linkedin.com/in/ links found on the site, matched to team members where possible
LinkedIn People Search Enrichment
Finding a team member's name on a website is useful. Finding their LinkedIn profile is more useful. After the crawl completes, AutoReach takes up to three discovered team members and searches for their LinkedIn profiles.
The people search uses Google Custom Search Engine (or SerpAPI as a fallback) with a query like site:linkedin.com/in/ "Jane Smith" "Acme Corp". If a profile is found, the LinkedIn URL and headline are attached to the team member record.
This feature degrades gracefully. If no search API keys are configured, the crawl still extracts team member names from the website. The LinkedIn enrichment is a bonus layer, not a dependency.
Crawl Visibility
You can watch the crawl happen. Every page visit emits a progress event that is logged and displayed in the "Gather & Crawl History" panel on the lead detail page. Each event shows:
- The URL visited
- The crawl depth
- The status (ok, error, or disallowed)
- The HTTP status code
- How long the page took to fetch
What This Costs in Credits
A deep crawl costs 4 credits per lead (the DEEP_CRAWL credit cost). This is more than the 2-credit single-page scan it replaced, but it extracts dramatically more data. If a user does not have enough credits for a deep crawl, the system falls back to the cheaper single-page scan automatically so workflows do not stall.
The optional LinkedIn people search adds 2 credits per person searched (up to 3 people per lead). This is only charged when team members are found and search API keys are configured.
Why This Matters for Outreach Quality
The entire point of lead enrichment is to give your outreach enough context to feel personal. "I noticed you run a marketing agency in Austin" is a generic opener anyone can write after a single homepage scan. "Saw on your team page that Sarah joined as Head of Growth last month, congrats, that usually means you're scaling outbound" is the kind of opener that gets a reply.
Deep crawling is what makes the second kind possible at scale. The AI agent does the research a human SDR would do, clicks through to the pages that matter, and uses what it finds to write outreach that does not sound like a template.
FAQ
How many pages does the crawler visit per lead?
Up to 8 pages, with a maximum depth of 2 clicks from the homepage. The crawl prioritizes contact, about, and team pages over other content.
Does the crawler respect robots.txt?
Yes. Every page is checked against the site's robots.txt rules before fetching. Disallowed pages are logged and skipped.
What happens if a crawl fails?
The crawler logs which pages failed and continues with the rest of the priority queue. Whatever intelligence was gathered from successful pages is saved. A total timeout of 30 seconds ensures no single lead blocks the workflow.
How does this compare to a single-page scan?
A deep crawl costs 4 credits vs 2 for a single-page scan, but extracts team members, social links, multiple contact methods, and LinkedIn profiles. If credits run low, the system automatically falls back to the single-page scan.
Do I need API keys for LinkedIn people search?
No. The crawl extracts team member names without any external API. LinkedIn profile matching via Google CSE or SerpAPI is an optional enrichment layer that only runs if keys are configured.
The Bottom Line
Lead enrichment is only as good as the research behind it. A single homepage scan gives you a description. A deep crawl gives you the names, titles, contact methods, and social profiles you need to write outreach that actually gets opened. That is the difference between a lead list and a pipeline.