Website Intelligence
Fetch, snapshot, and analyze a client's public website to build the structured evidence base GeoScript reasons over.
Private beta — available to agency partners
This capability is live and working for agencies in GeoScript's private beta program. Learn about agency access →
What Website Intelligence does
Website Intelligence is GeoScript's web crawling and snapshotting capability. It fetches a client's public website on a schedule, stores deterministic content-hash snapshots of each page, and feeds those snapshots into the Content Genome for structured analysis. The crawl is the foundation of GeoScript's understanding of what a business actually says about itself on its own site — the raw material that optimization recommendations are built from.
How it works
Static HTTP crawler with safety rules
GeoScript uses a real static HTTP crawler — not a headless browser or JavaScript-rendering engine in the current version. Pages are fetched over HTTP with SSRF hardening: the crawler refuses to follow redirects to private IP ranges or internal network addresses. Politeness rules prevent aggressive crawling: rate limits and delays are applied per domain to respect server resources.
Deterministic content-hash snapshots
Every page fetch produces a snapshot identified by a deterministic content hash. If a page has not changed since the last crawl, GeoScript detects this via the hash comparison and does not store a redundant snapshot. When a page does change, the new snapshot is stored alongside the old, preserving the change history. This means GeoScript can detect when a client's content changed and attribute recommendation outcomes to specific content states.
Crawl scope and depth
GeoScript crawls the public-facing pages of a client's website within the scope defined by the agency. Internal pages, password-protected sections, and pages disallowed by the site's robots.txt are excluded. The crawl scope is configurable — agencies can focus crawling on high-priority pages.
How crawls feed the rest of the system
Website Intelligence outputs feed directly into the Content Genome, which parses each snapshot into typed content blocks. From there, the Optimization Engine produces typed recommendations. The entire chain — crawl → genome → recommendations — is deterministic: the same page content produces the same genome and the same recommendations every time.
What Website Intelligence does not do
Clear boundaries increase trust. These limitations are documented explicitly, not buried in footnotes.
Website Intelligence does not render JavaScript — it is a static HTTP crawler. Pages that require JavaScript execution to display their primary content may not be fully captured.
GeoScript does not crawl pages behind authentication or paywalls.
GeoScript does not crawl pages disallowed by a site's robots.txt.
The crawler does not monitor third-party sites, backlinks, or references to a client from other domains.
GeoScript does not modify a client's website during crawling — it is read-only.
Related
Other features
Content Genome
Private betaDeterministically extract typed content blocks, structured data, and FAQs from every page into a stable typed representation.
Optimization Engine
Private betaTurn Content Genome findings into typed, deterministic recommendations proposing exact DOM-level changes to improve AI understanding.
GeoScript is in private beta
Agency partners get access to the full capability set — acquisition, monitoring, website intelligence, optimization, deployment, and reporting — for $299/month per active client.