Settings Configuration
Settings: Robots.txt, Crawler Authentication, Search & Crawlers, and Website
The Settings tab is the configuration layer behind your Supertab Connect deployment. There are four sections in the left-hand menu: Robots.txt, Crawler Authentication, Search & Crawlers, and Website. Most of these are set once and rarely touched again — but it's useful to know where each one lives.
Robots.txt
This is how compliant AI agents discover that your site has a license at all, before they ever try to access your content.
The dashboard shows you the exact line to add to your robots.txt file, for example:
License: https://yourdomain.com/license.xml
Copy and paste this into your robots.txt file, ideally at the top, before any User-agent directives.
Look for the green Verified badge in the top right of this card. It means Supertab has checked your robots.txt and confirmed the License directive is live and pointing to the right place. If this ever shows as unverified, use the Verify button to re-run the check on demand.
This is a one-time setup step — once it's verified, you don't need to touch it again unless your domain changes.
Crawler Authentication
This is where the Crawler Authentication Protocol (CAP) is managed — the enforcement layer that validates whether a bot is holding a valid license token before deciding to allow or block a request.
There's a short explanation at the top of this page describing exactly how token verification works and what happens if CAP is disabled — worth reading if you're new to the concept.
For WordPress users: CAP is handled entirely through the plugin. You can see your current plugin version here, along with a banner if a newer version is available. Keeping the plugin updated is worth doing, since updates typically include performance improvements and compatibility fixes — you can enable automatic updates so you don't have to check manually.
The Edit CAP Configuration button takes you to plugin settings where you can adjust enforcement mode.
API Keys: below the CAP explanation is the API Keys section — this is your Merchant API key, which is what the Supertab Connect SDK uses at the CDN edge to authenticate. For each key you'll see its name, prefix, status, when it was last used, and when it was created.
Security tip: treat your API key like a password — don't hardcode it anywhere. Your CDN's secret store is the right place for it. If you ever need to rotate a key (for security reasons, or if you suspect it's been compromised), click Generate New Key, update the key in your CDN or origin configuration, and delete the old key once the new one is confirmed working.
Search & Crawlers
This section contains a single, important toggle: Allow search crawlers to access the website without a license token. It's marked Recommended.
This setting ensures that standard search engines like Google can still crawl and index your content freely, without needing a license token. Some search crawlers are also used for AI features like summaries and answer generation — turning this toggle off may prevent your content from appearing in search results entirely, so leave it on unless you have a specific reason not to.
Think of this as your SEO safety switch: it ensures your enforcement targets AI training and licensed data use, not regular search indexing.
Website
This section holds your website's core identity in the system:
- Website name — the name you gave your property during setup.
- Base URL — your registered domain.
- Website URN — a unique identifier embedded in your
license.xmlURL path. If you're setting up CDN enforcement manually, this is the value you'll need. A copy button sits right next to it. - Delete Website — permanently removes the website along with all its licenses, deployments, and customer data. This action can't be undone, so use it only when decommissioning a property.