Importing your website into an AI knowledge base: which pages to exclude first

To import a website into a chatbot knowledge base, you enter a site or sitemap address and wait. The catch: the crawler may also collect the cart, tag pages, numbered blog listings and last year's promotions. The assistant may then answer from outdated terms or from a page with no real content. Before you add the source, spend an hour on an address review and an exclusion list.

This guide covers the import mechanics: address groups, exclusion rules, canonical tags, robots.txt and post-import tests. Which kinds of content belong in the knowledge at all is covered in the guide to choosing data for chatbot knowledge. Example addresses come from Greenleaf, a fictional garden shop.

Which pages to exclude from a website import

Exclude addresses that do not answer a customer question or describe something no longer valid. The table shows typical groups for the fictional Greenleaf shop; treat it as a starting point.

↔ Scroll the table sideways to see all columns.

Do not exclude the whole blog by reflex. A plant care guide may answer pre-purchase questions well, while a category listing of posts does not. Decide per address group, which is why you start from the sitemap.

Review the sitemap and group addresses by pattern

A sitemap is the quickest list of addresses the site itself declares as important. Open it in a browser, usually at /sitemap.xml or the address named in robots.txt, and copy the URLs into a spreadsheet. SEO plugins often split it into files for posts, pages, products and tags. That split already suggests your first groups.

1. List the addresses.Copy the URLs from the sitemap and its sub-files into one spreadsheet column. Without a sitemap, walk through the menu, footer and a few categories.
2. Extract patterns.Next to each URL, note the first path segment and any parameters, such as /tag/, /offers/ or ?orderby=. Sort the column so repeated prefixes sit together.
3. Decide per group.Mark each group import or exclude, with a short reason. Split mixed groups such as /offers/ into current and expired campaigns.
4. Write the rule.Next to each exclude decision, enter the rule type and value. This column helps at the next refresh and at any handover.

Watch for addresses that are missing from the sitemap but appear in on-page links, such as product filters. The crawler follows links even when it starts from a sitemap, so these variants can be fetched too.

Prefix, phrase or pattern: how to write an exclusion

In Meet AI, a website or sitemap source has an Address exclusions field with three rule types. “Starts with” needs a full address including https:// and excludes everything beginning with it. “Contains” checks whether a phrase appears anywhere in the address, parameters included. “Matches pattern” supports an asterisk that stands for any string of characters.

A pattern must match the whole address or the whole path. So /tag without an asterisk covers only the /tag page, while /tag/* covers every tag page. Choose “Contains” phrases carefully: the bare word “account” would also exclude a post about opening a loyalty account. A phrase with a slash, such as /my-account, is safer.

Canonical and robots.txt: what the crawler handles itself

The Meet AI crawler respects robots.txt and skips addresses blocked by a Disallow rule. Do not add robots.txt blocks just to slim down the assistant's knowledge. A rule for all bots also affects search engines, while a source exclusion applies only to the knowledge base. Do not assume a noindex tag keeps a page out of the import either; add a rule instead.

Google explains that robots.txt manages crawler traffic and is not a way to keep pages out of search. The document describes Google Search, not how the Meet AI crawler works. Google Search Central: robots.txt.

The crawler stores a page under its canonical address and skips later variants with the same canonical. It also fetches only addresses on the same domain and protocol, so a shop subdomain needs its own source. If sorted listings have no canonical, each one can enter the knowledge as a separate document. Check the code of two variants; if each points to itself, add an exclusion.

Google describes rel=canonical as a strong signal for the preferred version of duplicate pages, not an absolute directive. That does not show how other tools treat canonical tags. Google Search Central: duplicate URLs.

The 250-page limit and depth of 5: where to spend it

One website or sitemap source fetches at most 250 pages and goes no deeper than five link levels from the start address. Addresses excluded by a rule or blocked in robots.txt are not fetched, so they do not use up the limit. Variants sharing a canonical are still fetched and count. In a shop with a large blog, dozens of tag and pagination pages can exhaust the budget before the crawler reaches delivery or returns information.

The crawler works outward level by level, so pages closest to the start get priority. If your key content sits in a help section, consider starting from that section rather than the home page. One site's knowledge base holds up to 100 sources, Help Center articles included. Add PDF price lists and terms as files; the guide to PDF price lists in the knowledge base explains how to prepare them.

Private tests after the import: what to look for in citations

A new source starts working after a knowledge refresh, and the new version becomes active only when publication fully succeeds. Then run private tests. The result shows citations of the sources actually used, and a test has no effect on customers. Ask questions where excluded content would answer differently from current content.

• A question about an expired promotion does not cite a campaign archive page.
• A question about returns cites the current terms, not an archived version.
• No citation points to the cart, account, search results or a tag listing.
• The cited passage holds real content, not just the menu, footer or a no-products notice.
• An off-topic question ends with no match or a handoff to a person.

If a citation points to an excluded group, check that the rule matches the address exactly as written, trailing slash included. If an answer is missing despite a correct source, use the diagnosis in the guide on why a chatbot cannot find an answer. A full pre-launch question set is in the guide to testing a chatbot before launch.

Manual refresh after major site changes

The knowledge base does not pull website changes automatically. A site administrator can run a full refresh manually, at most once every 7 days. The 7-day wait starts even if the refresh fails, so finish the exclusion list before the first one. A corrected rule takes effect only at the next one, and until then the assistant uses the active knowledge version.

Plan a refresh after changes that affect URL structure: a new shop theme, a moved blog, a new promotions section or new terms. Before refreshing, review the sitemap again and compare it with the spreadsheet from the first import. A new group such as /outlet/ needs a decision, just as at launch. Content owners and publishing order are covered in updating chatbot knowledge, and editing the answers themselves in preparing a chatbot knowledge base.

Questions before your first website import

Should I import the whole website or the sitemap? A sitemap gives a predictable start, because it lists the addresses the site declares itself. A website import starts from one address and follows links, so it hits filters and URL variants more easily. Both use the same exclusion rules, limits and robots.txt.

Can I exclude one page and keep the rest of its section? Yes. Use a pattern with the exact path and no asterisk, or a “Starts with” rule with the page's full address. “Starts with” also covers addresses that extend that beginning, such as /terms-2022-annex. After the refresh, confirm the result with a private test about that page's content.