One builder, many knowledge bases

From one FAQ to anything#

The first assistant only knew our FAQ. The next step was a builder: a page where you name a knowledge base, add files or links, and get a new assistant. Any content, not just ours.

Step 1: getting the text out#

Every source has to become plain text first. Each type needs a different tool:

Cleaning a web page#

A web page is mostly menus, scripts and footers. We:

1. Remove <script>, <style>, <nav>, <footer>, <form>…
2. Prefer the <main> or <article> part if the page has one.
3. Turn block tags (headings, paragraphs, list items) into line breaks, so questions and answers stay on separate lines.
4. Add a space after links and buttons, otherwise side-by-side tabs glue together ("Accessing a loungeBringing guests").

Step 2: chunking with overlap#

Long text is split into passages of about 800 characters, broken at paragraph and sentence boundaries. Each passage repeats the last ~150 characters of the previous one.

Passage 1: [..................answer begins | ...]
Passage 2:               [...answer begins | answer ends.....]
                          └── overlap ──┘

Why 800 characters? Small enough that a match is precise (the passage is mostly about one thing), big enough to hold a full question and answer. It's a tunable setting, not a law.

Step 3: the knowledge base file#

Each knowledge base is one JSON file on the server:

{
  "title": "Priority Pass Lounge FAQ",
  "model": "text-embedding-3-small",
  "samples": ["Are guests allowed?", "..."],
  "sources": [{ "name": "Airport Lounge Access FAQ", "type": "Web", "text": "full text..." }],
  "chunks":  [{ "source": "...", "text": "passage...", "e": "base64 of 1,536 float32 numbers" }]
}

No link back to the website#

A key decision: the knowledge base stores the extracted text, not the URL. The assistant never goes back to the original site, so if the page changes or disappears, your copy still works. (Using someone else's content publicly still needs care: summarise or link, don't republish.)

Sample questions per knowledge base#

Each base gets five sample questions written by the AI from its own content, refreshed whenever a source is added. A Priority Pass base suggests "Are guests allowed?", not questions about a different business.

Step 4: taking it with you#

Three downloads per knowledge base:

The offline assistant: two kinds of search#

1. Keyword search (no internet, no key) using BM25, the classic search-engine formula. It scores passages by how often your words appear, giving extra weight to rare words and adjusting for passage length.
2. AI answers (needs internet + your key): the same RAG as the website, but run from the file: embed the question, compare against the embedded vectors, ask the model.

Key takeaways#

• Every source becomes plain text first; pick the right extractor per format.
• A server that fetches URLs must block private addresses (SSRF).
• Chunk with overlap so no answer is split in half.
• Store the knowledge, not the link: your copy survives the source changing.
• Keyword search (BM25) works offline; embeddings understand meaning. Both are useful.

Quick quiz#

Try it yourself#