Sources
This section describes the Sources tab in your AI Assistant / Chatbot.
Sources
Sources are your AI Assistant's data. Your AI can have many different sources of data, including your website, files and folders, plain text and more.
Website & URLs
How to use your website's data in your AI Assistants.
Crawling
One convenient and fast way to ingest data is to let us crawl your website. Just enter your website url and we fetch all your pages and retrieve the data for your AI Assistant.
By default, if you enter your website's root URL, i.e. https://my-site.com/, we will start looking for
ALL the pages in your website, starting at the root level.
You can also restrict the crawler to only fetch pages at a given "level" or "base url", for example
https://my-site-com/blog/ will only fetch pages that has the base url you provided, in this case, only
pages that are in the /blog/ area of your website. With this functionality, you have more control over
which pages you want our crawler to fetch.
Recrawling existing links
To recrawl your links, simply use the dropdown menu and choose between recrawling selected links, or recrawl all links. Use this functionality if you know you have some updated content. This will not crawl for new urls, but will fetch updated content on your already added url sources.
Files
Another source of data you can give your AI Assistants is uploading files from your computer. When you upload files, we will fetch all the text in the file and use this as a data source for your AI Assistant.
We currently support the following types of files:
| File format | Mime type |
|---|---|
| .c | text/x-c |
| .cs | text/x-csharp |
| .cpp | text/x-c++ |
| .doc | application/msword |
| .docx | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| .html | text/html |
| .java | text/x-java |
| .json | application/json |
| .md | text/markdown |
| application/pdf | |
| .php | text/x-php |
| .pptx | application/vnd.openxmlformats-officedocument.presentationml.presentation |
| .py | text/x-python |
| .py | text/x-script.python |
| .rb | text/x-ruby |
| .tex | text/x-tex |
| .txt | text/plain |
| .css | text/css |
| .js | text/javascript |
| .sh | application/x-sh |
| .ts | application/typescript |
Plain text & Markdown
Another option to ingest data is to simply copy & paste or just type your text into our markdown editor. You can use plain text or markdown.
Training your AI Assistant
We will automatically parse, chunk, embed and store your files and content, so it is immediately available in your assistants after the source is created. If you update a source (upload a file with same name, recrawl a url etc.), with different content, we automatically update the dataset with your new content.
After you add your sources, you can start asking your assistant questions and it will reply with answers based on your own data. AI answers may vary from time to time and on the quality of the sources you provided and the chosen AI model.