Crawl Scripts

To fetch a lot of data from a website — all orders, all records of a year — the AI writes a crawl script from the website's API Doc.

When to use it

When you need a lot of data from a website, such as every record of a month or a year.

How to use it

1. Make sure the website has an API Doc (see Drill).
2. Ask: "Crawl all orders of this month from this site".
3. The AI writes the script. A second AI checks it before it may run.
4. You approve the run on a card (Run / Cancel).
5. Ask "How far is it?" at any time. The data is saved on your computer.

Good to know

• The script uses your sign-in from the open tab; it never sees your password or tokens.
• It requests slowly (at least one second between requests) so the website is not overloaded.
• If the connection drops, run it again: it continues where it stopped.