The Crawl Tool used to be a SaaS product. It crawled your website, and gave you a lot of technical information so you could fix issues and improve your SEO. The target market was web professionals working with small to medium websites. It operated at a price point well below that of competitors.
But there was an issue. To be competitive it still needed to pack more and more data into charts and have more and more guides and instructions on how to interpret it. And established competitors with significantly higher costs justified them by also packing more and more data into charts.
And here’s the net result: The data gets harder and harder for people to interpret and people paying for these products are simply using very tiny parts of it. Often the industry heavily compresses data into value scores that are marketed as ‘actionable’, but actually say nothing at all.
So people were increasingly turning to things like Claude Desktop to interpret data for them.
Agentic AIs
Claude is part of something called “Agentic AI”. This is an extension of LLM AI models that used to be trained just to chat to you. An agentic AI can use tools and work in a loop using different tools until it has completed a task. This makes AI LLMs much more powerful as they can now perform tasks as an ‘agent’ that people needed to do.
Basically you go from simple chat to being able to answer questions like “Is problem x fixed on the website?”
That is clearly a more powerful way to interpret and action data. Not just compressed scores like “your site scores 78/100” but actual insights and answers. Which explains why people have started the process of moving.
The Problems with Agentic AI
But why haven’t people moved fully? There are essentially two problems. Access to data and “tokens, cost and sustainability”.
Access to Data
An agent like Claude Desktop has a limited amount of data available to it originally. So when you ask it something you can often see it fetching that data, for example by web search. But this data isn’t the whole site you’re asking it about and this creates a problem.
Claude Desktop for example, can do an amazing job if you ask it if a particular url is missing a canonical tag. It’ll fetch it, check for you, understand the html, and let you know.
Now imagine your website is 100 pages and you just want to know which pages are missing canonical tags. Suddenly it needs to fetch 100 pages and it will likely refuse or just check a few.
Part of this problem can be solved by having the right tools available to Claude to access summarised data. But then you’re limited by the summaries and can no longer ask any question. You become trapped into the same compressed data view that is the reason for considering Agentic AI to begin with.
Tokens, Cost, and Sustainability
The text an AI LLM produces is made up of tokens. These could be words (which is commonly used as a simplification of what tokens are) or more likely parts of words. How many tokens an AI provider (like Anthropic for Claude) can provide depends on their compute resources (how many GPUs they have and how fast they are).
This matters for Agentic AI because agentic AI requires the model to reason which takes a lot of tokens and to use and process the responses, such as the html of a web page. Both of these things make the much more powerful modern models we have today more advanced and usable, but they also contribute to an explosion in token usage. The providers cost for producing AI input/outputs has exploded.
Because of the competitive nature of several large companies wanting a dominating early position in the market, when you use an AI LLM then the costs are nearly always heavily subsidised. Often you can subscribe to “plans” that are non-specific about usage quotas which can change at any time. And if you’ve been watching for a while you’ll have noticed that whilst prices for plans have generally stayed the same, quotas are becoming more and more restrictive.
No AI company can subsidise indefinitely and eventually the customer (you) will need to pay the true cost – either by a quota becoming more restrictive or an increase in prices.
So a big issue with things like Claude Desktop is that you end up building processes based on resource usage that actually isn’t sustainable.
How The Crawl Tool Solves These Problems
When you crawl your website with the crawl tool it stores the pages and generates associated indexes and data. Then it provides the AI LLM with generic tools it can crawl.
So instead of the AI having to look at 100 pages to individually find out which are missing a canonical tag it can write a query that looks for the canonical tag and call a tool that returns which pages don’t match.
These AI tools are largely invisible to you as you just want answers to questions. But it is a massive shift. Our product development plain is based on making and adding efficient AI tool based on a specific domain (small/medium crawled websites). That both makes an AI LLM better at helping you but also reduces token usage (for example, you’re no longer trying to put 100 pages of html into the LLM).
A desktop app is hands-down the best way to do an implement this tooling for the small to medium websites that represent our target market.
The second way we solve the problem is by using Local AI with a second choice of Open Router as an LLM marketplace.
Local AI
Local AI does what it sounds like – it lets you run models locally. Tools like LM Studio, Ollama, or llama.cpp are becoming increasingly popular for running AI LLMs locally. Which models will run and at what speeds will depend on your exact hardware. But an 8GB GPU or something like an M2 MacBook Pro can run very capable models. These will be slower than cloud models, but recent models like Qwen 3.6 35B A3B are very capable for the kind of tasks we are talking about here. The chances are high that you have a computer capable of running a local model.
Whilst slower and not quite as ‘bright’ (you might need to rephrase how you ask things sometimes) what running a local AI model does is gives you infinite cost free tokens. You don’t need to worry about how many tokens you use, you don’t need to worry about hitting any invisible changing quotas, because you have infinite token capability for free! You also gain the advantage that it’s private – nobody knows what you’re doing and the AI isn’t getting trained on your processes.
But the biggest gain is that once you have worked out your processes and methods for whatever you want to do, you know you can rely on at least that local model being available to you in the future. A key thing to note is that small models like Qwen have being getting better and better and better at a surprising scale. Whilst part of the AI industry focuses on huge models trained on fantastical amounts of data, there seems to be a push currently to also have capable models that can run on your own machine.
LLM Marketplace
But what if your computer is a dinosaur or you can’t always run a local LLM because you’re a tech geek training your own models? We added in Cloud functionality from an LLM marketplace called Open Router. You can choose to use one of these. You might just do these to test The Crawl Tool before committing to setting up local AI like LM Studio, Ollama, or llama.cpp if you haven’t already.
There are models that have been available for free for a while. Like Minimax M2.5. In The Crawl Tool you can try searching for ‘free’ in the models box. These models tend to be slow and at times unreliable. But they are there for testing or if you need to (for example, you could use The Crawl Tool free and one of these models and pay nothing but maybe some patience).
Top up a small balance into Open Router and a new world opens up to you. Models like: Deepseek v4 Flash, Minimax M2.7, Deepseek v4 Pro, and Qwen 3.6 Pro all do fantastic jobs at a very very low cost. Of course here’s you’re effectively quota’d by your budget again but your budget can be surprisingly small for amazing results because there’s instantly more competition and you’re crossing international borders.
The Future
With those issues solved, the future is clear. At least for small to medium websites, this is the way forward. It’s not only an easier path, it’s a better path to those ‘actionable insights’ everybody claims, and it’s the sustainable path of the future. It’s also a path that literally everybody can take.