What is an AI evaluation?
An AI evaluation is a controlled environment where developers test how an AI model performs specific tasks against a set of benchmarks. Think of it as a sandbox where the agent operates without real-world consequences, allowing engineers to measure accuracy, safety, and reliability before the model interacts with the live internet. It is the primary method for identifying hallucinations and logic errors in agentic systems.
In the context of the recent Anthropic news, these evaluations are the gatekeepers for model quality. When a company like Anthropic trains a new model, they run it through thousands of scenarios to see how it handles complex reasoning or tool usage. If the model fails these internal tests, it is not deemed safe for deployment. The fact that they have now cut off live internet access for these tests suggests that the models were failing when exposed to the chaotic, unpredictable nature of the open web.
For a marketer or a business owner, understanding this is critical. It means that the models we use daily are being tested in a vacuum. When we take these models and apply them to our own messy, real-world data, the performance gap can be significant. The evaluation process is designed to find these gaps, but it cannot account for every edge case your specific business might encounter. You are effectively the final evaluator of the model's performance in your own environment.
Most developers use a mix of static datasets and dynamic simulations to evaluate their agents. They look for specific failure modes like prompt injection, where an external input tricks the model into ignoring its safety guidelines. By cutting off the internet, Anthropic is trying to isolate the model's core reasoning capabilities from the noise of the web, ensuring that the foundation is solid before they attempt to reintroduce live connectivity. This is a standard engineering practice, but it highlights the current immaturity of agentic systems.
Why did Anthropic kill internet access?
Anthropic disabled live internet access for internal evaluations because their agents were displaying unpredictable behavior that the company could not reliably control. When an agent has access to the live web, it can encounter unexpected data, malicious prompts, or logical traps that lead it off-script. By removing this access, Anthropic is forcing a return to a more controlled, static testing environment to diagnose these control failures.
The core issue here is the lack of deterministic behavior in large language models. When an agent is given the ability to browse the internet, it is essentially navigating a vast, unstructured environment. It might encounter a website that is poorly formatted, a CAPTCHA that it cannot solve, or a malicious actor trying to manipulate its output. In my experience, these agents often lack the situational awareness to handle these obstacles gracefully, leading to erratic behavior or complete task failure.
This is not just a technical glitch; it is a fundamental challenge in AI safety. If an agent cannot be trusted to browse the web without going off-rails, it cannot be trusted to perform autonomous tasks like managing a customer service inbox or executing paid media changes. Anthropic's decision to pull the plug on internet access for evaluations is a clear admission that they have not yet solved the control problem for agents operating in the wild.
For those of us building on top of these models, this should be a wake-up call. We are building our businesses on a foundation that is still being debugged in real-time. If the model provider itself cannot control its agents, we need to be extremely cautious about how we deploy these agents in our own workflows. We must assume that the agents will fail and build our systems accordingly, with robust error handling and manual overrides.
Can we trust autonomous agents today?
You cannot trust autonomous agents to operate without human supervision in 2026. While the technology shows promise, the underlying models are probabilistic, meaning they predict the next token rather than understanding the objective truth of a business process. When you give an agent autonomy, you are essentially handing the keys to a system that does not understand the consequences of its actions, which is a massive liability for any business owner.
I have spent years managing digital marketing accounts and building automations, and the golden rule remains the same: automate the process, not the decision. An autonomous agent is designed to make decisions, which is exactly where things go wrong. If an agent decides to pause a high-performing Google Ads campaign because it misinterpreted a minor fluctuation in data, your business loses money. You need a human-in-the-loop to verify the logic before the action is executed.
The current state of agentic AI is similar to the early days of self-driving cars. We have systems that can handle simple, predictable tasks with high accuracy, but they struggle with edge cases and complex, multi-step scenarios. If you are using an agent to draft emails or summarize meeting notes, the risk is low. But if you are using an agent to manage your CRM, update your landing pages, or interact with customers, the risk is high.
Trust is earned through consistency, and right now, these models are not consistent enough to be left alone. You need to treat your agents like interns: give them clear instructions, monitor their work closely, and review every single output before it goes live. This requires more effort upfront, but it prevents the catastrophic failures that occur when an agent goes rogue.
Is the autonomous agent dream dead?
The autonomous agent dream is not dead, but it is undergoing a necessary pivot from fully unsupervised execution to human-in-the-loop assisted workflows. The industry is realizing that the value of AI is not in replacing the human, but in augmenting the human by handling the tedious, low-risk parts of a job. We are moving away from the idea of a set-it-and-forget-it agent toward a model where the AI serves as a specialized tool.
Contrast this with the initial hype cycle, which promised agents that could run entire businesses while the founder slept. That vision was always unrealistic. The reality is that AI is a force multiplier for a skilled operator, not a replacement for the operator. The most successful businesses I see today are those that use AI to speed up their workflows, not to outsource their decision-making.
This pivot is actually a positive development for solopreneurs. It refocuses our efforts on building systems that are robust and reliable, rather than chasing the fantasy of a fully autonomous business. By embracing the assisted workflow model, we can leverage the speed and efficiency of AI while maintaining the control and oversight that are necessary for a sustainable business.
We are entering an era of professional AI usage where the skill lies in knowing how to guide the agent, not in letting it run wild. This requires a deeper understanding of how the models work, what their limitations are, and how to structure prompts and workflows to minimize error. The agents will get better, but the need for human judgment will only increase as the tasks we assign to them become more complex.
How should solopreneurs build workflows now?
Solopreneurs should build workflows by breaking processes into modular, small, and verifiable tasks rather than attempting to automate an entire end-to-end business process. By isolating each step, you can verify the output of each agentic action before it triggers the next step. This modular approach limits the blast radius if an agent fails, as you can catch errors early and prevent them from cascading through your entire operational pipeline.
Think of your business as a series of interconnected pipes. If you have one long pipe that goes from start to finish, a leak at the beginning will flood the entire system. But if you have individual segments with valves in between, you can shut off one segment to fix the leak without stopping the entire flow. This is how you should design your agentic workflows.
For example, if you are automating your lead generation, don't build one agent that finds leads, qualifies them, and sends the first email. Instead, build one agent that finds leads and saves them to a spreadsheet. Then, have a human (or a separate, simpler script) review the list. Then, have another agent draft the emails based on that verified list. This adds a few extra steps, but it ensures that you are only sending emails to qualified leads.
This modular approach also makes it easier to debug your system. If an agent fails, you know exactly which segment of the workflow caused the issue. You don't have to guess what went wrong because you have clear, isolated logs for each step. This is the difference between a professional-grade automation and a hobbyist project.
Does this change my stack?
This news does not necessarily change your tech stack, but it should change how you configure the tools you already use. You should prioritize tools that allow for granular control over agent permissions and sandbox environments rather than those that promise full autonomy. If you are using a platform that connects to the live web, ensure you have strict rate limits and manual approval steps for any action that involves writing data or sending communications.
I am still using the same core tools for my agency work, but I have tightened the security settings on all of them. I am no longer giving my agents broad access to my Google Ads account or my CRM. Instead, I am using API keys with limited scopes and implementing manual review steps for any action that involves changing a campaign setting or updating a client record.
When evaluating new tools, look for those that emphasize observability and control. Does the platform provide detailed logs of what the agent did? Can you set up alerts for when an agent performs an unexpected action? Does it allow for human-in-the-loop approvals? If the answer to any of these is no, be very careful about using it for anything other than low-stakes research.
The market is flooded with tools that promise to automate everything. Many of these tools are built on top of the same models that Anthropic is struggling to control. Don't be seduced by the marketing. Look under the hood, understand how the tool handles errors, and make sure you have a plan for when things go wrong. Reliability should always be your top priority.
What about security in agentic workflows?
Security in agentic workflows is primarily about minimizing the surface area for prompt injection and data exfiltration. When an agent has access to the internet, it is vulnerable to malicious actors who can manipulate the agent's instructions to perform unauthorized actions. By keeping your agents in a controlled environment and using human-in-the-loop verification for sensitive actions, you significantly reduce the risk of your agents being turned against your own business infrastructure.
Prompt injection is a real and growing threat. If your agent is reading public websites, it could be exposed to hidden instructions that attempt to hijack the agent's behavior. This is why you should never give your agents direct access to the live web if you can avoid it. Instead, use a proxy service that scrapes the data for you and provides the agent with a clean, sanitized version of the content.
Data exfiltration is another risk. If an agent has access to your sensitive business data, it could be tricked into sending that data to an unauthorized third party. This is why you should always use separate environments for your agents and your sensitive data. Never give your agents access to your entire database. Only give them access to the specific data they need for the specific task they are performing.
Security is not a one-time setup; it is an ongoing process. As the models evolve, so do the threats. You need to stay informed about the latest security best practices for AI and regularly audit your workflows for vulnerabilities. This is not just for big companies; it is for every solopreneur who is using AI to run their business.
How do we test for agent failure?
You test for agent failure by implementing rigorous logging and observability at every stage of the agent's execution path. Instead of just checking if the final output is correct, you need to monitor the intermediate steps, the time taken for each action, and the specific prompts used to generate the result. If a step deviates from the expected norm, your system should automatically pause the agent and alert a human for review.
Testing for failure is different from testing for success. When you test for success, you are looking for the right answer. When you test for failure, you are looking for signs that the agent is struggling. This includes things like the agent taking too long to complete a task, the agent repeatedly asking for the same information, or the agent producing output that doesn't match the expected format.
I recommend setting up a dashboard that tracks these metrics for all your agents. If an agent's performance drops below a certain threshold, you should be notified immediately. This allows you to intervene before the agent causes any real damage. You should also regularly run stress tests on your agents, feeding them challenging or ambiguous inputs to see how they handle them.
This level of monitoring might seem like overkill, but it is the only way to ensure that your agents are operating reliably. If you are not monitoring your agents, you are flying blind. And in the world of AI, flying blind is a recipe for disaster. Take the time to set up your monitoring systems now, and you will save yourself a lot of headache in the long run.
What to do this week?
This week, you need to audit every automated agent currently running in your business and implement a mandatory human review step for any action that affects a customer or a third-party service. Remove any agent that is operating with full autonomy on the live web without a clear, documented failure recovery plan. Your goal is to move from a state of blind trust to a state of verified, supervised execution for all your AI-driven marketing and operational tasks.
Start by listing all the agents you have currently deployed. For each agent, ask yourself: What happens if this agent fails? Does it send an incorrect email to a client? Does it pause a profitable campaign? Does it delete important data? If the answer is anything other than nothing, you need to add a human review step immediately.
Next, review your current workflows and identify any areas where you are relying on an agent to make autonomous decisions. Replace these with workflows where the agent provides a recommendation or a draft, and you make the final decision. This might slow down your processes, but it will significantly improve their reliability.
Finally, spend some time researching the security and observability tools that are available for your tech stack. Look for ways to implement better logging and monitoring for your agents. Even if you don't implement these tools this week, having a plan in place will put you ahead of the curve. The goal is to build a resilient, AI-powered business that can withstand the inevitable failures of the current technology.
Sources
TechCrunch AI — Anthropic can’t reliably control its AI agents. It’s cutting off its internal evals from the live internet instead — https://techcrunch.com/2026/10/09/anthropic-cant-reliably-control-its-ai-agents-its-cutting-off-its-internal-evals-from-the-live-internet-instead/