Why machines that reason for us raise the bar
on how well we reason about them
The Short Version
· The unit of risk changed. Agentic AI
chains many steps together, so the thing that fails is a sequence of decisions,
not a single answer.
· Our own reasoning is quietly weakening. A
2025 study of 666 people found heavier AI use correlates with weaker critical
thinking, driven by cognitive offloading: letting the tool do the reasoning,
not just the typing.
· Errors compound silently across steps. Component-level
accuracy doesn’t predict system-level reliability: 95% accuracy per step,
chained ten steps deep, lands around 60% end-to-end.
· Systems don’t crash, they drift. Real
deployments show agents drifting for months (skipping steps, changing behavior)
with every individual output still looking fine to reviewers.
· The fix needs both disciplines at once. Judge
individual outputs (critical thinking) AND watch how behavior evolves across
runs and time (systems thinking). Either one alone misses a different failure
mode.
The Ground Has Shifted
Generative AI answers a prompt and stops.
Agentic AI reasons across multiple steps, calls tools, retries, accumulates
context, and often acts before a human sees the result. That changes what
"reliable" even means.
· The unit of behavior is now a
sequence of decisions, not one prediction. That sequence can legitimately vary
run to run, by design, not by bug.
· McKinsey (Nov 2025): 39% of
organizations adopting agents are still experimenting; fewer than 10% have
scaled a solution across even one business capability.
· Gartner projects 40% of agentic
AI projects will be cancelled by end of 2027, citing cost overruns, unclear
ROI, and inadequate risk controls.
The gap here isn’t "does the
technology work": it’s the distance between a convincing demo and a
trustworthy production system. That gap is exactly what the next two sections
address, from two different angles.
Critical Thinking Under Offloading Pressure
The evidence on individual reasoning is
direct and concerning.
· A 2025 SBS Swiss Business
School study (Michael Gerlich, 666 participants) found a strong negative
correlation between frequency of AI use and critical-thinking performance.
· The mechanism is cognitive
offloading: people hand analysis and judgment to the tool, not just the
busywork, and the faculty weakens from disuse.
· Younger participants (17–25)
showed the highest AI dependence and the steepest decline; higher education
attainment acted as a partial buffer.
· Researchers studying AI in
education call this a genuine paradox, not a verdict: the same tools that can
scaffold deeper reasoning (surfacing counterarguments, structuring a problem)
also make it trivial to skip the reasoning entirely. The outcome depends on how
the tool is used, not just that it’s used.
Agentic systems sharpen this specific risk:
they’re fluent at every intermediate step, not just the final answer, so a
chain of reasoning that looks methodical is easy to mistake for reasoning
that’s correct. That’s classic automation bias, with a shrinking window to
catch it before the agent acts.
Systems Thinking: Why the Unit of Analysis Changes
Donella Meadows’ core argument: a system’s
behavior comes from its stocks, flows, feedback loops, and rules, not from any
single decision inside it. Agentic AI is a near-literal test case, because the
"system" now includes the model, its tools, its memory, other agents,
and the environment it acts on.
· The compounding math: 95%
accuracy per step, chained across 10 steps with no error-correction, lands
around 60% end-to-end success. Analysts call this the hallucination cascade:
improving one component barely moves system-level reliability once errors propagate
unchecked.
· Multi-agent setups add
coordination failures on top: conflicting assumptions between agents propagate
errors that never show up when each agent is tested alone.
· Production systems degrade
differently than expected. A 2026 CIO analysis found agentic systems rarely
fail suddenly: they drift, as prompts are refined, tools are added, and models
are upgraded, with each change looking reasonable in isolation.
· Case in point: a
credit-adjudication pilot ran a reliable income-verification step at launch.
Months and several unremarkable changes later, that step was being skipped in
20–30% of cases, not from a bug, but from the accumulated drift of small,
individually reasonable edits. Every single recommendation still looked fine to
reviewers.
· The Cloud Security Alliance now
frames this as cognitive degradation: systemic, gradual, and largely invisible
to evaluation built around single executions.
The operational takeaway: no single run of
an agent is representative. What matters is behavior across many runs, measured
against a baseline, distinguishing what persists from what’s just noise.
Reliability as an Engineering Discipline
This is where systems thinking becomes
architecture. LangGraph’s approach to agent reliability treats reliability as a
property of the whole execution graph, not any one model call:
· Checkpointing: every node’s
execution is persisted against a thread, so a crash or timeout resumes from the
last saved state instead of re-running everything; durable execution instead of
hoping nothing fails mid-run.
· interrupt(): a node can pause
indefinitely, persist its state, and wait for a human to approve, edit, or
reject a step before continuing, a designed pause point in the graph, not a
vague policy of "a person reviews this eventually."
· Explicit graph structure:
because the workflow is nodes and edges rather than an implicit loop, retries
and guardrail checks attach to specific edges instead of being scattered
through prompt text.
· The caveat that matters:
checkpointing guarantees the workflow resumes, not that side effects (a write,
an email, an external action) happen exactly once. Idempotency has to be
designed in separately. Conflating "it resumes" with "it’s correct"
is exactly the shallow fix Meadows warned about.
Where the Two Meet
Critical thinking and systems thinking ask
different questions, and each is blind to the failure mode the other catches:
|
|
Critical
Thinking |
Systems
Thinking |
|
Core question |
Should I believe this specific output,
right now? |
How does behavior propagate and change
through this system over time? |
|
Scope |
A
single point: one output, one decision |
A
trajectory: many runs, across weeks or months |
|
What it catches / misses |
Catches a wrong answer in front of you;
misses drift that’s invisible in any single run |
Catches that something changed; misses
whether the new behavior is actually wrong without a critical eye on it |
Meadows’ hierarchy of leverage points maps
onto this directly:
· Shallow leverage: patching a
prompt after a bad output. It treats a parameter, and the credit-adjudication
drift shows how easily it’s bypassed by the next round of edits.
· Deep leverage: changing what
gets measured, which decisions require a human checkpoint, and what the system
is actually optimized for. Teams that only ever do the former keep
rediscovering the same failure under a new name.
Practical Habits
For anyone building agentic systems or
relying on their output day to day:
·
Watch trajectories. Monitor behavior,
not just output: establish behavioral baselines and watch for persistent drift;
a single successful run proves almost nothing.
· Design for contained failure. You mostly
can’t prove an agent will always behave correctly. Bound what it can touch
(permissions, rate limits, approval gates on irreversible actions) so drift has
a small blast radius.
· Put humans at named checkpoints. "A
person reviews this" is a policy; interrupt() at a named decision point is
a mechanism. The difference is whether the checkpoint survives the next prompt
tweak.
·
Separate resumability from correctness. Durable
state (it resumes) and correct real-world effects (it doesn’t double-send,
double-write, double-charge) are different guarantees: treat both as
first-class requirements.
·
Protect the habit. Before accepting a
recommendation, ask both "do I actually believe this" and "what
changed to produce this, and would I have noticed if it hadn’t." The
offloading research says this habit degrades through disuse; practice it
deliberately, especially for less experienced team members.
Closing
The more autonomous these systems become,
the more discipline their supervisors need, not less, and in both directions at
once: skeptical of any single output, because fluency isn’t correctness; alert
to the system as a whole, because correctness at every step doesn’t guarantee
correct behavior in aggregate. Neither discipline is new. The agentic era just
stops letting you get away with practicing only one.
Annexure
Best Practices Across the SDLC: Developers, QA, and Beyond
Agentic workflows have already spread past "the coding tool" into planning, testing, review, and release. That means the critical-thinking-plus-systems-thinking discipline from earlier in this piece has to be applied role by role, not handed off to whoever owns the AI budget. Three roles carry most of that weight.
For Developers
The risks specific to code are concrete: agents can produce confident but insecure output, they can install dependencies that don’t exist, and they can be steered by instructions hidden in the repo itself.
· Review it like a new hire’s PR. Agent-authored code still needs full code review, security review, dependency and license checks, and tests before merge, the same gate a first PR from a new contributor would face. Competent-looking output is exactly what automation bias trades on.
· Verify every dependency an agent adds. In one large study, 19.7% of packages recommended by code-generating LLMs didn’t exist, and attackers register those exact names ahead of time ("slopsquatting"). Pin versions, confirm a package is real and maintained, and track provenance before an install step runs unattended.
· Keep agents sandboxed and scoped. A 2025 GitHub Copilot flaw (CVE-2025-53773) let a prompt hidden in a pull request description flip an auto-approve setting and grant full shell access. Treat broad "auto-approve" modes as standing production access, not a convenience toggle.
· \Put the ground rules in a file, not a prompt. A short, version-controlled AGENTS.md or CLAUDE.md changes the rules an agent operates under for every future session. That’s a deep-leverage fix in Meadows’ sense, not a one-off patch to a single bad output.
· Measure the aggregate, not today’s diff. AI-generated code carries roughly 2.74x more vulnerabilities than human-written code, and detection lags about six weeks behind. Track vulnerability and defect-escape rate for agent-authored code as a standing metric, the same way this piece argues for watching behavior instead of a single output.
For QA and Test Engineers
Testing is where volume is easiest to fake and hardest to audit, so the discipline has to shift from counting tests to auditing what they actually verify.
· Introduce autonomy in stages. Pilot agentic test generation and self-healing on regression or high-churn suites before letting it run unsupervised everywhere, the same staged-rollout logic that applies to any high-blast-radius change.
· Judge signal quality, not coverage volume. Agentic QA can grow coverage 5–10x at the same headcount, but a test that "heals" itself by quietly loosening what it checks isn’t passing, it’s a blind spot with a green checkmark. Review what changed in a healed test, not just whether it’s green again.
· Track defect leakage as the real metric. Raw test count is a vanity number. What escapes to production, and how much upkeep the suite demands, are what actually tell you whether agentic QA is working.
· Set the boundary before the agent does. Decide up front whether a test agent may touch product code as well as test code, and require sign-off when it does. The line between fixing a flaky test and changing what the system guarantees is the same shallow-versus-deep distinction raised earlier.
For Architects, Leads, and Whoever Governs the Pipeline
None of the above works without an owner. As implementation shifts to agents, someone still has to decide what the system is for and who answers when it breaks.
· Keep the deep-leverage decisions human. Architecture, security posture, and product judgment should stay explicitly owned by a named person, not covered by a general policy statement.
· Stage the rollout like any risky change. Start with low-risk repositories, write the default permission policy before wide adoption, and name an owner for what happens when it goes wrong, the same way you’d approach any change with real blast radius.
· Build checkpoints into the pipeline, not the policy. Schema and migration changes, auth and crypto code, production deploys, and anything touching kernel-mode or driver-level components are the high-blast-radius points from the reliability section above. Give each one a named human checkpoint instead of trusting that someone is watching.
The throughline is the same one running through this whole piece, just applied at the level of a single repo. Critical thinking asks whether this diff, this test, or this dependency is actually correct. Systems thinking asks whether the rates behind them, vulnerabilities per release, defects escaping to production, how often a checkpoint gets skipped, are moving the right direction. Splitting the work between humans and agents doesn’t make either question optional.
