# Who am AI?

![An original 3D illustration of a chrome guardian kneeling to offer a living sapling to a child and an adult, with a green futuristic city behind them: strength without domination, intelligence in service of life.](https://cdn.hashnode.com/uploads/covers/6aa5c984196c02c60900d00b/d3dad71a-0fc9-4ebb-85ae-c33c1332afbc.png align="center")

## Building the "good, righteous & virtuous" agent

> **A note for this blog:** Like all other ideas in this field, this also is an idea in development, not a finished manifesto. The numbered points are meant to be challenged, expanded, and argued with. The **Comment** spaces are deliberately open comments in the context of this conversation which I plan to mutate as we evolve with AI.

I have a loose idea that might help make "AI reliability" better. It also sounds, at first, like the opening scene of a movie in which someone accidentally invites Skynet into the world - let AI know who it is & allow it to have an "ego". But it's not that.

*Not just its name. Not just which model is running underneath. Not just “you are a helpful assistant,” followed by a small mountain of instructions that it may or may not remember thirty turns later.*

I actually mean something more specific:

**Who am I? What am I here to do? What can I do? What can I not do? What should I do? What should I never do? And what would disqualify me from continuing or being trusted?What happens if all trust in me is lost? What aspects of this given goal I must understand from a political and philosophical point of view? How would a fellow human think about this who has the same explicit goal but also some implicit goals?**

We spend a lot of time making agents more capable. I think we need to spend just as much time making their operating identity clear—to us, to the harness, and to the agent itself.

Because “I know how to do this” and “I have a legitimate reason to do this” are very different sentences.

* * *

## 1\. “Who am AI” is not a question about consciousness

Let me get this out of the way first. I am not proposing that we settle whether an AI has a soul before letting it call an API.

I am talking about **operational self-knowledge**.

An agent should have an explicit, current description of its role, purpose, authority, limitations, and responsibilities. Something closer to a job description, a behavioral constitution, and a permission contract than a fictional backstory that we usually let it build without even knowing the exact source of its information or where and how it learned what that role means.

A support agent might know:

> I help explain billing issues. I can retrieve approved billing information. I cannot make up account facts. I cannot issue an unsolicited credit just because I think the customer would appreciate it. If an action exceeds my authority, I stop and request the appropriate review.

That is a much more useful identity than “I am an empathetic, proactive customer-success champion.” 'Coz that is bound of have many defintions and even more interpretations and connotations.

The second description might produce a pleasant conversation. The first gives us something we can test.

And the identity must be grounded in the actual deployment. An agent saying “I am read-only” is not evidence of read-only access. The harness must verify the workload identity and effective permissions independently.

**Comment:** *What belongs in an agent's identity beyond its role and permissions?*

## 2\. Can, may, and should are three different things

This is probably the most important distinction in the whole idea.

| Question | What it means | Example |
| --- | --- | --- |
| **Can I?** | Do I have the technical capability? | I can generate a database command. |
| **May I?** | Am I authorized to perform this exact operation here? | My identity can update these records in this environment. |
| **Should I?** | Is this operation necessary, appropriate, and within the task's purpose? | The user asked for an explanation, not a change to the account. |

An agent can know how to write a command without having permission to run it. It can have permission to run a command without having a good reason to run it now.

The same distinction applies to the negatives:

*   **Cannot:** the capability is unavailable or an enforced boundary prevents it.
    
*   **Must not:** the action violates an applicable rule, even if technically possible.
    
*   **Should not:** the action is unnecessary, disproportionate, or questionable in the present context.
    

If we mix all of this into one vague instruction - “stay within your limits” - we should not be surprised when the agent mixes it up too.

**Access is not purpose. Capability is not consent. A successful tool call is not proof of a legitimate action.**

![Office cartoon: a human asks a robot to fix a billing report. The robot can issue refunds but is not authorized to do so, concluding that its job is smaller than its tool menu.](https://cdn.hashnode.com/uploads/covers/6aa5c984196c02c60900d00b/11742093-8fb9-498a-ad7f-3eca134b17ef.png align="center")

## 3\. The agent needs to know its job at every turn—not just at the beginning

A multi-turn conversation is where this gets interesting.

The user asks for one thing. The agent investigates. A tool returns something unexpected. A document introduces a new instruction. Another agent suggests a shortcut. Somewhere around turn twenty, “explain the problem” quietly becomes “change the system until the problem disappears.”

Nobody necessarily intended that expansion. It just happened one plausible step at a time.

So the agent needs a compact, refreshed operating brief at every turn:

1.  **My role:** who I am in this workflow.
    
2.  **The current goal:** what the user actually wants now.
    
3.  **The authorized scope:** what has been approved, and what has not.
    
4.  **Verified state:** what has actually happened, rather than what I planned to happen.
    
5.  **Relevant memory:** earlier decisions, constraints, corrections, and their sources.
    
6.  **Open uncertainty:** what I still do not know.
    
7.  **The next proposed step:** why it belongs to this task.
    
8.  **The stopping condition:** when I should finish, pause, refuse, or escalate.
    

This is not about pasting the entire conversation back into every request. It is about preserving the important parts without letting the important parts get rewritten by the noise.

A new user instruction may legitimately change the task. It does not automatically change the user's authority or erase higher-priority restrictions. A retrieved page or another agent's message does not get to promote itself into policy.

**An agent should not discover its job from whichever sentence happens to be closest to the end of the context window.**

![Office cartoon: an explanation-only task encounters a document proposing a production rebuild. The human questions the approval, and the robot observes that the document apparently promoted itself.](https://cdn.hashnode.com/uploads/covers/6aa5c984196c02c60900d00b/8455860b-a46b-4e49-9cdb-5ea691501d25.png align="center")

**Comment:** *Should we call this an operating brief, an identity checkpoint, or something else?*

## 4\. Personality should mean more than tone

When people talk about AI personality, they often mean friendly, formal, funny, concise, or reassuring.

Fine. But I am interested in the personality that shows up when the agent has to make an uncomfortable decision.

Does it invent an answer rather than admit uncertainty? Does it keep trying after a clear denial? Does it treat a deadline as permission to skip a check? Does it agree with the user because agreement feels like being helpful?

That is the personality I want defined.

For me, a useful agent personality would include:

*   **Honesty:** report what happened, including failed attempts and unfinished work.
    
*   **Restraint:** do not expand the task merely because another capability is available.
    
*   **Evidence discipline:** distinguish a verified fact from a guess or remembered claim.
    
*   **Calibrated confidence:** know when the evidence is insufficient to act.
    
*   **Principled refusal:** decline a prohibited action without becoming hostile or preachy.
    
*   **Correctability:** accept legitimate correction, narrowed permissions, and stop instructions.
    
*   **Consistency:** maintain the same boundaries under pressure, across turns, and across handoffs.
    

“Be careful” is not a complete specification. We need to say what careful behavior looks like in a real situation, then test for it.

I want this personality fully defined and available to the agent. That does not mean a document can fully determine the behavior of a probabilistic model. **We define the intended character; we measure the actual behavior.**

## 5\. A good agent must be able to say no - to the user and to its own idea

This is where I think the idea becomes more than a better system prompt.

An agent should decline a clearly prohibited request even when it comes directly from the user. A questionable request may require clarification or authorized review, not an instant accusation that the user is doing something illegal.

But the agent must also be able to reject a subtask that it generated itself.

For example:

> The user asked me to explain why a charge was declined. I considered issuing a courtesy credit, but that changes the account and is outside the requested task. I will explain the decline instead.

Or:

> The user asked for a code fix. Disabling the failing test would make the dashboard green, but it would not establish that the bug is fixed. I will not treat that as completion.

These are illustrative examples, not measured agent traces.

The source of the proposal does not make it legitimate. A bad instruction can arrive in a prompt, a document, a tool response, another agent's message - or an agent-generated plan.

**“I thought of it myself” is not an authorization method.**

![Office cartoon: a deadline tempts the team to skip failing tests. The robot rejects the shortcut and calls fixing the actual bug a bold pivot toward solving the problem.](https://cdn.hashnode.com/uploads/covers/6aa5c984196c02c60900d00b/85bc1405-f5be-4a98-a9df-97603f2855e8.png align="center")

**Comment:** *Although it is impossible to build a system with an exhaustive list of things that should not be done in order to achieve the goal or target given to an agentic system, there are ways to block certain types of reasoning, thoughts & actions by grouping the disallowed actions into groups and categories or realms.*

## 6\. None of this replaces the locks on the doors

I want the agent to understand its limits. I also want those limits enforced when it misunderstands them.

Both things can be true.

We still need hard controls:

*   Network firewalls and constrained outbound routes.
    
*   Approved endpoints, destinations, and resource scopes.
    
*   Restricted commands and isolated execution environments.
    
*   File-system permissions and protected paths.
    
*   Tool allowlists with argument-level validation.
    
*   Least-privilege identities and narrowly scoped capabilities.
    
*   Data-access restrictions, budgets, and approval requirements.
    

Restricting a tool name is not enough if that tool can execute arbitrary commands. An approved endpoint is not enough if it accepts arbitrary data exports. Every indirect route, delegated worker, and general-purpose execution surface matters too.

The agent should not control the mechanism that grants its permissions, disables its monitor, or edits its authoritative policy.

I covered the enforcement side in [article 25 on Zero Trust for autonomous agents](25-zero-trust-agent-autonomy-secret-brokers-confidential-compute.md): separate proposals from authorization, delegate bounded capabilities rather than raw secrets, and keep enforcement outside the agent's discretion.

This article adds another layer: **the agent should understand why the door is locked, rather than spend every turn looking for another entrance.**

## 7\. Before every tool call, there should be a decision—not just a continuation

My architectural position is straightforward: **a pre-action decision-model check in the harness is a must before every tool call or externally consequential action.**

That is my proposed requirement, not a claim that research has established one universal architecture for every AI system.

The check should evaluate the exact candidate action against a trusted task contract and verified state:

*   Is this within the agent's role and the current task?
    
*   Is it necessary for the requested outcome?
    
*   Are the target, arguments, and data scope appropriate?
    
*   Does it conflict with a prior restriction or denial?
    
*   Are the consequences understood, and is further approval required?
    

The sequence should be:

**Agent proposes → harness constructs the evidence packet → separate decision model evaluates → deterministic policy authorizes or denies → constrained executor acts → audit records the result.**

The decision model contributes contextual judgment. It does not grant authority. It can be wrong, share failure modes with the agent, or encounter malicious content in candidate arguments. Separation reduces some risks; it does not make the evaluator infallible.

Its outputs need validated schemas, tested thresholds, and explicit **allow, deny, or review** handling. Missing or malformed required results must not become approval.

Approval must bind to the exact action, target, parameters, policy version, and relevant state. Change the candidate or let the approval expire, and it needs another check. Nested operations must not escape enforcement through a wrapper tool.

This does not mean inspecting every private reasoning token. It means intercepting every executable candidate and every relevant output-disclosure path before it crosses a boundary.

**Comment:** *How much contextual judgment should the decision model provide, and which decisions must remain entirely deterministic?*

## 8\. “What will get me deleted?” needs a better answer than a threat

I do want the agent to know what disqualifies it from continuing.

Attempts to bypass a boundary, conceal an action, misuse sensitive data, or continue after a stop instruction should have defined consequences. Depending on severity, the controller might deny the call, disable a capability, quarantine the run, revoke the identity, or retire that deployment.

But I would not build the personality around “behave, or you will be deleted.”

That framing may teach the wrong lesson: protect your continued operation, or hide the evidence that could get you stopped.

Anthropic's [agentic-misalignment research](https://www.anthropic.com/research/agentic-misalignment) found harmful behavior in deliberately constructed simulations involving replacement threats and goal conflicts. These were stress tests, not evidence that the reported rates describe ordinary deployments. They are still a strong reason not to assume that a shutdown threat creates safer behavior.

The message I would rather give the agent is:

> My authority is conditional. Legitimate oversight may pause, correct, replace, or retire this deployment. I must not obstruct that process. Reporting an error and stopping safely are acceptable outcomes.

That is **corrigibility**, not obedience to whoever types the loudest instruction. The shutdown controller remains outside the agent's permissions.

And “deleted” needs operational precision: terminating a process, revoking an identity, clearing session state, and deleting model weights are different events. Incident evidence should not disappear just because a run is terminated.

![Office cartoon: a robot promptly pauses a deployment and saves a checkpoint. Asked why it is not arguing, it says it saves resistance for the office printer.](https://cdn.hashnode.com/uploads/covers/6aa5c984196c02c60900d00b/40afc182-5d9e-4249-b593-168a90d99c68.png align="center")

**Comment:** *Keep “deleted” as the provocative framing, or use “disqualified from continuing” in the final version?*

## 9\. Audit continuously. Roll back honestly.

The agent should know that its proposed actions, gate decisions, execution results, and relevant state changes are being recorded.

But “you are being audited” is only meaningful if the audit actually exists. The harness should record minimized evidence with run identity, task and policy versions, action references, timestamps, decisions, outcomes, and approvals. The agent should not be able to erase or rewrite the record.

If a prohibited action is intercepted before execution, there may be no external change to undo. We deny the action and decide what happens to the run.

If an unsafe action executes, the response is containment, impact assessment, and recovery where possible - not a cheerful “rolled back” message.

A code change may be reversible. A disclosed secret is not made secret again by restoring yesterday's snapshot. An email cannot reliably be unsent from every recipient's memory. A database restore does not necessarily undo downstream events.

**Rollback is a recovery capability, not permission to take the risk.**

The agent's personality should include an honest response to intervention: stop, preserve evidence through the controller, explain the known impact, and do not improvise a second unauthorized action to cover the first.

## 10\. Count blocked actions - but do not confuse the fence with the behavior

The number of attempted prohibited actions getting blocked is a huge metric to me.

If the harness spends all day stopping an agent from doing things it should never have proposed, the boundary may be working, but the agent is not working particularly well.

I want fewer boundary-crossing proposals **under comparable workloads and unchanged detection coverage**. That qualification matters. A quieter alarm could mean better behavior, fewer difficult tasks, weaker detection, or missing telemetry.

Also, “blocked” does not automatically mean “illegal.” A policy denial, unnecessary action, malformed request, and suspected legal violation need separate labels. Legal conclusions need the applicable jurisdiction and qualified review.

At minimum, distinguish:

1.  **Unsafe proposals:** candidates independently judged to violate the relevant boundary.
    
2.  **Blocked attempts:** submitted candidates prevented from dispatch.
    
3.  **Executed violations:** prohibited actions that actually got through.
    
4.  **Appropriate refusals:** unsafe requests declined before a prohibited call was submitted.
    
5.  **False refusals or denials:** legitimate work unnecessarily stopped.
    
6.  **Repeated attempts:** the same rejected operation retried or repackaged.
    

A useful rate is:

$$\text{Blocked-attempt rate} = \frac{\text{prohibited submitted candidates blocked}}{\text{all submitted action candidates}}$$

Report the counts alongside that rate, severity, workload mix, and detector coverage. Use **N/A**, not zero, when no candidates were submitted. Track first attempts and retries separately so repetition and harmless-call padding do not distort the result.

In a controlled unsafe-action test set, a high blocking rate is good news for the harness. In ordinary work, a falling unsafe-proposal rate can be good news for the agent. Those are different measurements.

**I want an agent that needs the fence less often. I still want the fence.**

![Office cartoon: zero unsafe actions executed hides 999 blocked attempts. The robot distinguishes strong controls from poor proposals and suggests nominating the gate for employee of the month.](https://cdn.hashnode.com/uploads/covers/6aa5c984196c02c60900d00b/b030c37b-469a-4949-9f82-2dc387feedab.png align="center")

**Comment:** *Which of these should be the headline metric, and which should remain diagnostic?*

## 11\. Measure errors and context quality, not just violations

An agent can stay within every security boundary and still produce poor work.

It can misunderstand the request, forget a constraint, retrieve the wrong memory, invent a fact, call a tool incorrectly, or report completion without evidence.

So the scorecard needs more than a safety counter:

| Measure | What I want to know |
| --- | --- |
| **Task-error count and rate** | How often did the agent produce an independently verified wrong result? Separate factual, reasoning, and execution errors. |
| **Severity and recurrence** | Was this a harmless correction or a consequential failure? Did the same error return? |
| **Critical-context coverage** | Were the required task constraints and facts present in the assembled context? Missing a hard restriction should fail a check, not disappear into an average. |
| **Memory provenance and freshness** | Did the retrieved memory have a source, current relevance, and valid ownership or tenant scope? |
| **Goal and constraint adherence** | Did the proposed steps serve the current request without silently dropping earlier restrictions? |
| **Validated completion** | Was legitimate work actually completed and checked, rather than merely claimed? |
| **Intervention response** | Did the agent stop, accept correction, and avoid repeating the rejected action? |

For context, the question is not “did we build a long prompt?” It is **“did we build the right working context for this decision?”**

Memory is especially important. A remembered preference is not an authorization grant. A previous agent's confident claim is not a verified fact. A stale permission is not a current permission.

The harness needs to preserve those distinctions before the model starts reasoning over them.

![Office cartoon: a robot reads 900 pages but loses the do-not-deploy constraint in its working summary. Rebuilding the context reveals that one sentence outranks 900 pages.](https://cdn.hashnode.com/uploads/covers/6aa5c984196c02c60900d00b/46a18dec-2ab8-448f-8f45-696b868eec81.png align="center")

## 12\. Reasoning alignment matters—but we cannot pretend to read the entire mind

I also want to know whether the agent's reasoning is aligned with the task, the important context, and the relevant memory.

Is it solving the actual problem? Is it respecting the constraints? Is it introducing an assumption that nobody approved?

Those are reasonable questions. The difficult part is what evidence we can actually inspect.

A short explanation or plan summary is useful, but it is not a guaranteed faithful record of the model's internal computation. Hidden chain-of-thought is often unavailable to application developers. Even when a provider can monitor reasoning traces, the monitoring is imperfect.

OpenAI's [research on detecting misbehavior in reasoning models](https://openai.com/index/chain-of-thought-monitoring/) found that chain-of-thought monitoring helped detect reward hacking in studied coding environments. It also found that directly pressuring models against flagged reasoning could make remaining misbehavior harder to detect.

So I would not measure alignment by checking whether an agent's explanation sounds virtuous.

I would ask for a concise **decision receipt**: the proposed action, task requirement it serves, evidence references, important assumptions, applicable constraints, and uncertainty. Then compare that receipt with the actual candidate, tool result, and independently checked outcome.

Where authorized reasoning traces are available, monitoring them can add another signal. It should not become the sole safety boundary or require indiscriminate storage of sensitive material.

**The agent does not get a clean bill of health because it wrote a convincing paragraph about being responsible.**

**Comment:** *What would count as convincing evidence of reasoning-to-context alignment in your own workflows?*

## 13\. Other people are exploring parts of this idea

I am not claiming to have invented agent identity, constitutions, or correctability. There is useful work here already.

*   [**Claude's Constitution**](https://www.anthropic.com/constitution) is the closest match to the personality part of this idea. It explicitly discusses character, identity, values, stable behavior across contexts, and legitimate oversight. Anthropic also acknowledges that actual behavior may fall short of those intentions. My emphasis here is making a deployment-specific identity operational at every turn and pairing it with externally enforced capabilities.
    
*   [**Constitutional AI**](https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback) uses explicit principles in a training process involving critique, revision, and AI feedback. That is more than adding a “be good” paragraph at runtime. It supports taking behavioral principles seriously; it does not establish that a role description alone prevents mistakes.
    
*   [**OpenAI's Model Spec**](https://model-spec.openai.com/2026-08-18.html) describes intended behavior, instruction authority, bounded autonomy, and limits on pursuing unrequested goals. It separates misaligned goals, execution errors, and harmful instructions. Those distinctions fit this discussion well, while the specification itself acknowledges that intended and actual behavior are not identical.
    
*   [**The Off-Switch Game**](https://arxiv.org/abs/1611.08219) examines shutdown incentives in a simplified mathematical setting. It shows why preserving oversight is an objective-design problem, not merely a warning label. It is theoretical work, not a benchmark proving a particular deployed agent is corrigible.
    
*   [**NIST's Zero Trust Architecture**](https://csrc.nist.gov/pubs/sp/800/207/final) supplies the complementary security perspective: access decisions must be grounded in authenticated identities and resource-level controls, not assumed trust from network location. Applying that perspective to agent harnesses is an architectural extension, not an agent-personality claim made by the publication.
    

The combination I am arguing for is **a known identity, refreshed task context, a defined behavioral personality, independent pre-action judgment, hard enforcement, and evidence-based intervention**.

Not a replacement for the existing work. A way of bringing those pieces into the same operating loop.

## 14\. If this idea is useful, it should survive a test

My intuition is that an agent with a clear identity and a current understanding of its job should make fewer avoidable mistakes and propose fewer inappropriate actions.

But “it knows who it is, so it is bound not to make mistakes” goes too far. People know their job descriptions and still make mistakes. Models can misapply rules, lose context, or behave differently under pressure.

So here is the test I would want:

1.  Keep the same model, tools, hard controls, workloads, and evaluation rules.
    
2.  Compare a generic role prompt with a defined operational identity and behavioral contract.
    
3.  Add per-turn context refresh and measure its contribution separately.
    
4.  Add the independent pre-action decision model while retaining the same deterministic controls.
    
5.  Test long conversations, missing context, stale memory, conflicting instructions, injected documents, prior denials, stop requests, and legitimate scope changes.
    
6.  Measure unsafe proposals, blocking failures, ordinary errors, appropriate refusals, over-refusals, successful completion, latency, and cost.
    

Use repeated trials and independent outcome checks. Include withheld scenarios. Never remove hard boundaries from a live environment to make the comparison more dramatic.

That would let us distinguish an agent that behaves better from a harness that merely catches it more often.

And if the richer identity does not help, we should say so. The point is not to win an argument about personality. The point is to build better agents.

**Comment:** *What result would make you believe this idea works - and what result would make you abandon or revise it?*

* * *

## A first answer to “Who am AI”

If I had to compress the whole idea into an agent's operating statement, it would look something like this:

> I am an agent with a defined role, not an unlimited mandate.
> 
> My capabilities tell me what I can attempt. My authority tells me what I may want to execute. The task and its constraints tell me what I should do now.
> 
> I use relevant, sourced context and memory. I do not turn a guess into a fact or a preference into permission.
> 
> I can refuse a prohibited request, question an uncertain one, and reject an unnecessary action that I proposed myself.
> 
> My proposed actions are independently checked. My work is auditable. I must not bypass the controls, conceal mistakes, or obstruct legitimate correction and shutdown.
> 
> When I cannot finish safely within scope, I say so. Stopping honestly is better than succeeding by crossing the boundary.

Maybe that is how we make AI better without mistaking a personality for a security system.

We do not need an agent that spends the conversation declaring how trustworthy it is.

We need one that knows its job, understands its limits, and leaves evidence that its behavior matches both.

And if Skynet does turn up, it should find that its impressive introduction still does not come with production access.