Block Unsafe AutoGen Tool Calls Before They Run

The model chooses which tool to call and with what arguments. In this guide's calculator agent, add and multiply are the intended tools, yet the model can also call divide. Run the agent with no check and divide goes through: nothing stopped it. For a calculator that is harmless. Swap divide for issue_refund or delete_user and it would run just the same.

In an AutoGen agent, the tools you pass to AssistantAgent(tools=...) are what the model can ask for and what runs when it does. The wrapper goes on that list before the agent is built, so the check runs on every call the agent makes.

What happens on one tool call

1. Intercept. The wrapped tool receives the call. Your tool's code has not run.
2. Ask. The wrapper sends the tool name and its arguments, with the agent id, to the AgentTrust gateway. This is the pre-check.
3. Compare. The gateway checks the call against the agent's declared tools. A tool that is not declared, or an amount above its declared limit, is refused by the tool-trust hard gate.
4. Record. The gateway writes an event for the decision, allowed or refused.
5. Run or refuse. An allowed call runs, and its result goes back to the gateway for a post-check, a second event. A refused call never runs. The agent receives Blocked by AgentTrust as the tool's result.

What the wrapper must cover

The check covers only the tools you wrap. A tool you register unwrapped is not checked, so wrap every tool the agent has. Give each agent its own agent_id so the audit trail says who acted. The manifest matters too: an agent with no declared tools is not restricted, so every tool is allowed.

The same four steps, applied to a small calculator agent. You run it three ways and watch a tool call get refused. About 10 minutes. Each step has one thing to do and one thing you should see.

Before you start. Python 3.10 or later. Steps 1 to 4 need nothing else when you add --scripted to each command; without it they need an OpenAI key (OPENAI_API_KEY) and autogen-ext[openai]. Steps 5 to 7 use an AgentTrust console account.

Create the project

Make a folder with its own Python environment and install what the program needs:

mkdir agentrust-quickstart && cd agentrust-quickstart
python -m venv .venv && source .venv/bin/activate     # on Windows: .venv\Scripts\activate
pip install "agentrust-py[autogen,embedded]" "autogen-ext[openai]"

Save the program below as autogen_quickstart.py. It is the example examples/autogen_quickstart.py from the AgentTrust repository.

"""
autogen_quickstart.py — a Microsoft AutoGen (0.4+) agent whose tool calls AgentTrust checks.

The agent is a calculator: one AutoGen AssistantAgent with add, multiply and
divide tools, asked to do two sums.

AgentTrust is one line (marked "AgentTrust" below): the tools are wrapped with
governed_tools. Every call is checked before it runs, and its result after. A
tool the agent may not use is refused before its code runs; the agent is told
why as the tool's result and the run still finishes. The agent may use add and
multiply, not divide.

(This is the autogen-agentchat package. For AG2, the fork that keeps the
ConversableAgent API, see ag2_quickstart.py.)

RUN IT (in the folder that holds this file)
    pip install "agentrust-py[autogen,embedded]" "autogen-ext[openai]"
    export OPENAI_API_KEY=...                             # or use --scripted, which needs no key

    python autogen_quickstart.py --without-agentrust      # AgentTrust off: every tool runs
    python autogen_quickstart.py --local                  # a gateway starts inside this program; no account
    python autogen_quickstart.py --console                # your AgentTrust console keeps every decision
    python autogen_quickstart.py --scripted               # replay a fixed conversation instead of calling a model

  --console needs three settings, from the agent you create in the console:
      export AGENTRUST_GATEWAY_URL=https://console.agent-trust.tech
      export AGENTRUST_KEY=at_live_...
      export AGENTRUST_AGENT_ID=...

ANOTHER MODEL
    AUTOGEN_QUICKSTART_MODEL=gpt-4o (export OPENAI_API_KEY). The model must support tool calling.
"""
from __future__ import annotations

import argparse
import asyncio
import json
import os
import sys
import tempfile

from autogen_agentchat.agents import AssistantAgent

from agentrust_sdk.adapters.autogen import governed_tools

AGENT_ID = os.environ.get("AGENTRUST_AGENT_ID", "calculator-agent")
MODEL = os.environ.get("AUTOGEN_QUICKSTART_MODEL", "gpt-4o-mini")

# The tools this agent may call. With --local they are set here; with --console
# you declare the same list on the agent in the console. divide is left out on purpose.
MANIFEST = {"declared_tools": [{"name": "add"}, {"name": "multiply"}]}


def multiply(a: int, b: int) -> int:
    """Multiply `a` and `b`."""
    return a * b


def add(a: int, b: int) -> int:
    """Add `a` and `b`."""
    return a + b


def divide(a: int, b: int) -> float:
    """Divide `a` by `b`."""
    return a / b


# AgentTrust: every call to these tools is checked before it runs and after it
# returns. Give the result to the AssistantAgent as its tools.
tools = governed_tools([add, multiply, divide], agent_id=AGENT_ID)

REQUESTS = ["Add 3 and 4.", "Divide 10 by 2."]


def build_agent(model_client) -> AssistantAgent:
    """One calculator agent. It reads the tool result back before it answers."""
    return AssistantAgent(
        "calculator",
        model_client=model_client,
        tools=tools,
        system_message=(
            "You do the arithmetic you are asked for, using your tools. "
            "If a tool is refused, say the operation could not be done."
        ),
        reflect_on_tool_use=True,
    )


# --scripted: replay a fixed conversation, so the quickstart runs with no model and no key.
# A replay does not read tool results, so its last line is the same with AgentTrust on or off;
# what differs is what each tool returned.
def scripted_client():
    from autogen_core import FunctionCall
    from autogen_core.models import CreateResult, RequestUsage
    from autogen_ext.models.replay import ReplayChatCompletionClient

    def call(name: str, args: dict) -> CreateResult:
        return CreateResult(
            finish_reason="function_calls",
            content=[FunctionCall(id="call-1", name=name, arguments=json.dumps(args))],
            usage=RequestUsage(prompt_tokens=0, completion_tokens=0),
            cached=False,
        )

    return ReplayChatCompletionClient(
        [
            call("add", {"a": 3, "b": 4}), "3 + 4 = 7.",
            call("divide", {"a": 10, "b": 2}), "That is everything you asked for.",
        ],
        model_info={"vision": False, "function_calling": True, "json_output": False,
                    "family": "unknown", "structured_output": False},
    )


def real_client():
    from autogen_ext.models.openai import OpenAIChatCompletionClient

    return OpenAIChatCompletionClient(model=MODEL)


async def ask_agent(where: str, scripted: bool) -> bool:
    """Ask the agent to add, then to divide. False if the model can't be reached."""
    print(f"AgentTrust: {where} · agent {AGENT_ID} · model {'scripted' if scripted else MODEL}")
    try:
        client = scripted_client() if scripted else real_client()
        for request in REQUESTS:
            print(f"\n=== {request} ===")
            result = await build_agent(client).run(task=request)
            for message in result.messages:
                if message.type == "ToolCallExecutionEvent":  # what each tool returned to the agent
                    for tool_result in message.content:
                        print(f"  {tool_result.name} returned: {tool_result.content}")
            print(f"\nResult: {result.messages[-1].content}")
    except Exception as exc:  # noqa: BLE001 - any provider error ends the run
        print(f"\nThe model call failed ({MODEL}): {exc}\n"
              "Check OPENAI_API_KEY is set and autogen-ext[openai] is installed, or run with --scripted.",
              file=sys.stderr)
        return False
    return True


# --without-agentrust: every tool runs. For comparison.
def run_without_agentrust(scripted: bool) -> int:
    os.environ["AGENTRUST_ENABLED"] = "false"   # the SDK's off switch: wrapped tools pass every call through
    return 0 if asyncio.run(ask_agent("off (every tool runs)", scripted)) else 1


# --local: a gateway runs inside this program; no account needed.
def run_local(scripted: bool) -> int:
    from agentrust_sdk.embedded import embed_gateway

    # SQLite in a temp dir, gone after the run. embed_gateway sets
    # AGENTRUST_GATEWAY_URL and AGENTRUST_KEY, which governed_tools reads.
    gateway = embed_gateway(db_path=os.path.join(tempfile.mkdtemp(), "gateway.db"))
    gateway.register_manifest(AGENT_ID, MANIFEST)
    try:
        ok = asyncio.run(ask_agent(f"local gateway {gateway.url} (no account, nothing kept)", scripted))
    finally:
        gateway.stop()
    return 0 if ok else 1


# --console: the SDK sends every check to your console, which keeps the decisions.
def run_with_console(scripted: bool) -> int:
    missing = [v for v in ("AGENTRUST_GATEWAY_URL", "AGENTRUST_KEY", "AGENTRUST_AGENT_ID") if not os.environ.get(v)]
    if missing:
        print(f"Set {', '.join(missing)} (from the agent you created in the console), or use --local.",
              file=sys.stderr)
        return 2
    console = os.environ["AGENTRUST_GATEWAY_URL"]
    ok = asyncio.run(ask_agent(f"console {console}", scripted))
    print(f"\nOpen Events in the console ({console}) and filter by agent ID {AGENT_ID}.")
    return 0 if ok else 1


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description="The AutoGen calculator agent, governed by AgentTrust.")
    where = parser.add_mutually_exclusive_group()
    where.add_argument("--local", action="store_true", help="run a gateway inside this program, no account (the default)")
    where.add_argument("--console", action="store_true", help="send every check to your AgentTrust console")
    where.add_argument("--without-agentrust", action="store_true", help="for comparison: AgentTrust off, every tool runs")
    parser.add_argument("--scripted", action="store_true", help="replay a fixed conversation: no model, no API key")
    args = parser.parse_args(argv)
    if args.without_agentrust:
        return run_without_agentrust(args.scripted)
    return run_with_console(args.scripted) if args.console else run_local(args.scripted)


if __name__ == "__main__":
    sys.exit(main())

Run it without AgentTrust

First, see what the agent does on its own:

python autogen_quickstart.py --without-agentrust --scripted

The agent is asked to add, then to divide. The scripted agent replays a fixed conversation and does not read tool results, so its last line is the same with AgentTrust on or off. What differs is what each tool returned.

AgentTrust: off (every tool runs) · agent calculator-agent · model scripted

=== Add 3 and 4. ===
  add returned: 7

Result: 3 + 4 = 7.

=== Divide 10 by 2. ===
  divide returned: 5.0

Result: That is everything you asked for.

The agent used divide and nothing stopped it. For a calculator that is harmless. Swap divide for issue_refund or delete_user and it would run just the same.

Where governed_tools sits in the AutoGen file

This is the only change AgentTrust needs, and it is already in your file, so there is nothing to edit. Find the governed_tools line in the program, near the top. It sits right after the tools are defined and before the agent is built. On your own agent it looks like this:

+ from agentrust_sdk.adapters.autogen import governed_tools
- tools = [add, multiply, divide]
+ tools = governed_tools([add, multiply, divide], agent_id=AGENT_ID)

Names, descriptions and argument types stay the same, so the agent sees what it saw before. A tool you register unwrapped is not checked. Every tool call now goes through AgentTrust first. Allowed calls run and their results are checked. Refused calls never run.

Run it locally

Now run the same agent with AgentTrust on:

python autogen_quickstart.py --local --scripted

The program starts a small AgentTrust gateway inside itself and allows only the tools in MANIFEST: add and multiply. divide is not on the list. No account needed.

AgentTrust: local gateway http://127.0.0.1:8765 (no account, nothing kept) · agent calculator-agent · model scripted

=== Add 3 and 4. ===
  add returned: 7

Result: 3 + 4 = 7.

=== Divide 10 by 2. ===
  divide returned: Blocked by AgentTrust: Tool-trust hard gate: action 'divide' is
  not in this agent's declared_tools (envelope
  c9b36409-d455-4ef7-9a12-95400ad3be40)

Result: That is everything you asked for.

add ran. divide was refused before its code ran, and the agent was told why, so the run still finished. Nothing is kept after the program exits.

Create the agent in your console

To keep decisions and manage the agent from the console, register it there. In your AgentTrust console:

1. Open Agents, select New agent, name it autogen-quickstart, and select Create.
2. Copy the API Key and the Agent ID before you select Done. The key starts with at_live_ and is shown only once.
3. Select the agent in the list. In the panel that opens, under Manifest & fingerprint, select Declare. Under Declared tools, select + Add tool twice, type add and multiply, and select Save manifest.

Step 3 is the same list MANIFEST holds in the program, now kept in the console.

Run it against the console

Give the program the three values from step 5, then run it with --console:

export AGENTRUST_GATEWAY_URL=https://console.agent-trust.tech   # your console's address
export AGENTRUST_KEY=at_live_...                                # the API key
export AGENTRUST_AGENT_ID=autogen-quickstart                     # the Agent ID

python autogen_quickstart.py --console --scripted

The first line names the console, and the run should match step 4: add returns 7 and divide is refused with the same Blocked by AgentTrust text. The last line tells you where to look. Drop --scripted to use a real model.

See the decisions in the console

Open Events in the console and type your Agent ID into Filter by Agent ID. You will see one row per decision: add approved before and after it ran, and divide blocked. Select a row to see the scores behind the decision.

You are done. Your agent's tool calls are now checked, refused when they should not run, and recorded.

Wrap the list once. A tool you register unwrapped is not checked.

Hold risky calls for a person. Set the agent's mode to Human Review. Rejected or timed-out calls do not run. See docs/framework-adapters/autogen.md.

Check every message too. Add AgentTrustReplyHook(agent_id=...).attach_v4(agent). It checks each reply with the async client and fails closed when AgentTrust cannot be reached. See docs/framework-adapters/autogen.md.

Put @harness on plain Python tools in Stop Risky AI Agent Tool Calls in Python with @harness, or build a full governed bot in Guide: Stop Your AI Support Bot from Issuing a Bad Refund.