Agent Smith

Agent Smith is a late core Python project (milestone M5; placement Reported by students) where you build an AI agent from scratch. The agent runs a loop: the model writes a short thought and a block of Python, your sandbox executes it, and the output becomes the next observation, until the model calls a final answer. You evaluate it on two benchmarks: MBPP, small functions with tests, and SWE-bench, real GitHub issues that must be fixed with a patch. No agent framework is allowed; the loop, the tools and the sandbox are yours. Details below come from student repositories (Reported by students); check them against your subject.

What you will learn

• The ReAct style loop: thought, action as code, observation
• Prompting an LLM through an OpenAI-compatible API and parsing its replies robustly
• Sandboxing untrusted code: separate processes, resource limits, timeouts, restricted imports, no network
• Exposing tools to a model, including through the Model Context Protocol
• Benchmarking: success rate, iterations, tokens, time, and ablations
• Producing patches as unified diffs and running a repository's tests in Docker

Before you start

• Previous nodes: RAG against the machine and Call Me Maybe (working with LLM outputs), plus Inception-level Docker comfort for SWE-bench.
• Tools: Python 3.10+, uv, flake8, mypy, Docker, and an API key for at least one OpenAI-compatible provider (students use several and rotate keys).
• Read the MBPP and SWE-bench task formats before writing code: they decide what your final answer looks like (function code for MBPP, a diff for SWE-bench).

Plan of attack

1. Day 1: minimal loop. Send a system prompt and a task, extract the first Python code block from the reply, run it in a subprocess, return stdout and stderr as the observation. Stop on a final_answer call or a maximum number of steps.
2. Day 1: robust parsing: models format code blocks differently; handle missing or several blocks.
3. Day 2: the sandbox. A separate process with CPU, memory and file descriptor limits, a timeout, an empty environment (no API keys), no network, and an import allowlist. Docker is the strongest boundary.
4. Day 2: MBPP: a run_tests tool, the final answer is the function source. Measure success on a subset.
5. Day 3: tools for SWE-bench: read, list and search files, edit, run tests, and produce the patch. Reported implementations expose them through MCP servers.
6. Day 4: SWE-bench runs in Docker within the reported time budgets, writing one result file per task with success, iterations, tokens and time.
7. Day 5: benchmark several models and an ablation (for example without a given tool), then lint and write the README with the AI usage section and the results table.

A sketch of the loop (helper names are placeholders):

def run_agent(task: str, max_steps: int = 10) -> str | None:
    history = [system_prompt(), user_message(task)]
    for _ in range(max_steps):
        reply = complete(history)
        code = extract_code(reply)
        if is_final(code):
            return final_value(code)
        history += [reply, observation(execute_in_sandbox(code))]
    return None

Common pitfalls

• Sandbox escapes through introspection (__subclasses__, __builtins__): an import blocklist alone is not a sandbox.
• API keys leaking into the executed code's environment.
• Unbounded loops that burn the token budget.
• Treating the model's text as trusted, for example running shell commands it suggests without limits.
• Results that cannot be reproduced because prompts, models or seeds are not recorded.
• Untyped JSON handling failing mypy --strict.

How it is evaluated

• Peer evaluation plus an automatic check of the result files (Reported by students).
• The evaluator runs the linters, runs the agent on MBPP and SWE-bench tasks, inspects the sandbox and tries to break it.
• Reported requirements include per-task deadlines, a benchmark across several models and an ablation study.
• You explain the loop, the sandbox boundaries and the trade-offs of each tool.

Resources

• ReAct paper on arXiv
• MBPP dataset on Hugging Face
• SWE-bench
• Model Context Protocol
• Python resource module
• Student repo: sousampere 42_agent_smith