ENGINEERING NOTEBOOK / PROJECT 01 / AUTONOMOUS AGENTS

Trajecta.

the agent that remembers ↘

A desktop AI agent that learns from experience.

Trajecta is a local-first desktop agent designed to carry useful context across tasks, use tools in a controlled workspace, and turn evidence from previous work into reviewable knowledge. It combines a stateful Deep Agents runtime with a human-controlled memory and skill lifecycle.

Active developmentMIT-licensed sourceLocal-first desktop agent
FIG. 01
Trajecta app iconTRAJECTAStateful AI agent
Built to improve,
not to act without permission.
InterfaceTauri 2 + React 19
AgentDeep Agents + LangGraph
MemorySQLite FTS5 + Qdrant
Model accessBifrost + Ollama
01
THE IDEA

Why build Trajecta?

The problem

An AI assistant can complete a task and still forget the decisions behind it. Giving that assistant filesystem, browser, and external-service tools also creates risk: the model should not be the authority for sensitive actions, and an apparently completed task should not automatically count as proven success.

The approach

Separate execution, retrieval, evidence capture, reflection, and skill activation. Keep agent state durable, keep permissions deterministic, and require explicit review or evaluation before new procedures become executable skills.

01

Act with tools

Multi-step agent tasks can use local utilities, selected workspaces, document tools, web tools, subagents and MCP integrations. Risky actions pass through permission checks.

A tool error is an observation for the agent to recover from, not necessarily the end of the whole conversation.
02

Remember selectively

Retain semantic facts, evidence-linked episodes and approved procedures across sessions. Retrieve relevant context with keyword and vector search instead of loading everything.

Unverified experiences and conflicting instructions are treated conservatively.
03

Improve through evidence

Capture execution trajectories and explicit corrections. A bounded reflection worker proposes lessons for human review; reusable skills have a separate, explicit evaluation path.

Completion does not mean verified success, and a suggestion is not an active skill.
04

Keep people in control

A deterministic allow / ask / deny policy controls tool use, while LangGraph checkpoints support interruptions, approvals, and safe resumptions.

Folder boundaries and optional Docker isolation further narrow what tools can access.
02
SYSTEM DESIGN

Inside the architecture

A thin desktop application talks to a FastAPI backend. The backend prepares a stateful agent and mediates model calls, tool permissions, retrieval, and learning. The important boundaries remain visible in the diagram.

01 / INTERFACETauri 2 + React 19Chat · Knowledge Center · Tasks · Settings
02 / APPLICATIONFastAPI + SSEConversations · Files · Models · Review APIs
03 / AGENT RUNTIMEDeep Agents + LangGraphPlanning · Checkpoints · Interrupt / Resume
04 / LLM ROUTEBifrost gatewayOpenAI · Anthropic · 9Router · Ollama · compatible APIs
05 / TOOL ROUTEAllow / Ask / DenyLocal tools · MCP · HITL · optional Docker
06 / RECALLHybrid retrievalSemantic · Episodic · Approved skills · Documents
07 / IMPROVELearning + ReviewTrajectories · Reflection · Optional evaluation
08 / DURABLE STORAGESQLite · FTS5 · Embedded Qdrant · LangGraph checkpoints
Fig. 02 — Simplified architecture based on the uploaded source repository; the diagram groups related services rather than representing every internal module.
Design decision: responsibilities are separated.

Model routing does not grant tool permissions. A memory hit does not bypass workspace scope. A completed run does not automatically activate a skill. Each boundary is enforced by its own subsystem.

03
REQUEST LIFECYCLE

What happens when you send a task?

The process is designed to stream progress to the UI while retaining enough state to resume after a user decision or regenerate from an earlier turn.

  1. 01

    Validate the request

    FastAPI receives the conversation or scheduled task. Input checks and preflight validation run before SSE headers are emitted; conversation and workspace settings are resolved.

    server/src/chat/service.py
  2. 02

    Select a model and tools

    The runtime resolves the selected model through Bifrost, loads enabled local and MCP tools, takes a permission snapshot, and binds all workspace-sensitive tools to one validated folder.

    server/src/chat/runtime.py
  3. 03

    Retrieve relevant context

    Bounded hybrid retrieval draws from semantic facts, scoped episodes, approved experiences, and active version-matched skills. Retrieval failure is advisory rather than fatal to a chat turn.

    server/src/memory/retrieval.py
  4. 04

    Execute a stateful agent

    Deep Agents and LangGraph handle planning, tools, checkpoints, and branching. Streaming events distinguish output, tool activity, errors, and interruptions.

    server/src/chat/runtime.py
  5. 05

    Ask before sensitive actions

    The policy can hide denied tools or interrupt calls requiring approval. Pending decisions survive through a checkpoint and resume after the user decides.

    server/src/guardrails/policy.py
  6. 06

    Persist the evidence

    The conversation, checkpoint, final outcome, and agent trajectory are recorded. Meaningful tool-using runs can produce scoped episodes and optional reflection jobs.

    server/src/chat/service.py
04
KNOWLEDGE LAYER

A memory system with boundaries

Trajecta separates ephemeral execution state from durable facts, evidence-linked episodes and executable procedures. Raw trajectories are stored independently so useful history can be inspected without treating every event as reliable memory.

01 / Short-term

Working state

Live conversation/thread state, active plan, checkpoints, recent actions, and temporary results. It is the context of a task, not a permanent user fact.

server/src/memory/short_term/store.py
02 / Semantic

Durable knowledge

Useful preferences, project facts, and remembered instructions. The agent store is the source of truth, with searchable SQLite FTS5 and Qdrant mirrors.

server/src/memory/semantic/store.py
03 / Episodic

Evidence-linked history

Bounded summaries from meaningful, finished trajectories reference the original execution evidence. Each episode remains unverified until supported by independent evidence.

server/src/memory/episodic/store.py
04 / Procedural

Approved workflows

Reusable procedures and explicitly promoted skills become loadable instructions. Pending, rejected, or obsolete suggestions do not automatically become executable skills.

server/src/memory/procedural/store.py
WHY SQLITE + QDRANT?

Two search strategies, one local source of truth.

SQLite stores relational records and audit history; FTS5 provides keyword search; embedded Qdrant adds vector similarity when configured. Search is scoped and re-checked against authoritative records. When vector search is unavailable, the application can fall back to local keyword retrieval.

Workspace-scoped episodes, bounded ranking, conflicting-memory suppression, and user-controlled automatic memory are handled in the retrieval path.
05
EXPERIENCE FIRST

How Trajecta learns

The learning design is intentionally not a self-modifying loop that silently rewrites the agent. It distinguishes observed events, suggested lessons, reviewed knowledge and evaluated executable skills.

01Observe

Execution becomes evidence

Completed work leaves append-only trajectory events: tool calls, errors, runtime events, and task outcomes. The evidence is useful even when success is not independently proven.

02Extract

Keep meaningful information

Explicit preferences and corrections can be recorded. Finished tool-using work can be consolidated into episodes; a configured background worker processes meaningful events within budgets.

03Reflect

Suggest, do not silently rewrite

Structured Bifrost-powered reflection generates evidence-grounded lessons and procedure suggestions. The queue supports leases, retries, and restart recovery.

04Review

Human decides

Uncertain corrections and procedure proposals are sent for approve / reject review. Approval of a knowledge note is different from activation of executable code or a skill.

05Evaluate

Optional skill candidate pathway

A separate, explicitly started workflow can mine repeated patterns, replay candidate skills against baseline and held-out tasks, then assess correctness, regressions, costs, and other outcomes.

06Promote

Version, activate, or roll back

A verified candidate can be promoted with approval; the registry tracks versions and supports rejection and rollback. Only active, version-matched skill descriptions enter retrieval.

Important: evaluation is not automatic after every chat.

Normal conversations contribute evidence and, when enabled, reflection suggestions. Creating, benchmarking, and activating an executable candidate is a separate manual or opt-in workflow. This is the behavior documented in the current source.

What the evaluation pathway measures

Task success & correctnessBaseline vs. candidateHeld-out task coverageTool-call accuracyToken & execution costRetries, failures & guardrails

These are evaluation dimensions and checks provided by the project—not published claims of a measured improvement or automatic skill-promotion rate.

06
TRUST & CONTROL

Useful autonomy needs limits

✓ALLOWExecute the enabled tool
?ASKInterrupt for approval
×DENYDo not expose capability

Permissions are stored and enforced by the application. Input and model-context checks handle untrusted content, while LangGraph provides resumable human-in-the-loop decisions for tool calls and questions. Docker-based isolation is an optional execution mode, not a blanket guarantee for every connected service.

Permission boundary

Policy outside the model

Tool access is determined using persisted allow, ask, or deny settings. This is a deterministic application decision rather than a text instruction asking the model to behave safely.

server/src/guardrails/policy.py
Workspace boundary

One folder per conversation

Validated workspace selection is bound at run preparation to file tools, local utilities, and sandbox mounts. Model arguments cannot choose another user’s workspace scope.

server/src/chat/workspace.py
Resilient retrieval

Keyword search survives vector failure

SQLite and FTS5 keep local records available when embedded Qdrant is unavailable. Search checks scopes and authoritative SQLite records; a vector hit alone grants no access.

server/src/memory/storage/vector.py
Agent lifecycle

Checkpoint-backed branching

Editing, resending, and regenerating a message forks a conversation at a recorded checkpoint instead of silently overwriting earlier execution history.

server/src/chat/service.py
07
UNDER THE HOOD

Built in code, not just diagrams

These short, annotated excerpts come from the project source. The GitHub repository is not currently publicly accessible, so source paths are presented as references rather than broken external links. They show three design decisions: bind tools to a workspace, fail softly on retrieval errors, and keep review decisions explicit.

EXCERPT 01Source inspected locally

Binding tools to the validated workspace

The agent does not get to choose a different root for each tool. A permission snapshot and a resolved workspace are used to construct the set of tools for that run.

Python · excerpt
workspace_root = conversation_workspace(conversation, self._settings)
personal_tools = self._personal_tools.for_workspace(workspace_root)

native_tools = personal_tools.filter_tools(
    personal_tools.get_tools(),
    policy,
)
server/src/chat/runtime.py
EXCERPT 02Source inspected locally

Fail-soft memory retrieval

Vector or memory outages should reduce available context, not prevent someone from chatting or asking the agent to complete a task.

Python · excerpt
except Exception:
    # Memory search is advisory; it must never break a chat turn.
    logger.warning("Unified memory retrieval unavailable", exc_info=True)
server/src/chat/runtime.py
EXCERPT 03Source inspected locally

Review before a lesson becomes active

Approved experiences may be retrieved as contextual guidance. This status change alone does not promote the item as an executable skill.

Python · excerpt
if decision not in {"approve", "reject"}:
    raise ValueError("decision must be approve or reject")

new_status = "active" if decision == "approve" else "rejected"
server/src/skills/learning/experience.py

Explore the modules

Runtime & streamingAgent preparation, tool binding, SSE events, HITL resume
server/src/chat/runtime.py
Conversation lifecycleBranching, checkpoints, runs, workspace state
server/src/chat/service.py
Unified memory retrievalScoped search, ranking, conflict handling
server/src/memory/retrieval.py
Reflection workerEvidence constraints, durable jobs, budget controls
server/src/memory/reflection/worker.py
Experience learningFeedback, corrections, reviewed knowledge
server/src/skills/learning/experience.py
Skill evaluationBaseline comparison and evaluation reports
server/src/skills/evaluation/evaluator.py
Permission policyPersisted allow / ask / deny permissions
server/src/guardrails/policy.py
Desktop applicationTauri and React workspace interface
apps/desktop/src
08
ENGINEERING HONESTY

Trade-offs & current scope

Trajecta is under active development. The useful engineering story includes what the system deliberately does not assume.

01

Local-first ≠ model-independent

Core state and search live locally, but hosted models and connected services may still receive data according to their configuration. Ollama offers an optional local inference path.

02

Knowledge ≠ executable skill

An approved experience note can inform context. It does not bypass the candidate, evaluation, and promotion checks required for a reusable executable skill.

03

Reflection ≠ proof

The reflection model is constrained by evidence and budgets, but its observations still need review. Successful task completion and independent outcome verification remain separate.

04

Optional isolation ≠ blanket sandboxing

Docker execution depends on the selected setup and permissions. Trajecta does not claim that every tool or connected action always runs in an isolated container.

PROJECT 01 / TRAJECTA

Agents should learn.
People should stay in control.

Read the implementation, explore the rest of my engineering work, or get in touch to discuss the architecture.