Skip to main content

The Autonomous Brain - Engineering AI for Continuous Learning and Memory Enrichment

• 8 min read
AI Memory LLM

We’ve built an AI with short-term recall, deep semantic long-term memory, and dynamic prompt engineering that curates context for every conversation. The AI can use its memory with real finesse.

But what if it could also manage and enrich its own memory? What if it could autonomously process new information, turning raw conversational data into structured, reusable knowledge?

That’s where we shift from an intelligent AI to a continuously learning AI system.

Today we’re looking at the background processors: autonomous agents that work behind the scenes, using the AI’s own capabilities to build and refine its Long-Term Semantic Memory (LTSM). The whole system gets progressively smarter, more knowledgeable, and more personalised over time.

AI to Self-Improve: The Role of Background Processors

In a serious AI system, memory management isn’t a purely human task. It’s baked into the AI’s operational loop. Our system uses two key background processors, both driven by LLMs:

  1. The SummaryProcessor: The diligent “historian”, responsible for abstracting conversation segments into concise, token-efficient summaries.
  2. The CriticalInfoProcessor: The sharp-eyed “private investigator”, tasked with extracting and categorising crucial facts about the user and their environment.

Both run asynchronously. They don’t block the main conversation flow. (Another key AI UX decision: the user gets an immediate response while the system quietly gets smarter in the background.)

The Historian Revisited: SummaryProcessor for LTSM Enrichment

We first met the SummaryProcessor in Part 2, where it pruned our Short-Term Conversational Memory (STCM) by replacing verbose messages with token-efficient summaries. But its mission goes beyond housekeeping. Each summary it generates isn’t just for the STCM; it’s also a vital piece of enduring knowledge for the LTSM.

// processing/summaryProcessor.js

// ... constructor receives shortTermMemory, longTermMemory, and the LLM 'model' ...

  async summarizeAndCleanup(conversationId) {
    const recentMessages = await this.shortTermMemory.getMessagesForSummarization(conversationId);

    if (recentMessages.length === 0) {
      return;
    }

    // AI-driven summarisation: Use an LLM to abstract the messages.
    // The prompt used here is key for guiding the LLM to create an effective summary.
    const summary = await this.model.summarize(recentMessages);

    // CRITICAL for LTSM: Store the summary in both memory layers.
    await this.shortTermMemory.saveSummary(conversationId, summary);
    await this.longTermMemory.storeSummary(conversationId, summary); // This populates our semantic store!

    // Clean up STCM, effectively replacing raw messages with their abstract summary.
    const messageIds = recentMessages.map(m => m.id);
    await this.shortTermMemory.deleteMessages(messageIds);

    console.log(`Successfully summarized and cleaned up ${messageIds.length} messages for conversation ${conversationId}.`);
  }
$

Summarisation as Implicit Knowledge Graph Construction

Every summary stored in the LTSM (with its corresponding vector embedding) is essentially a node in an implicit knowledge graph. When a user interacts, the SessionManager’s semantic search traverses this graph to find relevant nodes.

Raw conversation gets transformed into discoverable, reusable semantic units. No human intervention required. This is a foundational pattern for building self-improving AI.

The Private Investigator: CriticalInfoProcessor for Personalisation

Summaries capture the essence of conversation segments, but they don’t always pinpoint highly specific, reusable facts about the user. For that, we need the CriticalInfoProcessor, an AI agent built for personalised knowledge extraction. It listens to user messages, identifies important details, and explicitly stores them in the LTSM. Things like:

  • “My dog’s name is Buster.”
  • “I work as a software engineer.”
  • “My favourite holiday destination is Thailand.”

These seemingly small details are gold for personalisation.

The processor uses a two-stage AI-in-the-loop pipeline to balance accuracy with computational cost:

Stage 1: The AI Triage, Cost-Effective Pre-Screening

Every user message first goes through a lightweight, cost-effective LLM check. Its sole purpose: act as a “triage nurse” asking, “Does this message likely contain a new, critical, durable piece of information about the user?”

// processing/criticalInfoProcessor.js

// ... constructor receives longTermMemory and two LLM 'models' ...

  async processUserMessage(message, conversationId) {
    // isInfoCritical uses a cheaper, faster LLM for a quick check.
    const isCritical = await this.isInfoCritical(message);

    if (isCritical) {
      // If the triage nurse says 'yes', escalate to the specialist.
      const facts = await this.extractCriticalInformation(message);
      if (facts && facts.length > 0) {
        for (const fact of facts) {
          // Store each extracted fact in the LTSM.
          await this.longTermMemory.storeCriticalFact(conversationId, fact);
        }
      }
    }
  }

This pre-screening is a vital token economy optimisation. Most user messages are simple acknowledgements (“okay,” “thanks,” “got it”) and don’t contain extractable facts. By using a cheaper model for the initial check, we avoid running expensive, more powerful LLMs unnecessarily. Saves both cost and latency.

Stage 2: The AI Specialist, Detailed Fact Extraction and Structuring

If the triage nurse flags a message as potentially containing critical information, it gets escalated to a more powerful LLM. This “specialist” model is prompted to precisely extract, categorise, and score the facts.

// processing/criticalInfoProcessor.js

  async extractCriticalInformation(message) {
    // The prompt is crucial for instructing the LLM on what and how to extract.
    const prompt = `
      Analyze the following user message. If it contains any new, factual, and durable information about the user (e.g., their name, profession, preferences, plans, personal details, etc.), extract it.
      Format the output as a JSON array of objects, where each object has:
      - "fact": A concise, standalone statement of the fact.
      - "category": A classification (e.g., "Personal", "Professional", "Preference", "Plan", "Location").
      - "confidence": A score from 1-10 indicating certainty.
      If no critical information is found, return an empty JSON array [].

      User Message: "${message}"
      JSON Output:
    `;

    // this.extractionModel is a more capable LLM
    const response = await this.extractionModel.generateContent(prompt);

    // Robust parsing and validation of the LLM's JSON output is essential here.
    try {
      const parsedFacts = JSON.parse(response.text);
      if (Array.isArray(parsedFacts)) {
        return parsedFacts;
      }
    } catch (e) {
      console.error("Failed to parse critical info extraction from LLM:", response.text, e);
    }
    return [];
  }

For a user saying, “My name is John, and I’m a big fan of sci-fi novels,” this might yield:

[
  { "fact": "User's name is John.", "category": "Personal", "confidence": 10 },
  { "fact": "User is a big fan of sci-fi novels.", "category": "Preference", "confidence": 9 }
]

These structured facts get stored in the LTSM, complete with vector embeddings, making them immediately available for semantic retrieval in future conversations.

Structured Extraction for Reusability

Instructing the LLM to return JSON is a powerful technique for structured information extraction. The fact can be directly injected into a prompt. The category can be used for filtering. The confidence score could influence how strongly the AI relies on that fact.

This goes beyond summarisation. It’s about building a structured knowledge base directly from raw text, all powered by AI.

The Continuously Evolving AI

With both the SummaryProcessor and CriticalInfoProcessor working in concert, the system becomes truly dynamic. Every conversation contributes. Summaries enrich the general knowledge graph, while critical facts build a detailed, evolving profile of the user.

The AI isn’t just remembering; it’s learning. The more you interact with it, the more personalised and contextually aware it becomes. That’s Context Engineering doing its job.

What’s Next? The Window into the Mind

We’ve built a self-improving AI system. Now, how do we actually interact with it? How do users (and developers) peer into its memory and control its functions?

In the final instalment, we’ll tour the Command-Line Interface (CLI), the direct window into the AI’s mind and its capabilities.