Skip to main content

The Autonomous Brain - Engineering AI for Continuous Learning and Memory Enrichment

• 10 min read
AI Memory LLM

Alright, welcome back! So far, we’ve built a remarkably sophisticated AI capable of both short-term recall and deep, semantic long-term memory. We’ve mastered the art of dynamic prompt engineering, ensuring our LLM always receives a perfectly curated context for every conversation. Our AI can use its memory with impressive finesse.

But what if the AI could also manage and enrich its own memory? What if it could autonomously learn and process new information, turning raw conversational data into structured, reusable knowledge? This is where we move from an intelligent AI to a continuously learning AI system.

Today, we’re pulling back the curtain on the AI’s internal scholars: the background processors. These intelligent agents work tirelessly behind the scenes, leveraging the AI’s own capabilities to build and refine its Long-Term Semantic Memory (LTSM), making the entire system progressively smarter, more knowledgeable, and more personalised over time.

AI to Self-Improve: The Role of Background Processors

In a truly advanced AI system, memory management isn’t a purely human task. It’s an integral part of the AI’s operational loop. Our system employs two key background processors, both driven by LLMs, to achieve this:

  1. The SummaryProcessor: Our diligent “historian”, responsible for abstracting conversation segments into concise, token-efficient summaries.
  2. The CriticalInfoProcessor: Our astute “private investigator”, tasked with extracting and categorising crucial facts about the user and their environment.

These processes run asynchronously, meaning they don’t block the main conversation flow. This is another crucial AI UX decision: the user gets an immediate response, while the system quietly gets smarter in the background.

The Historian Revisited: SummaryProcessor for LTSM Enrichment

We first encountered the SummaryProcessor in Part 2, where its primary role was to prune our Short-Term Conversational Memory (STCM) by replacing verbose messages with token-efficient summaries. However, its mission extends far beyond simple housekeeping. Each summary it generates isn’t just for the STCM; it’s also a vital piece of enduring knowledge for the LTSM.

// processing/summaryProcessor.js

// ... constructor receives shortTermMemory, longTermMemory, and the LLM 'model' ...

  async summarizeAndCleanup(conversationId) {
    const recentMessages = await this.shortTermMemory.getMessagesForSummarization(conversationId);

    if (recentMessages.length === 0) {
      return;
    }

    // AI-driven summarisation: Use an LLM to abstract the messages.
    // The prompt used here is key for guiding the LLM to create an effective summary.
    const summary = await this.model.summarize(recentMessages);

    // CRITICAL for LTSM: Store the summary in both memory layers.
    await this.shortTermMemory.saveSummary(conversationId, summary);
    await this.longTermMemory.storeSummary(conversationId, summary); // This populates our semantic store!

    // Clean up STCM, effectively replacing raw messages with their abstract summary.
    const messageIds = recentMessages.map(m => m.id);
    await this.shortTermMemory.deleteMessages(messageIds);

    console.log(`Successfully summarized and cleaned up ${messageIds.length} messages for conversation ${conversationId}.`);
  }

Deep Dive: Summarisation as Knowledge Graph Construction (Implicitly)

Every summary stored in the LTSM (with its corresponding vector embedding) is essentially a node in a nascent, implicit knowledge graph. When a user interacts, the SessionManager’s semantic search capability effectively traverses this graph to find relevant nodes. This transforms raw conversation into discoverable, reusable semantic units, continuously enriching the AI’s understanding without direct human intervention. This is a foundational pattern for building self-improving AI.

The Private Investigator: CriticalInfoProcessor for Personalisation

While summaries capture the essence of conversation segments, they don’t always pinpoint highly specific, reusable facts about the user. For that, we need our CriticalInfoProcessor, an AI agent designed for personalised knowledge extraction. Its purpose is to listen to user messages, identify important details, and explicitly store them in the LTSM. This allows the AI to learn things like:

  • “My dog’s name is Buster.”
  • “I work as a software engineer.”
  • “My favourite holiday destination is Thailand.”

These seemingly small details are gold for building a truly personalised AI experience.

This processor employs a rather clever, two-stage AI-in-the-loop pipeline to balance accuracy with computational cost:

Stage 1: The AI Triage - Cost-Effective Pre-Screening

Every user message first goes through a lightweight, cost-effective LLM check. Its sole purpose is to act as a “triage nurse,” asking: “Does this message likely contain a new, critical, and durable piece of information about the user or their environment?”

// processing/criticalInfoProcessor.js

// ... constructor receives longTermMemory and two LLM 'models' ...

  async processUserMessage(message, conversationId) {
    // isInfoCritical uses a cheaper, faster LLM for a quick check.
    const isCritical = await this.isInfoCritical(message);

    if (isCritical) {
      // If the triage nurse says 'yes', escalate to the specialist.
      const facts = await this.extractCriticalInformation(message);
      if (facts && facts.length > 0) {
        for (const fact of facts) {
          // Store each extracted fact in the LTSM.
          await this.longTermMemory.storeCriticalFact(conversationId, fact);
        }
      }
    }
  }

This pre-screening is a vital token economy optimisation. Most user messages are simple acknowledgements (“okay,” “thanks,” “got it”) and don’t contain extractable facts. By using a cheaper model for the initial check, we avoid running expensive, more powerful LLMs unnecessarily, saving both cost and latency.

Stage 2: The AI Specialist - Detailed Fact Extraction and Structuring

If the triage nurse identifies a message as potentially containing critical information, we then escalate it to a more powerful LLM. This “specialist” model is prompted with a much more detailed task: to precisely extract, categorise, and even score the facts.

// processing/criticalInfoProcessor.js

  async extractCriticalInformation(message) {
    // The prompt is crucial for instructing the LLM on what and how to extract.
    const prompt = `
      Analyze the following user message. If it contains any new, factual, and durable information about the user (e.g., their name, profession, preferences, plans, personal details, etc.), extract it.
      Format the output as a JSON array of objects, where each object has:
      - "fact": A concise, standalone statement of the fact.
      - "category": A classification (e.g., "Personal", "Professional", "Preference", "Plan", "Location").
      - "confidence": A score from 1-10 indicating certainty.
      If no critical information is found, return an empty JSON array [].

      User Message: "${message}"
      JSON Output:
    `;
    
    // this.extractionModel is a more capable LLM
    const response = await this.extractionModel.generateContent(prompt);
    
    // Robust parsing and validation of the LLM's JSON output is essential here.
    try {
      const parsedFacts = JSON.parse(response.text);
      if (Array.isArray(parsedFacts)) {
        return parsedFacts;
      }
    } catch (e) {
      console.error("Failed to parse critical info extraction from LLM:", response.text, e);
    }
    return [];
  }

For a user saying, “My name is John, and I’m a big fan of sci-fi novels,” this might yield structured facts like:

[
  { "fact": "User's name is John.", "category": "Personal", "confidence": 10 },
  { "fact": "User is a big fan of sci-fi novels.", "category": "Preference", "confidence": 9 }
]

These highly structured facts are then stored in the LTSM, complete with their vector embeddings, making them immediately available for semantic retrieval in future conversations.

Technical Callout: Structured Extraction for Reusability

Instructing the LLM to return JSON is a powerful technique for structured information extraction. This isn’t just about pretty output; it makes the extracted facts machine-readable and directly reusable by other parts of our AI system. The fact can be directly injected into a prompt, the category can be used for filtering, and confidence could influence how strongly the AI relies on that fact. This is a leap beyond just summarisation; it’s about building a structured knowledge base directly from raw text, all powered by AI.

The Continuously Evolving AI

With both the SummaryProcessor and CriticalInfoProcessor working in concert, our AI system becomes truly dynamic. Every conversation doesn’t just pass; it contributes. Summaries enrich the general knowledge graph, while critical facts build a detailed, evolving profile of the user. This means the AI isn’t just remembering; it’s learning. The more you interact with it, the more personalised, contextually aware, and effective it becomes. This is the promise of Context Engineering fully realised.

What’s Next? The Window into the Mind

We’ve built a truly intelligent, self-improving AI system. Now, how do we interact with it? How do users (and developers) peer into its sophisticated memory and control its functions? In our final instalment, we’ll take a tour of the Command-Line Interface (CLI), showcasing how it provides a direct window into the AI’s mind and its powerful capabilities. Get ready for the grand tour!