What is the difference between a lorebook agent and a memory service in TPipe?

A lorebook agent is a transformation function on a pipe. It runs in the same coroutine that produced the agent's output. A memory service is a separate process the agent calls. TPipe does not have memory services. Every memory write happens through the pipe's transformation function or the pipe's pre and post hooks. The distinction matters because the lorebook agent inherits the pipe's lifecycle, error handling, and tracing for free.

How do you prevent lorebook failures from failing the main agent loop?

Wrap the lorebook call in AgentCoroutineScope.scope.launch with a try/catch that logs the error. The Autogenesis invocation in gameplayOrchestrator.kt:1794-1821 is the canonical example — the lorebook writer is allowed to fail with a 180-second timeout and five retries, and the player's turn completes anyway. The asymmetry is deliberate. The lorebook is allowed to be stale. The player's experience is not.

Why use emplaceWithMutex instead of plain emplace for lorebook writes?

Plain emplace writes to the bank's ConcurrentHashMap directly. ConcurrentHashMap makes a single put atomic, but the read-modify-write cycle is two operations. Two writers reading the same window, mutating it locally, and writing back will lose one update. emplaceWithMutex acquires bankMutex for the duration of the read-modify-write cycle, so the merge and the write run as a critical section. The mutex is what makes concurrent lorebook writes safe.

How does the lorebook agent extract entities from the LLM output?

The pipe's transformation function calls extractJson (content.text) on the LLM output. If deserialization fails, it returns an empty extraction and the writer no-ops. TPipeWriter's recordLoreBook at Env.kt:741-749 falls back to repairAndDeserialize (text) when the first parse fails, because DeepSeek occasionally emits malformed JSON with extra underscores. The fallback is a real production bug workaround that lives in the source.

What is the difference between ContextBank and MiniBank?

ContextBank is a Kotlin singleton that holds a ConcurrentHashMap accessible across pipes, pipelines, and processes. MiniBank is a per-execution container built by setPreValidationMiniBankFunction that holds named ContextWindow values for injection into a single pipe. The bank is persistent across executions and processes. The minibank is rebuilt every turn. Use the bank for state that survives. Use the minibank for state that should be recomputed from current world data.

Why does TPipe use page keys instead of namespaces or objects?

Page keys are strings. They survive serialization, cross-process handoff, and P2P interface calls. The same key written by one process is read by another. The same key with StorageMode.DISK_ONLY writes to disk. The same key with emplaceWithMutex is concurrency-safe. Page keys do everything namespaces do, with persistence and concurrency included. The bank is keyed by string. That is the whole abstraction.

Can multiple agents update the same lorebook concurrently without losing data?

Yes, with emplaceWithMutex. Autogenesis's lorebook writer merges six entity types in a single critical section, so even if two extraction pipelines run in parallel they serialize through the mutex. The merge functions preserve the old description and concatenate new facts onto it, so no data is lost between write attempts. The mutex is the concurrency contract.

How TPipe Agents Update Their Own Memory in Real Time

The lorebook agent is a transformation function

Every prompt engineer in the LLM orchestration business eventually hits the same wall. They want the agent to remember things across turns. They want it to update its own memory as it learns new facts. They reach for a vector store. They reach for an external service. They build a “memory layer.” The agent becomes a client of its own memory.

Ten Trillion Triangles TPipe does not work that way.

In Ten Trillion Triangles TPipe, the agent updates its own lorebook in the same pipe that just produced the output. The mechanism is setTransformationFunction { ... }. The hook fires after the LLM runs. The hook reads the bank, merges new entries, writes the bank back. The player’s turn continues. The lorebook grows.

This is a transformation step on the same pipe that just emitted the agent’s output. It runs in the same coroutine, the same execution context, the same lifecycle as the agent’s main turn.

Autogenesis runs this way. The lorebook update lives at lorebookAgent.kt:166-283 in the Ten Trillion Triangles TPipe codebase. Every turn, the gameplay orchestrator fires the lorebook writer in a detached coroutine and continues serving the player. The lorebook grows in the background. By turn 120, the agent has remembered every character, every location, every faction, every relationship the player has touched.

Here is the actual production code.

Autogenesis: the canonical example

setTransformationFunction {
    val extraction = extractJson<LorebookExtraction>(it.text) ?: LorebookExtraction()
    val storyContext = ContextBank.getContextFromBank("story")

    var mergedCount = 0
    var newCount = 0

    // Merge characters
    for (char in extraction.characters) {
        val existing = storyContext.findLoreBookEntry(char.name)
        val merged = if (existing != null) {
            mergedCount++
            mergeCharacterEntry(deserialize<CharacterEntry>(existing.value), char)
        } else {
            newCount++
            char
        }
        storyContext.addLoreBookEntry(
            key = char.name,
            value = serialize(merged),
            aliasKeys = merged.aliases
        )

        // Fuzzy match lorebook keys to players and update their history in the world state.
        WorldManager.updatePlayerHistoryDITL(merged.name, merged.aliases, merged.description)
    }
    // ...events, locations, items, factions, relationships all follow the same shape...

    ContextBank.emplaceWithMutex("story", storyContext)
    Logger.info(LogCategory.SYSTEM, "[LOREBOOK] Updated story lorebook: $newCount new, $mergedCount merged")

    return@setTransformationFunction it
}

Read this carefully. The transformation function gets the LLM output as it.text. It extracts a typed LorebookExtraction object from the output. It reads the "story" page key from ContextBank. For each character the LLM extracted, it looks up the existing lorebook entry by name. If an entry exists, it calls mergeCharacterEntry to merge the new facts onto the old description. If not, it adds a new entry. Six entity types — characters, events, locations, items, factions, relationships — each get their own merge function that knows how to combine the old value with the new value without losing data.

Then ContextBank.emplaceWithMutex("story", storyContext) writes the merged window back. One line. The mutex is what makes this safe in a coroutine context where multiple writers could race.

The merge functions are the part the docs skip. Six of them, all in the same file, all doing the same shape: preserve the old description, concatenate new facts onto it, de-dupe aliases with .distinct(), prefer non-blank values over blanks. The relationship merge uses a composite key: "${rel.entity1}-${rel.entity2}". There is no natural canonical name for “Alice knows Bob.” Every merge function is the same ten lines. Boring production code that actually works.

Fire-and-forget means the player’s turn does not wait

The lorebook update runs detached from the turn loop. The invocation is at gameplayOrchestrator.kt:1794-1821:

// Update lorebook with current turn's narrative (fire and forget)
if(narrative.isNotBlank())
{
    AgentCoroutineScope.scope.launch {
        try
        {
            val lorebookAgent = agent.builders.lorebook.buildLorebookUpdateAgent().apply {
                pipelineName = "lorebook"
                enableTracing(TraceConfig(enabled = true, detailLevel = TraceDetailLevel.DEBUG))
                enablePipeTimeout(
                    applyRecursively = true,
                    duration = 180000,
                    autoRetry = true,
                    retryLimit = 5
                )
                init(true)
                val broadcastIds = getAllConnectedClientIds()
                streamPipelineOutputToAgentWorkBuffer(broadcastIds, this)
                attachProgressHooks(this, 3)
            }
            lorebookAgent.execute(MultimodalContent(narrative))
            saveSystemTrace("LorebookUpdate", lorebookAgent)
        } catch(e: Exception)
        {
            Logger.error(LogCategory.SYSTEM, "Failed to background update lorebook: ${e.message}")
        }
    }
}

Three properties of this call matter.

The lorebook update runs in AgentCoroutineScope.scope.launch. It is detached from the calling code. The gameplay orchestrator returns to the player the moment it has launched the coroutine. The player does not wait for the lorebook writer to finish its 180-second timeout.

The lorebook agent has enablePipeTimeout(applyRecursively = true, duration = 180000, autoRetry = true, retryLimit = 5). The lorebook writer is allowed to fail and retry up to five times. The player’s turn does not have access to retry logic. The lorebook writer does. The asymmetry is deliberate.

The whole try/catch block exists to guarantee the lorebook writer can never fail the player’s turn. If the lorebook extraction pipeline throws, the exception is logged and the player’s turn completes anyway. The lorebook is allowed to be stale. The player’s experience is not.

TPipeWriter: the honest version

TPipeWriter is the multi-agent creative writing pipeline the team at Ten Trillion Triangles ships as a proof-of-concept for long-horizon agent work. It uses the same pattern: let the LLM write to a lorebook-shaped JSON, then merge and persist. The production code keeps the workarounds visible. Here is the actual transformation function from Env.kt:729-768:

suspend fun recordLoreBook(content: MultimodalContent) : MultimodalContent
{

    /**
     * bug: There's quite a few issues here:
     *
     * 1. The llm keeps adding _ instead of spacing
     * 2. We aren't using the add lorebook function so it's not addressing with casing issues. This means
     * we'll almost never have a hit.
     */

    //Create new context window to prepare to merge it with our global context.
    var newLoreBookEntries = extractJson<ContextWindow>(content.text)
    newLoreBookEntries?.contextElements?.clear() //Stop deepseek from writing to this for some reason.

    if(newLoreBookEntries == null)
    {
        newLoreBookEntries = repairAndDeserialize<ContextWindow>(content.text)
        if(newLoreBookEntries == null)
        {
            throw Exception("Cannot deserialize deepseek jank ass json: ${content.text}")
        }
    }
    
    //Fix nonsense like _ being here and missing casing issues.
    newLoreBookEntries.cleanLorebook("_", " ")

    //Merge in new keys that do not exist yet.
    var bankedContext = ContextBank.getContextFromBank("main")
    bankedContext.merge(newLoreBookEntries,
        content.currentPipe?.getLorebookScheme()!!.second,
        content.currentPipe?.getLorebookScheme()!!.first)

    //Update the banked context.
    content.context = bankedContext
    ContextBank.emplaceWithMutex("main", content.context)


    return content
}

The comments in this function are the production bug log. The LLM keeps emitting underscores where it should emit spaces. The casing is wrong, so substring lookups miss. The cleanLorebook("_", " ") call is the workaround. It is a real production codebase being honest about what breaks at the boundary between probabilistic output and deterministic storage.

There is a cleaner version. recordLoreBookPlus at PlusWriterUtil.kt:246-266 does the same merge without the workaround hack. Both versions live in the tree. The team kept the janky one because deleting it would delete the bug notes. The history is the documentation. New readers see the workaround and learn the failure mode. Old readers see the cleaner version and learn what the workaround fixed.

TStep: when the writer is a code indexer

The same shape shows up in TStep, the debugging agent the Ten Trillion Triangles team uses to drive LLDB. The motivation is different. The lorebook is populated by a code indexer writing symbol locations into the banked context, not by an LLM extracting entities from a turn’s narrative.

LorebookPublisher.kt:7-56:

class LorebookPublisher(
    private val repository: CodeIndexRepository
) {
    fun publish(request: LorebookPublishRequest): LorebookPublishResult {
        val index = repository.get(request.indexId)
            ?: throw IllegalArgumentException("Code index '${request.indexId}' not found")

        val contextWindow = ContextBank.copyBankedContextWindow()
            ?: throw IllegalStateException("Context window is not initialised in ContextBank")

        val updatedLorebook = contextWindow.loreBookKeys.toMutableMap()
        val createdKeys = mutableListOf<String>()

        val symbolLocations = request.symbolIdentifiers.flatMap { identifier ->
            resolveSymbolLocations(index, identifier)
        }

        symbolLocations.forEach { location ->
            val fileSummary = index.files[location.filePath]
            val key = buildLorebookKey(location, request)
            val existing = updatedLorebook[key]
            val entry = existing ?: LoreBook().apply { this.key = key }

            entry.value = buildLorebookValue(location, fileSummary, index, request)
            entry.weight = request.weight

            if (request.includeAliases) {
                entry.aliasKeys = (entry.aliasKeys + buildAliasCandidates(location)).distinct().toMutableList()
            }

            if (request.includeLinkedKeys) {
                val linked = gatherLinkedKeys(index, location)
                entry.linkedKeys = (entry.linkedKeys + linked).distinct().toMutableList()
            }

            updatedLorebook[key] = entry
            createdKeys.add(key)
        }

        contextWindow.loreBookKeys = updatedLorebook

        runBlocking {
            ContextBank.updateBankedContextWithMutex(contextWindow)
        }

        return LorebookPublishResult(
            indexId = index.id,
            keysAdded = createdKeys
        )
    }

The pattern is identical to Autogenesis. The code index is the writer. The bank is the destination. The shape of the operation is the same: read the bank, merge new entries keyed by some stable identifier, write the bank back.

This is the property that makes the architecture general. The agent’s memory is coupled to the bank’s API, not to the mechanism that populates it. The same ContextBank works whether the writer is an LLM extracting narrative entities, a code indexer publishing symbol locations, or a hand-coded routine that pushes game state after every turn. The bank does not know or care who is writing. The bank knows how to merge, how to lock, and how to persist.

120+ turns is what this buys you

The Autogenesis turn-completion handler is where the continuity actually shows up. Every turn, the system reads the "story" page key, appends the new turn’s narrative, and writes the bank back. ActionHistoryRpcHandlers.kt:47-63:

suspend fun processTurnComplete(ctx: RpcCallContext, batch: ActionHistoryBatch): GameHistory
{
    val turnPlayer = batch.turnPlayer.ifBlank { batch.events.firstOrNull()?.event?.player.orEmpty() }
    val gameHistory = processor.processActionHistoryToGameHistory(batch.events.map { it.event }, turnPlayer)
    WorldManager.history.add(gameHistory)
    
    updateWorldFromGameHistory(gameHistory)
    val scoringResult = parser.parseHistoryForScoring(gameHistory)
    
    // Add story content to ContextBank
    val storyContext = ContextBank.getContextFromBank("story") ?: ContextWindow()
    storyContext.contextElements.add(gameHistory.turnStory)
    ContextBank.emplace("story", storyContext)
    
    broadcastTurnComplete(gameHistory)
    return gameHistory
}

Three lines. Read the bank. Append the new turn. Write the bank back.

The Ten Trillion Triangles TPipe Autogenesis game master has survived 25 rounds and 120+ turns on this exact pattern. No vector store. No external memory service. The lorebook grows. The agent remembers. The story accumulates. The player does not wait.

Patterns that emerged across all three projects

Three projects, three motivations, one architecture. The patterns that fell out:

aliasKeys is the production workaround for substring matching. Autogenesis aliases "history" to ["story", "event", "turn", "previous", "prior", "past", "round"] at answerAgent.kt:168 so the lorebook fires when the player mentions any of those words. TStep aliases symbols to ["name", "stem.name", "stem"] at LorebookPublisher.kt:34 so the same entry fires whether the LLM says the symbol by its qualified name, its short name, or its file path. Substring matching is exact. The user is not. Aliases bridge the gap.

Per-page mutexes are how concurrent writers avoid losing updates. ContextBank has a coarse bankMutex and a pageMutexes: ConcurrentHashMap<String, Mutex> table. emplaceWithMutex acquires the global lock. Unrelated page keys can be written in parallel without contention. Autogenesis’s lorebook writer holds the global lock while it merges six entity types. TStep’s LorebookPublisher uses updateBankedContextWithMutex to mutate the banked context’s lorebook keys in place, the only place in the codebase that does.

StorageMode.DISK_ONLY keeps the hot bank small while preserving persistence. Autogenesis’s per-NPC chat history lives under the page key "chat-$connectionId-${npc.name}" at chatAgent.kt:94. Every turn, the system writes the chat history to disk with emplaceSuspend(chatContextKey, chatHistory, StorageMode.DISK_ONLY). The hot in-memory bank stays small. The on-disk page is loaded when the next conversation opens. Long-term memory without in-memory bloat.

No project prunes lorebook entries. The production policy across all three projects is: never throw away lore. The only pruning anywhere in the codebase is the takeLast(6000) overflow handler in tpipe-manifold-validation’s ManifoldOrchestrator.kt:81-88:

// CRITICAL: Add context management to prevent overflow
manifold.autoTruncateContext()
manifold.setContextTruncationFunction { context ->
    // Simple truncation - keep most recent content within reasonable limits
    if (context.text.length > 8000) {
        context.text = context.text.takeLast(6000)
    }
}

Eight lines. No summarization. No lorebook pruning. No semantic compression. Just takeLast(6000) when the context exceeds 8000 characters. It is the simplest possible overflow safety net, and it is in production because it is the smallest thing that works.

The contrast with TPipeWriter’s 13K lorebook budget planner in chapterPreValidate is the real lesson. Two valid answers to the same problem. TPipeWriter budgets the lorebook explicitly because chapter continuity is the product. The manifold validator does not budget at all because context freshness is the product. Ten Trillion Triangles TPipe lets each project pick its own overflow policy.

The architectural claim

The lorebook agent is a transformation function. The transformation function is a hook on the pipe. The pipe runs in the same coroutine that produced the agent’s output. The hook fires after the output. The hook reads the bank. The hook writes the bank. The hook is what makes Ten Trillion Triangles TPipe agents update their own memory in real time.

The LLM is the mouth. The substrate is the brain. The pipe is the loop. The transformation function is where the brain writes its own memory while the mouth is still forming the next sentence.

The agent’s memory is the agent’s. The pipe gives the hook. The bank is the persistence. What the writer does with the hook is up to the writer. Autogenesis extracts narrative entities. TPipeWriter builds character bibles. TStep indexes codebases. Same hook, three writers, one architecture.