feat: consume cogito for agent reasoning by mudler · Pull Request #320 · mudler/LocalAGI

mudler · 2025-10-10T20:36:37Z

Blocked by: mudler/cogito#9
Depends on : mudler/cogito#13

some TODO:

Dropping methods used to extract tools/gaps
Write a Action -> cogito.Tool converter
Wire MCP servers with cogito.WithMCPs() and pass by MCP client sessions
For re-evaluating with goals, we can put that under a flag and use the ContentReview on the resulting fragment of the tool execution loop
QA

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

richiejp · 2025-10-19T12:05:26Z

-				obs.Progress = append(obs.Progress, types.Progress{
-					ActionResult: res.Result,
-				})
-				a.observer.Update(*obs)


Have lines like this been recreated in the new version?

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Copilot

Pull Request Overview

This PR integrates the Cogito library to drive agent reasoning and tool execution, and removes the legacy “old” web UI in favor of the new React UI.

Replace in-house planning/reasoning and MCP tool wrapping with Cogito-based execution and evaluation
Remove legacy server-side templates and HTMX-based UI; simplify Fiber app initialization for React UI
Add Cogito integration points (status callbacks, tool callbacks, MCP sessions); update knowledgebase summarization to use Cogito

Reviewed Changes

Copilot reviewed 34 out of 37 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
webui/routes.go	Removes old embedded views/public routes; keeps React UI assets served.
webui/old/... (multiple)	Deletes legacy HTML/CSS/JS views and assets.
webui/app.go	Drops HTMX/templates engine usage; simplifies Fiber initialization.
go.mod	Adds cogito dependency; updates Go version/toolchain; adjusts deps.
core/types/job.go	Removes nextAction/pastActions; adds cogito.Fragment field.
core/types/actions.go	Adds cogito wrapper to expose Actions as Cogito tools; removes Plannable from interface.
core/agent/templates.go	Simplifies templates to align with Cogito-driven flow.
core/agent/mcp.go	Stops generating Actions for MCP tools; keeps MCP client sessions for Cogito.
core/agent/knowledgebase.go	Uses Cogito LLM to summarize conversation.
core/agent/evaluation.go	Removes custom evaluation; Cogito now handles evaluation.
core/agent/agent.go	Major refactor to Cogito execution, planning, and callbacks; cleans old codepaths.
core/agent/actions.go	Removes custom decision/planning utilities; simplifies available actions.
core/action/* (several)	Removes obsolete actions (reply, reasoning, plan, intention, goal).

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-19T19:54:26Z

+				chosenAction := availableActions.Find(tc.Name)
+
+				var obs *types.Observable
+				if job.Obs != nil {
+					obs = a.observer.NewObservable()
+					obs.Name = "action"
+					obs.Icon = "bolt"
+					obs.ParentID = job.Obs.ID
+					obs.Creation = &types.Creation{
+						FunctionDefinition: chosenAction.Definition().ToFunctionDefinition(),
+						FunctionParams:     types.ActionParams(tc.Arguments),
+					}


Possible nil pointer dereference if chosenAction is nil (e.g., for MCP tools or tools not in availableActions). Guard chosenAction before accessing Definition() and before later uses (job.Callback, SetResult). If nil, skip FunctionDefinition, and either construct a minimal placeholder for observer/callback or branch handling for non-local tools.

Copilot · 2025-10-19T19:54:26Z

+				cont := job.Callback(types.ActionCurrentState{
+					Job:       job,
+					Action:    chosenAction,
+					Params:    types.ActionParams(tc.Arguments),
+					Reasoning: reasoning})
+
+				if !cont {
+					job.Result.SetResult(
+						types.ActionState{
+							ActionCurrentState: types.ActionCurrentState{
+								Job:       job,
+								Action:    chosenAction,
+								Params:    types.ActionParams(tc.Arguments),
+								Reasoning: reasoning,
+							},
+							ActionResult: types.ActionResult{Result: "stopped by callback"},
+						})


Same nil dereference risk: chosenAction may be nil for MCP/external tools. Add a nil check and handle callback/result construction accordingly to avoid panics.

Copilot · 2025-10-19T19:54:26Z

+		toolToCall := fragment.Messages[len(fragment.Messages)-2].ToolCalls[0].Function.Name
+		switch toolToCall {
+		case action.StopActionName:
+			job.Result.Finish(nil)
 			return


Potential out-of-bounds access if fragment.Messages has fewer than 2 entries or if the assistant message has no ToolCalls. Guard lengths before indexing and verify ToolCalls is non-empty to avoid panics.

Suggested change

toolToCall := fragment.Messages[len(fragment.Messages)-2].ToolCalls[0].Function.Name

switch toolToCall {

case action.StopActionName:

job.Result.Finish(nil)

return

// Guard against out-of-bounds access

if len(fragment.Messages) >= 2 {

toolCalls := fragment.Messages[len(fragment.Messages)-2].ToolCalls

if len(toolCalls) > 0 {

toolToCall := toolCalls[0].Function.Name

switch toolToCall {

case action.StopActionName:

job.Result.Finish(nil)

return

}

}

Copilot · 2025-10-19T19:54:27Z

+		fragment := cogito.NewEmptyFragment().AddStartMessage("user", "Summarize the conversation below, keep the highlights as a bullet list:\n"+Messages(conv).String())
+		fragment, err := a.llm.Ask(a.context.Context, fragment)
 		if err != nil {
 			xlog.Error("Error summarizing conversation", "error", err)


If Ask returns an error, fragment might be nil or incomplete; calling fragment.LastMessage() can panic. Return early on error (or ensure fragment is valid) before accessing LastMessage.

Suggested change

xlog.Error("Error summarizing conversation", "error", err)

xlog.Error("Error summarizing conversation", "error", err)

return

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

richiejp · 2025-10-20T08:41:29Z

+								},
+							},
+						},
+					})


This looks quite suspicious. It seems like this is creating and completing an observable for a decision step upon receiving the result of that decision. Really it should be created when the first request is submitted to the LLM as part of the decision process.

mmm you are right, I think I've did some mistake when porting the logic, can you check again?

I think you need to create the observable when making the request to the LLM and close it when you get the result. I don't know if you have the callbacks for this in Cogito? It's the kind of thing only needed for observability.

Presently the user will just see the decision appear after completion with the whole of conv as the query to the LLM which may or may not be what is actually passed to the LLM.

there is WithStatusCallback which is called with the reasoning, and WithToolCallResultCallback with the tool call result. WithToolCallBack instead accepts a func with a boolean, which if false will stop the tool execution and the agent.

I think I've misread a bit things initially, but now I've moved the update and the closing of the observable to use the callbacks above

Possibly there should be WithReasoningCallback and WithReasoningResultCallback?

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler · 2025-10-25T20:33:32Z

tests are passing and manual tests looks good - the only issues I have now are with the observables @richiejp - looks like these are stuck in progress, but can't really pinpoint what's wrong

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler mentioned this pull request Oct 10, 2025

feat(tools): allow to automatically plan mudler/cogito#9

Merged

mudler linked an issue Oct 10, 2025 that may be closed by this pull request

Consume cogito #313

Closed

WIP

8c7cf3f

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler force-pushed the cogito branch from b119fc5 to 8c7cf3f Compare October 15, 2025 16:40

mudler added 6 commits October 15, 2025 19:35

Drop old webui

8b7453a

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Almost there

a381208

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

It builds

58f44a4

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Make it build

024b2b3

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

fixups, still doesn't work

71d747a

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

unused now

fae0641

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler force-pushed the cogito branch from 988ccf1 to fae0641 Compare October 18, 2025 21:08

richiejp reviewed Oct 19, 2025

View reviewed changes

mudler added 3 commits October 19, 2025 21:32

Send result before closing

910e642

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Fix observability

c088eef

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Drop MCP code and wire-up in cogito

77f68bf

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler marked this pull request as ready for review October 19, 2025 19:51

Copilot AI review requested due to automatic review settings October 19, 2025 19:51

Drop some templates

6c62819

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Copilot AI reviewed Oct 19, 2025

View reviewed changes

Keep reporting into conv

14950e0

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

richiejp reviewed Oct 20, 2025

View reviewed changes

mudler added 4 commits October 20, 2025 11:11

tests fixups

467a66d

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

tests fixups

e51407c

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

fixups

3ffc8cd

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Do not complete observable during thought process

e6929d8

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

richiejp reviewed Oct 21, 2025

View reviewed changes

Comment thread core/agent/agent.go

richiejp reviewed Oct 21, 2025

View reviewed changes

Comment thread core/agent/agent.go

richiejp reviewed Oct 21, 2025

View reviewed changes

Comment thread core/agent/agent.go

Update cogito

dc9fb71

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler force-pushed the cogito branch from fc12376 to a1f37e6 Compare October 21, 2025 14:18

mudler force-pushed the cogito branch 11 times, most recently from d0887fd to 38dbaa7 Compare October 24, 2025 19:21

tKeep indexing MCP actions

733220a

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler force-pushed the cogito branch from 38dbaa7 to 733220a Compare October 24, 2025 19:33

Split CI jobs to improve speed

cc1fbe8

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler force-pushed the cogito branch 2 times, most recently from 8c2cd2e to e7ba8b5 Compare October 25, 2025 17:37

CI optimizations

0d18613

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler force-pushed the cogito branch from e7ba8b5 to 0d18613 Compare October 25, 2025 19:47

Bump timeout

b632c37

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Bump cogito

e127e9c

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler mentioned this pull request Oct 26, 2025

chore(deps): bump cogito mudler/LocalAI#6785

Merged

1 task

fix: always commit last progress

eac621a

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler force-pushed the cogito branch 3 times, most recently from 06e4d0d to cdabab8 Compare October 27, 2025 16:37

chore: better management of observables

e6ba293

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

mudler force-pushed the cogito branch from cdabab8 to e6ba293 Compare October 27, 2025 16:46

mudler merged commit 73a6be8 into main Oct 27, 2025
2 checks passed

mudler deleted the cogito branch October 27, 2025 17:41

	xlog.Error("Error summarizing conversation", "error", err)
	xlog.Error("Error summarizing conversation", "error", err)
	return

Conversation

mudler commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

richiejp Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

richiejp Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

mudler Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

richiejp Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

mudler Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

richiejp Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mudler commented Oct 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mudler commented Oct 10, 2025 •

edited

Loading

mudler Oct 21, 2025 •

edited

Loading