Skip to content

Commit b8795bf

Browse files
fuzhengweifuzhengwei
committed
feat：《AI Agent 脚手架》第2-16节：fix-多模态能力使用
1 parent 7a2f44e commit b8795bf

10 files changed

Lines changed: 10 additions & 111 deletions

‎docs/md/project/ai-agent-scaffold/part-2/第2-10节：装配域节点-RunnerNode.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: 第2-10节：装配域节点-RunnerNode
2+
title: 【更】第2-10节：装配域节点-RunnerNode
33
pay: https://t.zsxq.com/pEBw8
44
---
55

‎docs/md/project/ai-agent-scaffold/part-2/第2-11节：智能体加载使用验证.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: 第2-11节：智能体加载使用验证
2+
title: 【更】第2-11节：智能体加载使用验证
33
pay: https://t.zsxq.com/CyuXv
44
---
55

‎docs/md/project/ai-agent-scaffold/part-2/第2-12节：增强装配-RunnerNode.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: 第2-12节：增强装配-RunnerNode
2+
title: 【更】第2-12节：增强装配-RunnerNode
33
pay: https://t.zsxq.com/jxbVS
44
---
55

‎docs/md/project/ai-agent-scaffold/part-2/第2-13节：增强装配-AgentWorkflowNode.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: 第2-13：增强装配-AgentWorkflowNode
2+
title: 【更】第2-13：增强装配-AgentWorkflowNode
33
pay: https://t.zsxq.com/qjhUb
44
---
55

‎docs/md/project/ai-agent-scaffold/part-2/第2-14节：增强装配-本地mcp.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: 第2-14节：增强装配-本地mcp
2+
title: 【更】第2-14节：增强装配-本地mcp
33
pay: https://t.zsxq.com/KYNyM
44
---
55

‎docs/md/project/ai-agent-scaffold/part-2/第2-15节：增强装配-回调plugin.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: 第2-15节：增强装配-回调plugin
2+
title: 【更】第2-15节：增强装配-回调plugin
33
pay: https://t.zsxq.com/U1p49
44
---
55

‎docs/md/project/ai-agent-scaffold/part-2/第2-16节：fix-多模态能力使用.md‎

Lines changed: 1 addition & 102 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: 第2-16节：fix-多模态能力使用
2+
title: 【更】第2-16节：fix-多模态能力使用
33
pay: https://t.zsxq.com/wF32j
44
---
55

@@ -121,104 +121,3 @@ public class SpringAiTest {
121121

122122
- 效果；运行结果依然是识别不了，告诉我要上传图片，它才能识别。
123123
- 猜想；这说明单独按照官网案例构建 Agent 并测试依然不行，再细化验证。
124-
125-
### 步骤3，单个功能
126-
127-
```java
128-
import com.alibaba.fastjson.JSON;
129-
import com.google.adk.agents.LlmAgent;
130-
import com.google.adk.events.Event;
131-
import com.google.adk.models.springai.SpringAI;
132-
import com.google.adk.runner.InMemoryRunner;
133-
import com.google.adk.sessions.Session;
134-
import com.google.genai.types.Content;
135-
import com.google.genai.types.Part;
136-
import io.reactivex.rxjava3.core.Flowable;
137-
import org.springframework.ai.chat.messages.UserMessage;
138-
import org.springframework.ai.chat.model.ChatModel;
139-
import org.springframework.ai.chat.model.ChatResponse;
140-
import org.springframework.ai.chat.prompt.Prompt;
141-
import org.springframework.ai.content.Media;
142-
import org.springframework.ai.openai.OpenAiChatModel;
143-
import org.springframework.ai.openai.OpenAiChatOptions;
144-
import org.springframework.ai.openai.api.OpenAiApi;
145-
146-
import org.springframework.core.io.ClassPathResource;
147-
import org.springframework.core.io.Resource;
148-
import org.springframework.util.MimeType;
149-
import org.springframework.util.MimeTypeUtils;
150-
151-
import java.io.InputStream;
152-
153-
public class SpringAiApiTest {
154-
155-
public static void main(String[] args) throws Exception {
156-
ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
157-
InputStream resourceAsStream = classLoader.getResourceAsStream("dog.png");
158-
Resource resource = new ClassPathResource("dog.png", classLoader);
159-
assert resourceAsStream != null;
160-
161-
OpenAiApi openAiApi = OpenAiApi.builder()
162-
.baseUrl("https://apis.xxx")
163-
.apiKey("sk-zahsFUzQcpOauNQUD3918eEe95194d...*****")
164-
.completionsPath("v1/chat/completions")
165-
.embeddingsPath("v1/embeddings")
166-
.build();
167-
168-
ChatModel chatModel = OpenAiChatModel.builder()
169-
.openAiApi(openAiApi)
170-
.defaultOptions(OpenAiChatOptions.builder()
171-
.model("gpt-4o")
172-
.build())
173-
.build();
174-
175-
// 模型测试，没问题可以识别图片
176-
ChatResponse response = chatModel.call(new Prompt(
177-
UserMessage.builder()
178-
.text("请描述这张图片的主要内容，并说明图中物品的可能用途。")
179-
.media(Media.builder()
180-
.mimeType(MimeType.valueOf(MimeTypeUtils.IMAGE_PNG_VALUE))
181-
.data(resource)
182-
.build())
183-
.build(),
184-
OpenAiChatOptions.builder()
185-
.model("gpt-4o")
186-
.build()));
187-
188-
System.out.println("测试结果" + JSON.toJSONString(response));
189-
190-
}
191-
192-
}
193-
194-
```
195-
196-
<div align="center">
197-
<img src="https://bugstack.cn/images/article/project/ai-agent-scaffold/part-2/2-16/images/ai-agent-scaffold-2-16-02.png" width="950px"/>
198-
</div>
199-
200-
- 效果；直接用 Spring AI 原有的功能进行验证，验证通过，可以识别图片。
201-
- 猜想；大概率是 Google ADK 和 Spring AI 对接的问题，需要debug调试验证，找到对接点的参数传递。
202-
203-
### 步骤4，升级框架
204-
205-
```java
206-
<google.adk.version>0.5.0</google.adk.version>
207-
208-
<dependency>
209-
<groupId>com.google.adk</groupId>
210-
<artifactId>google-adk-spring-ai</artifactId>
211-
<version>${google.adk.version}</version>
212-
</dependency>
213-
```
214-
215-
- 升级框架到 0.5.0 版本，依然不行。
216-
217-
### 步骤4，找到问题
218-
219-
<div align="center">
220-
<img src="https://bugstack.cn/images/article/project/ai-agent-scaffold/part-2/2-16/images/ai-agent-scaffold-2-16-03.png" width="950px"/>
221-
</div>
222-
223-
- 当 Debug 到 MessageConverter 消息转换的时候发现，它压根就没处理图片类的东西，写了个 TODO。
224-
- **那么，接下来，小傅哥再带着大家看下，这样的问题怎么排查。**

‎docs/md/project/ai-agent-scaffold/part-2/第2-7节：装配域节点-AgentNode.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: 第2-7节：装配域节点-AgentNode
2+
title: 【更】第2-7节：装配域节点-AgentNode
33
pay: https://t.zsxq.com/908Fo
44
---
55

‎docs/md/project/ai-agent-scaffold/part-2/第2-8节：装配域节点-AgentWorkflowNode.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: 第2-8节：装配域节点-AgentWorkflowNode
2+
title: 【更】第2-8节：装配域节点-AgentWorkflowNode
33
pay: https://t.zsxq.com/3mjk1
44
---
55

‎docs/md/project/ai-agent-scaffold/part-2/第2-9节：装配域节点-Loop、Parallel、Sequential.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: 第2-9节：装配域节点-Loop、Parallel、Sequential
2+
title: 【更】第2-9节：装配域节点-Loop、Parallel、Sequential
33
pay: https://t.zsxq.com/t29lj
44
---
55

0 commit comments

Comments
 (0)