-
Notifications
You must be signed in to change notification settings - Fork 0
feat: 로컬 RAG 파일 업로더 MCP 추가 #11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,17 @@ | ||
| { | ||
| "$schema": "https://agent-plugins.org/schemas/1.0.0/mcp.schema.json", | ||
| "mcpServers": { | ||
| "ennoia": {"type": "streamable-http", "url": "https://mcp.ennoia.so/mcp"} | ||
| "ennoia": { | ||
| "type": "streamable-http", | ||
| "url": "https://mcp.ennoia.so/mcp" | ||
| }, | ||
| "ennoia-file-uploader": { | ||
| "command": "node", | ||
| "cwd": ".", | ||
| "args": [ | ||
| "-e", | ||
| "import(require('node:url').pathToFileURL(require('node:path').join(process.env.CLAUDE_PLUGIN_ROOT || process.cwd(), 'mcp/file-uploader.mjs')).href).then(m => m.serve()).catch(() => { process.exitCode = 1; })" | ||
| ] | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,232 @@ | ||
| #!/usr/bin/env node | ||
| // 로컬 파일 byte는 JSON-RPC를 거치지 않고 HTTPS PUT stream으로 전송한다. | ||
| import { constants } from 'node:fs'; | ||
| import { lstat, open, realpath } from 'node:fs/promises'; | ||
| import { extname, resolve } from 'node:path'; | ||
| import { pathToFileURL } from 'node:url'; | ||
| import { request as httpsRequest } from 'node:https'; | ||
| import { Transform } from 'node:stream'; | ||
| import { pipeline } from 'node:stream/promises'; | ||
|
|
||
| const MAX_FILE_SIZE = 104857600; | ||
| const MAX_RESPONSE_SIZE = 65536; | ||
| const MAX_RPC_SIZE = 65536; | ||
| const UPLOAD_TIMEOUT_MS = 120000; | ||
| const EXTENSIONS = new Set(['.csv', '.txt', '.md', '.pdf', '.docx', '.pptx', '.xlsx', '.xls', '.zip']); | ||
| const HOSTS = new Set(['mcp.ennoia.so', 'dev-mcp-server.ennoia.so']); | ||
| const REQUIRED_HEADERS = new Set(['content-length', 'content-type', 'x-ennoia-upload-token']); | ||
| class UploadError extends Error {} | ||
| const fail = code => { throw new UploadError(code); }; | ||
|
|
||
| export const uploadTool = { | ||
| name: 'upload_ennoia_rag_file', | ||
| description: 'prepare_rag_document_upload가 반환한 upload_url과 exact headers로 로컬 파일을 PUT합니다. local_path만 지정하고 파일 byte/base64는 MCP JSON에 넣지 않습니다. 성공 후 collection_code/file_seq로 처리 상태를 확인하세요.', | ||
| inputSchema: { | ||
| type: 'object', additionalProperties: false, | ||
| properties: { | ||
| local_path: { type: 'string', description: '사용자가 업로드를 요청한 로컬 파일 경로' }, | ||
| upload_url: { type: 'string', description: 'prepare_rag_document_upload 응답의 upload_url' }, | ||
| headers: { type: 'object', additionalProperties: { type: 'string' }, description: 'prepare 응답의 headers를 수정 없이 전달' }, | ||
| }, | ||
| required: ['local_path', 'upload_url', 'headers'], | ||
| }, | ||
| annotations: { readOnlyHint: false, destructiveHint: false, idempotentHint: false, openWorldHint: true }, | ||
| }; | ||
|
|
||
| function validateArguments(args) { | ||
| if (!args || typeof args !== 'object' || Array.isArray(args) | ||
| || Object.keys(args).some(key => !['local_path', 'upload_url', 'headers'].includes(key)) | ||
| || typeof args.local_path !== 'string' || !args.local_path || args.local_path.includes('\0')) fail('ARGUMENTS_INVALID'); | ||
| if (typeof args.upload_url !== 'string') fail('UPLOAD_URL_INVALID'); | ||
| let url; | ||
| try { url = new URL(args.upload_url); } catch { fail('UPLOAD_URL_INVALID'); } | ||
| // URL parser의 경로 정규화, escape, query를 통한 allowlist 우회를 차단한다. | ||
| if (url.protocol !== 'https:' || !HOSTS.has(url.hostname) || url.port || url.username || url.password | ||
| || url.search || url.hash || !/^\/rag\/uploads\/[A-Za-z0-9_-]{1,128}$/.test(url.pathname) | ||
| || args.upload_url !== url.href) fail('UPLOAD_URL_INVALID'); | ||
| if (!EXTENSIONS.has(extname(args.local_path).toLowerCase())) fail('FILE_TYPE_UNSUPPORTED'); | ||
| if (!args.headers || typeof args.headers !== 'object' || Array.isArray(args.headers)) fail('HEADERS_INVALID'); | ||
| const names = new Set(); | ||
| let length; | ||
| for (const [key, value] of Object.entries(args.headers)) { | ||
| const name = key.toLowerCase(); | ||
| if (!REQUIRED_HEADERS.has(name) || names.has(name) || typeof value !== 'string' | ||
| || !value || /[^\x20-\x7e]|,/.test(value)) fail('HEADERS_INVALID'); | ||
| names.add(name); | ||
| if (name === 'content-length') length = value; | ||
| } | ||
| if (names.size !== REQUIRED_HEADERS.size || !/^[1-9][0-9]*$/.test(length)) fail('HEADERS_INVALID'); | ||
| return { url, length }; | ||
| } | ||
|
|
||
| async function readResponse(response) { | ||
| if (response.statusCode >= 300 && response.statusCode < 400) { | ||
| response.destroy(); fail('REDIRECT_REJECTED'); | ||
| } | ||
| if (!(response.statusCode >= 200 && response.statusCode < 300)) { | ||
| response.destroy(); fail(`UPLOAD_HTTP_${response.statusCode || 0}`); | ||
| } | ||
| let size = 0; | ||
| const chunks = []; | ||
| for await (const chunk of response) { | ||
| size += chunk.length; | ||
| if (size > MAX_RESPONSE_SIZE) { response.destroy(); fail('RESPONSE_TOO_LARGE'); } | ||
| chunks.push(chunk); | ||
| } | ||
| // 응답 본문은 token/URL이 반사될 수 있으므로 안전한 식별자만 반환한다. | ||
| const result = { uploaded: true, status_code: response.statusCode }; | ||
| try { | ||
| const data = JSON.parse(Buffer.concat(chunks).toString('utf8')); | ||
| const seq = data?.data?.file_seq ?? data?.file_seq; | ||
| if (Number.isSafeInteger(seq) && seq > 0) result.file_seq = seq; | ||
| } catch { /* 비 JSON 성공 응답은 HTTP 전송 성공만 기록한다. */ } | ||
| return result; | ||
| } | ||
|
|
||
| export async function uploadFile(args, request = httpsRequest, { signal, openFile = open } = {}) { | ||
| const { url, length } = validateArguments(args); | ||
| const checkCancellation = () => { if (signal?.aborted) fail('UPLOAD_CANCELLED'); }; | ||
| let file; | ||
| try { | ||
| checkCancellation(); | ||
| const before = await lstat(args.local_path); | ||
| if (!before.isFile() || before.isSymbolicLink()) fail('FILE_INVALID'); | ||
| checkCancellation(); | ||
| // NOFOLLOW와 fd 검증으로 lstat/open 사이의 symlink 교체도 거부한다. | ||
| file = await openFile(args.local_path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); | ||
| const stat = await file.stat(); | ||
| checkCancellation(); | ||
| if (!stat.isFile() || stat.dev !== before.dev || stat.ino !== before.ino) fail('FILE_INVALID'); | ||
| if (stat.size < 1 || stat.size > MAX_FILE_SIZE) fail('FILE_SIZE_INVALID'); | ||
| if (length !== String(stat.size)) fail('CONTENT_LENGTH_MISMATCH'); | ||
| let req, source, incoming, timer, transfer, rejectResponse; | ||
| const abortTransfer = code => { | ||
| rejectResponse?.(new UploadError(code)); | ||
| source?.destroy(); req?.destroy(); incoming?.destroy(); | ||
| }; | ||
| const cancel = () => abortTransfer('UPLOAD_CANCELLED'); | ||
| try { | ||
| const response = new Promise((resolveResponse, reject) => { | ||
| rejectResponse = reject; | ||
| req = request(url, { method: 'PUT', headers: args.headers }, received => { | ||
| incoming = received; | ||
| readResponse(incoming).then(resolveResponse, reject); | ||
| }); | ||
| req.once('error', () => reject(new UploadError('UPLOAD_NETWORK_ERROR'))); | ||
| timer = setTimeout(() => abortTransfer('UPLOAD_TIMEOUT'), UPLOAD_TIMEOUT_MS); | ||
| }); | ||
| let sent = 0; | ||
| const counter = new Transform({ | ||
| transform(chunk, encoding, callback) { sent += chunk.length; callback(null, chunk); }, | ||
| flush(callback) { callback(sent === stat.size ? null : new UploadError('FILE_SIZE_CHANGED')); }, | ||
| }); | ||
| source = file.createReadStream({ autoClose: false, start: 0, end: stat.size - 1 }); | ||
| transfer = pipeline(source, counter, req); | ||
| const completed = Promise.all([response, transfer]); | ||
| signal?.addEventListener('abort', cancel, { once: true }); | ||
| if (signal?.aborted) cancel(); | ||
| const [result] = await completed; | ||
| checkCancellation(); | ||
| if ((await file.stat()).size !== stat.size) fail('FILE_SIZE_CHANGED'); | ||
| return result; | ||
| } catch (error) { | ||
| checkCancellation(); | ||
| if (error instanceof UploadError) throw error; | ||
| fail('UPLOAD_NETWORK_ERROR'); | ||
| } finally { | ||
| signal?.removeEventListener('abort', cancel); | ||
| clearTimeout(timer); | ||
| source?.destroy(); req?.destroy(); incoming?.destroy(); | ||
| // fd close 전에 진행 중인 read와 pipeline 정리를 기다린다. | ||
| await transfer?.catch(() => {}); | ||
| } | ||
| } catch (error) { | ||
| checkCancellation(); | ||
| if (error instanceof UploadError) throw error; | ||
| fail('FILE_INVALID'); | ||
| } finally { | ||
| await file?.close().catch(() => {}); | ||
| } | ||
| } | ||
|
|
||
| async function handleRpc(message, upload = uploadFile) { | ||
| const id = message?.id ?? null; | ||
| const error = (code, text) => ({ jsonrpc: '2.0', id, error: { code, message: text } }); | ||
| if (!message || Array.isArray(message) || message.jsonrpc !== '2.0' || typeof message.method !== 'string') return error(-32600, 'Invalid Request'); | ||
| if (!Object.hasOwn(message, 'id')) return undefined; | ||
| const reply = result => ({ jsonrpc: '2.0', id, result }); | ||
| if (message.method === 'initialize') return reply({ | ||
| protocolVersion: ['2024-11-05', '2025-03-26', '2025-06-18'].includes(message.params?.protocolVersion) ? message.params.protocolVersion : '2025-06-18', | ||
| capabilities: { tools: {} }, serverInfo: { name: 'ennoia-file-uploader', version: '1.4.0' }, | ||
| }); | ||
| if (message.method === 'ping') return reply({}); | ||
| if (message.method === 'tools/list') return reply({ tools: [uploadTool] }); | ||
| if (message.method !== 'tools/call') return error(-32601, 'Method not found'); | ||
| if (message.params?.name !== uploadTool.name) return error(-32602, 'Unknown tool'); | ||
| try { | ||
| const result = await upload(message.params.arguments); | ||
| return reply({ content: [{ type: 'text', text: JSON.stringify(result) }], structuredContent: result }); | ||
| } catch (err) { | ||
| return reply({ isError: true, content: [{ type: 'text', text: err instanceof UploadError ? err.message : 'UPLOAD_FAILED' }] }); | ||
| } | ||
| } | ||
|
|
||
| export async function serve(input = process.stdin, output = process.stdout, { request = httpsRequest, openFile = open } = {}) { | ||
| let pending = '', oversized = false, closed = false; | ||
| // 업로드는 하나만 허용한다. Parser와 ping은 전송 완료를 기다리지 않는다. | ||
| const active = new Map(); | ||
| const write = value => { if (value && !closed) output.write(JSON.stringify(value) + '\n'); }; | ||
| const shutdown = () => { | ||
| closed = true; | ||
| for (const job of active.values()) job.controller.abort(); | ||
| }; | ||
| const dispatch = message => { | ||
| if (message?.jsonrpc === '2.0' && message.method === 'notifications/cancelled' && !Object.hasOwn(message, 'id')) { | ||
| active.get(message.params?.requestId)?.controller.abort(); | ||
| return; | ||
| } | ||
| const isUpload = message?.jsonrpc === '2.0' && message.method === 'tools/call' | ||
| && message.params?.name === uploadTool.name && Object.hasOwn(message, 'id'); | ||
| if (!isUpload) { void handleRpc(message).then(write); return; } | ||
| if (active.size) { | ||
| write({ jsonrpc: '2.0', id: message.id, result: { isError: true, content: [{ type: 'text', text: 'UPLOAD_BUSY' }] } }); | ||
| return; | ||
| } | ||
| const controller = new AbortController(); | ||
| const job = { controller, promise: undefined }; | ||
| active.set(message.id, job); | ||
| job.promise = handleRpc(message, args => uploadFile(args, request, { signal: controller.signal, openFile })) | ||
| .then(value => { if (!controller.signal.aborted) write(value); }) | ||
| .finally(() => active.delete(message.id)); | ||
| }; | ||
| input.setEncoding('utf8'); | ||
| input.on('end', shutdown); input.on('close', shutdown); input.on('error', shutdown); | ||
| try { | ||
| for await (const chunk of input) { | ||
| for (const [index, part] of chunk.split('\n').entries()) { | ||
| if (index > 0) { | ||
| if (oversized) write({ jsonrpc: '2.0', id: null, error: { code: -32600, message: 'Request too large' } }); | ||
| else if (pending.trim()) { | ||
| let message; | ||
| try { message = JSON.parse(pending); } | ||
| catch { write({ jsonrpc: '2.0', id: null, error: { code: -32700, message: 'Parse error' } }); } | ||
| if (message !== undefined) dispatch(message); | ||
| } | ||
| pending = ''; oversized = false; | ||
| } | ||
| if (!oversized) { | ||
| pending += part; | ||
| if (Buffer.byteLength(pending) > MAX_RPC_SIZE) { pending = ''; oversized = true; } | ||
| } | ||
| } | ||
| } | ||
| } finally { | ||
| shutdown(); | ||
| await Promise.allSettled([...active.values()].map(job => job.promise)); | ||
| input.off('end', shutdown); input.off('close', shutdown); input.off('error', shutdown); | ||
| } | ||
| } | ||
|
|
||
| if (process.argv[1] && import.meta.url === pathToFileURL(await realpath(resolve(process.argv[1]))).href) { | ||
| serve().catch(() => { process.exitCode = 1; }); | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
Repository: wanteddev/ennoia-plugin
Length of output: 6107
🏁 Script executed:
Repository: wanteddev/ennoia-plugin
Length of output: 43751
원격 PUT 성공 후
FILE_SIZE_CHANGED를 반환하지 마십시오.uploadFile은Promise.all([response, transfer])로 PUT 전송과 응답을 기다립니다.readResponse가 2xx 응답을 성공으로 처리한 뒤에도 line 130의file.stat()이 파일 크기 변경을 발견하면FILE_SIZE_CHANGED를 던집니다. 따라서 호출자는 원격 PUT 성공 후 실패를 받습니다.prepare_rag_document_upload가 발급한 upload header는 1회용이며, 문서도 업로드를 반복하지 말고 실패 전용 재시도를 사용하도록 규정합니다. 이 오류는 원격 결과가 이미 생성되었을 수 있는 상태에서 재시도 여부를 판단할 수 없게 만듭니다.파일 변경을 업로드 전에 검출하려면 요청을 종료하기 전에 검사하십시오. 마지막 chunk를 보류하는 방식은 선언된
Content-Length만큼 정확히 전송하고 검사 후 요청을 종료할 때만 사용할 수 있습니다. 그러나 검사 직후 파일이 변경될 수 있으므로, 마지막 chunk 보류만으로는 race를 제거할 수 없습니다. 원격 성공 후의 크기 불일치는 실패로 변환하지 말고, 필요한 경우 별도의 진단 정보로 처리하십시오.🤖 Prompt for AI Agents