[AI] [FEAT]: 독립 기업 뉴스 수집 코어 추가 - #378
Open
Dublecheckk wants to merge 1 commit into
Open
Conversation
Walkthrough미국 기업 뉴스 수집을 위한 데이터 계약, Google News RSS 공급자, 관련성·중복 처리, 단일·배치 파이프라인, CLI, S&P 100 유니버스, 호환 façade, 테스트와 문서가 추가되었습니다. Changes미국 기업 뉴스 수집
Estimated code review effort: 5 (Critical) | ~90 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant collect_company_news
participant GoogleNewsRssProvider
participant CompanyNewsCollector
Operator->>collect_company_news: CLI 옵션 전달
collect_company_news->>GoogleNewsRssProvider: 기업별 RSS 조회
GoogleNewsRssProvider-->>CompanyNewsCollector: ProviderFetchResult 전달
CompanyNewsCollector-->>collect_company_news: CollectionResult 집계
collect_company_news-->>Operator: JSON 결과와 종료 코드 출력
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (15)
AI/modules/data_collector/components/news/config.py (1)
10-13: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
parents[5]하드코딩 인덱스는 파일 위치 변경 시 조용히 잘못된 경로를 만듭니다.인덱스가 틀어져도 예외 없이 잘못된
PROJECT_ROOT가 계산되고, 이후FileNotFoundError로만 드러나 원인 파악이 어렵습니다. 저장소 표식(예: 상위 경로의AI디렉터리) 기준으로 도출하거나, 최소한 예상 경로 존재 여부를 단정하는 방식이 더 견고합니다.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/modules/data_collector/components/news/config.py` around lines 10 - 13, Update the PROJECT_ROOT calculation near DEFAULT_CONFIG_PATH to locate the repository root by searching ancestor paths for the expected AI directory instead of relying on the hardcoded parents[5] index, and validate that the root is found before constructing DEFAULT_CONFIG_PATH.AI/modules/data_collector/components/news/__init__.py (1)
42-74: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRuff RUF022:
__all__정렬 필요.CI에서 Ruff 검증을 수행한다고 PR 설명에 명시돼 있으니 isort 스타일 정렬로 맞추는 편이 안전합니다.
ruff check --fix로 자동 정렬됩니다.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/modules/data_collector/components/news/__init__.py` around lines 42 - 74, Reorder the entries in the module-level __all__ list using isort-style alphabetical grouping so Ruff RUF022 passes, without changing which symbols are exported.Source: Linters/SAST tools
AI/modules/data_collector/components/news/dedup.py (2)
228-235: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRuff RUF001(EN DASH) 경고가 CI를 막을 수 있습니다.
[-–—|]문자 클래스는 Google RSS 제목의 다양한 대시를 잡기 위한 의도적 선택이지만, PR에 Ruff 검증이 포함돼 있어 이 경고가 실패로 이어질 수 있습니다. 의도된 문자라면noqa: RUF001로 명시하거나 유니코드 이스케이프(\u2013\u2014)로 표기해 의도를 드러내는 편이 좋습니다.♻️ 제안: 유니코드 이스케이프로 의도 명시
candidate_title = re.sub( - rf"\s+[-–—|]\s+{re.escape(article.source)}\s*$", + rf"\s+[-\u2013\u2014|]\s+{re.escape(article.source)}\s*$", "", candidate_title, flags=re.IGNORECASE, )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/modules/data_collector/components/news/dedup.py` around lines 228 - 235, Update the regular expression in the candidate_title normalization block to represent the en dash and em dash through Unicode escapes (such as \u2013 and \u2014) instead of literal characters, while preserving matching for hyphen, both Unicode dashes, and the pipe separator.Source: Linters/SAST tools
221-237: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value고정 버킷 경계 때문에 몇 분 차이의 동일 기사가 서로 다른 그룹으로 갈립니다.
int(timestamp) // bucket_seconds는 절대 경계 기준이므로 05:59와 06:01에 재전송된 동일 원문은 절대 같은 그룹에 들어가지 않습니다. 그룹 ID가 downstream 힌트로만 쓰인다는 docstring을 감안하면 지금은 수용 가능하지만, 인접 버킷도 함께 조회하거나 후보 그룹을 2개(현재/이전 버킷) 발급하는 방식이 재현율을 높입니다. 현재 한계를 docstring에 명시해 두는 것만으로도 충분할 수 있습니다.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/modules/data_collector/components/news/dedup.py` around lines 221 - 237, Update the documentation for the function containing the bucket calculation to explicitly state that fixed absolute bucket boundaries can place near-simultaneous syndicated articles in adjacent groups, and that the group ID is only a downstream hint. Keep the existing candidate_title normalization, bucket calculation, and returned group ID behavior unchanged.AI/modules/data_collector/components/__init__.py (1)
48-55: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value지연 import 구현은 타당합니다. 정적 분석/IDE 지원을 위한 보완만 고려하세요.
__getattr__기반 지연 로딩은 옵션 의존성 문제를 잘 회피하지만, 타입 체커와 IDE는_EXPORTS의 문자열 매핑을 해석하지 못해 심볼을 미해결로 표시합니다.TYPE_CHECKING블록에 실제 import를 추가하면 런타임 동작을 그대로 유지하면서 정적 해석을 복원할 수 있습니다.♻️ 제안: TYPE_CHECKING 블록 추가
from importlib import import_module -from typing import Any +from typing import TYPE_CHECKING, Any + +if TYPE_CHECKING: # pragma: no cover + from .market_data import MarketDataCollector + from .stock_info_collector import StockInfoCollector + # ... 나머지 공개 심볼🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/modules/data_collector/components/__init__.py` around lines 48 - 55, 보완: TYPE_CHECKING 블록에 _EXPORTS가 노출하는 실제 심볼을 정적 import로 추가해 타입 체커와 IDE가 해당 API를 해석하도록 하세요. 런타임 지연 로딩은 __getattr__과 import_module 흐름을 그대로 유지하고, TYPE_CHECKING에서만 import가 실행되도록 구성하세요.AI/modules/data_collector/components/news/windows.py (1)
27-43: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
max_recovery_hours=72기본값이NewsCollectionConfig와 중복 정의돼 드리프트 위험이 있습니다.
config.py의max_recovery_hours: int = 72와 값이 이중으로 존재해, 한쪽만 바뀌면 CLI 검증과 이 함수의 상한이 어긋납니다. 호출자가 항상 설정값을 넘기도록 기본값을 제거하거나, 공용 상수를 참조하는 편이 안전합니다. 아울러require_aware_utc는contracts._utc_datetime과 동일 로직이라 한쪽으로 통합할 여지가 있습니다.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/modules/data_collector/components/news/windows.py` around lines 27 - 43, Remove the duplicated max_recovery_hours default from forward_collection_window and require callers to pass the value from NewsCollectionConfig, keeping validation based on that supplied configuration. Do not change the existing lookback or UTC validation behavior; leave require_aware_utc integration out of this focused change.AI/modules/data_collector/components/news/contracts.py (1)
375-375: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valuefrozen 계약인데
metadata는 가변 dict로 보관됩니다.
dict(self.metadata)로 얕은 복사는 하지만 반환된 인스턴스의metadata는 여전히 변경 가능해, "immutable collection contracts"라는 계약 의도와 어긋납니다.MappingProxyType으로 감싸면 의도를 코드로 강제할 수 있습니다.♻️ 제안: MappingProxyType 사용
+from types import MappingProxyType ... - object.__setattr__(self, "metadata", dict(self.metadata)) + object.__setattr__(self, "metadata", MappingProxyType(dict(self.metadata)))Also applies to: 432-432
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/modules/data_collector/components/news/contracts.py` at line 375, Make the frozen contract’s metadata immutable by storing a shallow-copied dictionary wrapped with MappingProxyType rather than a mutable dict. Update the metadata field declarations and their initialization/default handling at both affected locations, preserving the existing Mapping[str, Any] interface and empty default.AI/modules/data_collector/components/news/providers/google_news_rss.py (3)
48-66: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win인자 검증을 속성 할당보다 먼저 수행하세요.
현재는 잘못된 timeout/limit 값이 들어와도
requests.Session()이 먼저 생성되고 나서ValueError가 발생하므로, 닫히지 않은 세션 객체가 남습니다. 검증을__init__최상단으로 올리면 이 경로가 사라집니다.♻️ 제안 리팩터
- self.session = session or requests.Session() - self.endpoint = endpoint + if ( + connect_timeout_seconds <= 0 + or read_timeout_seconds <= 0 + or max_response_bytes <= 0 + or max_feed_items <= 0 + or max_snippet_chars <= 0 + or saturation_item_threshold <= 0 + ): + raise ValueError("timeout/응답/feed item/snippet 제한은 0보다 커야 합니다.") + self.session = session or requests.Session() + self.endpoint = endpoint self.language = language self.country = country self.edition = edition self.timeout = (connect_timeout_seconds, read_timeout_seconds) self.max_response_bytes = max_response_bytes self.max_feed_items = max_feed_items self.max_snippet_chars = max_snippet_chars self.saturation_item_threshold = saturation_item_threshold - if ( - connect_timeout_seconds <= 0 - or read_timeout_seconds <= 0 - or self.max_response_bytes <= 0 - or self.max_feed_items <= 0 - or self.max_snippet_chars <= 0 - or self.saturation_item_threshold <= 0 - ): - raise ValueError("timeout/응답/feed item/snippet 제한은 0보다 커야 합니다.")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/modules/data_collector/components/news/providers/google_news_rss.py` around lines 48 - 66, Move the timeout and limit validation in the provider __init__ before creating requests.Session() or assigning instance attributes, so invalid arguments raise ValueError without constructing a session. Preserve the existing validation conditions and error message.
218-218: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low valueRuff S314:
defusedxml사용 검토.
_read_limited_response가<!doctype/<!entity를 사전 차단하고 크기·item 상한도 두었으므로 실질 XXE/entity expansion 위험은 낮습니다. 다만 방어가 문자열 검사에 의존하고 있어,defusedxml.ElementTree.fromstring으로 대체하면 파서 수준에서 보장됩니다. 현 구조를 유지한다면# noqa: S314와 근거 주석을 남겨 의도를 명시하는 편이 좋습니다.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/modules/data_collector/components/news/providers/google_news_rss.py` at line 218, Update the XML parsing at ElementTree.fromstring in the response-reading flow to use defusedxml.ElementTree.fromstring for parser-level protection, while preserving the existing _read_limited_response safeguards and parsing behavior.Source: Linters/SAST tools
138-142: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
locals()검사 대신 사전 초기화가 더 명확합니다.
response = None을 try 앞에서 선언하고finally에서if response is not None: response.close()로 처리하거나,with contextlib.closing(...)패턴을 쓰면"response" in locals()와getattr(..., "close")이중 방어가 필요 없습니다. 테스트 fake가close를 제공하므로 동작 변화도 없습니다.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/modules/data_collector/components/news/providers/google_news_rss.py` around lines 138 - 142, 뉴스 RSS 요청 처리 함수의 finally 블록에서 locals() 검사와 getattr() 방어를 제거하세요. try 시작 전에 response를 None으로 초기화한 뒤, finally에서 response가 None이 아닐 때 close()를 호출하도록 정리하고 기존 fake 응답의 종료 동작은 유지하세요.AI/modules/data_collector/components/news/pipeline.py (1)
128-131: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value
zip(..., strict=True)로 길이 불일치를 조기에 드러내세요.
deduplicate_exact가articles와identities를 1:1로 반환한다는 전제가 있으므로, 향후 dedup 로직이 바뀔 때 조용히 잘리지 않도록strict=True를 명시하는 편이 안전합니다. (Ruff B905)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/modules/data_collector/components/news/pipeline.py` around lines 128 - 131, Update the zip call in the article-processing loop to pass strict=True, ensuring deduplicated.articles and deduplicated.identities must have equal lengths while preserving the existing iteration behavior.Source: Linters/SAST tools
AI/modules/data_collector/scripts/collect_company_news.py (2)
96-115: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win요청한 티커가
enabled=false인 경우 조용히 누락됩니다.
unknown판정은 universe 존재 여부만 보므로, 사용자가 명시한 티커가 비활성 issuer면 경고 없이 결과에서 빠집니다(전부 비활성일 때만 Line 166에서 실패). 비활성으로 스킵된 티커를 별도로 알려주는 편이 CLI 사용성에 좋습니다.♻️ 제안 변경
unknown = requested.difference(known) if unknown: raise ValueError(f"universe에 없는 ticker: {', '.join(sorted(unknown))}") + selected_tickers = {ticker for company in selected for ticker in company.tickers} + disabled = requested.difference(selected_tickers) + if disabled: + raise ValueError( + f"universe에서 비활성(enabled=false)인 ticker: {', '.join(sorted(disabled))}" + ) return selected🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/modules/data_collector/scripts/collect_company_news.py` around lines 96 - 115, Update _select_companies to track requested tickers that exist only on disabled companies and explicitly report them as skipped, while preserving the existing unknown-ticker error for tickers absent from the universe. Ensure the notification occurs even when some enabled companies are selected, not only when the final selection is empty.
33-36: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
EXIT_PARTIAL_SUCCESS = 2가 argparse의 사용법 오류 종료 코드와 겹칩니다.argparse는 잘못된 인자에 대해
SystemExit(2)로 종료하므로, 자동화 측에서 "부분 성공"과 "CLI 인자 오류"를 종료 코드로 구분할 수 없습니다. 부분 성공을 5 이상 등 미사용 코드로 옮기는 것을 권장합니다.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/modules/data_collector/scripts/collect_company_news.py` around lines 33 - 36, Update the EXIT_PARTIAL_SUCCESS constant to an unused exit code of 5 or higher, keeping it distinct from argparse’s usage-error code 2; preserve the other exit-code constants and all existing references to EXIT_PARTIAL_SUCCESS.AI/modules/data_collector/components/news_data.py (1)
54-55: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winuniverse 로딩 결과를 캐시하는 편이 좋습니다.
_temporary_target호출마다news_collection.json과 100개 issuer JSON을 디스크에서 다시 파싱합니다. 티커 루프에서 legacy 함수를 반복 호출하면 불필요한 I/O가 누적됩니다. 모듈 수준functools.lru_cache로 감싼 로더를 두고 티커→회사 dict를 만들어 두면 선형 스캔도 함께 제거됩니다.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/modules/data_collector/components/news_data.py` around lines 54 - 55, Update the universe-loading flow around NewsCollectionConfig.from_file and load_company_universe to use a module-level functools.lru_cache loader that builds and reuses a ticker-to-company dictionary, so repeated _temporary_target calls avoid reparsing files and linear scans.AI/tests/verify_company_news.py (1)
477-490: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win호환 파사드의 공개 함수에 대한 테스트가 없습니다.
테스트는 private
_temporary_target만 검증하고, 실제 legacy 호출 경로인fetch_news_links(provider 실패 시NewsCollectionError승격, 반환 스키마)와collect_news(lookback_hours경계 1/72)는 커버되지 않습니다.company_name을 함께 넘기는 호출이 universe 정보를 유지하는지도 함께 고정해 두면news_data.pyLine 53-68 동작 변경 시 회귀를 잡을 수 있습니다.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AI/tests/verify_company_news.py` around lines 477 - 490, Expand UniverseConfigTest to cover the public compatibility facade: test fetch_news_links for provider failure promotion to NewsCollectionError and its return schema, and test collect_news at the lookback_hours boundaries of 1 and 72. Include a call that supplies company_name and assert the returned universe company information is preserved, using the public functions rather than only _temporary_target.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@AI/modules/data_collector/components/news_data.py`:
- Around line 53-68: Update the company-target resolution flow around the
universe lookup so it always attempts to match symbol against
universe.companies, regardless of whether company_name is provided. Return the
matched company with its full CIK, approved aliases, and all tickers, merging
any supplied aliases as currently done; only use the company_name-based
temporary target as fallback when no universe match exists.
In `@AI/modules/data_collector/components/news/config.py`:
- Around line 131-142: Update the company parsing loop that constructs
CompanyTarget to validate required company_key and legal_name fields before
accessing them, raising a descriptive ValueError that identifies the invalid
company entry and missing field, consistent with the existing metadata
validation. Preserve the current optional-field handling and CIK normalization.
In `@AI/modules/data_collector/components/news/relevance.py`:
- Around line 245-261: Prevent the market-context evidence from being counted
twice for the ambiguous_ticker:uppercase_business_context path: update the later
scoring logic near the best_ticker_score adjustment so this branch does not
receive the additional context bonus after _contains_market_context already
enabled its 0.60 score. Preserve the existing scoring and bonus behavior for
other ticker-matching branches.
In `@AI/modules/data_collector/config/sp100_companies.json`:
- Around line 738-741: Add "PM" to the ambiguous_tickers array for the Philip
Morris International entry, matching the existing handling of CAT, LOW, and NOW
while leaving the other company configuration unchanged.
- Around line 924-931: Update the ExxonMobil entry identified by company_key
"exxonmobil" so legal_name is "Exxon Mobil Corporation" and cik is "0000034088",
matching the standard SEC mapping for ticker XOM; leave its other fields
unchanged.
In `@AI/modules/data_collector/scripts/collect_company_news.py`:
- Around line 214-234: Move args = parse_args(argv) outside the try block in
main so args is always available when handling collection errors. Keep run(args)
inside the existing exception handling and preserve the current failure payload
and exit code behavior.
---
Nitpick comments:
In `@AI/modules/data_collector/components/__init__.py`:
- Around line 48-55: 보완: TYPE_CHECKING 블록에 _EXPORTS가 노출하는 실제 심볼을 정적 import로 추가해
타입 체커와 IDE가 해당 API를 해석하도록 하세요. 런타임 지연 로딩은 __getattr__과 import_module 흐름을 그대로
유지하고, TYPE_CHECKING에서만 import가 실행되도록 구성하세요.
In `@AI/modules/data_collector/components/news_data.py`:
- Around line 54-55: Update the universe-loading flow around
NewsCollectionConfig.from_file and load_company_universe to use a module-level
functools.lru_cache loader that builds and reuses a ticker-to-company
dictionary, so repeated _temporary_target calls avoid reparsing files and linear
scans.
In `@AI/modules/data_collector/components/news/__init__.py`:
- Around line 42-74: Reorder the entries in the module-level __all__ list using
isort-style alphabetical grouping so Ruff RUF022 passes, without changing which
symbols are exported.
In `@AI/modules/data_collector/components/news/config.py`:
- Around line 10-13: Update the PROJECT_ROOT calculation near
DEFAULT_CONFIG_PATH to locate the repository root by searching ancestor paths
for the expected AI directory instead of relying on the hardcoded parents[5]
index, and validate that the root is found before constructing
DEFAULT_CONFIG_PATH.
In `@AI/modules/data_collector/components/news/contracts.py`:
- Line 375: Make the frozen contract’s metadata immutable by storing a
shallow-copied dictionary wrapped with MappingProxyType rather than a mutable
dict. Update the metadata field declarations and their initialization/default
handling at both affected locations, preserving the existing Mapping[str, Any]
interface and empty default.
In `@AI/modules/data_collector/components/news/dedup.py`:
- Around line 228-235: Update the regular expression in the candidate_title
normalization block to represent the en dash and em dash through Unicode escapes
(such as \u2013 and \u2014) instead of literal characters, while preserving
matching for hyphen, both Unicode dashes, and the pipe separator.
- Around line 221-237: Update the documentation for the function containing the
bucket calculation to explicitly state that fixed absolute bucket boundaries can
place near-simultaneous syndicated articles in adjacent groups, and that the
group ID is only a downstream hint. Keep the existing candidate_title
normalization, bucket calculation, and returned group ID behavior unchanged.
In `@AI/modules/data_collector/components/news/pipeline.py`:
- Around line 128-131: Update the zip call in the article-processing loop to
pass strict=True, ensuring deduplicated.articles and deduplicated.identities
must have equal lengths while preserving the existing iteration behavior.
In `@AI/modules/data_collector/components/news/providers/google_news_rss.py`:
- Around line 48-66: Move the timeout and limit validation in the provider
__init__ before creating requests.Session() or assigning instance attributes, so
invalid arguments raise ValueError without constructing a session. Preserve the
existing validation conditions and error message.
- Line 218: Update the XML parsing at ElementTree.fromstring in the
response-reading flow to use defusedxml.ElementTree.fromstring for parser-level
protection, while preserving the existing _read_limited_response safeguards and
parsing behavior.
- Around line 138-142: 뉴스 RSS 요청 처리 함수의 finally 블록에서 locals() 검사와 getattr() 방어를
제거하세요. try 시작 전에 response를 None으로 초기화한 뒤, finally에서 response가 None이 아닐 때
close()를 호출하도록 정리하고 기존 fake 응답의 종료 동작은 유지하세요.
In `@AI/modules/data_collector/components/news/windows.py`:
- Around line 27-43: Remove the duplicated max_recovery_hours default from
forward_collection_window and require callers to pass the value from
NewsCollectionConfig, keeping validation based on that supplied configuration.
Do not change the existing lookback or UTC validation behavior; leave
require_aware_utc integration out of this focused change.
In `@AI/modules/data_collector/scripts/collect_company_news.py`:
- Around line 96-115: Update _select_companies to track requested tickers that
exist only on disabled companies and explicitly report them as skipped, while
preserving the existing unknown-ticker error for tickers absent from the
universe. Ensure the notification occurs even when some enabled companies are
selected, not only when the final selection is empty.
- Around line 33-36: Update the EXIT_PARTIAL_SUCCESS constant to an unused exit
code of 5 or higher, keeping it distinct from argparse’s usage-error code 2;
preserve the other exit-code constants and all existing references to
EXIT_PARTIAL_SUCCESS.
In `@AI/tests/verify_company_news.py`:
- Around line 477-490: Expand UniverseConfigTest to cover the public
compatibility facade: test fetch_news_links for provider failure promotion to
NewsCollectionError and its return schema, and test collect_news at the
lookback_hours boundaries of 1 and 72. Include a call that supplies company_name
and assert the returned universe company information is preserved, using the
public functions rather than only _temporary_target.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 84eeb033-b0d9-4a13-a12f-070e46863852
📒 Files selected for processing (18)
AI/modules/data_collector/README.mdAI/modules/data_collector/components/__init__.pyAI/modules/data_collector/components/news/__init__.pyAI/modules/data_collector/components/news/config.pyAI/modules/data_collector/components/news/contracts.pyAI/modules/data_collector/components/news/dedup.pyAI/modules/data_collector/components/news/pipeline.pyAI/modules/data_collector/components/news/providers/__init__.pyAI/modules/data_collector/components/news/providers/base.pyAI/modules/data_collector/components/news/providers/google_news_rss.pyAI/modules/data_collector/components/news/relevance.pyAI/modules/data_collector/components/news/windows.pyAI/modules/data_collector/components/news_data.pyAI/modules/data_collector/config/news_collection.jsonAI/modules/data_collector/config/sp100_companies.jsonAI/modules/data_collector/scripts/collect_company_news.pyAI/tests/fixtures/news/google_news_rss.xmlAI/tests/verify_company_news.py
Comment on lines
+53
to
+68
| if company_name is None: | ||
| config = NewsCollectionConfig.from_file(DEFAULT_CONFIG_PATH) | ||
| universe = load_company_universe(config.universe_file) | ||
| for company in universe.companies: | ||
| if symbol in company.tickers: | ||
| if not aliases: | ||
| return company | ||
| return CompanyTarget( | ||
| company_key=company.company_key, | ||
| cik=company.cik, | ||
| legal_name=company.legal_name, | ||
| tickers=company.tickers, | ||
| aliases=(*company.aliases, *aliases), | ||
| enabled=company.enabled, | ||
| ambiguous_tickers=company.ambiguous_tickers, | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
company_name이 주어지면 universe 정보를 통째로 버립니다.
if company_name is None: 조건 때문에 회사명을 함께 넘기는 호출(파일 하단 collect_news("AAPL", company_name="Apple Inc.") 포함)은 universe에 존재하는 issuer라도 cik, 승인 별칭, 복수 티커(GOOG/GOOGL)를 모두 잃고 임시 타깃으로 처리됩니다. 결과적으로 관련도 점수와 식별 메타데이터 품질이 legacy 경로에서만 낮아집니다. universe 매칭을 먼저 시도하고, company_name은 매칭 실패 시의 fallback 또는 별칭 보강 용도로 쓰는 편이 일관됩니다.
♻️ 제안 변경
- if company_name is None:
- config = NewsCollectionConfig.from_file(DEFAULT_CONFIG_PATH)
- universe = load_company_universe(config.universe_file)
- for company in universe.companies:
- if symbol in company.tickers:
- if not aliases:
- return company
- return CompanyTarget(
- company_key=company.company_key,
- cik=company.cik,
- legal_name=company.legal_name,
- tickers=company.tickers,
- aliases=(*company.aliases, *aliases),
- enabled=company.enabled,
- ambiguous_tickers=company.ambiguous_tickers,
- )
+ config = NewsCollectionConfig.from_file(DEFAULT_CONFIG_PATH)
+ universe = load_company_universe(config.universe_file)
+ for company in universe.companies:
+ if symbol in company.tickers:
+ extra_aliases = tuple(
+ value for value in (*aliases, company_name) if value
+ )
+ if not extra_aliases:
+ return company
+ return CompanyTarget(
+ company_key=company.company_key,
+ cik=company.cik,
+ legal_name=company.legal_name,
+ tickers=company.tickers,
+ aliases=(*company.aliases, *extra_aliases),
+ enabled=company.enabled,
+ ambiguous_tickers=company.ambiguous_tickers,
+ )📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if company_name is None: | |
| config = NewsCollectionConfig.from_file(DEFAULT_CONFIG_PATH) | |
| universe = load_company_universe(config.universe_file) | |
| for company in universe.companies: | |
| if symbol in company.tickers: | |
| if not aliases: | |
| return company | |
| return CompanyTarget( | |
| company_key=company.company_key, | |
| cik=company.cik, | |
| legal_name=company.legal_name, | |
| tickers=company.tickers, | |
| aliases=(*company.aliases, *aliases), | |
| enabled=company.enabled, | |
| ambiguous_tickers=company.ambiguous_tickers, | |
| ) | |
| config = NewsCollectionConfig.from_file(DEFAULT_CONFIG_PATH) | |
| universe = load_company_universe(config.universe_file) | |
| for company in universe.companies: | |
| if symbol in company.tickers: | |
| extra_aliases = tuple( | |
| value for value in (*aliases, company_name) if value | |
| ) | |
| if not extra_aliases: | |
| return company | |
| return CompanyTarget( | |
| company_key=company.company_key, | |
| cik=company.cik, | |
| legal_name=company.legal_name, | |
| tickers=company.tickers, | |
| aliases=(*company.aliases, *extra_aliases), | |
| enabled=company.enabled, | |
| ambiguous_tickers=company.ambiguous_tickers, | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@AI/modules/data_collector/components/news_data.py` around lines 53 - 68,
Update the company-target resolution flow around the universe lookup so it
always attempts to match symbol against universe.companies, regardless of
whether company_name is provided. Return the matched company with its full CIK,
approved aliases, and all tickers, merging any supplied aliases as currently
done; only use the company_name-based temporary target as fallback when no
universe match exists.
Comment on lines
+131
to
+142
| for item in raw.get("companies", []): | ||
| company = CompanyTarget( | ||
| company_key=str(item["company_key"]), | ||
| cik=str(item["cik"]).zfill(10) if item.get("cik") else None, | ||
| legal_name=str(item["legal_name"]), | ||
| tickers=tuple(str(value).upper() for value in item.get("tickers", [])), | ||
| aliases=tuple(str(value) for value in item.get("aliases", [])), | ||
| enabled=item.get("enabled", True), | ||
| ambiguous_tickers=tuple( | ||
| str(value).upper() for value in item.get("ambiguous_tickers", []) | ||
| ), | ||
| ) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
필수 필드 누락 시 KeyError가 그대로 전파됩니다.
company_key, legal_name은 item[...]로 직접 접근하므로 universe JSON에 오타가 있으면 메타데이터 검증(라인 123-126)처럼 친절한 ValueError가 아니라 KeyError: 'legal_name'이 올라옵니다. 어느 항목에서 문제가 났는지도 알 수 없습니다.
🛡️ 제안: 항목 단위 필수 필드 검증
- for item in raw.get("companies", []):
+ for index, item in enumerate(raw.get("companies", [])):
+ missing_fields = [
+ key for key in ("company_key", "legal_name") if not item.get(key)
+ ]
+ if missing_fields:
+ raise ValueError(
+ f"companies[{index}] 필수 필드 누락: {', '.join(missing_fields)}"
+ )
company = CompanyTarget(
company_key=str(item["company_key"]),📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| for item in raw.get("companies", []): | |
| company = CompanyTarget( | |
| company_key=str(item["company_key"]), | |
| cik=str(item["cik"]).zfill(10) if item.get("cik") else None, | |
| legal_name=str(item["legal_name"]), | |
| tickers=tuple(str(value).upper() for value in item.get("tickers", [])), | |
| aliases=tuple(str(value) for value in item.get("aliases", [])), | |
| enabled=item.get("enabled", True), | |
| ambiguous_tickers=tuple( | |
| str(value).upper() for value in item.get("ambiguous_tickers", []) | |
| ), | |
| ) | |
| for index, item in enumerate(raw.get("companies", [])): | |
| missing_fields = [ | |
| key for key in ("company_key", "legal_name") if not item.get(key) | |
| ] | |
| if missing_fields: | |
| raise ValueError( | |
| f"companies[{index}] 필수 필드 누락: {', '.join(missing_fields)}" | |
| ) | |
| company = CompanyTarget( | |
| company_key=str(item["company_key"]), | |
| cik=str(item["cik"]).zfill(10) if item.get("cik") else None, | |
| legal_name=str(item["legal_name"]), | |
| tickers=tuple(str(value).upper() for value in item.get("tickers", [])), | |
| aliases=tuple(str(value) for value in item.get("aliases", [])), | |
| enabled=item.get("enabled", True), | |
| ambiguous_tickers=tuple( | |
| str(value).upper() for value in item.get("ambiguous_tickers", []) | |
| ), | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@AI/modules/data_collector/components/news/config.py` around lines 131 - 142,
Update the company parsing loop that constructs CompanyTarget to validate
required company_key and legal_name fields before accessing them, raising a
descriptive ValueError that identifies the invalid company entry and missing
field, consistent with the existing metadata validation. Preserve the current
optional-field handling and CIK normalization.
Comment on lines
+245
to
+261
| elif ( | ||
| len(ticker) >= 3 | ||
| and _contains_market_context((article.title, snippet)) | ||
| and ( | ||
| _uppercase_ticker_token_matches(article.title, ticker) | ||
| or _uppercase_ticker_token_matches(snippet, ticker) | ||
| ) | ||
| ): | ||
| # CAT/LOW/NOW 같은 사전 단어형 ticker는 대문자 토큰과 | ||
| # 금융·사업 문맥이 함께 있을 때만 제한적으로 인정합니다. | ||
| if 0.60 > best_ticker_score: | ||
| best_ticker_score = 0.60 | ||
| best_ticker_reason = ( | ||
| f"ambiguous_ticker:uppercase_business_context:{ticker}" | ||
| ) | ||
| best_ticker = ticker | ||
| continue |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
문맥 증거가 두 번 가산됩니다.
ambiguous_ticker:uppercase_business_context 분기는 진입 조건 자체가 _contains_market_context(...)인데, 그렇게 얻은 0.60에 라인 292-294에서 동일한 문맥 조건으로 다시 +0.20이 더해져 0.80이 됩니다. 이름 증거가 전혀 없는 사전 단어형 ticker(CAT/LOW/NOW)가 임계값 0.60을 크게 상회해 통과하므로, 보수적으로 설계한 의도와 어긋나고 오탐이 확대됩니다.
🐛 제안: 문맥 가산 중복 방지
+ context_already_credited = False
...
if 0.60 > best_ticker_score:
best_ticker_score = 0.60
best_ticker_reason = (
f"ambiguous_ticker:uppercase_business_context:{ticker}"
)
best_ticker = ticker
+ context_already_credited = True
...
- if score > 0.0 and _contains_market_context((article.title, snippet)):
+ if (
+ score > 0.0
+ and not context_already_credited
+ and _contains_market_context((article.title, snippet))
+ ):
score = min(1.0, score + 0.20)
reasons.append("business_or_market_context")Also applies to: 292-294
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@AI/modules/data_collector/components/news/relevance.py` around lines 245 -
261, Prevent the market-context evidence from being counted twice for the
ambiguous_ticker:uppercase_business_context path: update the later scoring logic
near the best_ticker_score adjustment so this branch does not receive the
additional context bonus after _contains_market_context already enabled its 0.60
score. Preserve the existing scoring and bonus behavior for other
ticker-matching branches.
Comment on lines
+738
to
+741
| "tickers": ["PM"], | ||
| "aliases": ["Philip Morris International"], | ||
| "enabled": true, | ||
| "ambiguous_tickers": [] |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
fd relevance.py AI/modules/data_collector/components/news --exec cat -nRepository: SISC-IT/sisc-web
Length of output: 11662
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== company config entry =="
sed -n '733,743p' AI/modules/data_collector/config/sp100_companies.json
echo
echo "== nearby ticker entries with ambiguous_tickers =="
python3 - <<'PY'
import json
from pathlib import Path
data=json.loads(Path("AI/modules/data_collector/config/sp100_companies.json").read_text())
for item in data:
if "PM" in item.get("tickers", []) or item.get("ambiguous_tickers"):
print(item["legal_name"])
print(item)
print()
PY
echo
echo "== read-only relevance matcher behavior for candidate headlines =="
python3 - <<'PY'
import json
from pathlib import Path
import ast
src=Path("AI/modules/data_collector/components/news/relevance.py").read_text()
mod=ast.parse(src)
for cls in "RelevanceResult", "ProviderArticle":
print(cl := [n.name for n in mod.body if isinstance(n, ast.ClassDef) and n.name == cls][0])
PY
python3 - <<'PY'
import json
import re
import unicodedata
from dataclasses import dataclass
from pathlib import Path
src = Path("AI/modules/data_collector/config/sp100_companies.json").read_text()
data = json.loads(src)
companies = [x for x in data if x.get("tickers") == ["PM"]]
print("PM config:", companies)
PM_CONFIG = companies[0] if len(companies) == 1 else None
AMBIGUOUS = set(PM_CONFIG["ambiguous_tickers"]) if PM_CONFIG else set()
print("ambiguous_tickers:", AMBIGUOUS)
patterns = [
rf"\${re.escape('PM')}(?![A-Z0-9])",
rf"\({re.escape('PM')}\)",
rf"\b(?:NASDAQ|NYSE|NYSEARCA|AMEX)\s*:\s*{re.escape('PM')}\b",
]
def _ticker_matches(text, ticker):
escaped = re.escape(ticker.upper())
uppercase_text = unicodedata.normalize("NFKC", text).upper()
decorated = any(re.search(pattern, uppercase_text) for pattern in patterns)
bare = bool(re.search(rf"(?<![A-Z0-9]){escaped}(?![A-Z0-9])", uppercase_text))
return decorated, bare
def _uppercase_ticker_token_matches(text, ticker):
normalized = unicodedata.normalize("NFKC", text)
escaped = re.escape(ticker.upper())
return bool(re.search(rf"(?<![A-Z0-9]){escaped}(?![A-Z0-9])", normalized))
samples = [
("PM: stock rises", "after-hours trading"),
("Philip Morris International rises", "after-hours trading"),
("PM closes at $120", "after-hours trading"),
("PM closes at 120 pm", "after-hours trading"),
("PM closes at 120 PM", "after-hours trading"),
("PM closes at 120 PM ET", "after-hours trading"),
("PM closes at 3 PM", "after-hours trading"),
("PM closes at 3 pm", "after-hours trading"),
("PM sells in NYSEPM test", "after-hours trading"),
]
for title, snippet in samples:
print(title, "=>", _ticker_matches(title, "PM"), _ticker_matches(snippet, "PM"),
"upper title token", _uppercase_ticker_token_matches(title, "PM"))
PY
echo
echo "== search for GE/MU samples and ambiguous entries =="
python3 - <<'PY'
import json
from pathlib import Path
data=json.loads(Path("AI/modules/data_collector/config/sp100_companies.json").read_text())
for key in ["GE","GM","AM","PM","MU"]:
print(key, [x["legal_name"] for x in data if key in x.get("tickers", []) or key in x.get("aliases", [])])
for item in data:
if item.get("legal_name") in ["General Electric Company","Alcoa Inc.","Morgan Stanley","Philip Morris International","Micron Technology Inc."]:
print("==", item["legal_name"], "==")
print({"tickers": item.get("tickers"), "aliases": item.get("aliases"), "ambiguous_tickers": item.get("ambiguous_tickers")})
PYRepository: SISC-IT/sisc-web
Length of output: 647
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
src = Path("AI/modules/data_collector/config/sp100_companies.json").read_text()
data = json.loads(src)
for item in data:
if "PM" in item.get("tickers", []) or item.get("ambiguous_tickers"):
print(f"legal_name={item.get('legal_name')!r} tickers={item.get('tickers')!r} ambiguous_tickers={item.get('ambiguous_tickers')!r}")
PY
python3 - <<'PY'
import json
import re
import unicodedata
from pathlib import Path
src = Path("AI/modules/data_collector/config/sp100_companies.json").read_text()
data = json.loads(src)
PM_CONFIG = next(x for x in data if x.get("tickers") == ["PM"])
print("PM ambiguous_tickers:", PM_CONFIG["ambiguous_tickers"])
def _ticker_matches(text, ticker):
escaped = re.escape(ticker.upper())
uppercase_text = unicodedata.normalize("NFKC", text).upper()
decorators = (
rf"\${escaped}(?![A-Z0-9])",
rf"\({escaped}\)",
rf"\b(?:NASDAQ|NYSE|NYSEARCA|AMEX)\s*:\s*{escaped}\b",
)
decorated = any(re.search(pattern, uppercase_text) for pattern in decorators)
bare = bool(re.search(rf"(?<![A-Z0-9]){escaped}(?![A-Z0-9])", uppercase_text))
return decorated, bare
samples = [
"PM: stock rises",
"Philip Morris International rises",
"PM closes at $120",
"PM closes at 120 pm",
"PM closes at 120 PM",
"PM closes at 120 PM ET",
"PM closes at 3 PM",
"PM closes at 3 pm",
"PM closes at 3 P.M.",
"PM closes at 3 P.M.",
"PM sells in NYSEPM test",
"PM trades, Philip Morris International sells",
"PM sales fall",
"PM, Philip Morris International reported",
"Philip Morris reported",
]
print("PM ticker/relevance probe:")
for sample in samples:
decorated, bare = _ticker_matches(sample, "PM")
print(sample[:80].ljust(82), repr(f"deco={decorated:1} bare={bare:1}"))
# Simulate score_company_relevance for PM config with no aliases/name evidence
def business_context(text):
words = re.sub(r"[^\w]+", " ", text.casefold()).split()
terms = {"acquisition","acquire","antitrust","business","ceo","cfo","chief executive","company","demand","dividend",
"stock","stocks","share","shares","market cap","valuation","nasdaq","nyse","earnings","revenue","sales",
"profit","guidance","forecast","outlook","quarter","production","supply","launch","expands","expansion",
"services","regulator","regulation","lawsuit","settlement","layoff","workforce","executive","merger",
"investment","financial","results","tariff","recall","outage","investor"}
return any(r"$words", terms)
for sample in samples:
deco, bare = _ticker_matches(sample, "PM")
is_relevant = (deco or bare) # enough to enter score loop; real function has thresholds, see relevance.py
print("===", sample)
PYRepository: SISC-IT/sisc-web
Length of output: 279
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
src = Path("AI/modules/data_collector/config/sp100_companies.json").read_text()
data = json.loads(src)
print("top_level_type=", type(data).__name__)
if isinstance(data, dict):
print("top_level_keys=", list(data.keys())[:10])
elif isinstance(data, list):
print("top_level_length=", len(data))
for item in data:
if "PM" in item.get("tickers", []) or item.get("ambiguous_tickers"):
print(f"legal_name={item.get('legal_name')!r} tickers={item.get('tickers')!r} ambiguous_tickers={item.get('ambiguous_tickers')!r}")
pm = next((x for x in data if x.get("tickers") == ["PM"]), None)
print("PM entry found", pm.get("legal_name") if pm else "NO PM")
if pm:
print("PM ambiguous_tickers", pm.get("ambiguous_tickers"))
PYRepository: SISC-IT/sisc-web
Length of output: 508
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
src = Path("AI/modules/data_collector/config/sp100_companies.json").read_text()
data = json.loads(src)
companies = data.get("companies", [])
print("companies_type=", type(companies).__name__)
print("companies_length=", len(companies) if isinstance(companies, list) else "N/A")
for item in companies if isinstance(companies, list) else []:
if type(item) is dict and ("PM" in item.get("tickers", []) or item.get("ambiguous_tickers")):
print(f"legal_name={item.get('legal_name')!r} tickers={item.get('tickers')!r} aliases={item.get('aliases')!r} ambiguous_tickers={item.get('ambiguous_tickers')!r}")
pm = next((x for x in companies if isinstance(x, dict) and x.get("tickers") == ["PM"]), None)
print("PM entry found:", pm.get("legal_name") if pm else "NO PM")
if pm:
print("PM ambiguous_tickers", pm.get("ambiguous_tickers"))
PYRepository: SISC-IT/sisc-web
Length of output: 1903
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import json
import re
import unicodedata
from pathlib import Path
src = Path("AI/modules/data_collector/config/sp100_companies.json").read_text()
data = json.loads(src)
companies = data.get("companies", [])
PM_CONFIG = next((x for x in companies if isinstance(x, dict) and x.get("tickers") == ["PM"]), None)
print("PM ambiguous_tickers", PM_CONFIG.get("ambiguous_tickers") if PM_CONFIG else "ERROR")
def _ticker_matches(text, ticker):
escaped = re.escape(ticker.upper())
uppercase_text = unicodedata.normalize("NFKC", text).upper()
decorators = (
rf"\${escaped}(?![A-Z0-9])",
rf"\({escaped}\)",
rf"\b(?:NASDAQ|NYSE|NYSEARCA|AMEX)\s*:\s*{escaped}\b",
)
decorated = any(re.search(pattern, uppercase_text) for pattern in decorators)
bare = bool(re.search(rf"(?<![A-Z0-9]){escaped}(?![A-Z0-9])", uppercase_text))
return decorated, bare
def _contains_market_context(values):
values_text = " ".join(values)
normalized = re.sub(r"[^\w]+", " ", values_text.casefold()).split()
terms = {"stock","stocks","trading","trades","shares","earnings","revenue","sales","profit","guidance","market","nasdaq","nyse"}
return any(word in terms for word in normalized)
def _uppercase_ticker_token_matches(text, ticker):
normalized = unicodedata.normalize("NFKC", text)
escaped = re.escape(ticker.upper())
return bool(re.search(rf"(?<![A-Z0-9]){escaped}(?![A-Z0-9])", normalized))
samples = [
"PM: stock rises",
"Philip Morris International rises",
"PM closes at $120",
"PM closes at 120 pm",
"PM closes at 120 PM",
"PM closes at 120 PM ET",
"PM closes at 3 PM",
"PM closes at 3 pm",
"PM trades 3 PM",
"PM sales fall",
"PM reported earnings",
"PM sells",
"PM, Philip Morris International reported",
"Philip Morris International closes at 3 PM",
]
print("PM headlined token/context probe:")
for title in samples:
deco, bare = _ticker_matches(title, "PM")
print(f"{title[:60].ljust(60)} deco={deco} bare={bare}")
print(f"{(' ' + ' '.join(title.split())).split()[-5:]} market_context={_contains_market_context([title])} upper_token={_uppercase_ticker_token_matches(title, 'PM')}")
PYRepository: SISC-IT/sisc-web
Length of output: 2371
PM를 ambiguous_tickers에 추가하세요.
score_company_relevance()는 3자 티커도 대문자 토큰과 비즈니스 문맥이 있으면 0.60 점까지 인정하므로, Philip Morris International closes at 3 PM 같은 헤드라인이 PM 관련으로 오탐될 수 있습니다. CAT/LOW/NOW 처리 방식과 맞추려면 PM도 모호 티커로 두고, 회사명·마켓 문맥에서 보강해야만 인정되도록 해야 합니다.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@AI/modules/data_collector/config/sp100_companies.json` around lines 738 -
741, Add "PM" to the ambiguous_tickers array for the Philip Morris International
entry, matching the existing handling of CAT, LOW, and NOW while leaving the
other company configuration unchanged.
Comment on lines
+924
to
+931
| "company_key": "exxonmobil", | ||
| "cik": "0002115436", | ||
| "legal_name": "ExxonMobil Holdings Corp", | ||
| "tickers": ["XOM"], | ||
| "aliases": ["ExxonMobil", "Exxon Mobil"], | ||
| "enabled": true, | ||
| "ambiguous_tickers": [] | ||
| } |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
curl -sS -A "review-check/1.0" https://www.sec.gov/files/company_tickers.json -o /tmp/sec_tickers.json
python - <<'PY'
import json
sec = json.load(open("/tmp/sec_tickers.json"))
by_ticker = {v["ticker"].upper(): (str(v["cik_str"]).zfill(10), v["title"]) for v in sec.values()}
snap = json.load(open("AI/modules/data_collector/config/sp100_companies.json"))
print(f"companies={len(snap['companies'])} tickers={sum(len(c['tickers']) for c in snap['companies'])}")
for c in snap["companies"]:
for t in c["tickers"]:
hit = by_ticker.get(t)
if hit is None:
print(f"[MISSING] {t} {c['legal_name']}")
elif hit[0] != c["cik"]:
print(f"[CIK-DIFF] {t} snapshot={c['cik']}/{c['legal_name']} sec={hit[0]}/{hit[1]}")
elif hit[1].strip().lower() != c["legal_name"].strip().lower():
print(f"[NAME-DIFF] {t} snapshot={c['legal_name']} sec={hit[1]}")
PYRepository: SISC-IT/sisc-web
Length of output: 883
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the relevant config lines and confirm file shape without running repo code.
python3 - <<'PY'
import json
from pathlib import Path
path = Path("AI/modules/data_collector/config/sp100_companies.json")
data = json.loads(path.read_text())
companies = data.get("companies", [])
for idx, c in enumerate(companies):
if c.get("legal_name") in {
"ExxonMobil Holdings Corp",
"ExxonMobil Corporation",
"Exxon Mobil Corporation",
"The BlackRock Inc.",
"The BlackRock Inc",
"BlackRock, Inc.",
"Bank of New York Mellon Corp",
"Bank of New York Mellon Corporation",
"Honeywell Aerospace Inc",
}:
print("index", idx, "key", c.get("company_key"), "cik", c.get("cik"), "name", c.get("legal_name"), "tickers", c.get("tickers"))
sec = json.loads(Path("/tmp/sec_tickers.json").read_text())
by_ticker = {str(k).upper(): (str(v["cik_str"]).zfill(10), v["title"]) for k, v in sec.items()}
targets = ["XOM", "HONA", "BNY", "BLK"]
for t in targets:
print("SEC", t, by_ticker.get(t, "MISSING"))
PYRepository: SISC-IT/sisc-web
Length of output: 826
🌐 Web query:
SEC company_tickers.json XOM CIK BlackRock HONA BNY current
💡 Result:
The SEC provides the company_tickers.json file as a public, static resource that maps stock ticker symbols, CIK (Central Index Key) numbers, and official EDGAR entity names [1][2][3]. This file is a critical utility for developers to translate human-readable tickers into the numeric CIKs required for accessing SEC EDGAR API endpoints [4][3]. When using these CIKs in API requests (e.g., to the data.sec.gov domain), they must be zero-padded to a 10-digit string (e.g., CIK0000034088) to avoid 404 errors [4][5][3]. The requested identifiers are as follows: XOM: The ticker symbol for Exxon Mobil Corporation, which has the CIK 0000034088 [6][7][8]. BlackRock: BlackRock, Inc. has used different CIKs over time due to corporate structure changes [9][10]. As of July 2026, the primary CIK associated with recent BlackRock, Inc. filings is 0002012383 [10]. Other entities, such as BlackRock Finance, Inc. (CIK 0001364742), also exist within the EDGAR database [9][11]. HONA: This is the ticker for Honeywell Aerospace Inc. [12][13][14]. It was assigned the CIK 2089271 [14][15]. BNY: This ticker is associated with The Bank of New York Mellon Corporation, which has the CIK 1390777 [16]. Note that "BNY" also appears in various fund-related entries (e.g., BNY Mellon Investment Adviser funds) that utilize different, fund-specific CIKs [17][18][19][20]. For the most accurate and current mapping, you should always consult the official company_tickers.json file directly from the SEC website [1][4][3].
Citations:
- 1: https://www.sec.gov/file/company-tickers
- 2: https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data
- 3: https://dev.to/odeeb/the-sec-edgar-api-a-practical-guide-to-free-filing-data-in-python-15b
- 4: https://edgarscout.com/edgar-api/
- 5: https://blog.greenflux.us/so-you-want-to-integrate-with-the-sec-api/
- 6: https://www.sec.gov/cgi-bin/browse-edgar?CIK=XOM&Country=&Find=Find+Companies&SIC=&action=getcompany&company=&filenum=&match=&owner=exclude
- 7: https://www.sec.gov/cgi-bin/browse-edgar?CIK=XOM&type=4
- 8: https://app.edgar.tools/companies/XOM
- 9: https://www.sec.gov/cgi-bin/browse-edgar?CIK=BLK&action=getcompany
- 10: https://www.sec.gov/cgi-bin/browse-edgar?CIK=0002012383
- 11: https://www.sec.gov/cgi-bin/browse-edgar?CIK=0001364742&owner=exclude
- 12: https://finviz.com/stock?t=HONA
- 13: https://uk.finance.yahoo.com/quote/HONA/
- 14: https://stockanalysis.com/stocks/hona/company/
- 15: https://last10k.com/sec-filings/hona/0001628280-26-045874.htm
- 16: https://stockanalysis.com/stocks/bny/company/
- 17: https://www.sec.gov/cgi-bin/browse-edgar?CIK=C000001368&action=getcompany
- 18: https://www.sec.gov/cgi-bin/browse-edgar?CIK=C000158972&action=getcompany
- 19: https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&filenum=033-31809
- 20: https://www.sec.gov/cgi-bin/browse-edgar?CIK=C000173298&action=getcompany
🏁 Script executed:
#!/bin/bash
set -euo pipefail
SEC_URL="https://www.sec.gov/files/company_tickers.json"
CONFIG="AI/modules/data_collector/config/sp100_companies.json"
SEC_BODY="$(curl -sS -L -A "review-check/1.0" -H "Accept-Encoding: gzip, deflate, br" "$SEC_URL")"
printf '%s\n' "$SEC_BODY" | python3 - <<'PY'
import json, sys
try:
sec = json.load(sys.stdin)
except Exception as e:
print("SEC_JSON_PARSE_ERROR", repr(str(e)[:160]))
sys.exit(0)
with open("$CONFIG") as f:
snap = json.load(f)
by_ticker = {str(k).upper(): (str(v["cik_str"]).zfill(10), v["title"]) for k, v in sec.items()}
targets = {
"XOM": "0000034088",
"HONA": "0001628280",
"BNY": "0001368",
"BLK": "0002012383",
}
print("SEC_JSON_OK", "count", len(sec))
for t, expected_cik in targets.items():
hit = by_ticker.get(t)
if not hit:
print("MISSING", t)
continue
print(t, "hit_cik", hit[0], "target_cik", expected_cik, "name", hit[1])
for c in snap["companies"]:
if c.get("company_key") in {"exxonmobil", "honeywell-aerospace", "bank-of-new-york-mellon", "blackrock"}:
print("SNAP", c.get("company_key"), "cik", c.get("cik"), "name", c.get("legal_name"), "tickers", c.get("tickers"))
PYRepository: SISC-IT/sisc-web
Length of output: 219
🏁 Script executed:
#!/bin/bash
set -euo pipefail
SEC_URL="https://www.sec.gov/files/company_tickers.json"
CONFIG="AI/modules/data_collector/config/sp100_companies.json"
SEC_BODY="$(curl -sS --connect-timeout 20 --max-time 40 -L -A "review-check/1.0" -H "Accept-Encoding: gzip, deflate, br" "$SEC_URL")"
printf 'SEC_CONTENT_LINES=%s\n' "$(printf '%s\n' "$SEC_BODY" | wc -l)"
printf 'FIRST_BYTES=%s\n' "$(printf '%s' "$SEC_BODY" | head -c 200 | tr '\n' '\n')"
CONFIG="$(node -e 'console.log(require("fs").readFileSync(process.argv[1],"utf8"))' "$CONFIG")"
SEC="$SEC_BODY"
node - <<'JS' "$CONFIG" "$SEC"
const [snap, sec] = process.argv.slice(2);
try {
const snapData = JSON.parse(snap);
const secData = JSON.parse(sec);
const byTicker = {};
for (const [k, v] of Object.entries(secData)) byTicker[String(k).toUpperCase()] = [String(v.cik_str).padStart(10, '0'), v.title];
const targets = {
"XOM": "0000034088",
"HONA": "0001628280",
"BNY": "0001368",
"BLK": "0002012383",
};
for (const [t, expectedCik] of Object.entries(targets)) {
const hit = byTicker[t];
console.log("SEC", t, hit ? `cik=${hit[0]} name=${hit[1]}` : "MISSING", "expected_cik", expectedCik);
}
for (const c of snapData.companies) {
if (["exxonmobil", "honeywell-aerospace", "bank-of-new-york-mellon", "blackrock"].includes(c.company_key)) {
console.log("SNAP", c.company_key, "cik=", c.cik, "legal_name=", c.legal_name, "tickers=", JSON.stringify(c.tickers));
}
}
} catch (e) {
console.log("PARSE_ERROR", e.message);
}
JSRepository: SISC-IT/sisc-web
Length of output: 456
🏁 Script executed:
#!/bin/bash
set -euo pipefail
SEC_URL="https://www.sec.gov/files/company_tickers.json"
SEC_BODY="$(curl -sS --connect-timeout 20 --max-time 40 -L -A "review-check/1.0" "$SEC_URL" --compressed)"
printf 'CONTENT_LENGTH=%s\n' "${`#SEC_BODY`}"
printf 'HTTP_CONTENT_TYPE_CHECK=%s\n' "$SEC_BODY" | head -c 50
[ "${SEC_BODY:0:1}" = "{" ] && printf 'STARTS_WITH_JSON=%s\n' 'yes' || printf 'STARTS_WITH_JSON=%s\n' 'no'
[ "${SEC_BODY:0:1}" = "{" ] || exit 0
SEC_TOOL_PATH="$(mktemp)"
printf '%s\n' "$SEC_BODY" > "$SEC_TOOL_PATH"
node - <<'JS' "$SEC_TOOL_PATH" "AI/modules/data_collector/config/sp100_companies.json"
const [secPath, snapPath] = process.argv.slice(2);
const sec = require(secPath);
const snap = require(snapPath.replace(/\.json$/, '') + (snapPath.endsWith('.json') ? '' : '.json'));
const secData = typeof sec === 'object' ? sec : { sec };
const byTicker = {};
for (const [k, v] of Object.entries(secData)) {
if (typeof v === 'object' && typeof v.cik_str === 'number' && typeof v.ticker === 'string') {
byTicker[String(k).toUpperCase()] = [String(v.cik_str).padStart(10, '0'), v.title];
}
}
const targets = { "XOM": "0000034088", "HONA": "0001628280", "BNY": "0001368", "BLK": "0002012383" };
for (const [t, expectedCik] of Object.entries(targets)) {
const hit = byTicker[t] || byTicker[t.toUpperCase()] || Object.values(byTicker).find(([c]) => c.endsWith(expectedCik) || c === expectedCik || String(expectedCik).replace(/^0+/, '') === String(c).replace(/^0+/, ''));
if (!hit) continue;
console.log("SEC", t, "cik=", hit[0], "name=", hit[1], "expectedCIK=", expectedCik, "cikMatch=", hit[0] === expectedCik);
}
for (const c of snap.data_collector.config.sp100_companies.companies || snap.companies || []) {
if (["exxonmobil", "honeywell-aerospace", "bank-of-new-york-mellon", "blackrock"].includes(c.company_key)) {
console.log("SNAP", c.company_key, "cik=", c.cik, "legal_name=", c.legal_name, "tickers=", JSON.stringify(c.tickers));
}
}
JSRepository: SISC-IT/sisc-web
Length of output: 244
🏁 Script executed:
#!/bin/bash
set -euo pipefail
SEC_URL="https://www.sec.gov/cgi-bin/browse-edgar?Company=Exxon&type=Corporate+name&Search=Search"
SEC_BODY="$(curl -sS --connect-timeout 20 --max-time 40 -L -A "review-check/1.0" --compressed "$SEC_URL")"
SEC_MATCH="$(printf '%s\n' "$SEC_BODY" | grep -oE "0000034088|CIK0002115<PASSWORD>|Exxon Mobil Corporation|ExxonMobil Holdings Corp|Honeywell Aerospace Inc|The Bank[\s\S]{0,100}BNY|BlackRock, Inc" | head -20 || true)"
CONFIG_MATCH="$(python3 - <<'PY'
import json
with open("AI/modules/data_collector/config/sp100_companies.json") as f:
data=json.load(f)
for c in data["companies"]:
if c.get("company_key") in {"exxonmobil","honeywell-aerospace","bank-of-new-york-mellon","blackrock"}:
print(c["company_key"], "CIK=", c["cik"], "legal_name=", c["legal_name"], "tickers=", c["tickers"])
PY
)"
printf 'SEC_RESULTS=%s\n' "$SEC_MATCH"
printf 'CONFIG_RESULTS=%s\n' "$CONFIG_MATCH"Repository: SISC-IT/sisc-web
Length of output: 534
ExxonMobil 항목의 legal_name/cik를 SEC 매핑과 맞춰 주세요.
"ExxonMobil Holdings Corp" / 0002115436은 XOM의 일반 SEC 매핑인 Exxon Mobil Corporation / 0000034088과 다르므로, Google RSS 검색식과 수집 대상 판정에 의도치 않은 issuer로 연결될 수 있습니다.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@AI/modules/data_collector/config/sp100_companies.json` around lines 924 -
931, Update the ExxonMobil entry identified by company_key "exxonmobil" so
legal_name is "Exxon Mobil Corporation" and cik is "0000034088", matching the
standard SEC mapping for ticker XOM; leave its other fields unchanged.
Comment on lines
+214
to
+234
| def main(argv: list[str] | None = None) -> int: | ||
| try: | ||
| args = parse_args(argv) | ||
| payload, exit_code = run(args) | ||
| except (OSError, ValueError, json.JSONDecodeError) as exc: | ||
| payload = { | ||
| "collection_status": "failed", | ||
| "collection_outcome": CollectionStatus.INVALID_CONFIG.value, | ||
| "failure_reason": f"{type(exc).__name__}: {exc}", | ||
| "relevant_article_count": None, | ||
| } | ||
| exit_code = EXIT_INVALID_CONFIG | ||
|
|
||
| serialized = json.dumps(payload, ensure_ascii=False, indent=2) | ||
| if args.output: | ||
| output_path = Path(args.output) | ||
| output_path.parent.mkdir(parents=True, exist_ok=True) | ||
| output_path.write_text(serialized + "\n", encoding="utf-8") | ||
| else: | ||
| print(serialized) | ||
| return exit_code |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
예외 경로에서 args가 바인딩되지 않을 수 있습니다.
args = parse_args(argv)가 try 안에 있어, 인자 파싱 단계에서 ValueError/OSError가 발생하면 Line 228의 args.output 접근이 NameError로 이어집니다(원래 오류가 가려짐). 파싱은 try 밖으로 옮기는 편이 안전합니다.
🛡️ 제안 변경
def main(argv: list[str] | None = None) -> int:
+ args = parse_args(argv)
try:
- args = parse_args(argv)
payload, exit_code = run(args)
except (OSError, ValueError, json.JSONDecodeError) as exc:🧰 Tools
🪛 ast-grep (0.45.0)
[info] 226-226: use jsonify instead of json.dumps for JSON output
Context: json.dumps(payload, ensure_ascii=False, indent=2)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@AI/modules/data_collector/scripts/collect_company_news.py` around lines 214 -
234, Move args = parse_args(argv) outside the try block in main so args is
always available when handling collection errors. Keep run(args) inside the
existing exception handling and preserve the current failure payload and exit
code behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
변경 내용
SEC EDGAR 이벤트 저장 여부와 관계없이 실행할 수 있는 미국 기업 뉴스 수집 코어를 추가했습니다.
뉴스 수집 구조
news_data.py에 섞여 있던 RSS 수집, 기사 본문 접근, LLM 요약 로직을 분리했습니다.components/news/패키지로 분리했습니다.news_data.pyimport 경로는 호환 façade로 유지했습니다.대상 기업
GOOG,GOOGL처럼 동일 기업의 복수 주식 클래스를 하나의company_key와 CIK로 묶습니다.company_keyGoogle News RSS 수집
{ticker} stock을 하나의 OR 검색식으로 조회합니다.시각 처리
pubDate원문을 그대로 보존합니다.published_at_utc를 별도로 생성합니다.accepted_at을 기준으로 다음 구간을 계산할 수 있는 순수 함수를 추가했습니다.URL 처리 및 정확 중복 제거
정확 중복은 다음 강한 키 우선순위로 처리합니다.
(provider, provider_article_id)(provider, provider_url_hash)제목이 같다는 이유만으로 기사 행을 삭제하지 않습니다.
재전송 기사 후보 그룹
회사 관련도 계산
company_relevance_score만 계산합니다.수집 상태 구분
최상위 실행 상태와 수집 결과를 분리했습니다.
collection_statussuccesspartial_successfailedcollection_outcomeresultsno_resultsno_relevant_newspartial_resultsprovider_errorinvalid_config따라서 다음 두 상황을 구분할 수 있습니다.
{ "collection_status": "success", "collection_outcome": "no_relevant_news", "relevant_article_count": 0 }{ "collection_status": "failed", "collection_outcome": "provider_error", "failure_reason": "source_timeout", "relevant_article_count": null }S&P 100 기업 중 일부만 실패하면 다른 기업의 수집은 계속 진행하고
partial_success와 non-zero 종료 코드로 알립니다.수집 메트릭
다음 관측 지표를 기록합니다.
정답 기사 집합이 없으면 실제 기사 누락률을 계산할 수 없으므로
article_omission_rate는null로 유지합니다.본문 및 fallback 정책
실행 CLI
S&P 100 전체 또는 선택한 ticker만 수집할 수 있는 one-shot CLI를 추가했습니다.
변경 이유
기업 뉴스는 SEC 이벤트와 독립적으로 먼저 수집할 수 있습니다. 이후 SEC 이벤트가 저장되면 기존 뉴스 중 CIK·기업·시간 범위가 일치하는 기사만 별도 linker에서 연결하는 구조가 더 안전합니다.
이번 변경으로 다음 작업을 SEC 구현과 독립적으로 검증할 수 있게 되었습니다.
Part of #377
영향 범위
AI/modules/data_collector/components/news/AI/modules/data_collector/components/news_data.pyAI/modules/data_collector/components/__init__.pyAI/modules/data_collector/config/news_collection.jsonAI/modules/data_collector/config/sp100_companies.jsonAI/modules/data_collector/scripts/collect_company_news.pyAI/modules/data_collector/README.mdAI/tests/verify_company_news.pyAI/tests/fixtures/news/google_news_rss.xml다음 영역은 변경하지 않았습니다.
schema.sql검증 결과
accepted_at기준 −24시간/+48시간 구간 계산 테스트참고 사항 및 현재 제한
company_relevance_score이며 SEC 공시 관련도가 아닙니다.다음 단계: SEC 이벤트 연결
SEC 담당자가 다음 필드를 저장한 후 별도 event-news linker를 구현합니다.
{ "event_id": "AAPL_2026_000123", "ticker": "AAPL", "cik": "0000320193", "accession_number": "0001234567-26-000123", "event_type": "8-K_2.02", "accepted_at": "2026-07-22T16:12:00-04:00", "updated_at": "...", "status": "active" }연결 순서는 다음과 같습니다.
company_key를 연결합니다.accepted_at을 UTC로 변환합니다.company_relevance_score와 별도로event_relevance_score를 계산합니다.DIRECT: 공시·실적 발표를 직접 다루는 기사CONTEXT: 같은 기업의 이벤트 전후 사업·시장 문맥 기사UNRELATED: 기업은 같지만 이벤트 문맥과 무관한 기사event_id + article_id를 unique key로 사용해 idempotent upsert합니다.updated_at, 상태 또는 version이 변경되면 연결 결과를 다시 계산합니다.accepted_at + 48시간이 지날 때까지 증분 reconcile합니다.권장 저장 구조는 다음과 같습니다.
DB migration 번호는 구현 시점의 최신
main을 다시 확인한 후 정합니다. 현재 저장소에는 이미V4__post_like_concurrency.sql이 있으므로 뉴스 migration을V4로 생성하면 안 됩니다.후속 구현 순서