How did this take so long to identify...
collect_github_messages decides whether a repo is "small" (by checking if issue_pr_sum < 10) and then switches how it collects messages (comments i think)
|
if is_repo_small(repo_id): |
|
message_data = fast_retrieve_all_pr_and_issue_messages(repo_git, logger, manifest.key_auth, task_name, core_data_last_collected) |
|
|
|
if message_data: |
|
process_messages(message_data, task_name, repo_id, logger, db_session) |
|
|
|
else: |
|
logger.info(f"{owner}/{repo} has no messages") |
|
|
|
else: |
|
process_large_issue_and_pr_message_collection(repo_id, repo_git, logger, manifest.key_auth, task_name, db_session, core_data_last_collected) |
The "fast" path uses a call to /repos/{owner}/{repo}/issues/comments with the since= parameter
the slow path does an individual call per issue or comment:
|
for comment_url in comment_urls: |
|
try: |
|
messages = list(github_data_access.paginate_resource(comment_url)) |
|
all_data += messages |
additionally, according to Sonnet 4.6 medium:
The large repo path filters by parent issue/PR updated_at in the DB query
The unified endpoint supports ?since= which filters by comment's own updated_at.
These semantics differ: if someone changes a label on a 3-year-old issue (updating the issue's updated_at), the large repo path fetches ALL of that issue's old comments again. The unified endpoint with since would correctly fetch only comments that were actually new or edited since the cutoff.
The large repo approach is actually less correct for incremental collection, not more correct — it re-fetches stale comment data that's already in the DB whenever a parent issue is touched for any reason.
This is very likely significantly contributing to #391
How did this take so long to identify...
collect_github_messages decides whether a repo is "small" (by checking if
issue_pr_sum< 10) and then switches how it collects messages (comments i think)CollectOSS/collectoss/tasks/github/messages.py
Lines 42 to 52 in e2e7513
The "fast" path uses a call to
/repos/{owner}/{repo}/issues/commentswith thesince=parameterthe slow path does an individual call per issue or comment:
CollectOSS/collectoss/tasks/github/messages.py
Lines 123 to 126 in e2e7513
additionally, according to Sonnet 4.6 medium:
This is very likely significantly contributing to #391