Spring futureeval 2026 updates

.gitignore

@@ -5,4 +5,5 @@ __pycache__
 latest.log
 logs/
 sandbox.py
-local/
+local/
+temp/

README.md

@@ -1,11 +1,8 @@
-The main entry point to the project is `aib_analysis/main.py`
-
 ## Installing Dependencies
-To set up the project please run:
-Installing dependencies
-Make sure you have python and poetry installed (poetry is a python package manager).
 
-If you don't have poetry installed run the below:
+Make sure you have Python and Poetry installed (Poetry is a Python package manager).
+
+If you don't have Poetry installed:
 ```
 sudo apt update -y
 sudo apt install -y pipx
@@ -15,42 +12,47 @@ pipx install poetry
 poetry config virtualenvs.in-project true
 ```
 
-Inside the terminal, go to the directory you cloned the repository into and run the following command:
-
+From the repo root:
 ```
 poetry install
 ```
 
 to install all required dependencies.
 
-You may also need to follow instructions [here](https://git-lfs.com/) for setting up git management for large files.
+## Input data
+
+Forecast CSVs live under `local/private_input_data/` (gitignored). For Spring 2026 the simulation script expects:
+
+- `local/private_input_data/pro_forecasts_2026_spring.csv`
+- `local/private_input_data/bot_forecasts_2026_spring.csv`
+
+You can:
 
-## Running the Analysis
+- If you have access, download prior analysis outputs (and sometimes inputs) from [this Google Drive folder](https://drive.google.com/drive/folders/1m7e8AQd4M-Y4oPuj--dDAEYwxO-J2kth?usp=drive_link), or
+- Request access to analysis data from the instructions shared [here](https://www.metaculus.com/notebooks/38928/ai-benchmark-resources/#what-data-do-i-have-access-to-via-api-how-can-i-get-access-to-more) which will show you how to use the [Metaculus Data Needs Form](https://docs.google.com/forms/d/e/1FAIpQLSeJhtZzHl5qMvBjbXbatyaqoS4IU7RE0GGw_vlhs6I9syqn1g/viewform?usp=pp_url&entry.192763438=https://www.metaculus.com/api/)
 
-To generate the analysis data run `aib_analysis/run_q1_q1_simulations.py`.
+## Running the analysis
 
-Check the logs for warnings to see edge cases that have come up.
+For the current (Spring 2026) tournament:
 
-To display the analysis please execute:
 ```
-poetry run streamlit run aib_analysis/front_end.py
+poetry run python aib_analysis/run_spring_simulations.py
 ```
 
-This will bring up the visuals for the analysis. Make sure you have chosen the right input data in this script. Front end uses caching, so restart streamlit if you update the underlying data.
+That writes JSON artifacts to `local/spring_2026_simulations_teams_comparison/`.
+
+Check `logs/latest_info.log` for warnings (annulled questions, weight mismatches, unmatched questions, missing control bots, etc.).
 
+For older seasons (Q1–Q4), use `aib_analysis/run_q1_q2_simulations.py` and edit the `__main__` paths/folders as needed.
 
-Make sure to restart the site or click "clear cache" in the triple dot menu on the site if you change the underlying data or simulated_tournament loading/intialization code.
+## Viewing results
 
+```
+poetry run streamlit run aib_analysis/front_end.py
+```
 
-You can see/download/use previous analysis results here https://drive.google.com/drive/folders/1m7e8AQd4M-Y4oPuj--dDAEYwxO-J2kth?usp=drive_link
+In the UI, set the tournaments folder to the simulation output path (e.g. `local/spring_2026_simulations_teams_comparison`). Restart Streamlit or use **Clear cache** in the ⋮ menu after changing underlying JSON or loading/scoring code.
 
 ## Structure
-The project is focused around the SimulatedTournament object. This is initialized with a number of Forecast objects (see `data_models.py`) and used to create other parts of a tournament (Users, Scores, etc). Every data analysis item we care about is just a tournament of some sort. Often this is a filter, aggregation, intersection, etc of forecasts from another tournament, but even a comparison of 2 people is a tournament.
-
-Refactor advantages
-- Pydantic model validation (type safety, and consistency checks at initialization time)
-- Easier to unit test
-- More reusable (e.g. We can use this structure for regular peer score comparisons in the future)
-- Git diffs are easier to read
-- We can eventually publish our results as an interactive app if we want to (and if not, we can more easily check and interact with the data ourselves)
 
+The project is built around `SimulatedTournament` (see `data_models.py` / `simulated_tournament.py`). It is initialized from `Forecast` objects and derives users, scores, etc. Most analyses are filtered, aggregated, or combined tournaments built from those forecasts.

aib_analysis/data_structures/custom_types.py

@@ -4,25 +4,26 @@
 BinaryResolutionType = bool
 MCResolutionType = str
 NumericResolutionType = float
-ResolutionType = BinaryResolutionType | MCResolutionType | NumericResolutionType | AnnulledAmbiguousResolutionType
+DiscreteResolutionType = float
+ResolutionType = BinaryResolutionType | MCResolutionType | NumericResolutionType | DiscreteResolutionType | AnnulledAmbiguousResolutionType
 
 BinaryForecastType = list[float] # binary: [p_yes, p_no]
 MCForecastType = list[float] # multiple choice: [p_option_a, p_option_b, p_option_c],
 NumericForecastType = list[float] # numeric: [p_0, p_1, p_2, ..., p_200] (201 value cdf)
-ForecastType = BinaryForecastType | MCForecastType | NumericForecastType | None
-
+DiscreteForecastType = list[float] # discrete: cdf of variable length
+ForecastType = BinaryForecastType | MCForecastType | NumericForecastType | DiscreteForecastType | None
 
 class UserType(Enum):
     PRO = "pro"
     BOT = "bot"
     CP = "community_prediction"
     AGGREGATE = "aggregate"
 
-
 class QuestionType(Enum):
     BINARY = "binary"
     MULTIPLE_CHOICE = "multiple_choice"
     NUMERIC = "numeric"
+    DISCRETE = "discrete"
 
 
 class ScoreType(Enum):

aib_analysis/data_structures/data_models.py

@@ -3,7 +3,7 @@
 import logging
 import random
 from datetime import datetime, timedelta, timezone
-
+from typing import Literal
 from pydantic import BaseModel, ConfigDict, model_validator
 from typing_extensions import Self
 
@@ -51,7 +51,6 @@ def id(self) -> str:
         return self._id
 
     def get_spot_baseline_score(self, resolution: ResolutionType) -> Score:
-        self._error_if_need_zero_point()
         q = self.question
         score_value = calculate_baseline_score(
             forecast=self.prediction,
@@ -63,6 +62,7 @@ def get_spot_baseline_score(self, resolution: ResolutionType) -> Score:
             range_max=q.range_max,
             open_upper_bound=q.open_upper_bound,
             open_lower_bound=q.open_lower_bound,
+            zero_point=q.zero_point,
         )
 
         return Score(
@@ -74,13 +74,9 @@ def get_spot_baseline_score(self, resolution: ResolutionType) -> Score:
     def get_spot_peer_score(
         self, resolution: ResolutionType, other_users_forecasts: list[Forecast]
     ) -> Score:
-        self._error_if_need_zero_point()
+        # Metaculus includes the scored forecaster in the geometric mean baseline and
+        # applies N/(N-1) as a leave-one-out approximation, so self may appear here.
         other_preds = [f.prediction for f in other_users_forecasts]
-        users_used_in_scoring = [f.user for f in other_users_forecasts]
-        if self.user in users_used_in_scoring:
-            raise ValueError(
-                "Forecast Author cannot be in other users forecasts list for peer score"
-            )
         q = self.question
         score_value = calculate_peer_score(
             forecast=self.prediction,
@@ -91,17 +87,14 @@ def get_spot_peer_score(
             options=list(q.options) if q.options is not None else None,
             range_min=q.range_min,
             range_max=q.range_max,
+            zero_point=q.zero_point,
         )
         return Score(
             score=score_value,
             type=ScoreType.SPOT_PEER,
             forecast=self,
         )
 
-    def _error_if_need_zero_point(self) -> None:
-        if self.question.is_log_scale:
-            raise NotImplementedError(f"Numeric question {self.question.question_id} has a zero point. Log Scall is currently not supported")
-
     @model_validator(mode="after")
     def check_prediction_type_matches(self) -> Self:
         if self.prediction is None:
@@ -126,9 +119,9 @@ def check_prediction_type_matches(self) -> Self:
                 raise ValueError(
                     "Prediction must be a list of floats (length >= 2) for multiple choice questions."
                 )
-            if abs(sum(self.prediction) - 1) > 1e-6:
+            if abs(sum(self.prediction) - 1) > 1e-5:
                 raise ValueError(
-                    "Prediction must sum to 1 for multiple choice questions."
+                    f"Prediction must sum to 1 for multiple choice questions, got {sum(self.prediction)} for {self.prediction}"
                 )
         elif q_type == QuestionType.NUMERIC:
             is_list = isinstance(self.prediction, list)
@@ -139,6 +132,19 @@ def check_prediction_type_matches(self) -> Self:
                 raise ValueError(
                     "Prediction must be a list of 201 floats for numeric questions."
                 )
+        elif q_type == QuestionType.DISCRETE:
+            if self.question.inbound_outcome_count is None:
+                raise ValueError("inbound_outcome_count must be set for discrete questions.")
+
+            is_list = isinstance(self.prediction, list)
+            expected_points = self.question.inbound_outcome_count + 1
+            has_expected_points = len(self.prediction) == expected_points
+            non_float_items = [p for p in self.prediction if not isinstance(p, float)]
+            all_floats = len(non_float_items) == 0
+            if not (is_list and has_expected_points and all_floats):
+                raise ValueError(
+                    f"Prediction must be a list of {expected_points} floats for discrete questions, got {len(self.prediction) if is_list else 'non-list'}."
+                )
 
         # verify predictions are between 0 and 1
         for p in self.prediction:
@@ -207,12 +213,14 @@ class Question(BaseModel, frozen=True):
     open_upper_bound: bool | None
     open_lower_bound: bool | None
     zero_point: float | None = None
+    inbound_outcome_count: int | None = None
     weight: float
     post_id: int
     created_at: datetime
     spot_scoring_time: datetime
     project: str | None = None
     notes: str | None = None
+    unscored_resolution_reason: Literal["annulled", "ambiguous", "blank"] | None = None     # When resolution is None: "annulled" / "ambiguous" from CSV, or "blank" for null resolution (often deleted posts). None if unknown (e.g. older JSON).
     model_config = ConfigDict(frozen=True)
 
     @property
@@ -236,6 +244,10 @@ def check_resolution_type_matches(self) -> Self:
                 self.resolution, float
             ):
                 raise ValueError("Resolution must be a float for numeric questions.")
+            if self.type == QuestionType.DISCRETE and not isinstance(
+                self.resolution, float
+            ):
+                raise ValueError("Resolution must be a float for discrete questions.")
         return self
 
     @model_validator(mode="after")
@@ -267,6 +279,17 @@ def check_question_type_has_right_fields(self) -> Self:
                 raise ValueError(
                     "Numeric questions must have all bound information (upper_bound, lower_bound, open_upper_bound, open_lower_bound)."
                 )
+        if self.type == QuestionType.DISCRETE:
+            if (
+                self.range_max is None
+                or self.range_min is None
+                or self.open_upper_bound is None
+                or self.open_lower_bound is None
+                or self.inbound_outcome_count is None
+            ):
+                raise ValueError(
+                    "Discrete questions must have all bound information (upper_bound, lower_bound, open_upper_bound, open_lower_bound) and inbound_outcome_count."
+                )
         if self.type == QuestionType.MULTIPLE_CHOICE:
             if not self.options or len(self.options) < 2:
                 raise ValueError(
@@ -339,6 +362,7 @@ def get_hash_for_tournament_matching(self) -> str:
             self.open_upper_bound,
             self.open_lower_bound,
             self.zero_point,
+            self.inbound_outcome_count,
             spot_scoring_window,
         )
         return str(hash(hash_fields))
@@ -348,12 +372,23 @@ class User(BaseModel):
     name: str
     type: UserType
     aggregated_users: list[User]
+    is_primary_bot: bool | None = None
+    exclude_from_aggregations: bool | None = None
     model_config = ConfigDict(frozen=True)
 
     @property
     def is_metac_bot(self) -> bool:
         return "metac-" in self.name.lower() or "mf-bot-" in self.name.lower()
 
+    @property
+    def contributes_to_peer_baseline(self) -> bool:
+        """Match Metaculus exclude_non_primary_bots + blacklist filtering."""
+        if self.exclude_from_aggregations is True:
+            return False # None for exclude_from_aggregations is interpreted as a default of False
+        if self.type == UserType.BOT and self.is_primary_bot is False:
+            return False # None for is_primary_bot is interpreted as a default of True
+        return True
+
 
 class Leaderboard(BaseModel):
     entries: list[LeaderboardEntry]

aib_analysis/data_structures/problem_questions.py

@@ -20,6 +20,7 @@ class Tournament(Enum):
     Q1_2025_VS_CUP = "q1_2025_vs_cup"
     Q2_2025 = "q2_2025"
     Fall_2025 = "fall_2025"
+    Spring_2026 = "spring_2026"
 
 
 class ProblemQuestion(BaseModel):
@@ -97,6 +98,35 @@ def check_urls(self) -> Self:
         proposed_action="",
         tournament=Tournament.Q1_2025_VS_CUP,
     ),
+    # Spring 2026
+    ProblemQuestion(
+        question_text="How many commercial aircraft deliveries will Airbus report for March 2026 ?",
+        question_1_url="https://www.metaculus.com/questions/42109/",
+        question_2_url="https://www.metaculus.com/questions/42272/",
+        notes="Duplicate questions in the same tournament. One is discrete, one is numeric. Both resolved to 60.",
+        proposed_action="Remove from comparison",
+        tournament=Tournament.Spring_2026,
+    ),
+    ProblemQuestion(
+        question_text="What will be the legal status of an EU PPWR delegated act on recyclability performance classes ... May 1, 2026?",
+        question_1_url="https://www.metaculus.com/questions/42519/",
+        question_2_url="https://www.metaculus.com/questions/42329/",
+        notes="Title mismatch only: pro 'on May 1' vs bot 'before May 1'. Same options and resolution ('No publication').",
+        proposed_action="Force match",
+        tournament=Tournament.Spring_2026,
+    ),
+    ProblemQuestion(
+        question_text="How many new civil antitrust merger challenges will the DOJ file against 'Big Tech' firms before May 1, 2026?",
+        question_1_url="https://www.metaculus.com/questions/42094/",
+        question_2_url="https://www.metaculus.com/questions/42050/",
+        notes=(
+            "Already hash-matches; force-match documents the intended pair. "
+            "Do not pair with annulled pro duplicate https://www.metaculus.com/questions/42063/ "
+            "(different inbound_outcome_count / spot window / annulled resolution)."
+        ),
+        proposed_action="Force match (exclude annulled 42063)",
+        tournament=Tournament.Spring_2026,
+    ),
 ]
 
 class ProblemManager:
@@ -131,6 +161,39 @@ def _pair_matches_problem_question_in_list(
         return False
 
 
+def _all_unscored_title_match_message(title_matched_questions: list[Question]) -> str:
+    urls = [question.url for question in title_matched_questions]
+    reasons = {
+        question.unscored_resolution_reason for question in title_matched_questions
+    }
+    if None in reasons:
+        return (
+            "Matching question titles found, but all resolutions are blank "
+            f"(annulled or deleted): {urls}"
+        )
+    if reasons == {"annulled"}:
+        return f"Matching question titles found, but all are annulled: {urls}"
+    if reasons == {"ambiguous"}:
+        return f"Matching question titles found, but all are ambiguous: {urls}"
+    if reasons == {"blank"}:
+        return (
+            "Matching question titles found, but all have blank resolutions "
+            f"(deleted or unresolved): {urls}"
+        )
+    reason_counts = []
+    for reason in sorted(reasons):
+        count = sum(
+            1
+            for question in title_matched_questions
+            if question.unscored_resolution_reason == reason
+        )
+        reason_counts.append(f"{count} {reason}")
+    return (
+        "Matching question titles found, but all resolutions are blank "
+        f"(annulled or deleted): {urls} ({', '.join(reason_counts)})"
+    )
+
+
 def title_matched_questions_are_problematic(
     title_matched_questions: list[Question], log_results: bool
 ) -> bool:
@@ -148,7 +211,7 @@ def title_matched_questions_are_problematic(
 
     log_comparison_table = False
     if len(non_annulled_questions) == 0:
-        info_message = f"Matching question titles found, but all are annulled: {[q.url for q in title_matched_questions]}"
+        info_message = _all_unscored_title_match_message(title_matched_questions)
     elif len(non_annulled_questions) == 1:
         unique_projects = list(set([q.project for q in title_matched_questions]))
         if len(unique_projects) > 1: