Skip to content

feat(ocl): implement an event system for evaluation#356

Open
tachyonicClock wants to merge 1 commit into
adaptive-machine-learning:mainfrom
tachyonicClock:events
Open

feat(ocl): implement an event system for evaluation#356
tachyonicClock wants to merge 1 commit into
adaptive-machine-learning:mainfrom
tachyonicClock:events

Conversation

@tachyonicClock

@tachyonicClock tachyonicClock commented May 4, 2026

Copy link
Copy Markdown
Collaborator

An event/hook/plugin system similar (but more dynamic) to PyTorch lightning's Callback class or Avalanche's plugins. I needed a way to get the predictions of a learner from a replay buffer without having to modify the replay buffers owner. With the new system the replay buffer subscribes to TrainBatchPredict. Another use-case is when we have nested learners OuterLearner(InnerLearner) and the inner learner needs to know about tasks TestTaskBegin. At the moment we need to proxy the signals .on_train_task function calls through OuterLearner to the InnerLearner. With the new system we subscribe the inner learner first and then pass it into the outer learner. The system will also allow for custom evaluators and loggers.

A big chunk of the changes is spitting up the evaluation module into smaller files

@tachyonicClock

Copy link
Copy Markdown
Collaborator Author
image

@tachyonicClock tachyonicClock force-pushed the events branch 4 times, most recently from dffed48 to bcd4c9a Compare May 4, 2026 04:32
@tachyonicClock tachyonicClock force-pushed the events branch 4 times, most recently from f8e5761 to c8c6205 Compare July 9, 2026 04:42
@tachyonicClock tachyonicClock requested a review from Copilot July 9, 2026 04:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces an event/handler system for OCL evaluation, enabling decoupled hooks (callbacks/plugins) during training/evaluation, and refactors the prior monolithic evaluation module into a package of smaller components.

Changes:

  • Add a generic Dispatcher/Event/Handler system and define OCL evaluation loop events.
  • Rewrite ocl_train_eval_loop to be event-driven and move evaluation code into capymoa.ocl.evaluation submodules.
  • Update several OCL strategies and documentation/tutorial materials to use the new handler-based mechanism.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
tests/ocl/test_strategy.py Minor formatting update (blank line) after evaluation import.
src/capymoa/ocl/strategy/l2p.py Switch L2P to the new Handler model (subscribes to task-boundary events).
src/capymoa/ocl/strategy/_rar.py Removes old task-aware proxy methods (now needs event forwarding via Handler).
src/capymoa/ocl/strategy/_gdumb.py Switch GDumb to Handler to react to TestTaskBegin.
src/capymoa/ocl/strategy/_experience_replay.py Switch ExperienceReplay to Handler and forward attach_with to wrapped learners.
src/capymoa/ocl/strategy/_ewc.py Switch EWC to Handler and consume TrainTaskBegin/TestTaskBegin events.
src/capymoa/ocl/events.py New core event dispatcher + handler interface for the plugin system.
src/capymoa/ocl/evaluation/events.py New event definitions for the OCL evaluation lifecycle.
src/capymoa/ocl/evaluation/_metrics.py New OCL metrics data structures and helper functions extracted from legacy module.
src/capymoa/ocl/evaluation/_metrics_handler.py New default metrics sink that reproduces legacy metric collection via events.
src/capymoa/ocl/evaluation/_loop.py New event-driven OCL train/eval loop (supports optional custom dispatcher).
src/capymoa/ocl/evaluation/_evaluator.py New evaluator/collector that subscribes to eval events to build OCL metrics.
src/capymoa/ocl/evaluation/init.py New package exports for ocl_train_eval_loop, OCLMetrics, and events.
src/capymoa/ocl/evaluation.py Deleted legacy monolithic evaluation implementation (migrated to package).
src/capymoa/ocl/base.py Deleted legacy TrainTaskAware/TestTaskAware interfaces (replaced by events).
src/capymoa/ocl/init.py Update OCL package exports to include the new events module.
notebooks/ocl_event_system.ipynb New tutorial notebook demonstrating event usage in isolation and in the OCL loop.
docs/tutorials.rst Add the new notebook to the tutorials toctree.
docs/contributing/faq.md Update contributor guidance to reference the new Handler event subscription model.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/capymoa/ocl/events.py
Comment thread src/capymoa/ocl/evaluation/_loop.py Outdated
Comment thread src/capymoa/ocl/strategy/l2p.py Outdated
Comment thread src/capymoa/ocl/strategy/_experience_replay.py Outdated
Comment thread src/capymoa/ocl/strategy/_gdumb.py Outdated
Comment thread src/capymoa/ocl/strategy/_ewc.py Outdated
Comment on lines 1 to 11
import torch
from torch import Tensor

from capymoa.base import BatchClassifier
from capymoa.ocl.util._replay import ReservoirSampler
from capymoa.ocl.base import TrainTaskAware, TestTaskAware

from typing import Callable


class RAR(BatchClassifier, TrainTaskAware, TestTaskAware):
class RAR(BatchClassifier):
"""Repeated Augmented Rehearsal.
Comment on lines 120 to 123
@torch.no_grad()
def batch_predict_proba(self, x: Tensor) -> Tensor:
x = x.to(self.learner.device, self.learner.x_dtype)
return self.learner.batch_predict_proba(x)
Comment thread notebooks/ocl_event_system.ipynb Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants