Topological divergence#363
Conversation
Merge latest changes from main into topological_divergence
Add ripser and joblib in requirements.txt
…d register it in default calculators
|
Please specify the library versions in requirements.txt |
| fastchat | ||
| diskcache>=5.6.3 | ||
| joblib | ||
| ripser |
There was a problem hiding this comment.
please specify the library versions in requirements.txt
| "model_heads_cache": "model_heads_cache.yaml", | ||
| "max_heads": 6, | ||
| "n_jobs": -1, | ||
| "dataset": ["LM-polygraph/coqa", "continuation"], |
There was a problem hiding this comment.
Please specify the dataset parameters in configs
| "seed": [1], | ||
| "size": None, | ||
| }, | ||
| ) |
There was a problem hiding this comment.
Please move all these parameters into the configs and also specify this calculator there. Additionally, it might be much better to integrate the initialization of this calculator with TrainingStatisticExtractionCalculator to avoid multiple loadings of the training dataset.
For reference, we combine TrainingStatisticExtractionCalculator with other calculators for supervised methods in another PR. I think this case should be handled in a similar way.
| @@ -0,0 +1,63 @@ | |||
| import os | |||
There was a problem hiding this comment.
To avoid multiple loadings of the training dataset, it might be much better to integrate the initialization of this calculator with TrainingStatisticExtractionCalculator
| labels = [] | ||
|
|
||
| greedy_calc = GreedyProbsCalculator(False, False) | ||
| attn_forward_pass_calc = AttentionForwardPassCalculator() |
There was a problem hiding this comment.
This should be specified in __init__
| mtopdivs = np.concatenate(mtopdivs, axis=0) | ||
|
|
||
| labels = np.concatenate(labels) | ||
| labels = ~(labels > 0.3) |
There was a problem hiding this comment.
This hyperparameter should be specified in __init__ and in configs
| self.x = [self.x[i] for i in indices] | ||
| self.y = [self.y[i] for i in indices] | ||
| self.x = [self.x[int(i)] for i in indices] | ||
| self.y = [self.y[int(i)] for i in indices] |
| heads = np.unravel_index(heads, (num_layers, num_heads)) | ||
| heads = np.stack(heads, axis=1) | ||
|
|
||
| save_model_heads(self.cache_path, name_or_path, heads.tolist()) |
There was a problem hiding this comment.
It should be configurable, since saving all statistics is not always required
| n_jobs=-1, | ||
| ) | ||
| ue = estimate_uncertainty(model, estimator, INPUT) | ||
| get_reusable_executor().shutdown(wait=True) |
There was a problem hiding this comment.
The usage of get_reusable_executor is unclear
| def test_topological_divergence_select_heads(model): | ||
| estimator = TopologicalDivergence(n_jobs=-1) | ||
| ue = estimate_uncertainty(model, estimator, INPUT) | ||
| get_reusable_executor().shutdown(wait=True) |
There was a problem hiding this comment.
Can you clarify how this would work without a training dataset?
3 participants
Add TopologicalDivergence estimator based on the method proposed in arXiv:2504.10063