BayesPE integration#409
Conversation
ArtemVazh
left a comment
ArtemVazh
left a comment
There was a problem hiding this comment.
Please rebase the branch onto the latest main. Additionally, please add these methods to the README and the default_estimators config.
| ) | ||
|
|
||
| elif model_type == "Whitebox": | ||
| from lm_polygraph.stat_calculators.ensemble_probs import EnsembleProbsCalculator |
There was a problem hiding this comment.
EnsembleProbsCalculator should be specified in the __init__ of lm_polygraph.stat_calculators and removed from here.
| get_linear_schedule_with_warmup, | ||
| ) | ||
|
|
||
| try: # Transformers >=4.46 removes AdamW from top-level |
There was a problem hiding this comment.
This issue has already been resolved in main. from torch.optim import AdamW should work with the package versions in requirements.
| return (["ensemble_probs"], ["input_texts"]) | ||
|
|
||
|
|
||
| def load_stat_calculator(config, environment): |
There was a problem hiding this comment.
It should be placed in a separate file, as is done for all other calculators here: lm_polygraph/defaults/stat_calculator_builders
| return result_dict | ||
|
|
||
|
|
||
| def load_stat_calculator(config, environment): |
There was a problem hiding this comment.
Is this needed here? It doesn’t seem to be used.
| ) | ||
| if not ( | ||
| str(s).startswith("blackbox_") and s[len("blackbox_") :] in have_stats | ||
| ) # remove blackbox_X from stats only if X is already in stats to remove duplicated run of stat calculator |
There was a problem hiding this comment.
It’s unclear why this modification is necessary.
| from lm_polygraph.utils.model import WhiteboxModel | ||
|
|
||
|
|
||
| def test_bayespe_zero_shot_end_to_end(): |
There was a problem hiding this comment.
This test should be moved to test/local.
| estimator = Focus( | ||
| model_name=model_name, | ||
| path="../token_idf/{model_name}/token_idf.pkl", | ||
| path=f"../focus_data/{model_name}/token_idf.pkl", |
There was a problem hiding this comment.
It’s unclear why this modification is necessary.
| assert isinstance(ue.uncertainty, float) | ||
|
|
||
|
|
||
| def test_bayespe_zero_shot(): |
There was a problem hiding this comment.
It would be better to refactor the test_bayespe_zero_shot and test_bayespe_few_shot tests to use the standard estimate_uncertainty function, while keeping these tests as additional ones.
|
@rvz16 Hi! Thank you for addressing most of the requests. However, could you please clarify a few remaining modifications?
Could you also fix the lint issues, please? |
…condition in manager.py and fixed lint issue
2 participants
Add implementation of Bayesian Prompt Ensembles: Model Uncertainty Estimation for Black-Box Large Language Models (https://aclanthology.org/2024.findings-acl.728.pdf)