Reproducible results

[3505675604]

• I have recently attempted to reproduce the results reported in your paper, using the provided codebase. While the training and evaluation process runs smoothly, I noticed that the performance metrics I obtained (e.g., [specify metric, such as RMSE, AUC]) are consistently lower than those reported in the paper.
  {'task_name': 'bbbp', 'batch_size': 32, 'seed': 8, 'epochs': 100, 'patience': 10, 'userconfig': {'mpp': {'dataset_dir': './data/mpp/pkl', 'split_dir': './data/mpp/split/'}}, 'mode': 'finetune', 'split_type': 'scaffold', 'save_ckpt': 50, 'save_model': 'best_valid', 'pretrain_model_path': 'None', 'checkpoint': './weights/pretrain/pretrain.pth', 'DP': False, 'optim': {'type': 'adam', 'init_lr': 5e-05, 'init_base_lr': 0.0001, 'weight_decay': 0.0001, 'momentum': 0}, 'lr_scheduler': {'type': 'None', 'warm_up_epoch': 10, 'start_lr': 1e-05}, 'model': {'atom_embed_dim': 512, 'num_kernel': 128, 'layer_num': 6, 'num_heads': 16, 'hidden_size': 256, 'num_tasks': 'None', 'bond_embed_dim': 512}, 'root': './data/mpp/pkl', 'dataset_form': 'pkl', 'dropout': 0.1, 'dataloader_num_workers': 4, 'target': ['p_np'], 'task': 'classification', 'loss_type': 'bce', 'num_tasks': 1, 'fg_num_': 191, 'freeze_layers': 0, 'loss_style': ''} epoch:1 train_loss:0.7493732746909646 valid_loss:0.8314943867070335 test_loss:1.1792062180382865 train_metric:0.756949245929718 valid_metric:0.9647709727287292 test_metric:0.7031505703926086 epoch:2 train_loss:0.6238408895099864 valid_loss:0.755109578371048 test_loss:1.2924151590892248 train_metric:0.9079096913337708 valid_metric:0.9698175191879272 test_metric:0.7081606984138489 epoch:3 train_loss:0.5917834359056809 valid_loss:0.8472606582301003 test_loss:1.6364021812166487 train_metric:0.9389441609382629 valid_metric:0.9667119383811951 test_metric:0.7245399355888367 epoch:4 train_loss:0.6209826919378019 valid_loss:0.9208931922912598 test_loss:1.7931351491383143 train_metric:0.9587492346763611 valid_metric:0.966323733329773 test_metric:0.7291646599769592 epoch:5 train_loss:0.6613108515739441 valid_loss:1.0605963127953666 test_loss:2.1001366887773787 train_metric:0.9707706570625305 valid_metric:0.9732142686843872 test_metric:0.7351382374763489 epoch:6 train_loss:0.6578038229661829 valid_loss:1.10717739377703 test_loss:2.195648499897548 train_metric:0.9772943258285522 valid_metric:0.9685559272766113 test_metric:0.7432315349578857 epoch:7 train_loss:0.6512252413759044 valid_loss:1.222784468105861 test_loss:2.3698778663362776 train_metric:0.9844943284988403 valid_metric:0.9599184989929199 test_metric:0.7290683388710022 epoch:8 train_loss:0.6350641490197649 valid_loss:1.3815491114343916 test_loss:2.8549907888684953 train_metric:0.9889788627624512 valid_metric:0.961568295955658 test_metric:0.7203969359397888 epoch:9 train_loss:0.6052223595918393 valid_loss:1.4038536974361964 test_loss:2.7316887378692627 train_metric:0.9919977188110352 valid_metric:0.9516692757606506 test_metric:0.7225165963172913 epoch:10 train_loss:0.6034321890157812 valid_loss:1.3945110099656242 test_loss:2.775458608354841 train_metric:0.993455708026886 valid_metric:0.9546778202056885 test_metric:0.6967915892601013 epoch:11 train_loss:0.6054806084025139 valid_loss:1.4544918962887354 test_loss:3.0051942552839006 train_metric:0.9944766163825989 valid_metric:0.9496312141418457 test_metric:0.6921668648719788 epoch:12 train_loss:0.6050250249750474 valid_loss:1.663840115070343 test_loss:3.2077205181121826 train_metric:0.9956645965576172 valid_metric:0.9482725262641907 test_metric:0.7124000191688538 epoch:13 train_loss:0.5701212345385084 valid_loss:1.6229955639157976 test_loss:3.636416162763323 train_metric:0.9958754181861877 valid_metric:0.961277186870575 test_metric:0.711821973323822 epoch:14 train_loss:0.568649504698959 valid_loss:1.782990711075919 test_loss:3.5622428144727434 train_metric:0.9960888624191284 valid_metric:0.9530279636383057 test_metric:0.703921377658844 best_val_metric:0.9732142686843872 best_test_metric:0.7432315349578857 true_test_metric:0.7351382374763489

[KazeDog]

Hello, the reproduction results may fluctuate due to differences in different experimental environments. For bbbp, your run resulted in 0.735 and our reported value is 0.734, which is within a reasonable margin of error. Are you confusing the two divisions, scaffold and random scaffold?

Please do not hesitate to contact me if you have any other questions.

[3505675604]

@KazeDog I have a question. Is it normal that the results of the following dataset using different seeds are very different from those mentioned in the paper? The dataset name is clintox.
seed=10 best_val_metric:0.984024703502655 best_test_metric:0.8827121257781982 true_test_metric:0.8539950847625732 seed=11 best_val_metric:0.9894919395446777 best_test_metric:0.9059865474700928 true_test_metric:0.8528854846954346 seed=14 best_val_metric:0.9875458478927612 best_test_metric:0.9067111611366272 true_test_metric:0.8557354211807251 seed=32 best_val_metric:0.9905604124069214 best_test_metric:0.8904702663421631 true_test_metric:0.868484377861023 seed=35 best_val_metric:0.9911472201347351 best_test_metric:0.9083098769187927 true_test_metric:0.8416310548782349 seed=5 best_val_metric:0.9890655875205994 best_test_metric:0.9345167875289917 true_test_metric:0.8629783391952515

[mxqmxqmxq]

@KazeDog I have a question. Is it normal that the results of the following dataset using different seeds are very different from those mentioned in the paper? The dataset name is clintox. seed=10 best_val_metric:0.984024703502655 best_test_metric:0.8827121257781982 true_test_metric:0.8539950847625732 seed=11 best_val_metric:0.9894919395446777 best_test_metric:0.9059865474700928 true_test_metric:0.8528854846954346 seed=14 best_val_metric:0.9875458478927612 best_test_metric:0.9067111611366272 true_test_metric:0.8557354211807251 seed=32 best_val_metric:0.9905604124069214 best_test_metric:0.8904702663421631 true_test_metric:0.868484377861023 seed=35 best_val_metric:0.9911472201347351 best_test_metric:0.9083098769187927 true_test_metric:0.8416310548782349 seed=5 best_val_metric:0.9890655875205994 best_test_metric:0.9345167875289917 true_test_metric:0.8629783391952515

I’ve encountered the same issue. Could the author kindly share the fine-tuning hyperparameters, if available?

[KazeDog]

Hello, the results given in the article are hyper-parametrically tuned. You can tune the parameters as described in the supplementary information.

[mxqmxqmxq]

Hello, the results given in the article are hyper-parametrically tuned. You can tune the parameters as described in the supplementary information.

Thank you very much, do you have the best hyperparameters related to record?

[kingwzun]

Hello, the results given in the article are hyper-parametrically tuned. You can tune the parameters as described in the supplementary information.

I have the same query, can you give the fine tuning parameters for the rest of the dataset, as dealing with BBBP and lipophilicity was able to debug it, and the rest of the dataset debugged also differed greatly from the reported results.

[mxqmxqmxq]

Hello, the results given in the article are hyper-parametrically tuned. You can tune the parameters as described in the supplementary information.

I have the same query, can you give the fine tuning parameters for the rest of the dataset, as dealing with BBBP and lipophilicity was able to debug it, and the rest of the dataset debugged also differed greatly from the reported results.

My advice is to make the batch_size a little smaller and try more other seeds to get the results you want.

[kingwzun]

Hello, the results given in the article are hyper-parametrically tuned. You can tune the parameters as described in the supplementary information.

I have the same query, can you give the fine tuning parameters for the rest of the dataset, as dealing with BBBP and lipophilicity was able to debug it, and the rest of the dataset debugged also differed greatly from the reported results.

My advice is to make the batch_size a little smaller and try more other seeds to get the results you want.

If possible, could you share the parameters you used for the different datasets?