Hey all, so I am running into a problem. I am training massive ML models which take literally a day to fully train.
We want to run HPO to make it so that we can get the best parameters for the model and we require very high accuracy for the task so we need the HPO step.
Because the model takes a day to fully train, we reduced the number of epochs for the HPO part to take around 1 to 2 hours for each hPo trial.
With pruning we can get to under 30 minutes per. Now the thing is that we want to get these models and HPO trained about twice a month so I can’t be doing full training runs on the HPO and also we have 5 different models that we need to train and keep up to date.
We also change model architecture periodically so we need to do fresh hPo runs on those.
The main issue I am running into is that by reducing the HPO epochs below what is used for the full training runs, I fear my learning rate scheduler and other HPO params may be poorly optimized for a full training run.
How do you manage these massive training runs with HPO and ensure no parameter drift when needing to do a full training run vs small HPO run.
Also last question is does pruning reward model for converging fast and punish models that may converge closer to truth but slower. Because we prune with median pruner and I’m finding most models converge fast but don’t learn anything past a certain point.
I’m considering to restart my LR scheduler from the start after it stops learning and then this may help fix LR problem. Similar to early stopping but to start LR back up again when this happens. What do you think??
[link] [comments]
