scaler.scale(loss).backward() # scales loss, then backprop scaler.step(optimizer) # unscales gradients, updates weights scaler.update() # adjusts scale for next iteration
Loss scaling is a simple, elegant fix: