Actually, I have noticed a secondary problem here. Whenever I regenerate my audio because I had made small changes to the content and if the audio is longer than the previous text to speech audio, then the slide duration is adjusted to the exact length of the new audio. So every time I regenerate the text to speech audio, I need to skim through the presentation to check the timeline for all the slides and make sure none of the slides' duration span to the exact duration of the audio file.
This is a bit annoying though.