I like the way you have expressed your GPU behaviour it is very well written.
🙂What I think might be happening is that during these periods of idle your GPU fan is in fact powering off completly. And while that is ok for a short period of time, the heat will build (reasons why are many and the main one is air flow pattern in case) once a certain monitoring point is reached the fan kicks in and brings the temps down.
What you can do to check what is going on is download GPUz the latest version. Now open the monitoring tab, select continues monitoring (or background depending which version you use) and also log to file.
Now leave the monitor application open. And let your system just sit, and go watch a movie or some TV. The system will revert to idle and in the back ground the GPUZ monitoring data will be recorded. Once you hear the fan power up let it be for a while just to see what the out come is. Then go into the location where you log is kept and inspect.
Will show you what is really happening.
Possible fix is to have a set value running for your fan at all times and not a variable program.
To help us help you - please provide as much information about your system and the problem as possible.