Hi Wyndoor
have you tried MSI Afterburn? try to upper the Powerlimit alittle and try Heaven benchmark to check if it is stable.
All comes down to the silicon lottery my friend. If you can't get it stable on normal boost with higher power limit, you got a lemon.
1080 is a really bad overclocker. but to be honest , overclock isn't really necessary when it comes down to the performance it already can deliver.
My Gigabyte 1080 FE on stock boost clocks to 1968 core with everything on stock. that is decent , but to be honest i could'n care less. in reality we are talking about max gain of 5-15fps with a good/high overclock, and that is on 1080p / 1440p. 4K on the otherhand you get minimal gains like 1-5fps.
If you're not happy with your 1080 strix , try to replace it, if not just be happy with it and enjoy. 1080 is a monstercard , and plays the most games maxed out on 2k with much over 60fps.
/Sam