Are you sure you stress tested long enough?

Did you have HWinfo64 running in the background with the bottom sensors field "WHEA" section open at the same time to look for errors that Realbench won't pick up?

It's completely impossible for a *CPU* overclock to fail with a CPU/GPU overclock but NOT with a CPU only overclock. There is no relationship at all between the CPU and GPU in that fashion, unless a PSU component itself is failing in some way. I suggest using Realbench 2.56 and doing a 4 hour stress test and have HWinfo64 open with sensors only, and watching the very bottom WHEA errors section. But then you said "instability failed" (Luxmark) which is the GPU section of the test--the CPU is not related to this test at all. So setting your CPU down to 3600 mhz for example, should still trigger this same instability. That means your video card overclock is not stable.

If somehow 'downclocking' your CPU makes your video card able to overclock again ...I would look at the quality of your power supply. The 12v voltages could be dropping too low. On some higher quality PSU's, simply replacing the PCIE cable with a spare from the box (anything above 550W should have more than 2 PCIE cables) will fix a GPU +12v problem. Otherwise you just need to upgrade your PSU if it's worn out or low quality.

HWinfo64 should allow you to monitor the main +12v line. Some motherboards also have a VRM section with the EPS12v input line separately from the primary 24 pin 12v reading. And some video cards also have a VRM that will show the 12v going into the PCIE power connectors. If you have one of those, you're lucky. Use it.

Also disabling XMP And suddenly being stable points to a problem with the cache ratio being too high, as XMP and cache frequency are related to the IMC. You can try adjusting your VCCIO and VCCSA voltages (usually 3200-3600 mhz memory will work on 1.20 IO and 1.25v SA without any drawbacks).