It's the Mellanox that's taking half your GPU lanes. There's only one x16 bus on the CPU, so it's either x16/0 or x8/x8 for the 2 PCIEX16 slots. M.2_2 and M.2_3 also share the CPU's single x16, similarly reducing the top slot to x8. M.2_1 uses the separate x4 storage lanes from the CPU. DIMM.2 and PCIEX1 are through the chipset. See page xiii of the manual for the shared lanes and possible configurations.
You can happily use PCIEX16_1, M.2_1, and both DIMM.2 slots simultaneously; while getting the full x16 to the GPU.