sp4rk2, to Bleeping Computer Forums!
Thanks for choosing us to assist with your issue, which looks to me to be an OS issue or setting, rather than hardware one, based on your one statement above.
- Booted into a live Ubuntu 17.10 installation to check if all 3 monitors worked there. (They do)
While I haven't got into multi monitor setups yet, have the hardware (EVGA GTX 1070 & two 1060's), intend on doing so at some point. I do find it odd that Ubuntu 17.10 (although in Live Mode) can get the job done, yet can't when running Windows 10. I've found W10 to be more 'over my head' than some has, am fortunate just to have a single 4K monitor running on my main PC, the one with GTX 1070.
There has to be some adjustment in PC Settings (not Control Panel) that can help. The closest one I found was 'Rearranging multiple displays' (type it in the search box).
May not be the case, as I also have a PC with the same CPU as yours (FX-8350), while a powerful one, there's a chance that it could be the bottleneck of your obviously more powerful GTX 1080. Note that I stated a chance, not a certainty. Do you have any other computer to try this out on to rule out the CPU as the culprit? Many modern notebooks can push dual displays (actually three if the onboard is included).
One other thing, is the one Displayport cable itself, must be 3 meters (roughly 6.6 feet) or less & for best quality must have the DP logo (or DP++) on the cable ends. Knockoff cables can produce unpredictable results, I learned this lesson long ago with my first monitor with a DP connection, ordered a 10 feet cable of the Nippon brand from Newegg in late 2013. Monitor wasn't up to what other customers rankings, then read the reviews again more carefully & one recommended to use an Accell brand cable (can't find the model now, probably because it was rated for DP 1.2). That's one of the important basics of DP, making sure not to purchase cables with the logo, to be honest, found it quite surprising when my 4K Samsung UE510 arrived, the DP cable didn't have the logo, so used mine instead, by then had purchased another on eBay. Here's the Accell site that shows some multi monitor accessories.
Finally, the adapter to convert DP to HDMI, should be an active one, especially with two monitors with long cables, or have an active DP to HDMI cable for each. I now use the higher speed HDMI cables (the full 6Gbps/channel or total of 18Gbps) rather than the standard ones of around 11 Gbps for everything, other than testbeds. These are available at Monoprice at a low cost, the main difference that can be seen is packaging at first (Premium rather than silver or Standard background on the package), yet there are speed increases & a couple other differences the Standard didn't have. Here's the ones I last purchased, which are rated for higher speeds over the standard version, simply choose the desired length. these are very affordable & high quality HDMI cables, more so than the Standard ones now back in the original packaging & you'll see that between 3, 6 & 10 feet cables, little price difference, 10x less than brick & mortar stores. Much better than Amazon Basics cables, I had one of those to literally come apart at the end on one monitor, used for a folding@home setup.
Displayport is indeed a superior connection over HDMI in many ways, to include faster bandwidth, yet when going with two or more monitors, everything must be right to achieve the desired result. I'm still puzzled by the fact that you're able to accomplish this with a Live Ubuntu 17.10 session & won't do the same with Windows, it has to be something minor, Windows 10 has lots more settings than previous versions (Windows 8.1 & prior). Back then, everything was handled via the Control Panel, now one must use PC Settings for some operations, to include Windows Update.
Would had been great for both monitors to have a native DP connection to simplify things, most of the GTX 10 series cards has three DP 1.4 outputs, in that case, HDMI wouldn't be a factor. Maybe something to consider in your next display purchase, both of my last new ones has one DP connection (although the first is crippled by a DP 1.1 port ), I wasn't aware of the difference then, a $300 lesson. Made sure that the 4K one had DP 1.2, plus a full bandwidth HDMI (I believe 2.0), although the second is 1.4, typical for many models released in 2015.
Keep on pounding, you'll get this thing right, one way or the other.