It's not easy to troubleshoot such problems without being a gamer yourself. Are the problems game-specific? Do they occur all the time or only sometimes? Maybe someone is using up all the bandwidth at home at the time they are playing. It could very well be beyond your control, ISP or routing problems etc.
If you want to rule out configuration problems, connect a computer to that 2nd router and run WinMTR with a close-by, reliable server (e.g. google). That way you can check that there's no packet loss or jitter.
I'm not a console gamer, so I'm not sure if this is necessary, but it seems like Xbox Live requires you to forward ports to play online. http://support.xbox.com/en-US/xbox-one/networking/network-ports-used-xbox-live
In addition to these generic ports, there may be game-specific ports that need forwarding. Note that because you have two routers, you will need to forward twice; your internet-facing router should forward to the "second router" and from that, to the Xbox One.