Most likely, yes. Your internet connection (be it DSL or otherwise) isn't like your phone line that allows you to split it with a doubler, tripler, etc.
The very purpose of a router is to perform that function, taking that single connection to the outside world and "juggling the plates" such that multiple devices can use it without problem.
A router is always hooked to the modem, when a single modem is the current setup, and the wired computers are connected to the ethernet port(s) on the router. Wireless is its own side issue that the router handles along with the wired ports.
Brian AKA Bri the Tech Guy (website in my user profile) - Windows 10 Home, 64-Bit, Version 1803, Build 17134
Memory is a crazy woman that hoards rags and throws away food.
~ Austin O'Malley