Server distributions are set up (or should be set up) so only the needed services are running. Yes, you could manually enable the gui, and then manually disable the gui, but then you are constantly editing config files. The temptation after a while would be to just leave it running. Altering a server distro in any way is just a bad idea, unless you know linux inside and out, and then know the distro inside and out. Gnome is not just a stand-alone application, there are other applications attached to it. Each of those applications has a cascade effect of adding files, changing permissions, altering configuration files, etc.
Once a server is set up, there should not be that much to do. Ssh, cp, mv, less, nano (or vi), and maybe a handful of other commands are all you need. Again, do what you want: I am just of the opinion that since I don't know enough about how a distro is put together, then I can't possibly keep it secure. If I can't keep it secure, then there isn't much sense in me running it in the first place. I am also of the opinion that tools are meant to be used in the way which they were intended, so if I am going to use those tools, then I need to learn how to use them properly, otherwise there is no sense in me using them. After I understand how things work and their capabilities, then I can alter them.
If you merely want a server to host some web pages for friends, then it isn't such a big deal to re-image the server when it gets hacked. A real production server is a different problem because money is on the line when the server is down.
"Take the risk of thinking for yourself, much more happiness, truth, beauty, and wisdo
m will come to you that way" - Christopher Hitchens