No, it will not slow down your PC when you are not using it.
As you stated correctly, it will obviously take away some system resources when you do run it.
Be glad you can use it. My processor doesn't have HAV, so I can't run XP Mode. I have to do a regular XP virtual machine.
Just leave the setting in your BIOS enabled, then just use XP Mode when you need it. You should see no degradation in performance during normal usage (not running XP Mode).