Can you tell us something about your network? Server type how many computers?Do you have an own webserver?
He most likely has something installed like VPN ( Virtual Private Network) on his computer and your server.
The best way to do this is to switch of internet connection when you leave your office. This will disable access from outside.
Most importantly a "heart to heart "would be a better solution for the future. If you haven't given him permission that is crime...