The guy works out of his home along with his father.
This is not a bar to being an organised business. A friend and neighbour of mine ran a machinery agency until he retired selling equipment valued at up to £150,000 from his house. The critical distinction here is whether you bought the computer from a 'business' or from a 'private individual'. In general, legal protection is of a much lower standard for 'private' deals - where 'buyer beware' rules - than for deals with a business.
I am not a lawyer, and my familiarity is with consumer protection law in the UK and the EU, so I cannot advise on the detail of Georgia's business regulations, but I would have thought that replacing the tower with another HP one would be merely an administrative problem for him. He sends the faulty tower back to HP or whichever intermediary he got it from, and they send him a replacement which he passes on to you. The buck should eventually finish up with HP. Your other points :
"Should I expect him to repair it with all costs to him" - YES
"Should I be the one paying for them?" - NO
"I am not saying that you did not cause whatever happen to happen but I do not think you did."
This bit is hilarious. If you had thrown it out of an upstairs window, or spilt a bowl of soup all over the interior then nobody would expect that to be covered by any normal warranty, but you didn't. You just used it as a computer.
If this guy is trading as a business, and that is fairly easy to establish - adverts in the local press or on-line, mail shots, flyers and so on - then you have the full rights granted you by the State of Georgia and any applicable Federal regulations as well. Look them up in your local library, speak to Citizens Advice and to whatever organisation looks after trading standards in Georgia - here it would be my local County Council, and settle for nothing less than either a repair or replacement at no cost to you OR every cent you paid back.
Taking his offer of an Apple has other implications - the software you use in its Windows format will not natively work on an Apple OS. It is possible to install a Windows 'simulator' on an Apple machine but this is an extra cost and inconvenience to you.
If it turns out he can claim to be a 'private' individual, then you may have to settle for what you can get, but push as hard as you can.
Am I right in thinking that you have some form of small claims court in the US, I think we got the idea from you ? Here you can bring actions for small amounts very cheaply, you don't even have to have a lawyer. In the final resort, I would think about this line of action.
As you can gather, this one has got me annoyed ! You are being ripped off, and I don't like it. I can't give practical assistance at 3000 miles or so, but if you do finish up having to bite the bullet on this one - and you shouldn't - one thing is for sure. Never buy another computer from this guy, and tell your friends that ! I have sourced quite a few computers for friends over the years, but I have never supplied them myself - I take them to my local computer and components store who I have been dealing with for over twenty years.
Let us know how you get on.