I know Mac users can get Malware if they install something without checking it out, but is it possible for web browsers to install malware on a mac without user permission or knowlege?
If the correct sequence of vulnerabilities are in place, then yes. Drive by downloads are not impossible simply by virtue of running OSX.
Traditionally, most malware is ineffective against OSX. This is because of one fact, and one fact only: most malware is not written to attack OSX
OSX users still occupy a tiny market share of the OS market, with Windows having a much larger share. Therefore, it is in the best interest of malware writers to design their infections and attacks so that they can attack the greatest number of people.