If I had to pick one player that met all of your specs (and that I have personal experience with) it would be the Sansa Clip or Clip+ available at Amazon and other retailers. A new Clip with 4GB capacity should run you less than $40.
Some points to keep in mind are the compromises you'll be making.
If you just want to "Copy and Paste" music files into it, larger capacity drives will be hard to manage. Four GB of music is a lot of songs to try to keep track of. You can do it, but some software (a lot of it free) will make it a whole lot easier.
Voice recording on an MP3 player -- at least the ones I'm familiar with -- is pretty sketchy. If you're thinking of recording lectures, think about getting a dedicated recorder. I've never seen an MP3 player with a microphone jack, so you're further limited in not being able to use a directional mic. (I use a Sony TCM-200 standard cassette recorder for meetings -- it's not very impressive, but it works great.) I don't have any experience with digital voice recorders, but that might be the way to go if you need to move voice recordings via email, etc., but they'll be more expensive than the cassette.
If you really want to go with "Copy and Paste" and decide to stick with a smaller (1 or 2 GB) player, it's a commodity market. Places like Woot and Newegg offer refurb Sansa Clips in the $20 range pretty frequently (it's how I got mine). You can also find similar models from companies like Creative (such as the Creative Zen V+) in the same price range. I also own a couple "no name" players that use removable, rechargeable NiMH AAA batteries -- it can be convenient to keep an extra charged battery or two in your pocket in case the player dies mid-afternoon, and it's easier to recharge a AAA battery in places where USB ports are scarce. In this case, check out the cheap players at Newegg and Tiger Direct, as well as any big retailer.
And don't forget to budget for a decent
pair of headphones -- sooner or later you're going to decide you need them!
EDIT: Sorry -- I just noticed you're from "Down Under." I don't know what the retailers or prices in Oz are like, but I assume Creative is available, and I'd hope Sansa is. Good luck, and happy hunting!
Edited by Capn Easy, 30 June 2010 - 03:41 PM.