Two other ways to do it, the first is the way I usually do it.
(1) At the right hand end of the taskbar - it may be in 'hidden icons' behind the upside down chevrons - you will find a button which, in Win 10, looks like a white cyclinder with a blob on top of it. Clicking on this will bring up the 'Safely remove hardware' box. Click on the item you want to remove and a second or two later you will get a message. It will be either 'May be safely removed' or 'This device is still in use and cannot be removed'. In the latter case try and find out what is keeping it 'in use'.
Which, if I do it at all, is always how I do it since the devices connected, and only those, are enumerated upon right click. There's nothing else to confuse the issue.
There is a reason for this procedure. If the system is still writing to the device you can scramble the contents.
And you've hit the nail on the head as to when it's necessary, and why, with the very rarest of exceptions, I never use the "Eject" function and never have.
Windows has, for decades (I can't speak to OSX), been set up not to buffer anything waiting for an "Eject" to finish writing, and for precisely the reason you note. Many people were pulling out their USB connections and losing the last "buffer's worth" of the data being written to the device.
Since this feature is now turned off, when processes finish writing to an external USB device the buffer is flushed immediately and the write completed.
For reading it's entirely irrelevant as far as data damage on the USB device goes. Of course, if you pull it while it's being read then whatever is doing the reading will, of course, lose its incoming data source.
External USB devices are, by design, supposed to be hot swappable. If you know that you are not currently writing something to a given drive (which has never been tricky for me to determine - if I'm not using any program that's been writing to the device and the activity light is either out or not blinking) they're plug/unplug at will.
Eject is, for the most part, an anachronism. If you ejected in the middle of some program using the device and the data stream was very far from complete (as in it's in the middle of writing a ton of data) you're still screwing up that result, as it yanks the storage device out from under the process in mid-stream.
Here's a screen shot of the USB devices icon that shows up in the Notification Area/System Tray or hidden in the notification overflow area accessed by the notification chevron, ^
My recollection is that it looks pretty much the same in Windows 8 and forward, and in earlier versions of Windows it looks pretty much like a tiny rendering of a thumb drive.