Within the Windows XP architecture, there are two modes--user and kernel. In the kernel mode, there lies the Executive services, and within the Executive services lies a sub-service that is called the Plug and Play Manager. This sub-service is responsible for installing, uninstalling, managing, loading, and unloading drivers for Plug and Play devices. If it is responsible for the unloading of drivers and managing I/O with the USB device, why do we need to "Safely Remove Hardware" when USB devices are plug and play AND hot swappable? I think that we have all been brainwashed to "Safely Remove Hardware."
I regularly support customers over the phone and I tell them to "Safely Remove Hardware" (company policy that I tell them to do so) and they ask what that means, which possibly indicates that they never do it, yet their USB devices continue to function.
I think that feature is there just to grant us "peace of mind." Only time I see a need for it is when you are writing/reading from the USB device and you wish to unplug it, in that case, safely removing it would be a good idea to prevent data corruption.