Visioneer keeps the following available for automatic PaperPort software and scanner driver removal:
http://support.visioneer.com/utilities/ScannerSW_Uninstall.exe
Brother keeps driver removers on its web site for all of its scanning equipment, but the above is sometimes necessary in addition, to clean up a situation.