That's how I understand it too, binary should work for any thing. It sends bytes data in it's 8-bit format.
FTP was developed at a time when typical speeds were very low (at 110 to 300 bits per second). Since ASCII only uses 7 bits, long TEXT files could be transmitted more quickly by NOT sending all the unused bits. But now it doesn't matter at today's speeds.
So as far as I know, ASCII mode sends 7bits per byte neglecting the 8th bit. This works for simple ASCII text files. But for any other files (ie. .doc .pdf .exe etc) it won't work as you mentioned. Besides, ASCII mode does some character formatting when sending between different platforms. For example, a Windows machine receiving a file from a Unix machine will replace the line feeds with carriage return-line feed pairs.
You might as well keep it on AUTO mode, the FTP client will know what mode to use for each file.