Of your 8-bit character set, ASCII technically includes only the low-order 7 bits (0-127); there are numerous extension character sets that define codepoints 128-255 differently. However, ASCII has a number of non-printable characters in codepoints 0-31; these include things like the bell, field separator, row separator, null byte, etc that will do very strange things to your terminal.

The whole point of base-64 is to create a printable representation of binary data, so if it uses non-printable characters, it's failed in its job. Realistically this creates a range of codepoints 32-126 to work with, but it's much easier if it's an integer number of bits, and the largest power of 2 within those numbers is base-64.