This is a problem I noticed has been around for a while now but if you switch from ASCII to Binary (or the other way around) you actually have to restart FlashFXP for it to take effect but it does not tell you that you need to restart, I just found out by realising it was uploading my files in the wrong format.
To clarify a bit more, The setting in the preference is the default value that is set on startup, changing it has no effect on the current session. Change it via the session menu or the toolbar button if you have it activated.