This error occurs when windows runs out of memory.
Looking at your debug.log though I have to wonder about the integrity of your copy of FlashFXP. Your debug.log does not look correct. I highly recommend you re-download your copy of FlashFXP from our website
https://oss.azurewebsites.net/rc.php
Using a modified/tampered copy of FlashFXP can cause all sorts of unpredictable results.