Custom command improvement/discussion - Page 2

bigstar · 01-27-2015, 03:35 PM

I've made changes to the scripting engine to address these issues and now the sample script above works as intended.

Here's a direct link to the updated flashfxp.exe https://oss.azurewebsites.net/testr/5/ffxp5.0.0.3805-r2.zip

I have tested all of the scripts I have and they all seem to work as expected with these changes, please let me know if you encounter any situations or problems with existing scripts. I have to wonder if there's any existing scripts that were designed based on the flawed ordering. I sure hope not.

DayCuts · 01-31-2015, 02:06 AM

Thanks bigstar. If I remember correctly the initial ordering issue arose with the implementation of status bar logging for command progress. Perhaps there is something with the command progress messages at the heart of things?

I forgot to mention this in my last post but I am guessing it likely traces back to the same changes. When I was playing around with things I tried using /delay to slow things down so I could visual confirm exactly when different things were being done but what I got from '/delay 5' was a delay of some very big number like 45944364133whatever seconds. Changing the 5 in my command didn't make any difference.

DayCuts · 01-31-2015, 02:08 AM

Quote:

Originally Posted by bigstar

I've made changes to the scripting engine to address these issues and now the sample script above works as intended.

Here's a direct link to the updated flashfxp.exe https://oss.azurewebsites.net/testr/5/ffxp5.0.0.3805-r2.zip

I have tested all of the scripts I have and they all seem to work as expected with these changes, please let me know if you encounter any situations or problems with existing scripts. I have to wonder if there's any existing scripts that were designed based on the flawed ordering. I sure hope not.

Cool, heading to work shortly but should have time to check it out tomorrow.

DayCuts · 02-01-2015, 03:18 AM

Testing with 3805-r3

1. Repeating Example 3 from previous posts... everything within the initial CMD seems to be performed in the correct order now, however tasks from sub command CMD2 still do not function in the expected order. Sub command raws and echos (not sure if other commands are affected or not) are still misplaced.

Code:

#CMD
/echo -- Enqueue A
/enqueue A
A
{
/cd %p%f
[CMD2]
}
/uncd

#CMD2
/echo -- Enqueue B
/enqueue B
B

#LOG
[18:15:53] [L] [CMD] -- Enqueue A
#A is correctly enqueued
[18:15:53] [L] A
[18:15:54] [L] 500 'A': Command not understood.
[18:15:54] [L] [CMD] /cd /tools/bin/TESTING
[18:15:54] [L] CWD /tools/bin/TESTING
[18:15:54] [L] 250 CWD command successful.
[18:15:54] [L] PWD
[18:15:54] [L] 257 "/tools/bin/TESTING" is current directory.
[18:15:54] [L] STAT -l
[18:15:55] [L] List Complete: 549 bytes in 0.86 seconds (0.5 KB/s)
[18:15:55] [L] [CMD] CMD2
#B is correctly enqueued
[18:15:55] [L] [CMD] /cd /tools/bin/TESTING2
[18:15:55] [L] CWD /tools/bin/TESTING2
[18:15:56] [L] 250 CWD command successful.
[18:15:56] [L] PWD
[18:15:56] [L] 257 "/tools/bin/TESTING2" is current directory.
[18:15:56] [L] STAT -l
[18:15:57] [L] List Complete: 201 bytes in 0.86 seconds (0.2 KB/s)
[18:15:57] [L] [CMD] CMD2
#B is correctly enqueued again
[18:15:57] [L] [CMD] /uncd
[18:15:57] [L] CWD /tools/bin
[18:15:57] [L] 250 CWD command successful.
[18:15:57] [L] PWD
[18:15:58] [L] 257 "/tools/bin" is current directory.
[18:15:58] [L] STAT -l
[18:15:58] [L] List Complete: 7,618 bytes in 0.87 seconds (7.4 KB/s)
#Misplaced echos and raw commands
[18:15:58] [L] [CMD] -- Enqueue B
[18:15:58] [L] B
[18:15:59] [L] 500 'B': Command not understood.
[18:15:59] [L] [CMD] -- Enqueue B
[18:15:59] [L] B
[18:15:59] [L] 500 'B': Command not understood.

Expected behavior...

Code:

/echo -- Enqueue A
/enqueue A
A
/cd /tools/bin/TESTING
/echo -- Enqueue B
/enqueue B
B
/cd /tools/bin/TESTING2
/echo -- Enqueue B
/enqueue B
B
/uncd

Previous build evaluated to...

Code:

/echo -- Enqueue A
/enqueue A
/cd /tools/bin/TESTING
/enqueue B
/cd /tools/bin/TESTING2
/enqueue B
/uncd
A
/echo -- Enqueue B
B
/echo -- Enqueue B
B

This build evaluated to...

Code:

/echo -- Enqueue A
/enqueue A
A
/cd /tools/bin/TESTING
/enqueue B
/cd /tools/bin/TESTING2
/enqueue B
/uncd
/echo -- Enqueue B
B
/echo -- Enqueue B
B

2. /delay does not function, seems to ignore the numeric parameter completely. It always starts a countdown of ~4294968 seconds (almost 50 days).

3. While doing a few tests I have noticed that command progress status bar updates are bugged somehow. In the above 'Example 3' the X value in X of Y is probably mostly incorrect due to the ordering issue with CMD2 (so it just appears odd). However the limited testing i was able to do shows that initial count for Y is also wrong in some cases. Take the following example... Y is evaluated to 5, /cd is shown as 1 of 5 and /uncd is shown as 4 of 5. I believe X is correct but the initial 5 value for Y is not. Perhaps counting the close group }?

Code:

{
/cd %p%f
/select -d rx: (?-i).*\.min
/queue selected
}
/uncd

4. When an enqueued command (eg /ren) encounters a problem (eg 550 on RNTO) is it intentional that the enqueued command queue entry still be removed rather than marked as failed?

I was unable to find any old commands that did not continue to perform as expected with this build. This includes the command i wrote that started all this discussion which continues to function correctly (since the sub command only contains one /enqueue line). Since sub command support is relatively new I think it is safe to assume very minimal if any custom commands created by people are likely to malfunction as a result of fixes here - also since the tasks most affected are those that people are unlikely to mistakenly put in the wrong place / rely on being in the wrong place.

DayCuts · 02-01-2015, 03:46 PM

UPDATE: upon further testing this morning something is definitely wrong with this build. It seems I tested older custom commands mainly with just a single item selection so did not notice the problems. Some examples...

This works - performs task on each selection item.

Code:

site chmod 777 %p%f

This does not work - performs task only on the LAST selected item.

Code:

site chmod 777 %f

This enqueues all selected items but still performs task only on the LAST selected item. (previously the {} would not be necessary for this to function as one would expect, not sure if/when that may have changed)

Code:

/enqueue %f
site chmod 777 %f

This completely fails and instead enqueues "TESTING<cr>site chmod 777 TESTING<cr>" for each item.

Code:

{
/enqueue %f
site chmod 777 %f
}

This works.

Code:

{
/cd %f
site chmod 777 %p%f
/uncd
}

This does not work - performs 'site chmod' on each item first, then performs /cd and /uncd only once, again on the LAST selected item.

Code:

/cd %f
site chmod 777 %p%f
/uncd

bigstar · 02-01-2015, 04:35 PM

1. Please try this updated build ffxp5.0.0.3805-r4.zip

I changed the way linked scripts are executed with [] to address this problem, I will need to do additional testing to make sure that this doesn't introduce any unexpected issues.

2. I noticed that one of the variables used with /delay was signed, it should of been unsigned and I bet this resulted in a int overflow. I have updated this in the build above.

3. For now the the command X of total is wrong, this has to do with the way the flow of the engine was changed. I will correct this once I am satisfied that everything else is working as correctly, each change I make effects this.

4. When the command engine was run there is no success/failure result that could be passed to the transfer queue (atleast until now). I've added this ability so if any command is enqueued and that command fails then the whole queue is stopped, this makes the most sense especially when any following command may rely on the one before it being executed successfully.

DayCuts · 02-02-2015, 06:26 PM

Testing with 3805-r4

1. Repeating Example 3 - everything now seems to function in the expected order. (sorting was by date so TESTING2 was first in the list)

Code:

[10:05:16] [L] [CMD] -- Enqueue A
[10:05:16] [L] A
[10:05:17] [L] 500 'A': Command not understood.
[10:05:17] [L] [CMD] /cd /tools/bin/TESTING2
[10:05:17] [L] CWD /tools/bin/TESTING2
[10:05:17] [L] 250 CWD command successful.
[10:05:17] [L] PWD
[10:05:18] [L] 257 "/tools/bin/TESTING2" is current directory.
[10:05:18] [L] STAT -l
[10:05:18] [L] List Complete: 186 bytes in 0.92 seconds (0.2 KB/s)
[10:05:18] [L] [CMD] CMD2
[10:05:19] [L] [CMD] -- Enqueue B
[10:05:19] [L] B
[10:05:19] [L] 500 'B': Command not understood.
[10:05:19] [L] [CMD] /cd /tools/bin/TESTING
[10:05:19] [L] CWD /tools/bin/TESTING
[10:05:19] [L] 250 CWD command successful.
[10:05:19] [L] PWD
[10:05:20] [L] 257 "/tools/bin/TESTING" is current directory.
[10:05:20] [L] STAT -l
[10:05:21] [L] List Complete: 549 bytes in 0.94 seconds (0.5 KB/s)
[10:05:21] [L] [CMD] CMD2
[10:05:21] [L] [CMD] -- Enqueue B
[10:05:21] [L] B
[10:05:21] [L] 500 'B': Command not understood.
[10:05:21] [L] [CMD] /uncd
[10:05:21] [L] CWD /tools/bin
[10:05:21] [L] 250 CWD command successful.
[10:05:21] [L] PWD
[10:05:22] [L] 257 "/tools/bin" is current directory.
[10:05:22] [L] STAT -l
[10:05:23] [L] List Complete: 7,618 bytes in 0.92 seconds (7.4 KB/s)

2. I can confirm that /delay now functions correctly.

4. Great, this seems to function as intended.

Regarding the chmod examples in previous post, these all still function as described in that post.

I tested some other commands as well and everything so far seemed to function correctly, other than those impacted by the %f handing/etc (re chmod examples)

bigstar · 02-03-2015, 01:32 PM

Quote:

Regarding the chmod examples in previous post, these all still function as described in that post.

I tested some other commands as well and everything so far seemed to function correctly, other than those impacted by the %f handing/etc (re chmod examples)

After reviewing all my of recent changes I discovered a mistake where a variable was passed by reference to a sub-routine and modified; when should of been passed by value so that the original value outside of the sub-routine was preserved.

This mistake resulted in only some of the selected items being evaluated when the %f token was used.

As you discovered using the /enqueue command within a command block results in all of the following commands within the block being enqueued as well, I'm fairly sure that this was not the intended behavior but the /enqueue command is somewhat mysterious. It was originally added so that the user could add operations to the queue via the raw command line and then review the result to verify the output is as expected.

From another post I found this example

Code:

/enqueue /ren "%p%f" "%p$replacerx("%f", "^([0-9]+[\s][-]) (.*)\.(.*)$","\2.\3")"

I found a race condition in the hash related commands (/checksfv, /crc32../sha512, etc) that perform threaded operations.

There was also a problem that caused loading commands from within groups to sometimes fail [\group\sub-group\command] depending on the order and layout of the command groups.

Here's the latest revision with all my current fixes
ffxp5.0.0.3805-r5.zip

I think the only issue remaining is that the command progress counter might be wrong under some conditions.

DayCuts · 02-04-2015, 05:13 AM

Testing with 3805-r5

Almost everything seems to be acting as expected. As you say Y count for command progress isn't quite right (seems to count an extra increment for each iteration of a command group).

Only the very last chmod example acts strangely. I know its not a realistic command (or the way it should be done) just one of the test cases I was using. As you can see the order is lost when there are no surrounding {}. It seems to iterate over each line containing %f individually but not in the order they are presented.

Code:

/cd %f
site chmod 777 %p%f
/uncd

Code:

[20:45:25] [L] site chmod 777 /tools/bin/TESTING2
[20:45:25] [L] 200 CHMOD command successful.
[20:45:25] [L] site chmod 777 /tools/bin/fibseq/TESTING
[20:45:26] [L] 200 CHMOD command successful.
[20:45:26] [L] [CMD] /cd TESTING2
[20:45:26] [L] CWD TESTING2
[20:45:26] [L] 250 CWD command successful.
[20:45:26] [L] PWD
[20:45:26] [L] 257 "/tools/bin/TESTING2" is current directory.
[20:45:26] [L] STAT -l
[20:45:27] [L] List Complete: 282 bytes in 0.86 seconds (0.3 KB/s)
[20:45:27] [L] [CMD] /cd TESTING
[20:45:27] [L] CWD TESTING
[20:45:27] [L] 550 TESTING: No such file or directory.
[20:45:27] [L] CWD /tools/bin/TESTING2/TESTING
[20:45:28] [L] 550 /tools/bin/TESTING2/TESTING: No such file or directory.

DayCuts · 02-04-2015, 07:04 AM

Again tweaking my cleanup command I have noticed some things performed in an odd order, take the following example.

Code:

#CMD
{
/enqueue site chmod 777 %p%f
/cd %p%f
/select -f rx: ...
/delete selected
/enqueue mkd %p%f/Subdir
/select -f rx: ...
[SUBCMD]
}
/uncd

#SUBCMD
/enqueue /ren "%p%f" "%pSubdir/%f"

#SUBCMD V2
/enqueue mkd %p%f/Subdir
{
/enqueue /ren "%p%f" "%pSubdir/%f"
}

All of the /enqueue lines in the main {} block of CMD are performed first. Moving the /enqueue mkd to the sub command (re SUBCMD V2) is not good of course because then the wrong %f set is referenced. This just demonstrates that some things are still not performed in the logical order that they are written in. It causes the /enqueue commands in the above example to be iterated individually as if they were not inside the command block {} at all. This seems to be at odds with the purpose and function of a command block. Of course in the full command I am using, the /enqueue site chmod is intentionally outside of the bulk command block {} as I want it iterated/performed first, but this should be the users choice and I can see situations where not respecting its exact place inside the command block and iterating it as part of that group of commands could cause problems.

Also I noticed if I use the internal /mkd command instead of mkd as a raw command it following /enqueue it fails. A little testing confirms that the /mkd is non-functional under any circumstances. It is not being recognized by the parser at all and is sent as a raw command. Obviously this results in an error.

bigstar · 02-04-2015, 03:50 PM

I had mentioned this once or twice awhile back but even I neglected to notice my own mistake.

The script engine was using a priority level of each command being executed, this level determined whether or not the command could be executed safely now or if it needed to be executed later, with 3 stacks (input, processing, execution stack) commands are moved from one to the other, some specific commands skip the processing stack all together.

Once the execution stack contains an item certain processing can't occur until the execution stack is emptied.

An invalid command or a few others such as /enqueue, /queue selected, RAW command would yield until the execution stack was emptied and this caused the out of order sequences we both have noticed.

While this is clear to me now, it was not entirely obvious to me while trying to solve these problems.

This way makes everything overly complicated (as well as unclear and out of order); after several hours of testing I can't recall exactly why it was even done this way, I am fairly sure it had something to do with queuing the selected items and then deleting them but not entirely sure.

I am changing this and eliminating the priority based ordering for now.

With this change everything now appears to work exactly as intended based on all my test scripts as well as the ones you provided.

I am also changing the way the script handles failures so that if a failure occurs within a command block or included script then the failure only stops execution within that scope, allowing the remaining script to finish.

Code:

/cd /System/
/selectall -d
{
/cd %p%f
site chmod 777 %p%f
}
/uncd

CWD /System/Config.Msi
250 CWD command successful.
site chmod 777 /System/Config.Msi
...
CWD /System/Documents and Settings
550 CWD failed. No permission.
< /cd failed as a result site chmod is not performed >
...
CWD /System/DvTape
250 CWD command successful.
site chmod 777 /System/DvTape

I am performing a massive code cleanup of the script engine and I should have an updated build for you to test later tonight or tomorrow.

bigstar · 02-04-2015, 08:35 PM

With the new simplified execution order of the command engine I have successfully implemented nested command blocks

So now you can do something like this if you wanted to without needing to call an external script

Code:

/selectall -d
{
    /echo [BLOCK 1] %p%f
    /cd %p%f
    /selectall -d
    {
        /echo [BLOCK 2] %p%f
        /cd %p%f
        /selectall -d
        {
            /echo [BLOCK 3] %p%f
            /cd %p%f
        }
    }
}
/uncd

bigstar · 02-04-2015, 11:13 PM

Here's the new build ffxp5.0.0.3805-r6.zip

DayCuts · 02-05-2015, 02:30 AM

Ah, I was not aware of the intentional priority handling but that certainly explains a lot.

DayCuts · 02-05-2015, 08:59 AM

Testing 3805-r6... everything that had previously been discussed seems to work perfectly, I was unable to find or invent a command that broke things.

Playing with the new nested command blocks I did notice that space-indenting breaks [Name of command] and most likely [path to cmd file]. Presumably due to the internal check looking at chr 1 to identify these special commands. With new nested command block support indenting with spaces will be more common (at least for me) since it improves readability. Everything else functioned correctly regardless of space indenting.

Minor suggestion, the ctrl+a (select all) keyboard shortcut is not mapped within the custom command edit box, the other common keyboard shortcuts are.