Custom command improvement/discussion

[DayCuts]

First of all a potential bug/issue, unless I am misusing or missing something...

Take the following example...


%d[input]
{
command $replace(%f,"s","z",1) %1
}


Gives the following result...


[13:19:24] [L] command tezt testinput
[13:19:26] [L] 200- ...
[13:19:27] [L] 200 Command Successful.
[13:19:24] [L] command anothertezt testinput
[13:19:26] [L] 200- ...
[13:19:27] [L] 200 Command Successful.
[13:19:32] [L] testinput
[13:19:33] [L] 500 'testinput': Command not understood.


As you can see after the code blocks are performed on each item selection the raw string from %d[] is sent to the server. Not sure if this is intentional?

I realize in this basic example I could just replace the %1 with %d[] and remove the code block, and if i need to use the input multiple times within a code block i can reference %1 for the rest. Moving %d[] inside the code block will only prompt once (i assume intentionally) but will then send the raw string from %d[] to the server for each iteration.

I then tried doing things some other ways and came across some unexpected results...


command:
%d[input]
{
command $replace(%f,"s","z",1) $replace(%1,"e","o")
}
result:
[13:19:24] [L] command tezt
[13:19:26] [L] 200- ...
[13:19:27] [L] 200 Command Successful.
[13:19:24] [L] command anothertezt
[13:19:26] [L] 200- ...
[13:19:27] [L] 200 Command Successful.
[13:19:32] [L] testinput
[13:19:33] [L] 500 'testinput': Command not understood.


command:
command $replace(%f,"s","z",1) $replace(%d[input],"e","o")
result:
[13:19:24] [L] command tezt
[13:19:26] [L] 200- ...
[13:19:27] [L] 200 Command Successful.
[13:19:24] [L] command anothertezt
[13:19:26] [L] 200- ...
[13:19:27] [L] 200 Command Successful.


This indicates to me that multiples of the same token modifier can't be use in a single line of a custom command. I tried with $uppercase as the second modifier which worked.

In summery:
1. Should the string input from %d[], when placed at the start of the line, be sent to the server? And if so can this be solved by adding a /noop (no operation) command.
2. Is it intentional for a token modifier to be limited to one instance per line, or is this a bug that can be fixed? If it is intentional then a solution might be to add variable support to the custom command interface. /var <key> <value> and $var(<key>) (or %v[<key>] etc, whichever fits best with the syntax standard) with support for performing /var <key> $modifier(%v[<key>],...) (which might be a useful addition anyway)

[bigstar]

%d on a line by it self will send the result to the server, this is by design, you could do something like this

/echo %d[input]However I have discovered an a logic flow problem with the order of commands when mixing commands inside and outside of a command block

TEST 1
{
TEST 2 - %f
TEST 3 - %f
}
TEST 4


The commands are executed in the wrong order.. as seen below
TEST 2 - zz
TEST 3 - zz
TEST 1
TEST 4

Testing with /echo prefix like this and the result is

/echo TEST 1
{
/echo TEST 2 - %f
/echo TEST 3 - %f
}
/echo TEST 4


TEST 1
TEST 2 - zz
TEST 3 - zz
TEST 4

The problem is with the way "/prefixed" commands vs direct commands are executed when the command contains command blocks.

And if you use multiple command blocks the blocks execute in reverse order. Its a bit of a mess and it will take me some time to hammer out these problems.

It appears that my token parser fails to check for multiple identical modifiers within the same command line, I will have to re-work this.

I could add variables that can be set and then replaced later but the format will need to be unique so that it doesn't conflict with existing token elements I think I'd need to do something like this /set @<name>@ <value> where the name is encased with @'s I can't use $, %, #, !

[DayCuts]

I did notice the ordering issue but did not realize it was so extensive and forgot to mention it.

I was unaware that /echo existed, it is not listed in the help file that I could see. Though /noop would be more appropriate since the point is to prevent it being sent outside of the parser but /echo certainly suffices, thanks.

Variables was an afterthought really but for the sake of discussion is there a particular reason why %key or $var(key) would be an issue? I guess it comes down to how you parser is designed. If it allows nested 'token modifiers' (a kind of misleading name for them) then $var(localvar) could be used with minimal effort without needing much change to the parser. If not then %localvar might seem a better solution (and would comply more closely with the %globalvar form already used) but would obviously need a little error checking to avoid conflicts with global vars (or blacklist them from being used with local vars). Time/date vars shouldn't interfere because they already use a different form.

In any case would a prefix or encapsulation be needed at all for /set? wouldn't the code to process that command know that the second token on the line is the key and 3-N is the value, without the need to be told by some non-alphanumeric indicator? Again I suppose this all depends on how the parser is designed.

[bigstar]

The command ordering issue doesn't appear to be as bad as I initially thought, though it is an unforeseen problem.
The order is modified when using <tokens> i.e. %f within the command block and should now be fixed.

At this time not all commands and settings are currently documented in the help file, some might be mentioned somewhere in the forums.

The /echo command was added to make it easier for me to test the result of regex and token modifier operations, you can also use /test with the same result.

There are also quite a few undocumented /debug commands that I have defined for testing and experimenting. such as /debug cache which you may have seen me mention on the forums.

We currently use /set <var> <value> for setting several internal variables (with more planned) though I don't recall they've been documented in the help file.

/set information-bar-active-color <value>
/set information-bar-normal-color <value>
/set toolbar-bg-color <value>
Where value can be entered as a HTML color code i.e. #FFFFFF or RGB color code 255 255 255

For example in the Site Manager > Selected Site > "Perform these commands after login" I use /set toolbar-bg-color #C4E25A to set the toolbar background color to help me identify the session connected to the flashfxp.com server, often I'm opening and/or closing many instances of FlashFXP for testing and this makes it easier for me to avoid closing the wrong window at a glance.

The reason we can't use %key is because the %<token> variables are a fixed length of 1 character, to use a variable length token identifier the parser needs a way to determine where the identifier ends.

My original thought to use @<name>@ as the variable name seems like the best solution at this point, I experimented with a few different ideas and so far this is the only solution that appears to work 100% and didn't require any major design refactoring.

You can give this test build a try
https://oss.azurewebsites.net/testr/5/ffxp5.0.0.3799-r5.zip
(https://oss.azurewebsites.net/testr/5/ffxp5.0.0.3799-r5.zip)
1. Fixes the command block order when the block contains tokens.
2. Adds support for "/set @<name>@ <value>" and then you can use @<name>@ from that point forward. The variable/value is preserved until you close FlashFXP. I will add an /unset command when I have more time.

Some examples:

Store the value of %d as @x@

/set @x@ %d
/echo @x@


Store the current path %p as @p@

/set @p@ %p
/echo @p@


I haven't spent as much time on this as I would have liked, with the holidays my schedule has been quite full.

[DayCuts]

Hope you have enjoying the holiday season!

Re: /set, again I was unaware of this but that is easily solved by using /var instead (which is often used in scripting language to represent local variables that are automatically unset after the current script is done). I see your point about % tokens being of restricted length in the current implementation. The way most do it is to break on any non-word character. A word character is usually defined as any letter, number, or underscore (and sometimes a dash is included). This way any percentage sign, space, comma, period, newline, etc would indicate the local variable name boundary without the need for @'s etc.

How hard such an implementation change might be of course again depends on the parser design. Use of PCRE for example would make variable replacement of the entire script quite simple (in comparison to say, a step through approach). But as you say this would most likely require significant design refactoring of your parser.

[bigstar]

Thanks.

When defining the variable I could omit the trailing @ but not but not for the inline use of it. There may be instances where the variable is used in a way that doesn't word break.

At this point I don't feel comfortable refactoring the script/command engine because I don't want to accidentally change a behavior of how something currently works. I have quite a few test cases however I know that it doesn't cover everything and it would be very easy for something to break and go unnoticed. such as the issue with the command execution order.

My main goal is to make it as usable as possible without limitations and while I understand that this syntax might overly complicate the logic it does seem to work in many conditions where any way else could possibly fail.

I found an issue with the new variable code, if you assign a command to a variable such as /select it doesn't perform the command after replacing the variable with the value.

i.e. this doesn't execute the /select command.

/set @A@ /select -f *.txt
@A@

[DayCuts]

I think it would be perfectly acceptable for /set @A@ to treat "/select -f *.txt" as a literal string and not re-evaluate it to allow your usage example at all. Commands by nature are expected to be at the beginning of the line for the most part (in this type of set up), and certainly that is ALWAYS the expectation for custom command scripting in flashfxp.

What I would generally expect is for "/set @A@ $something()" to evaluate $something() and store the resulting string literal as the value of @A@, meaning that NOT evaluating /select in that same manner would be inconsistent unless by design (that is, /cmd's must always be at the start of a line therefor a / will be treated as a string literal thereafter).

I can not really think of situations where you example would be used? All that comes to mind is the hash calculation commands, but then I am wondering how /crc32 "<filename>" is different to /echo $crc32("<filename>") or just $crc32("<filename>") on its own? and by extension is there actually a need for /crc32 etc. The only other design method I can think of at the moment would possibly be to not support your example of putting a /cmd in a @var@ and running it with just @var@... but rather set the result of said /cmd as the string literal value of @var@. In the latter I can at least come up with usage situations, though I doubt they would actually be utilized. That also keeps things at a decent level of consistency.

Again the suggestion of variables was an afterthought. Though they would be useful - it is clear they push against the boundaries of what one would expect from a basic command design interface and are a more complex thing to implement in the type of parser you have designed. Mostly continued this discussion because I happen to enjoy software 'engine' design and have written various low level 'engines' and parsers myself in the past.

[bigstar]

I've corrected the issue with the way @var@ is evaluated because with the above mentioned issue behavior was inconsistent when the command was preparsed. just like the out of order command issue when mixing raw commands with /commands that I previously fixed.

In this example @A@ was echoed to the console window as if it were executed but in reality it was just echoed and not actually performed, since this is very misleading I considered this behavior wrong.
/set @A@ /select -f *.txt
@A@


In this example the is performed as expected.
/set @A@ SITE HELP
@A@


The reason for the two different commands is that /crc32 displays the result directly in the status window using a predefined format.


[07:33:02] [L] [CMD] /crc32 flashfxp.log
[07:33:02] CRC32: a9ce9e68 File: F:\flashfxp.log (0.1 KB/s)

with $crc32(<filename>) it would return only the crc32 hash and this could be used to do things like create a sfv file.

# This will create a SFV file of all selected files
# using the standard SFV hash file format
# <filename> <crc32>
/writeln "%pverify.sfv" %f $crc32("%p%f")

I think your idea of allowing the user to set/get variables is a great idea and I wish I had thought of it :)

I will link to an updated build shortly with my latest changes.

[DayCuts]

Hi, couple of things that are all sort of related...

1. Just wondering what the current implementation status is for variables? and for example will "/set @A@ /select -f *.txt" set @A@ to the literal string "/select -f *.txt" or the result of it (based on previous post i am assuming literal string).

EDIT: Testing and confirmed for myself that my assumption is correct.

2. While attempting to write a new command i found myself unable to figure out a way to do something. However I am not sure if I am just forgetting about something, and I have not tried using variables yet hence the question above (if my assumption is correct then variables wont help)

What I am attempting to do it move files (on the remote side) but I am selecting the files to be moved with /select and can not think of a way to use /ren /enqueue (or raw commands) with the selected item(s). Can you think of a way to do this that I am not seeing?

3. While working on another command I noticed some unexpected behavior. Consider the following example.

/select ...
/delete selected
/transfer queue

If their are previously failed items already in the queue then /transfer queue resets them even through there are unfailed items also in the queue (the enqueue commands). Maybe the global transfer routine (that checks the queue and resets if needed) is not taking enqueued items into account?

4. Suggestion to update /delete (and maybe similar commands) to accept a [now] parameter at the end eg /delete selected now. This would perform the delete operation immediately. I understand this may not be as simple as it sounds.

5. Linked to both 2 and 3... a way to reference selected items similar to %f. This could then be used in other commands (like /ren). Again I understand this may not be as simple as it sounds. For such a token I would expect that the command, eg /ren, would be iterated for each item in the /select result that the token is referencing.

[bigstar]

1. Yes, as you discovered. Though the variables feature and how it works isn't 100% set in stone.

2. If you want to move all the items in the current directory to a specific folder you could do something like this

/selectall
/ren "%p%f" "/archived/%f"
/list
If that doesn't help, please give me a specific example of what you're trying to do.

3. This is the behavior when the delete operation command is executed. It will not run if any items in the queue are marked as failed, FlashFXP will reset and move these failed operations to the top of the queue before allowing the delete operation to run.. if the failed operation cannot be performed (fails over and over) then once the retry limit is reached the delete operation is also marked as failed.

4. The only reliable way to delete files is to do it via the queue (which can't be performed immediately), if you needed to delete a file immediately you could use "DELE %p%f"

5. Did you use pass absolute path name when using /ren? in many cases the absolute path is required, sometimes FlashFXP will try to guess and use the current dir but since the path may change due to a loss of connection or another command this is not reliable, especially if you're restoring a saved queue.

%f should work in most commands, I tested it with /ren and it worked as expected.
Also remember that most /macro's parameters must be double-quoted.

[bigstar]

/enqueue is a way for a script to include operations in the queue that would otherwise be performed immediately.

If you use for example the /ren command then the operation is performed immediately (not added to the queue), now depending on the situation this may not give the desired result.

Take a look at this code, since the /ren command is performed immediately the rename operation takes place before the files are transferred. FAILURE..

/selectall
/transfer selected
/ren "%p%f" "%p%f.downloaded"
/start
To get the desired result we need to enqueue the /ren command like this

/selectall
/transfer selected
/enqueue /ren "%p%f" "%p%f.downloaded"
/start
The reason I used /transfer selected above and not /queue selected is because the /queue selected command adds the items to the end of the queue where-as the /transfer selected command inserts the items in-line. This is somewhat confusing, I know :confused:

[DayCuts]

Re 3+4. Thank you, I see why it would be designed that way. I just happened to have items marked as failed when I was working on the command. I shall just remember to open a clean session to avoid this.

Re 2. Sorry I wasn't very clear about the selection process here... I am attempting to perform an action on a /select*ed set of items from within a sub folder. These items can not be known prior to entering the folder as the folder is itself entered using /cd %p%f. So %f is already populated. I tried using a sub command to break things up and test the internal handing of %f and got the following results...

Command TEST

/echo -A
{
/cd %p%f
/echo -B
/select -f rx: .*\.tmp$
/echo -C
/delete selected
/echo -D
[SUBTEST]
/echo -E
}
/echo -F
/uncd
/echo -G


Command SUBTEST

/echo -D1
/select -f rx: .*\.log$
/echo -D2
/enqueue /ren "%p%f" "%pLogs/%f"
/echo -D3


Manually select folder TESTING2 and perform command TEST.

[19:05:38] [L] [CMD] -A
[19:05:38] [L] [CMD] /cd /tools/bin/TESTING2
[19:05:38] [L] CWD /tools/bin/TESTING2
[19:05:38] [L] 250 CWD command successful.
[19:05:38] [L] PWD
[19:05:39] [L] 257 "/tools/bin/TESTING2" is current directory.
[19:05:39] [L] STAT -l
[19:05:40] [L] List Complete: 294 bytes in 0.88 seconds (0.3 KB/s)
[19:05:40] [L] [CMD] -B
[19:05:40] [L] [CMD] /select -f rx: .*\.tmp$
[19:05:40] [L] [CMD] -C
[19:05:40] [L] [CMD] /delete selected
#Correct item added to queue for deletion... "/tools/bin/TESTING2/rebuild.tmp"
[19:05:40] [L] [CMD] -D
[19:05:40] [L] [CMD] SUBTEST
[19:05:40] [L] [CMD] -E
[19:05:40] [L] [CMD] -F
[19:05:40] [L] [CMD] /uncd
[19:05:40] [L] CWD /tools/bin
[19:05:40] [L] 250 CWD command successful.
[19:05:40] [L] PWD
[19:05:40] [L] 257 "/tools/bin" is current directory.
[19:05:40] [L] STAT -l
[19:05:41] [L] List Complete: 7,618 bytes in 0.88 seconds (7.4 KB/s)
[19:05:41] [L] [CMD] -G
[19:05:41] [L] [CMD] -D1
[19:05:41] [L] [CMD] /select -f rx: .*\.log$
[19:05:41] [L] [CMD] -D2
#Unexpected item added to queue for rename... "/tools/bin/TESTING2/rebuild.tmp" to "/tools/bin/TESTING2/Logs/rebuild.tmp"
[19:05:41] [L] [CMD] -D3


Manually select folders TESTING and TESTING2 and perform command TEST.

[19:35:07] [L] [CMD] -A
[19:35:07] [L] [CMD] /cd /tools/bin/TESTING
[19:35:07] [L] CWD /tools/bin/TESTING
[19:35:07] [L] 250 CWD command successful.
[19:35:07] [L] PWD
[19:35:07] [L] 257 "/tools/bin/TESTING" is current directory.
[19:35:07] [L] STAT -l
[19:35:08] [L] List Complete: 294 bytes in 0.89 seconds (0.3 KB/s)
[19:35:08] [L] [CMD] -B
[19:35:08] [L] [CMD] /select -f rx: .*\.tmp$
[19:35:08] [L] [CMD] -C
[19:35:08] [L] [CMD] /delete selected
#Correct item added to queue for deletion... "/tools/bin/TESTING/rebuild.tmp"
[19:35:08] [L] [CMD] -D
[19:35:08] [L] [CMD] SUBTEST
[19:35:08] [L] [CMD] -E
[19:35:08] [L] [CMD] /cd /tools/bin/TESTING2
[19:35:08] [L] CWD /tools/bin/TESTING2
[19:35:09] [L] 250 CWD command successful.
[19:35:09] [L] PWD
[19:35:09] [L] 257 "/tools/bin/TESTING2" is current directory.
[19:35:09] [L] STAT -l
[19:35:10] [L] List Complete: 294 bytes in 0.87 seconds (0.3 KB/s)
[19:35:10] [L] [CMD] -B
[19:35:10] [L] [CMD] /select -f rx: .*\.tmp$
[19:35:10] [L] [CMD] -C
[19:35:10] [L] [CMD] /delete selected
#Correct item added to queue for deletion... "/tools/bin/TESTING2/rebuild.tmp"
[19:35:10] [L] [CMD] -D
[19:35:10] [L] [CMD] SUBTEST
[19:35:10] [L] [CMD] -E
[19:35:10] [L] [CMD] -F
[19:35:10] [L] [CMD] /uncd
[19:35:10] [L] CWD /tools/bin
[19:35:10] [L] 250 CWD command successful.
[19:35:10] [L] PWD
[19:35:11] [L] 257 "/tools/bin" is current directory.
[19:35:11] [L] STAT -l
[19:35:12] [L] List Complete: 7,618 bytes in 0.89 seconds (7.4 KB/s)
[19:35:12] [L] [CMD] -G
[19:35:12] [L] [CMD] -D1
[19:35:12] [L] [CMD] /select -f rx: .*\.log$
[19:35:12] [L] [CMD] -D2
#Unexpected item added to queue for rename... "/tools/bin/TESTING/rebuild.tmp" to "/tools/bin/TESTING/Logs/rebuild.tmp"
[19:35:12] [L] [CMD] -D3
[19:35:12] [L] [CMD] -D1
[19:35:12] [L] [CMD] /select -f rx: .*\.log$
[19:35:12] [L] [CMD] -D2
#Unexpected item added to queue for rename... "/tools/bin/TESTING2/rebuild.tmp" to "/tools/bin/TESTING2/Logs/rebuild.tmp"
[19:35:12] [L] [CMD] -D3


There are a few things to note in the results...
First of all I see that %f can in fact refer to a sub selection if outside of a command group already iterating %f. So %f is only static within its current command group {}. I had wondered about this hence the testing with a sub command.
Secondly the SUBTEST command is not performed until after everything in the TEST command, I am assuming this is not the intended behavior. As you can see from the second set of results it iterates over the command block for each manually selected %f when the command was launched, then continues to finish off the rest of TEST, then performs each iteration of the SUBTEST command.

I will continue to play around with the logic and ordering to see if I can get this to work, including maybe something like below, though I may just write it in C instead and load it as a raw command on the server. The latter is easy enough I just thought it would be more handy to have it with my client for better portability... and now that I have started the challenge to get it to work as expected has sucked me in lol.

{
[MARK ITEMS FOR DELE]
}
{
[SELECT MARKED ITEMS AND ENQUEUE /delete]
}
/markedclear
{
[MARK ITEMS FOR MOVE]
}
{
[SELECT MARKED ITEMS AND ENQUEUE /ren]
}
/markedclear

[DayCuts]

Upon further investigation it seems that the sub command in the above examples are in fact performed at the correct time but the echo's are not. Using the simplest examples i could to demonstrate...

Example 1 - raw commands are not performed at the expected times.

#CMD
A
{
/cd %p%f
[CMD2]
}
/uncd

#CMD2
B

#LOG
[21:01:27] [L] [CMD] /cd /tools/bin/TESTING
[21:01:27] [L] CWD /tools/bin/TESTING
[21:01:27] [L] 250 CWD command successful.
[21:01:27] [L] PWD
[21:01:28] [L] 257 "/tools/bin/TESTING" is current directory.
[21:01:28] [L] STAT -l
[21:01:29] [L] List Complete: 294 bytes in 0.94 seconds (0.3 KB/s)
[21:01:29] [L] [CMD] CMD2
[21:01:29] [L] [CMD] /cd /tools/bin/TESTING2
[21:01:29] [L] CWD /tools/bin/TESTING2
[21:01:29] [L] 250 CWD command successful.
[21:01:29] [L] PWD
[21:01:29] [L] 257 "/tools/bin/TESTING2" is current directory.
[21:01:29] [L] STAT -l
[21:01:30] [L] List Complete: 294 bytes in 0.93 seconds (0.3 KB/s)
[21:01:30] [L] [CMD] CMD2
[21:01:30] [L] A
[21:01:31] [L] 500 'A': Command not understood.
[21:01:31] [L] [CMD] /uncd
[21:01:31] [L] CWD /tools/bin
[21:01:31] [L] 250 CWD command successful.
[21:01:31] [L] PWD
[21:01:32] [L] 257 "/tools/bin" is current directory.
[21:01:32] [L] STAT -l
[21:01:32] [L] List Complete: 7,618 bytes in 0.95 seconds (7.4 KB/s)
[21:01:32] [L] B
[21:01:33] [L] 500 'B': Command not understood.
[21:01:33] [L] B
[21:01:33] [L] 500 'B': Command not understood.


Example 2 - /enqueue commands are performed at the expected times but sub command /echo's are not.

#CMD
/echo -- Enqueue A
/enqueue A
{
/cd %p%f
[CMD2]
}
/uncd

#CMD2
/echo -- Enqueue B
/enqueue B

#LOG
[21:11:57] [L] [CMD] -- Enqueue A
#A is correctly enqueued
[21:11:57] [L] [CMD] /cd /tools/bin/TESTING
[21:11:57] [L] CWD /tools/bin/TESTING
[21:11:57] [L] 250 CWD command successful.
[21:11:57] [L] PWD
[21:11:58] [L] 257 "/tools/bin/TESTING" is current directory.
[21:11:58] [L] STAT -l
[21:11:59] [L] List Complete: 294 bytes in 0.93 seconds (0.3 KB/s)
[21:11:59] [L] [CMD] CMD2
#B is correctly enqueued
[21:11:59] [L] [CMD] /cd /tools/bin/TESTING2
[21:11:59] [L] CWD /tools/bin/TESTING2
[21:11:59] [L] 250 CWD command successful.
[21:11:59] [L] PWD
[21:11:59] [L] 257 "/tools/bin/TESTING2" is current directory.
[21:11:59] [L] STAT -l
[21:12:00] [L] List Complete: 294 bytes in 0.93 seconds (0.3 KB/s)
[21:12:00] [L] [CMD] CMD2
#A is correctly enqueued again
[21:12:00] [L] [CMD] /uncd
[21:12:00] [L] CWD /tools/bin
[21:12:01] [L] 250 CWD command successful.
[21:12:01] [L] PWD
[21:12:01] [L] 257 "/tools/bin" is current directory.
[21:12:01] [L] STAT -l
[21:12:02] [L] List Complete: 7,618 bytes in 0.94 seconds (7.4 KB/s)
#These echos are misplaced.
[21:12:02] [L] [CMD] -- Enqueue B
[21:12:02] [L] [CMD] -- Enqueue B


Example 3 - /enqueue commands are performed at the expected times but sub command /echo's AND raw command are not

#CMD
/echo -- Enqueue A
/enqueue A
A
{
/cd %p%f
[CMD2]
}
/uncd

#CMD2
/echo -- Enqueue B
/enqueue B
B

#LOG
[21:20:39] [L] [CMD] -- Enqueue A
#A is correctly enqueued
[21:20:39] [L] [CMD] /cd /tools/bin/TESTING
[21:20:39] [L] CWD /tools/bin/TESTING
[21:20:39] [L] 250 CWD command successful.
[21:20:39] [L] PWD
[21:20:39] [L] 257 "/tools/bin/TESTING" is current directory.
[21:20:39] [L] STAT -l
[21:20:40] [L] List Complete: 294 bytes in 0.93 seconds (0.3 KB/s)
[21:20:40] [L] [CMD] CMD2
#B is correctly enqueued
[21:20:40] [L] [CMD] /cd /tools/bin/TESTING2
[21:20:40] [L] CWD /tools/bin/TESTING2
[21:20:41] [L] 250 CWD command successful.
[21:20:41] [L] PWD
[21:20:41] [L] 257 "/tools/bin/TESTING2" is current directory.
[21:20:41] [L] STAT -l
[21:20:42] [L] List Complete: 294 bytes in 0.94 seconds (0.3 KB/s)
[21:20:42] [L] [CMD] CMD2
#B is correctly enqueued again
#Misplaced raw command
[21:20:42] [L] A
[21:20:42] [L] 500 'A': Command not understood.
[21:20:42] [L] [CMD] /uncd
[21:20:42] [L] CWD /tools/bin
[21:20:43] [L] 250 CWD command successful.
[21:20:43] [L] PWD
[21:20:43] [L] 257 "/tools/bin" is current directory.
[21:20:43] [L] STAT -l
[21:20:44] [L] List Complete: 7,618 bytes in 0.94 seconds (7.4 KB/s)
#Misplaced echos and raw commands
[21:20:44] [L] [CMD] -- Enqueue B
[21:20:44] [L] B
[21:20:45] [L] 500 'B': Command not understood.
[21:20:45] [L] [CMD] -- Enqueue B
[21:20:45] [L] B
[21:20:45] [L] 500 'B': Command not understood.


As you can see order is not maintained correctly for certain things. What I would expect this to have evaluated to (internally) is...

/echo -- Enqueue A
/enqueue A
A
/cd /tools/bin/TESTING
/echo -- Enqueue B
/enqueue B
B
/cd /tools/bin/TESTING2
/echo -- Enqueue B
/enqueue B
B
/uncd


Instead it evaluated to

/echo -- Enqueue A
/enqueue A
/cd /tools/bin/TESTING
/enqueue B
/cd /tools/bin/TESTING2
/enqueue B
/uncd
A
/echo -- Enqueue B
B
/echo -- Enqueue B
B

[DayCuts]

Ok I figured out the mechanics behind the use of multiple %f lists and got my command working quite simply. %f simply must be set outside of the command group or sub command that iterates it. Logical enough... command groups or sub commands can be thought of as a 'foreach' in these circumstances. Not sure why it didn't occur to me before (as that is pretty standard for most programming languages) but for some reason I was thinking of %f in a static-global-variable nature and ignoring the obvious. When in fact %f is tied to its stack-level.

All the messing around did at least expose some oddities (above).

One other thing that I noticed after getting my command right and doing some error tolerance was that if there is a 550 error when trying to perform the remote rename the queue entry is still removed (rather than marked as failed). Is this intentional? I have not done a lot with enqueue commands so I am not entirely familiar with the expected behavior.

[bigstar]

Thank you for your feedback.

I attempted to reply to you several times today but each time I tried to write an example and show you some references I ran into some unexpected issues.

There are some problems with the order in which the scripts are being executed.

Ironically most of these problems seem to originate back to a single problem I tried to resolve awhile back where the order of the files in %f was reversed, it would appear that my fix and then additional fixes and changes have opened a huge can of worms.

I use a multi-stage partially recursive parsing and processing engine and it appears that somewhere along the lines I've overly complicated things because nothing works as expected.

I've spent most of the day trying to work things out but every time I fix one issue it introduces a new problem or changes the behavior in a way that may cause existing scripts to malfunction, though at this point I'm not entirely sure whats working correctly and what's not. I am pretty sure that I'll be rolling back all of my changes from today and starting fresh tomorrow.

To quickly answer some of your questions, yes when you use tokens such as %f in a command that command becomes a FOR-EACH LOOP or if the command is within a command block the entire command block becomes a FOR-EACH LOOP.

Chaining another command via [\sub\func] containing a command block can create its own FOR-EACH LOOP that is parsed within the sub context of the main loop but the the commands aren't actually executed until after the main FOR-EACH LOOP has been parsed. (If that makes any sense)

I have the following example that I was going to use to show you how things worked but at the moment it doesn't actually work as expected.

[WARNING - CODE DOESN'T WORK]

main function

/echo start
{
/cd %p%f
/selectall -f
[\sub\func1]
}
/clear cache %p
/cd %p
/echo end


sub function named "\sub\func1"

{
/move "%p%f" "../%f"
}
RMD %p



Below is a simplified output of the way the script should appear when run

[CMD] start
[CMD] /cd /base/sub1/
[CMD] /selectall -f
[CMD] \sub\func1
[CMD] /move "/base/sub1/sub1-file1.bin" "../sub1-file1.bin"
[CMD] /move "/base/sub1/sub1-file2.bin" "../sub1-file2.bin"
RMD /base/sub1/
[CMD] /cd /base/sub2
[CMD] /selectall -f
[CMD] \sub\func1
[CMD] /move "/base/sub1/sub2-file1.bin" "../sub2-file1.bin"
[CMD] /move "/base/sub1/sub2-file2.bin" "../sub2-file2.bin"
RMD /base/sub2/
[CMD] /clear cache /base/
[CMD] /cd /base/
[CMD] end

[bigstar]

I've made changes to the scripting engine to address these issues and now the sample script above works as intended.

Here's a direct link to the updated flashfxp.exe https://oss.azurewebsites.net/testr/5/ffxp5.0.0.3805-r2.zip

I have tested all of the scripts I have and they all seem to work as expected with these changes, please let me know if you encounter any situations or problems with existing scripts. I have to wonder if there's any existing scripts that were designed based on the flawed ordering. I sure hope not.

[DayCuts]

Thanks bigstar. If I remember correctly the initial ordering issue arose with the implementation of status bar logging for command progress. Perhaps there is something with the command progress messages at the heart of things?

I forgot to mention this in my last post but I am guessing it likely traces back to the same changes. When I was playing around with things I tried using /delay to slow things down so I could visual confirm exactly when different things were being done but what I got from '/delay 5' was a delay of some very big number like 45944364133whatever seconds. Changing the 5 in my command didn't make any difference.

[DayCuts]

I've made changes to the scripting engine to address these issues and now the sample script above works as intended.

Here's a direct link to the updated flashfxp.exe https://oss.azurewebsites.net/testr/5/ffxp5.0.0.3805-r2.zip

I have tested all of the scripts I have and they all seem to work as expected with these changes, please let me know if you encounter any situations or problems with existing scripts. I have to wonder if there's any existing scripts that were designed based on the flawed ordering. I sure hope not.

Cool, heading to work shortly but should have time to check it out tomorrow.

[DayCuts]

Testing with 3805-r3

1. Repeating Example 3 from previous posts... everything within the initial CMD seems to be performed in the correct order now, however tasks from sub command CMD2 still do not function in the expected order. Sub command raws and echos (not sure if other commands are affected or not) are still misplaced.

#CMD
/echo -- Enqueue A
/enqueue A
A
{
/cd %p%f
[CMD2]
}
/uncd

#CMD2
/echo -- Enqueue B
/enqueue B
B

#LOG
[18:15:53] [L] [CMD] -- Enqueue A
#A is correctly enqueued
[18:15:53] [L] A
[18:15:54] [L] 500 'A': Command not understood.
[18:15:54] [L] [CMD] /cd /tools/bin/TESTING
[18:15:54] [L] CWD /tools/bin/TESTING
[18:15:54] [L] 250 CWD command successful.
[18:15:54] [L] PWD
[18:15:54] [L] 257 "/tools/bin/TESTING" is current directory.
[18:15:54] [L] STAT -l
[18:15:55] [L] List Complete: 549 bytes in 0.86 seconds (0.5 KB/s)
[18:15:55] [L] [CMD] CMD2
#B is correctly enqueued
[18:15:55] [L] [CMD] /cd /tools/bin/TESTING2
[18:15:55] [L] CWD /tools/bin/TESTING2
[18:15:56] [L] 250 CWD command successful.
[18:15:56] [L] PWD
[18:15:56] [L] 257 "/tools/bin/TESTING2" is current directory.
[18:15:56] [L] STAT -l
[18:15:57] [L] List Complete: 201 bytes in 0.86 seconds (0.2 KB/s)
[18:15:57] [L] [CMD] CMD2
#B is correctly enqueued again
[18:15:57] [L] [CMD] /uncd
[18:15:57] [L] CWD /tools/bin
[18:15:57] [L] 250 CWD command successful.
[18:15:57] [L] PWD
[18:15:58] [L] 257 "/tools/bin" is current directory.
[18:15:58] [L] STAT -l
[18:15:58] [L] List Complete: 7,618 bytes in 0.87 seconds (7.4 KB/s)
#Misplaced echos and raw commands
[18:15:58] [L] [CMD] -- Enqueue B
[18:15:58] [L] B
[18:15:59] [L] 500 'B': Command not understood.
[18:15:59] [L] [CMD] -- Enqueue B
[18:15:59] [L] B
[18:15:59] [L] 500 'B': Command not understood.


Expected behavior...

/echo -- Enqueue A
/enqueue A
A
/cd /tools/bin/TESTING
/echo -- Enqueue B
/enqueue B
B
/cd /tools/bin/TESTING2
/echo -- Enqueue B
/enqueue B
B
/uncd


Previous build evaluated to...

/echo -- Enqueue A
/enqueue A
/cd /tools/bin/TESTING
/enqueue B
/cd /tools/bin/TESTING2
/enqueue B
/uncd
A
/echo -- Enqueue B
B
/echo -- Enqueue B
B


This build evaluated to...

/echo -- Enqueue A
/enqueue A
A
/cd /tools/bin/TESTING
/enqueue B
/cd /tools/bin/TESTING2
/enqueue B
/uncd
/echo -- Enqueue B
B
/echo -- Enqueue B
B


2. /delay does not function, seems to ignore the numeric parameter completely. It always starts a countdown of ~4294968 seconds (almost 50 days).

3. While doing a few tests I have noticed that command progress status bar updates are bugged somehow. In the above 'Example 3' the X value in X of Y is probably mostly incorrect due to the ordering issue with CMD2 (so it just appears odd). However the limited testing i was able to do shows that initial count for Y is also wrong in some cases. Take the following example... Y is evaluated to 5, /cd is shown as 1 of 5 and /uncd is shown as 4 of 5. I believe X is correct but the initial 5 value for Y is not. Perhaps counting the close group }?


{
/cd %p%f
/select -d rx: (?-i).*\.min
/queue selected
}
/uncd


4. When an enqueued command (eg /ren) encounters a problem (eg 550 on RNTO) is it intentional that the enqueued command queue entry still be removed rather than marked as failed?

I was unable to find any old commands that did not continue to perform as expected with this build. This includes the command i wrote that started all this discussion which continues to function correctly (since the sub command only contains one /enqueue line). Since sub command support is relatively new I think it is safe to assume very minimal if any custom commands created by people are likely to malfunction as a result of fixes here - also since the tasks most affected are those that people are unlikely to mistakenly put in the wrong place / rely on being in the wrong place.

[DayCuts]

UPDATE: upon further testing this morning something is definitely wrong with this build. It seems I tested older custom commands mainly with just a single item selection so did not notice the problems. Some examples...

This works - performs task on each selection item.
site chmod 777 %p%f

This does not work - performs task only on the LAST selected item.
site chmod 777 %f

This enqueues all selected items but still performs task only on the LAST selected item. (previously the {} would not be necessary for this to function as one would expect, not sure if/when that may have changed)
/enqueue %f
site chmod 777 %f

This completely fails and instead enqueues "TESTING<cr>site chmod 777 TESTING<cr>" for each item.
{
/enqueue %f
site chmod 777 %f
}

This works.
{
/cd %f
site chmod 777 %p%f
/uncd
}

This does not work - performs 'site chmod' on each item first, then performs /cd and /uncd only once, again on the LAST selected item.
/cd %f
site chmod 777 %p%f
/uncd

[bigstar]

1. Please try this updated build ffxp5.0.0.3805-r4.zip (https://oss.azurewebsites.net/testr/5/ffxp5.0.0.3805-r4.zip)

I changed the way linked scripts are executed with [] to address this problem, I will need to do additional testing to make sure that this doesn't introduce any unexpected issues.

2. I noticed that one of the variables used with /delay was signed, it should of been unsigned and I bet this resulted in a int overflow. I have updated this in the build above.

3. For now the the command X of total is wrong, this has to do with the way the flow of the engine was changed. I will correct this once I am satisfied that everything else is working as correctly, each change I make effects this.

4. When the command engine was run there is no success/failure result that could be passed to the transfer queue (atleast until now). I've added this ability so if any command is enqueued and that command fails then the whole queue is stopped, this makes the most sense especially when any following command may rely on the one before it being executed successfully.

[DayCuts]

Testing with 3805-r4

1. Repeating Example 3 - everything now seems to function in the expected order. (sorting was by date so TESTING2 was first in the list)
[10:05:16] [L] [CMD] -- Enqueue A
[10:05:16] [L] A
[10:05:17] [L] 500 'A': Command not understood.
[10:05:17] [L] [CMD] /cd /tools/bin/TESTING2
[10:05:17] [L] CWD /tools/bin/TESTING2
[10:05:17] [L] 250 CWD command successful.
[10:05:17] [L] PWD
[10:05:18] [L] 257 "/tools/bin/TESTING2" is current directory.
[10:05:18] [L] STAT -l
[10:05:18] [L] List Complete: 186 bytes in 0.92 seconds (0.2 KB/s)
[10:05:18] [L] [CMD] CMD2
[10:05:19] [L] [CMD] -- Enqueue B
[10:05:19] [L] B
[10:05:19] [L] 500 'B': Command not understood.
[10:05:19] [L] [CMD] /cd /tools/bin/TESTING
[10:05:19] [L] CWD /tools/bin/TESTING
[10:05:19] [L] 250 CWD command successful.
[10:05:19] [L] PWD
[10:05:20] [L] 257 "/tools/bin/TESTING" is current directory.
[10:05:20] [L] STAT -l
[10:05:21] [L] List Complete: 549 bytes in 0.94 seconds (0.5 KB/s)
[10:05:21] [L] [CMD] CMD2
[10:05:21] [L] [CMD] -- Enqueue B
[10:05:21] [L] B
[10:05:21] [L] 500 'B': Command not understood.
[10:05:21] [L] [CMD] /uncd
[10:05:21] [L] CWD /tools/bin
[10:05:21] [L] 250 CWD command successful.
[10:05:21] [L] PWD
[10:05:22] [L] 257 "/tools/bin" is current directory.
[10:05:22] [L] STAT -l
[10:05:23] [L] List Complete: 7,618 bytes in 0.92 seconds (7.4 KB/s)

2. I can confirm that /delay now functions correctly.

4. Great, this seems to function as intended.

Regarding the chmod examples in previous post, these all still function as described in that post.

I tested some other commands as well and everything so far seemed to function correctly, other than those impacted by the %f handing/etc (re chmod examples)

[bigstar]

Regarding the chmod examples in previous post, these all still function as described in that post.

I tested some other commands as well and everything so far seemed to function correctly, other than those impacted by the %f handing/etc (re chmod examples) After reviewing all my of recent changes I discovered a mistake where a variable was passed by reference to a sub-routine and modified; when should of been passed by value so that the original value outside of the sub-routine was preserved.

This mistake resulted in only some of the selected items being evaluated when the %f token was used.

As you discovered using the /enqueue command within a command block results in all of the following commands within the block being enqueued as well, I'm fairly sure that this was not the intended behavior but the /enqueue command is somewhat mysterious. It was originally added so that the user could add operations to the queue via the raw command line and then review the result to verify the output is as expected.

From another post I found this example
/enqueue /ren "%p%f" "%p$replacerx("%f", "^([0-9]+[\s][-]) (.*)\.(.*)$","\2.\3")"

I found a race condition in the hash related commands (/checksfv, /crc32../sha512, etc) that perform threaded operations.

There was also a problem that caused loading commands from within groups to sometimes fail [\group\sub-group\command] depending on the order and layout of the command groups.

Here's the latest revision with all my current fixes
ffxp5.0.0.3805-r5.zip (https://oss.azurewebsites.net/testr/5/ffxp5.0.0.3805-r5.zip)

I think the only issue remaining is that the command progress counter might be wrong under some conditions.

[DayCuts]

Testing with 3805-r5

Almost everything seems to be acting as expected. As you say Y count for command progress isn't quite right (seems to count an extra increment for each iteration of a command group).

Only the very last chmod example acts strangely. I know its not a realistic command (or the way it should be done) just one of the test cases I was using. As you can see the order is lost when there are no surrounding {}. It seems to iterate over each line containing %f individually but not in the order they are presented.

/cd %f
site chmod 777 %p%f
/uncd

[20:45:25] [L] site chmod 777 /tools/bin/TESTING2
[20:45:25] [L] 200 CHMOD command successful.
[20:45:25] [L] site chmod 777 /tools/bin/fibseq/TESTING
[20:45:26] [L] 200 CHMOD command successful.
[20:45:26] [L] [CMD] /cd TESTING2
[20:45:26] [L] CWD TESTING2
[20:45:26] [L] 250 CWD command successful.
[20:45:26] [L] PWD
[20:45:26] [L] 257 "/tools/bin/TESTING2" is current directory.
[20:45:26] [L] STAT -l
[20:45:27] [L] List Complete: 282 bytes in 0.86 seconds (0.3 KB/s)
[20:45:27] [L] [CMD] /cd TESTING
[20:45:27] [L] CWD TESTING
[20:45:27] [L] 550 TESTING: No such file or directory.
[20:45:27] [L] CWD /tools/bin/TESTING2/TESTING
[20:45:28] [L] 550 /tools/bin/TESTING2/TESTING: No such file or directory.

[DayCuts]

Again tweaking my cleanup command I have noticed some things performed in an odd order, take the following example.

#CMD
{
/enqueue site chmod 777 %p%f
/cd %p%f
/select -f rx: ...
/delete selected
/enqueue mkd %p%f/Subdir
/select -f rx: ...
[SUBCMD]
}
/uncd

#SUBCMD
/enqueue /ren "%p%f" "%pSubdir/%f"

#SUBCMD V2
/enqueue mkd %p%f/Subdir
{
/enqueue /ren "%p%f" "%pSubdir/%f"
}


All of the /enqueue lines in the main {} block of CMD are performed first. Moving the /enqueue mkd to the sub command (re SUBCMD V2) is not good of course because then the wrong %f set is referenced. This just demonstrates that some things are still not performed in the logical order that they are written in. It causes the /enqueue commands in the above example to be iterated individually as if they were not inside the command block {} at all. This seems to be at odds with the purpose and function of a command block. Of course in the full command I am using, the /enqueue site chmod is intentionally outside of the bulk command block {} as I want it iterated/performed first, but this should be the users choice and I can see situations where not respecting its exact place inside the command block and iterating it as part of that group of commands could cause problems.

Also I noticed if I use the internal /mkd command instead of mkd as a raw command it following /enqueue it fails. A little testing confirms that the /mkd is non-functional under any circumstances. It is not being recognized by the parser at all and is sent as a raw command. Obviously this results in an error.

[bigstar]

I had mentioned this once or twice awhile back but even I neglected to notice my own mistake.

The script engine was using a priority level of each command being executed, this level determined whether or not the command could be executed safely now or if it needed to be executed later, with 3 stacks (input, processing, execution stack) commands are moved from one to the other, some specific commands skip the processing stack all together.

Once the execution stack contains an item certain processing can't occur until the execution stack is emptied.

An invalid command or a few others such as /enqueue, /queue selected, RAW command would yield until the execution stack was emptied and this caused the out of order sequences we both have noticed.

While this is clear to me now, it was not entirely obvious to me while trying to solve these problems.

This way makes everything overly complicated (as well as unclear and out of order); after several hours of testing I can't recall exactly why it was even done this way, I am fairly sure it had something to do with queuing the selected items and then deleting them but not entirely sure.

I am changing this and eliminating the priority based ordering for now.

With this change everything now appears to work exactly as intended based on all my test scripts as well as the ones you provided.

I am also changing the way the script handles failures so that if a failure occurs within a command block or included script then the failure only stops execution within that scope, allowing the remaining script to finish.


/cd /System/
/selectall -d
{
/cd %p%f
site chmod 777 %p%f
}
/uncd
CWD /System/Config.Msi
250 CWD command successful.
site chmod 777 /System/Config.Msi
...
CWD /System/Documents and Settings
550 CWD failed. No permission.
< /cd failed as a result site chmod is not performed >
...
CWD /System/DvTape
250 CWD command successful.
site chmod 777 /System/DvTape

I am performing a massive code cleanup of the script engine and I should have an updated build for you to test later tonight or tomorrow.

[bigstar]

With the new simplified execution order of the command engine I have successfully implemented nested command blocks

So now you can do something like this if you wanted to without needing to call an external script

/selectall -d
{
/echo [BLOCK 1] %p%f
/cd %p%f
/selectall -d
{
/echo [BLOCK 2] %p%f
/cd %p%f
/selectall -d
{
/echo [BLOCK 3] %p%f
/cd %p%f
}
}
}
/uncd

[bigstar]

Here's the new build ffxp5.0.0.3805-r6.zip (https://oss.azurewebsites.net/testr/5/ffxp5.0.0.3805-r6.zip)

[DayCuts]

Ah, I was not aware of the intentional priority handling but that certainly explains a lot.

[DayCuts]

Testing 3805-r6... everything that had previously been discussed seems to work perfectly, I was unable to find or invent a command that broke things.

Playing with the new nested command blocks I did notice that space-indenting breaks [Name of command] and most likely [path to cmd file]. Presumably due to the internal check looking at chr 1 to identify these special commands. With new nested command block support indenting with spaces will be more common (at least for me) since it improves readability. Everything else functioned correctly regardless of space indenting.

Minor suggestion, the ctrl+a (select all) keyboard shortcut is not mapped within the custom command edit box, the other common keyboard shortcuts are.

[bigstar]

That's awesome :D


Playing with the new nested command blocks I did notice that space-indenting breaks [Name of command] and most likely [path to cmd file]. Presumably due to the internal check looking at chr 1 to identify these special commands. With new nested command block support indenting with spaces will be more common (at least for me) since it improves readability. Everything else functioned correctly regardless of space indenting.
This makes sense and I will change this so that leading spaces are no longer an issue. You can also use tabs as well.

Minor suggestion, the ctrl+a (select all) keyboard shortcut is not mapped within the custom command edit box, the other common keyboard shortcuts are. I will definitely add this.