Integration of "date and time" at Download in xferstats

[MONGi]

Hello community,

i use "xferstats" from Mouton to get a statistic.
All in all everything works as it should.

However, i would like a function in the script; Date- and timestamp for each download at the beginning of the line.
If it possible, someone can help me with the extension-code for the script "xferstats.itcl"?
My TCL-knowledge are very poor...

I´ve try to read the systemtime and write the value in the outputfile. Ok, thats fine but the systemtime updated always all lines. The old time entrys are overwrite with actual time.
Something are wrong...

Thank you

[MONGi]

Hello,

a question about the code.

I try to read the timestamp from "xferlog" and write it to "xferlog.result" with follow lines:

set date [clock seconds]   //integrated at top from the script
set date [clock format $date -format %Y.%m.%d]   //integrated at top from the script
set timestamp [lindex $log end-14]
puts $ofp "[lindex $date $timestamp]  |  User: [lindex $users $i]  |  Release: $rlsn"

The "xferlog" line looks like e.g.:

Fri May 23 22:12:00 2014 2 my.dyn-dns.org 857071 /path/DIRNAME/picture.jpg b _ o r TestAccount ftp 1 * l

Now "xferstats.itcl" should write the timestamp in "xferlog.result":

2014.05.23  |  User: TestAccount  |  Release: DIRNAME

The date i`ve set is the systemdate, not fine but for the begining good enough.

But i`m missing the timestamp behind the date...whats wrong? Using a wrong format?

Can someone give me an advice?

Greetings

[Yil]

Umm, your formatting the date variable as %Y.%m.%d using the current time. You'll want to add hours and min to the format I'm also not sure what the lindex's in the puts line are accomplishing...

[Yil]

I'd suggest going to Tcl Developer Site and see what documentation they offer online, and I'd grab plain old TCL from www.activestate.com/activetcl/downloads and install that and browse the documentation locally if that's easier for you. I use komodo from activestate but the docs for it are the same as you get with plain TCL from them.

[MONGi]

Hello Yil,

thank you very much for your replay...have been eagerly waiting for

Originally Posted by Yil

Umm, your formatting the date variable as %Y.%m.%d using the current time. You'll want to add hours and min to the format I'm also not sure what the lindex's in the puts line are accomplishing...

Yes, it`s a solution to take the date and time from my system, but i would like read the correct time from the Downloads out from the file "xferlog".
So I try (see the second post) to read the timestamp out from the "xferlog" file. Will take the same format as Mouton read the user(s) or Count and this is working.
I´m not sure and I have nothing found in the worldwideweb, if I can read out ":" with the command lindex or need a other format e.g. string.
If you're interested to see in my entire script, I can send you. I´m not sure if I can post it here completely.

Originally Posted by Yil

I'd suggest going to Tcl Developer Site and see what documentation they offer online, and I'd grab plain old TCL from www.activestate.com/activetcl/downloads and install that and browse the documentation locally if that's easier for you. I use komodo from activestate but the docs for it are the same as you get with plain TCL from them.

I googled for information's about TCL commands and there functions.

[Yil]

xferlog format: day-of-week(3) name-of-month(3) day-of-month(2) hour(2):minute(2):second(2) year(4) duration-in-seconds(rounded to nearest) hostname(or ip or "[hidden]" if Hide_Xfer_Host enabled) size-in-bytes filename(TRICKY!) encoding(b or a) "_" direction(o or i) "r" username "ftp 1" ident "l"

To grab the date use [clock scan -format] if you can find a -format that works. Else just use the [scan] tcl command which works like sscanf from C to grab the fields and put them into an order that [clock scan] likes better with [format]. Care must be taken with the xferlog entries from the filename on so use [lrange line end-9 end] or something to grab the tail end of the line and parse that apart if needed.

Now you have the filename to deal with. I'd start with using lreplace to chop off the front, then use lreplace with the end qualifiers to strip off the end and what's left should be the filename. The reason you need to be careful is the filename can contains spaces, brackets, curly-braces, etc and thus you can't trust TCL to think it's a single entry in the list. That may work, but I'd want to test it a bunch including things like multiple spaces in a row. If that trips it up you can [lrange] the front/end of the list, then use [string] to get their lengths and use some math and [string] to extract the filename or chop chars from it as needed such that TCL never interprets the filename as a list. I know that works, but not sure if you need to do that or not...

[MONGi]

Hello,

first a thanks to Yil for the support

The script are extended with the function I want.

If anyone can use it:

# settings

# xferlog location
set xfs(xferlog) "../logs/xferlog"

# output file location
set xfs(outfile) "../logs/xferlog.results"

# directory containing any of those will be ignored
# Case-sensitive
set xfs(skip)[list "_requests" "_incomplete"]

# part of the directory to strip
# Not case-sensitive
set xfs(strip)[list "cd1" "cd2" "cd3" "cd4" "cd5" "cd6" "cd7" "cd8" "cd9" "disc1" "disc2" "disc3" "extra" "covers" "samples" "sample" "vobsubs" "subtitles" "subtitle" "subs" "subpack" "codec"]

# display upload stats too ?
set xfs(upstats) false

# output format (csv,html,text)
set xfs(output) "text"

# only display release name ?
set xfs(rlsn_only) false

########################################################
########################################################
########################################################

proc xferstats {} {
    global xfs

    # init
    set i 0
    set rls ""
    set users ""
    set count ""
    set datetime ""

    set ifp [open $xfs(xferlog) r]
    while {![eof $ifp]} {
        set line [gets $ifp]
        regsub {  } $line { } line
        set log [split $line { }]
        incr i

        set direction [lindex $log end-6]
        set user [lindex $log end-4]
        set date [lrange $log 3 3]
        set date [clock format [clock scan $date] -format {%Y.%m.%d - %H:%M:%S}]	
        set filen [lrange $log 8 end-9]
        set filen [string range $filen 0 [expr [string last / $filen]-1]]

        # Check for skip
        set j 0
        set skipping false
        while {$j<[llength $xfs(skip)]} {
            if {[string match "*[lindex $xfs(skip) $j]*" $filen]} {
                set skipping true
                break
            }
            incr j
        }
        if {$skipping} { continue }

        # Check for stripping
        set j 0
        while {$j<[llength $xfs(strip)]} {
            regsub -nocase "/[lindex $xfs(strip) $j]" $filen {} filen
            incr j
        }

        regsub -all -- "\\\[" $filen "<" filen
        regsub -all -- "\\\]" $filen ">" filen

        set pos [lsearch $rls "$filen $direction"]
        if {$pos==-1} {
            # Not found - add it
            set rls [linsert $rls end "$filen $direction"]
            set users [linsert $users end "$user"]
            set count [linsert $count end 1]
            set datetime [linsert $datetime end "$date"]		
        } else {
            set users_test [lindex $users $pos]
            if {![string match "*$user*" $users_test]} {
                set users [lreplace $users $pos $pos "$users_test $user"]
                set count [lreplace $count $pos $pos "[expr [lindex $count $pos]+1]"]
            }
        }

        set size [lindex $log 7]

        if {$i%1000==0} {
            iputs -nobuffer "[llength $rls]"
        }
    }
    close $ifp

    iputs -nobuffer "[llength $rls] releases found (upload and download)"

    set i 0
    set ofp [open $xfs(outfile) w]	
    if {$xfs(output) == "html"} {
        puts $ofp "<table>"
    }
    while {$i<[llength $rls]} {
        set rlsn [lindex $rls $i]
        regsub -all -- "<" [lindex $rls $i] "\[" rlsn
        regsub -all -- ">" $rlsn "\]" rlsn
        regsub -all -- "\[{}\]" $rlsn "" rlsn
        set arr [split $rlsn]
        set rlsn [lindex $arr 0]
        if {$xfs(rlsn_only)} {
            set rlsn [string range $rlsn [expr [string last / $rlsn]+1] end]
        }
        set direction [lindex $arr 1]
        if {$rlsn!=""} {
            if {$xfs(output) == "text"} {
                if {$direction=="o"} {
                    puts $ofp "[lindex $datetime $i]  |  User: [lindex $users $i]  |  Count: [lindex $count $i]  |  Release: $rlsn"
                } elseif {$xfs(upstats)} {
                    puts $ofp "[lindex $datetime $i]  |  User: [lindex $users $i]  |  Count: [lindex $count $i] | Release: $rlsn"
                }
            } elseif {$xfs(output) == "csv"} {
                if {$direction=="o" && $xfs(upstats)} {
                    puts $ofp "download,$rlsn,[lindex $count $i],[lindex $users $i]"
                } elseif {$direction=="o"} {
                    puts $ofp "$rlsn,[lindex $count $i],[lindex $users $i]"
                } elseif {$xfs(upstats)} {
                    puts $ofp "upload,$rlsn,[lindex $count $i],[lindex $users $i]"
                }
            } elseif {$xfs(output) == "html"} {
                if {$direction=="o" && $xfs(upstats)} {
                    puts $ofp "<tr><td>$rlsn</td><td>download</td><td>[lindex $count $i]</td><td>[lindex $users $i]</td></tr>"
                } elseif {$direction=="o"} {
                    puts $ofp "<tr><td>$rlsn</td><td>[lindex $count $i]</td><td>[lindex $users $i]</td></tr>"
                } elseif {$xfs(upstats)} {
                    puts $ofp "<tr><td>$rlsn</td><td>upload</td><td>[lindex $count $i]</td><td>[lindex $users $i]</td></tr>"
                }
            }
        }
        incr i
    }
    if {$xfs(output) == "html"} {
        puts $ofp "</table>"
    }
    close $ofp

    global ioerror
    set ioerror 0
}

iputs -nobuffer "XFERStats started..."
xferstats