development weekly status

[_panic_]

as promised, here is the first of my weekly status reports.

i'm still getting fully ramped into development. in fact, i haven't been able to compile ioftpd yet, as i'm still missing headers for php and tcl. i'll locate these this week. that hasn't stopped me from reading and learning about the code, which at this point is as important as compiling it, as i'm still learning about major subsystems of ioftpd.

i've begun the process of documenting the UserModule interface. i won't be able to publish anything until i have ioftpd compiling, but i have the skeleton for a basic UserModule dll that i'll publish as source code.

i've also been organizing feature and bug requests, and trying to put that data together into a coherent list. here is my first stab at it:


source code for a sample UserModule, GroupModule, EventModule library.
documentation for the UserModule, GroupModule, and EventModule interfaces.
full ssl data (ioftpd<=>ioftpd) support.
ability to access and modify user/group/vfs settings from outside of ioftpd.
support for extended/custom user/group data fields.
support for extended/custom .ioFTPD data.
scripting as a module, so we can add lua, and php, as well as use the existing tcl interface.
script access to all builtin verbs, particularly LIST.
bugfixes to the chattr command.
bugfixes to the OnLogout/OnConnectionClose event handlers.


the most important things to me on this list are ssl support, and code/documentation for the module interfaces. i'll be working on bugfixes (including those on this list) at the same time as these larger projects.

hopefully i've summed up the feature requests i've seen at this point, let me know if i missed anything.

please direct all comments to the comment thread (http://www.inicom.net/forum/showthread.php?t=15630)
(thx for the idea EwarWoo)

[EwarWoo]

A quick request, include a link to a seperate thread for comments and keep this thread locked and clean then it remains easier to follow and know whats happening.
If thats OK with yourself also delete this reply please, or move to the linked thread whichever you want :)
And thanks for posting this, its great to be able to keep up with whats happenning and watch it develop. I'm certain I'm not the only one who will appreciate you taking time out to do so.

[_panic_]

as requested above, please direct all comments to the comment thread (http://www.inicom.net/forum/showthread.php?t=15630).

this thread is now locked.

[_panic_]

this is my second weekly status report. my first one is in this thread (http://www.inicom.net/forum/showthread.php?t=15629). it is looking like monday night is a good time for me to post these, so i expect this to be my pattern.

i spent all my ioftpd development time this week getting ioftpd to compile. as i mentioned last week, i didn't get all the stuff i needed from d1 before he disappeared, and recreating his development environment has been very time consuming.

i've gotten hold of the versions of php and tcl used in the current ioftpd beta, got them built, and now can use them to compile ioftpd. so definitely the good news is that i have a version of ioftpd compiling on my local machine!

it isn't currently behaving like the beta release of ioftpd, as i get an error immediately on startup. the error message is "failed to create creds." i haven't even searched the forums yet to see if this has come up before, but i think this is related to a missing library or something else wrong with my build environment. i have some debugging yet to figure this out.

when i *do* get all this worked out, i'm going to put the whole build environment inside a virtual machine. (probably vmware (http://www.vmware.com/), as i have a friend that works there who keeps me informed about what they are up to.) this way no one else will have to go through this mess again in the event of a bus error (i.e. if i get hit by a bus).

so this week i'll be debugging the errors i'm now getting and figuring out what is wrong with my build. my goal is to have a running ioftpd by next monday.

the version of php we're using is kinda old now, so i'd like to upgrade it at some point. i'll do the same for tcl. i'm not in a hurry for either of these, unless i hear otherwise.

i picked up and clarified some of my own notes on some existing bugs that came up this week in the forums. namely the 4k+.ioFTPD bug, and an issue with modifying the user/group files while ioftpd is running. i'm particularly annoyed at the 4k+.ioFTPD bug, so i might go hunting for that to settle into the code (after writing the UserModule docs).

so not very much activity is externally visible this week, but i'm very relieved to finally see ioftpd compiling. it was scary to think i might be missing something i couldn't replace. :(


on another note:

EwarWoo, i *think* your suggestion of keeping a separate comment thread was intended for me to continue posting to my original thread (http://www.inicom.net/forum/showthread.php?t=15629) and thereby keeping my status updates all in one place. alas, i don't have moderator status on the boards, so i can't open closed threads. heh, i don't even know if what i just explained was your intention anyway, so some clarification on what you think works best for keeping track of these reports is appreciated.

[_panic_]

this is my third weekly status report.

i didn't get my goal accomplished this week, which was having a working ioftpd compiled from source. i'm still getting the same error i had last week, "failed to create creds." the problem orginates from the call to |CertFindCertificateInStore|, which ultimately fails. the code doesn't try to determine why, it just aborts.

i've checked both version 0.5.82u, as well what code exists in version 1.0, and they both use the routine the same way. there are only a couple ways this routine can fail, so i just need to figure out which one it is and fix the problem.

in order to do this, i needed to start modifying the code, so i finally got my revision control system set up. i've selected darcs (http://abridgegame.org/darcs/), as i wanted a "distributed revision control system." this means i can carry a full repository around on my laptop as well as on my desktop. i move around a lot, so being able to check in code while on the move was very important. i'll probably talk about darcs (and distributed revision control) more in the future, since it really changes the way you can colaborate writing software.

i also spent some time looking at the 4k+.ioFTPD bug, since it was supposed to be a misuse of reallocating memory (and hence easy to fix). i actually couldn't find it in the directory cache/permission code, leading me to suspect it might actually be fixed in the code i've got. i guess we'll see when i get the next release out.

as usual, please direct any comments to the comment thread (http://www.inicom.net/forum/showthread.php?t=15630).

[_panic_]

this is my fourth weekly status report.

appologies for the late status update, i didn't manage to finish this post last night, and i choose to post it with a fresh mind this morning instead of groggy one last night.

huzzah! i have a working, compiled version of ioftpd. the problem i was having with |CertFindCertificateInStore| is now fixed! i went back through every source code snapshot of ioftpd we have watching the code in question change over time. from this more historical perspective, i got a much better understanding of what the code was trying to accomplish, made a few tweaks, and found myself staring at a login screen!

after doing a little dance and settling down off my excitement, i'm now preparing to make a new beta release of ioftpd. this is a "see where we're at release." to assess the completeness of the code and to ascertain which bugs have and have not been fixed. particularly, i think this code should have the 4k+.ioFTPD bug issue fixed.

stay tuned later this week for that release. i'm extremely excited to deliver it, and ecstatic to have something to build on going forward.

as usual, please direct any comments to the comment thread (http://www.inicom.net/forum/showthread.php?t=15630).

[_panic_]

this is my fifth weekly status report.

apologies for the late status update, monday night was a holiday. the same thing will happen next monday too (i'm going to a birthday party), so these updates might move to tuesday.

i was unexpectedly sucked into holiday planning last week, which delayed me in getting ioftpd prepared for release. i've compiled a new registered version, 5.8.6r, which i'm uploading right now.

we'll have an unregistered version available in a week or two, after some testing from registered users.

as usual, please direct any comments to the comment thread (http://www.inicom.net/forum/showthread.php?t=15630).

[_panic_]

this is my sixth weekly status report.

i must admit to being fairly frightened releasing a new version of ioFTPD last week. i didn't really know how much the source code i have deviated from the 5.8.5r release. i was prepared for massive numbers of reported problems over all areas of the codebase.

the reality turned out much better than that, and it appears most users are able to use 5.8.6r without problems. doing the release did get additional bug reports to me, which i will summarize below:


"site who" now crashes the server, instead of being unimplemented like it used to be.
in some cases (all cases?) an empty php.ini file should be included.
the !uptime command is not working like it used to.
the ioftpd icon reverted to the default command-line icon.


i think i got everything, but if i missed an issue that specifically came up with the 5.8.6r release, let me know.

one major issue that has come up is that the 5.8.6r release seems to be incompatibly with any compiled module. these are not the normal tcl scripts, but the dll files specified in the [Modules] section of ioftpd.ini. it could be that sharedb is the only actual program using this interface.

documenting this interface is high on my priority list, so it looks like i'll both get that stuff done as well as provide support to those of you working on modules using this interface.

also this week, a security issue was published concerning ioftpd. you can read about the issue at this site (http://www.security.nnov.ru/Fnews399.html). basically ioftpd uses different error messages for an unknown user and an unknown password. you can exploit this to determine whether a username is a valid account on the server. modern server design calls for this information to be hidden.

i actually noticed this issue writing some regression tests for ioftpd. i consider the behaviour poor form regardless of the security implications, expect a fix for this in the next release.

please direct all comments to the comment thread (http://www.inicom.net/forum/showthread.php?t=15630).

[_panic_]

this is my seventh weekly status report.

now that i've put most of the major setup hurdles behind me, i'm finally getting the time to dive into the code. there is a lot of stuff for me to learn. instead of trying to understand all of the codebase at once, i just work on exploring in the area around whatever change i'm making.

this does make modifying the code a slower process here in the beginning, but i get to feel much more comfortable with it and i even find a few surprises! (any two developers are going to have different ways of solving the same problem, and some of darkone's idioms are really fun--i've never seen them before.)

i'm currently working on moving all of the username/password error checking code until after the PASS command. it isn't as simple as moving code around, as there are scope inconsistencies that need to be fixed one by one as i discover them. (by crashing the program...) all of the slowness here is really just me exploring the code. there is some really fun stuff in this program.

please direct all comments to the comment thread (http://www.inicom.net/forum/showthread.php?t=15630).

[_panic_]

this is my eighth weekly status report.

i've got ioFTPD taken apart right now as i work on both a bug fix (http://www.inicom.net/forum/showthread.php?p=126507#post126507) and continue to rework the login code.

i didn't get much done this week, as my primary time for working on ioFTPD is the weekend, and this weekend i was racing my bike in the el tour de tucson (http://www.pbaa.com/!ETT/eltourmap.htm). this is a 109 mile bicycle race in tucson arizona. i drove out on friday, did the race on saturday, and drove back on sunday. this left no time for working on ioFTPD. :(

i did complete the race in 6 hours 30 minutes. this is the first time i've finished a century bike race. the time was good enough to get me a "silver," though if i can shave 30 minutes off my time and get under 6 hours, i can get a gold next year!

i might be in a similiar spot this weekend with the thanksgiving holiday, i'll just have to see how demanding my schedule is. appologies that real life interrupted, sometimes my todo list gets ahead of me.

please direct all comments to the comment thread (http://www.inicom.net/forum/showthread.php?t=15630).

[_panic_]

this is my ninth weekly status report.

this week, i fixed the account information leak pointed out here (http://www.security.nnov.ru/Fnews399.html), which i first talked about in my wk5 development status report (http://www.inicom.net/forum/showpost.php?p=126054&postcount=6). the code now accepts any username given, and waits for a PASS command before echoing an error message, which is always the same regardless of the nature of the error.

none of the logging information is changed, of course.

i'm going to work now on fixing site commands. i'll probably do another release once that issue is resolved.

please direct all comments to the comment thread (http://www.inicom.net/forum/showthread.php?t=15630).

[_panic_]

hello everyone, i'm very sorry for dropping offline. real life took a very unexpected turn and i've been dealing with that full time.

but i thought i'd pop in and leave a quick shout out. i will be right back in the swing of things very soon. (sometime in the next couple or few weeks.) i'll try to be on more often than i have been, but things are still settling out for me.

if everything works out, i'll have some very good news soon!

[_panic_]

if everything works out, i'll have some very good news soon!

last monday, january 23rd 2006, i gave two weeks notice at my current job. i will be joining inicom full time starting february 6th. this move has been in planning for a long time, but i have really not been able to talk about it much until the move was official.

both inicom and i have wanted this process to be faster. we both had timelines and issues we didn't have much control over, and this caused the process to stretch out into february this year. in fact, i was able to leave my job over a month sooner than we originally planned for, and inicom has been able to accomodate the accelerated schedule.

i'm going to be busy most of this week finishing up at my current job (my last day is february 3rd) and making the transition to working at inicom. i expect my first week at inicom to be filled with administration, office set up, and other non-development work.

after that point i'll be talking about what we're going to do going forward, including the already published feature list; timelines, and a return to weekly status updates.

particularly i'll have a much more regular schedule, and with it more time to work on ioftpd. i'm ecstatic that i'm finally about to make this transition. there have been so many things to get done that it feels like a major milestone to me.

working at inicom is going to be a new challenge with a new set of demands. a lot in my life is going to be changing around my new job, and i've got to work through these changes before i can fully ramp into development. please give me the next two weeks to get these things done. i'll be really glad to have them finished so i can much better focus on ioftpd.

talk to you soon!

[_panic_]

we've rented some new office space in downtown albuquerque (http://local.google.com/local?f=q&hl=en&q=500+4th+st+nw+ste+210,+albuquerque+nm,+87102&ll=35.089234,-106.650566&spn=0.001172,0.003819&t=k), and i reported here this morning.

we still don't have everything set up: my computer is plugged into an extension cord that is running across the room to one of the few working power outlets, and while our wireless network is running we haven't run any ethernet cable yet.

i've spent part of the day getting my computer set up and the other half getting the office set up. we got a *ton* done on saturday, but it will still be a few half days of work to get everything together. one of our techs spent the day with his laptop balanced on a stack of cardboard boxes. :)

not all of my equipment is here yet, so i've still got some computers to set up. i'll be bringing things online this week and getting my environment running. i have some very sweet hardware to play with. :)

i'll keep everyone updated as we get moved in!

[_panic_]

i just got back from visiting friends in san jose california. i had an amazing time, and saw some landmarks i've never had the chance to see before.

i walked across the golden gate bridge (http://www.google.com/local?hl=en&lr=&q=&ll=37.817819,-122.475672&spn=0.018103,0.047121&t=k). while i was walking back a helicopter flew under the bridge, only to turn around and fly between the towers at the low point of the suspension cables. maybe that happens every day, but it happened while i was on the bridge!

i made a quick pilgrimage to google (http://www.google.com/local?f=q&hl=en&q=1600+Amphitheatre+Pkwy,+mountain+view,+californi a+94043&ll=37.422841,-122.085035&spn=0.00455,0.015278&t=k), vmware (http://www.google.com/local?f=q&hl=en&q=3305+Hillview+Ave,+palo+alto,+california&ll=37.40713,-122.145167&spn=0.002275,0.007639&t=k) and, apple (http://www.google.com/local?f=q&hl=en&q=1+infinite+loop,+cupertino+california&ll=37.331856,-122.03075&spn=0.004555,0.015278&t=k).

as well, i got to see palo alto research center (http://www.parc.xerox.com/).

i got to eat free food in the vmware cafeteria, which was probably the hilight of my trip. :p

here at work, i've been working on setting up build hosts, revision control servers, wikis, blogs, etc. essentially, building our intranet and knowledge management infrastructure. i still need to spend some time on a bug reporting system and a test infrastructure.

i've been moving the ioftpd code base to these new systems, which hasn't left me much time to work on the actual code. fixing the critical bugs in the newest versions of ioftpd is my highest priority, as i need those releases working for more people.

i stuck my head in the #ioftpd channel of efnet for a bit, but some outstanding firewall problems keep me from staying connected. expect more of me there, at least to lurk. otherwise i'll see everyone around the fourms!

please direct all comments to the comment thread (http://www.inicom.net/forum/showthread.php?t=15630).

[_panic_]

well, i'm feeling w00t.

i finally got ioFTPD compiling again here at inicom. previously, everything was just running off my home computer, and well, i've not been spending that much time at home. :)

when i first started working with ioftpd, i inherited an incomplete build environment. it took me a while to figure out what i needed to do to get the program to compile. alas, i did not take very good notes, and when i came time to set that environment up again, i basically was starting over.

this time, i've got the machines, the wiki, and the support to actually document what i did and make the process repeatable. i've spend the last several days having more than one "ah ha! i remember how i did that!" moments as i remember what to do about these error messages that i *know* i've seen.

while doing this, i updated both tcl and php to the latest versions. i'm also using visual studio 8 now, which involved all sorts of tweaks to the code. mmm progress.

with any luck, my next update will actually be about what i'm doing *to* ioftpd, instead of what i'm doing *with* ioftpd. :p

please direct all comments to the comment thread (http://www.inicom.net/forum/showthread.php?t=15630).

[_panic_]

it has been very busy around here, in the growing pains sort of way. i've got a lot of projects on my plate, and i'm trying to divide my time between all of them. this works much better in theory. :/

on the ioftpd front, i've built the beginning of a regression test framework so i can actually reproduce the problems reported here on the forums. it is complete enough that i've written several basic tests. a test input (baseline) file looks something like this:


;;;
;;; a basic list of the README on the remote server, using the
;;; anonymous account.
;;;
<-- 220 FTP Server ready.
--> USER anonymous
<-- 331 Password required for anonymous.
--> PASS root@127.0.0.1
<-- 230- [0-9]{2}/[0-9]{2}/[0-9]{2}[0-9]{2}:[0-9]{2}:[0-9]{2}
<-- 230-User [a-zA-Z]+ from [a-zA-Z]+, welcome to our FTP server.
<-- 230-
<-- 230-Site layout:
<-- 230-
<-- 230- /home/ - Home directories
<-- 230- /mnt/ - Network mounts
<-- 230- /private/ - Private content
<-- 230- /pub/ - Public content
<-- 230-
<-- 230-
<-- 230-Site activity:
<-- 230-
<-- 230- Users online : [0-9]+
<-- 230- Active transfers: [0-9]+
<-- 230-
<-- 230-Enjoy your stay.
<-- 230 User anonymous logged in.
--> TYPE I
<-- 200 Type set to I.
--> PASV
<-- 227 Entering Passive Mode \([0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+\)
--> RETR README
<-- 150 Opening BINARY mode data connection for README.
this tarball should be uncompressed into the root of the ioFTPD
server you intend to test. it is designed to test the default
ioFTPD configuration, so make sure you haven't made any
configuration changes before running this test suite.
<-- 226 Transfer complete.
--> QUIT
<-- 221 Come back soon!


the test framework sends lines that start with '-->' to the ftp server, and compares what it gets back against the lines starting with '<--' in the baseline.

with the test suite i've got, i can unzip the files in the root of the ftp server and run the tests against the now configured "test" server. overall, it works pretty well. the one *major* problem is that the test suite currently runs on linux, as i had a bunch of tools developed for linux that i could leverage.

i'd like this to be something i can publish to the community so i can add user-contributed tests to the test suite. this would be especially useful when reporting problems... if it can be recreated it can be fixed.

what is the interest in writing tests? is the fact that the test suite runs on linux a problem? let me know what you think of the idea, i'm sure i didn't do it justice with the quick description.

please direct all comments to the comment thread (http://www.inicom.net/forum/showthread.php?t=15630).

[_panic_]

i just released ioFTPD 5.8.8r.

unlike my previous releases, this is a full release of the distribution, binary, library, configuration, site, etc. it is essentially a snapshot of my development tree, minus source code. ;)

i've added quite a bit of documentation to the configuration files, changed the default configuration to make it easier to get started with ioFTPD, and done some basic documentation of the install process, with references back to these forums and the user guide.

all of this is intended to make it easier to get started with ioFTPD. additionally, i'd really like to figure out what is causing site commands to fail. in all my testing i wasn't able to uncover anything, so i'm putting a new baseline out there for further testing.

this release includes several enhancements:


the server side of the testing framework is included in the "site/" directory. the client side isn't available yet, so this is a bit of a teaser/technology preview. it isn't very well documented, but have a look around.
several small issues found while doing testing have been fixed. i don't think anything is user visible.
tcl updated to version 8.4.2.12
php updated to version 4.4.2.2


i'm hoping that the update of both tcl and php will help with the site command problem, and if it doesn't that we can at least develop a test showing what is going on. i'd like to recreate the problem if it doesn't go away with this release. please test!

please direct all comments to the comment thread (http://www.inicom.net/forum/showthread.php?t=15630).