Go Back   FlashFXP Forums > > > >

NeoXed's scripts nxAutoNuke, nxTools, etc.

Reply
 
Thread Tools Rate Thread Display Modes
Old 09-01-2007, 03:56 PM   #1
neoxed
Too much time...
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: May 2003
Posts: 1,326
Default nxMyDB - Back from the Grave

After some encouragement from a certain user in the ioFTPD community, I decided to pick up my database module project again. Over the last few days I have rewritten roughly five thousand lines of code. There is still work to be done, but it's partially functional at the moment.

Completed:
  • MySQL connection support (with SSL and compression optional)
  • Persistent connection pool support for high volume servers
  • Write-only support for the user-file and group-file backend (user/group files are updated)
  • Reads default.user and default.group for user and group creation
  • Module operations: create, delete, rename, lock, unlock, open, write, and close.
  • Timer to synchronize the local user and group cache.
  • Smart-renames for users and groups when syncing.
  • Group rename support.
  • Spin-able locking mechanism.

Unfinished:
  • Higher precision clock for the updated times.

Updates:
  • Sep 1, 2007 - Post created.
  • Sep 2, 2007 - Implemented the create and write operations.
  • Sep 3, 2007 - Implemented the open operation for groups.
  • Sep 7, 2007 - Implemented the open operation for users.
  • Sep 8, 2007 - Alpha 1 released.
  • Sep 22, 2007 - Group synchronization.
  • Sep 23, 2007 - User synchronization.
  • Sep 24, 2007 - Event handler for start-up and shut-down events.
  • Sep 25, 2007 - Alpha 2 released.
  • Sep 30, 2007 - Independent logging system.
  • Oct 3, 2007 - Spin-lock based locking.
  • Oct 4, 2007 - Group rename support.

Last edited by neoxed; 10-05-2007 at 12:19 AM.
neoxed is offline   Reply With Quote
Old 09-01-2007, 04:04 PM   #2
neoxed
Too much time...
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: May 2003
Posts: 1,326
Default

So to answer the next big question: when will it be released?

My plan is to release an alpha version in the next week, but not feature-complete. As people assist me with testing, I will implement the more complex features in following alpha releases (refresh and smart-renames).

Alpha 1 - Single Site Support
The first alpha release will not synchronize the local user and group cache (the refresh feature of ioShareDb). This will take a bit of work to implement properly.

Alpha 2 - Multi-Site Support
The second alpha release will synchronize the local user and group cache.

Alpha 3 - Improved Locking
The third alpha release will implement a new locking mechanism which will greatly reduce the number of "user locking failed" messages.

Last edited by neoxed; 09-25-2007 at 07:38 PM.
neoxed is offline   Reply With Quote
Old 09-01-2007, 04:08 PM   #3
neoxed
Too much time...
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: May 2003
Posts: 1,326
Default

During the alpha phase, the database schema will most likely change between releases. So DO NOT use the alpha versions in a production environment - they are for testing only (a preview).

Wait for a beta or final release before you plan on rolling the module out in a production environment.
neoxed is offline   Reply With Quote
Old 09-02-2007, 02:56 AM   #4
hukker
Senior Member
FlashFXP Registered User
ioFTPD Foundation User
 
Join Date: Jun 2004
Posts: 165
Default

greate news man!... Iam so excited about this... this is prolly THEEE request of all times so, Iam really glad you are taking the time for it.

greate
hukker is offline   Reply With Quote
Old 09-08-2007, 02:51 PM   #5
neoxed
Too much time...
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: May 2003
Posts: 1,326
Default

I finished all basic module operations, the database is able to read/write and open/close users and groups stored on the database. It's mostly functional now at this point.

All that's left before I'll have alpha-1 ready is group renaming and a bit more testing.
neoxed is offline   Reply With Quote
Old 09-08-2007, 04:28 PM   #6
neoxed
Too much time...
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: May 2003
Posts: 1,326
Default

For those interested in the amount of code written so far:

Code:
wc -l include\* source\*
     82 include\backends.h
    147 include\base.h
     39 include\condvar.h
    121 include\database.h
     26 include\debug.h
    142 include\pool.h
    107 include\proctable.h
    247 include\queue.h
    154 source\condvar.c
    809 source\database.c
    113 source\debug.c
     54 source\dllentry.c
    392 source\group.c
    530 source\groupdb.c
    292 source\groupfile.c
    695 source\pool.c
     93 source\proctable.c
    395 source\user.c
   1779 source\userdb.c
    293 source\userfile.c
   6510 total
neoxed is offline   Reply With Quote
Old 09-08-2007, 07:12 PM   #7
oldhouse80
Member
ioFTPD Foundation User
 
Join Date: May 2003
Posts: 37
Default

Thx very much for your effort. Keep up the good work!
I will give it a try as soon as I have some spare time.
oldhouse80 is offline   Reply With Quote
Old 09-09-2007, 04:22 AM   #8
peep
Senior Member
FlashFXP Scripter
ioFTPD Foundation User
 
Join Date: Sep 2003
Posts: 132
Default

Nice one neoxed! Great to see people who are able to keep things up..
peep is offline   Reply With Quote
Old 09-24-2007, 12:12 AM   #9
neoxed
Too much time...
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: May 2003
Posts: 1,326
Default

Well it's been a while since you've heard from...it's because I've been busy coding. I implemented and tested different designs for syncing with the database and I've finally come up with one I like.

I've created a "changes" table that will be used for sites to see which users/groups have been created, renamed or deleted. The changes table is processed incrementally on configurable time intervals. During start-up a full synchronization is done since the module has no idea how long ioFTPD was offline.

The group synchronization is done, now I just need to do some copy and pasting to finish user synchronization (basically the same thing, just different structures).

Code:
E:\ioFTPD\modules>wc -l include\* source\*
     31 include\array.h
    128 include\backends.h
    161 include\base.h
     39 include\condvar.h
     92 include\database.h
     54 include\debug.h
     36 include\idlist.h
     39 include\namelist.h
    142 include\pool.h
    119 include\proctable.h
    585 include\queue.h
    189 source\array.c
    157 source\condvar.c
    956 source\database.c
    410 source\debug.c
     54 source\dllentry.c
    524 source\group.c
    691 source\groupdb.c
    612 source\groupdbsync.c
    292 source\groupfile.c
    103 source\idlist.c
    356 source\namelist.c
    696 source\pool.c
     99 source\proctable.c
    527 source\user.c
   1940 source\userdb.c
     30 source\userdbsync.c
    293 source\userfile.c
   9355 total

Last edited by neoxed; 09-24-2007 at 12:30 AM.
neoxed is offline   Reply With Quote
Old 10-05-2007, 12:06 AM   #10
neoxed
Too much time...
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: May 2003
Posts: 1,326
Default

Well I'm glad to say nxMyDB is near completion. I've finished implementing all the basic features required to make it a functional database module. The database table format has stabilized as well, the only changes for the next release is the addition of two stored procedures - no changes to the tables themselves.

I have implemented a few new things for the next release:
  • A new logging system to keep track of errors (no need for DebugView).
  • Spin-lock based locking, which greatly reduces the "user locking failed" messages.
  • Support for atomic group rename operations.

The change log for the next alpha release (v0.3.0), so far:

Code:
nxMyDB v0.3.0 (not released):
  NEW: Added configuration option "Log_Level" to control the log verbosity.
  NEW: Group rename operations are now implemented.
  NEW: Logging system to log errors, warnings, and information to a dedicated log file.
  NEW: Spin-lock based locking implemented using stored procedures, reduces failed locks.
  CHG: Renamed configuration option "SyncFirst" to "Sync_First".
  CHG: Renamed configuration option "SyncInterval" to "Sync_Interval".
  CHG: Requires MySQL Server v5.0.19, or newer.
  FIX: User rename operations failed during sync, since the new name was not present in the changes table.
The code statistics (as usual ):
Code:
E:\ioFTPD\modules>wc -l include\* source\*
     43 include\alloc.h
     31 include\array.h
    135 include\backends.h
    163 include\base.h
     40 include\condvar.h
     95 include\database.h
    110 include\logging.h
     39 include\namelist.h
    142 include\pool.h
    138 include\proctable.h
    585 include\queue.h
    315 source\alloc.c
    189 source\array.c
    160 source\condvar.c
   1041 source\database.c
     45 source\dllentry.c
    154 source\event.c
    528 source\group.c
    919 source\groupdb.c
    612 source\groupdbsync.c
    292 source\groupfile.c
    107 source\log.c
    115 source\logdebug.c
    299 source\logfile.c
    356 source\namelist.c
    694 source\pool.c
    104 source\proctable.c
    531 source\user.c
   1983 source\userdb.c
    757 source\userdbsync.c
    293 source\userfile.c
  11015 total
If no bugs come up in my testing, expect a release this weekend.
neoxed is offline   Reply With Quote
Old 10-05-2007, 11:25 AM   #11
ArtX
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Join Date: Jan 2004
Posts: 301
Default

good work
ArtX is offline   Reply With Quote
Old 10-05-2007, 09:58 PM   #12
monk-
Member
 
Join Date: Oct 2007
Posts: 32
Default

very nice work,
this is the new sharedb!
it runs much smoother already,
i hope Yill can fix the mod system on 6.x...
monk- is offline   Reply With Quote
Reply

Tags
2007, group, sep, support, user

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 12:32 PM.

Parts of this site powered by vBulletin Mods & Addons from DragonByte Technologies Ltd. (Details)