nxMyDB - Back from the Grave

neoxed · 09-01-2007, 03:56 PM

After some encouragement from a certain user in the ioFTPD community, I decided to pick up my database module project again. Over the last few days I have rewritten roughly five thousand lines of code. There is still work to be done, but it's partially functional at the moment.

Completed:

MySQL connection support (with SSL and compression optional)
Persistent connection pool support for high volume servers
Write-only support for the user-file and group-file backend (user/group files are updated)
Reads default.user and default.group for user and group creation
Module operations: create, delete, rename, lock, unlock, open, write, and close.
Timer to synchronize the local user and group cache.
Smart-renames for users and groups when syncing.
Group rename support.
Spin-able locking mechanism.

Unfinished:

Higher precision clock for the updated times.

Updates:

Sep 1, 2007 - Post created.
Sep 2, 2007 - Implemented the create and write operations.
Sep 3, 2007 - Implemented the open operation for groups.
Sep 7, 2007 - Implemented the open operation for users.
Sep 8, 2007 - Alpha 1 released.
Sep 22, 2007 - Group synchronization.
Sep 23, 2007 - User synchronization.
Sep 24, 2007 - Event handler for start-up and shut-down events.
Sep 25, 2007 - Alpha 2 released.
Sep 30, 2007 - Independent logging system.
Oct 3, 2007 - Spin-lock based locking.
Oct 4, 2007 - Group rename support.

neoxed · 09-01-2007, 04:04 PM

So to answer the next big question: when will it be released?

My plan is to release an alpha version in the next week, but not feature-complete. As people assist me with testing, I will implement the more complex features in following alpha releases (refresh and smart-renames).

Alpha 1 - Single Site Support
The first alpha release will not synchronize the local user and group cache (the refresh feature of ioShareDb). This will take a bit of work to implement properly.

Alpha 2 - Multi-Site Support
The second alpha release will synchronize the local user and group cache.

Alpha 3 - Improved Locking
The third alpha release will implement a new locking mechanism which will greatly reduce the number of "user locking failed" messages.

neoxed · 09-01-2007, 04:08 PM

During the alpha phase, the database schema will most likely change between releases. So DO NOT use the alpha versions in a production environment - they are for testing only (a preview).

Wait for a beta or final release before you plan on rolling the module out in a production environment.

hukker · 09-02-2007, 02:56 AM

greate news man!... Iam so excited about this... this is prolly THEEE request of all times so, Iam really glad you are taking the time for it.

greate

neoxed · 09-08-2007, 02:51 PM

I finished all basic module operations, the database is able to read/write and open/close users and groups stored on the database. It's mostly functional now at this point.

All that's left before I'll have alpha-1 ready is group renaming and a bit more testing.

neoxed · 09-08-2007, 04:28 PM

For those interested in the amount of code written so far:

Code:

wc -l include\* source\*
     82 include\backends.h
    147 include\base.h
     39 include\condvar.h
    121 include\database.h
     26 include\debug.h
    142 include\pool.h
    107 include\proctable.h
    247 include\queue.h
    154 source\condvar.c
    809 source\database.c
    113 source\debug.c
     54 source\dllentry.c
    392 source\group.c
    530 source\groupdb.c
    292 source\groupfile.c
    695 source\pool.c
     93 source\proctable.c
    395 source\user.c
   1779 source\userdb.c
    293 source\userfile.c
   6510 total

oldhouse80 · 09-08-2007, 07:12 PM

Thx very much for your effort. Keep up the good work!
I will give it a try as soon as I have some spare time.

peep · 09-09-2007, 04:22 AM

Nice one neoxed! Great to see people who are able to keep things up..

neoxed · 09-24-2007, 12:12 AM

Well it's been a while since you've heard from...it's because I've been busy coding. I implemented and tested different designs for syncing with the database and I've finally come up with one I like.

I've created a "changes" table that will be used for sites to see which users/groups have been created, renamed or deleted. The changes table is processed incrementally on configurable time intervals. During start-up a full synchronization is done since the module has no idea how long ioFTPD was offline.

The group synchronization is done, now I just need to do some copy and pasting to finish user synchronization (basically the same thing, just different structures).

Code:

E:\ioFTPD\modules>wc -l include\* source\*
     31 include\array.h
    128 include\backends.h
    161 include\base.h
     39 include\condvar.h
     92 include\database.h
     54 include\debug.h
     36 include\idlist.h
     39 include\namelist.h
    142 include\pool.h
    119 include\proctable.h
    585 include\queue.h
    189 source\array.c
    157 source\condvar.c
    956 source\database.c
    410 source\debug.c
     54 source\dllentry.c
    524 source\group.c
    691 source\groupdb.c
    612 source\groupdbsync.c
    292 source\groupfile.c
    103 source\idlist.c
    356 source\namelist.c
    696 source\pool.c
     99 source\proctable.c
    527 source\user.c
   1940 source\userdb.c
     30 source\userdbsync.c
    293 source\userfile.c
   9355 total

neoxed · 10-05-2007, 12:06 AM

Well I'm glad to say nxMyDB is near completion. I've finished implementing all the basic features required to make it a functional database module. The database table format has stabilized as well, the only changes for the next release is the addition of two stored procedures - no changes to the tables themselves.

I have implemented a few new things for the next release:

A new logging system to keep track of errors (no need for DebugView).
Spin-lock based locking, which greatly reduces the "user locking failed" messages.
Support for atomic group rename operations.

The change log for the next alpha release (v0.3.0), so far:

Code:

nxMyDB v0.3.0 (not released):
  NEW: Added configuration option "Log_Level" to control the log verbosity.
  NEW: Group rename operations are now implemented.
  NEW: Logging system to log errors, warnings, and information to a dedicated log file.
  NEW: Spin-lock based locking implemented using stored procedures, reduces failed locks.
  CHG: Renamed configuration option "SyncFirst" to "Sync_First".
  CHG: Renamed configuration option "SyncInterval" to "Sync_Interval".
  CHG: Requires MySQL Server v5.0.19, or newer.
  FIX: User rename operations failed during sync, since the new name was not present in the changes table.

The code statistics (as usual

):

Code:

E:\ioFTPD\modules>wc -l include\* source\*
     43 include\alloc.h
     31 include\array.h
    135 include\backends.h
    163 include\base.h
     40 include\condvar.h
     95 include\database.h
    110 include\logging.h
     39 include\namelist.h
    142 include\pool.h
    138 include\proctable.h
    585 include\queue.h
    315 source\alloc.c
    189 source\array.c
    160 source\condvar.c
   1041 source\database.c
     45 source\dllentry.c
    154 source\event.c
    528 source\group.c
    919 source\groupdb.c
    612 source\groupdbsync.c
    292 source\groupfile.c
    107 source\log.c
    115 source\logdebug.c
    299 source\logfile.c
    356 source\namelist.c
    694 source\pool.c
    104 source\proctable.c
    531 source\user.c
   1983 source\userdb.c
    757 source\userdbsync.c
    293 source\userfile.c
  11015 total

If no bugs come up in my testing, expect a release this weekend.

ArtX · 10-05-2007, 11:25 AM

good work

monk- · 10-05-2007, 09:58 PM

very nice work

,
this is the new sharedb!
it runs much smoother already,
i hope Yill can fix the mod system on 6.x...