ParallelVarFiler 1.31

[aminer]

Hello,


ParallelVarFiler was updated to version 1.31,
i have corrected some hard to find memory leaks,
and now it has no memory leaks...


Author: Amine Moulay Ramdane


Parallel Variable filer and streamer for Delphi and Freepascal that can
be used also as a parallel hashtable and that uses ParallelHashList
(Parallel Hashtable) with O(1) best case and O(log(n)) worst case access
that uses lock striping and lightweight
MREWs(multiple-readers-exclusive-writer) , this allows multiple threads
to write and read concurently. also ParallelHashList maintains an
independant counter , that counts the number of entries , for each
segment of the hashtable and uses a lock for each counter, this is also
for better scalability.
Description:

ParallelVarFiler is a Parallel HashTable that can be saved automatically
or manually to a file or to a stream or to a string and it can be
restored from a file or from a stream or from a string back to the
Hashtable in memory, and it's fault tolerant to power failures etc.
You can collect different types of variables and save them into one file
or stream. TParallelVarFiler reads and writes on files, streams and
Strings. You can collect Strings, Integers and Dates, in fact everything
that can be stored in a variant. In addition you can collect Streams.

You can use also ParallelVarFiler to send parameters over network. It
can be used for example to share data by any IPC mechanism.

And please look at test.pas and test1.pas a parallel variable filer
examples - compile and execute them...
Now you can use ParallelVarFiler as a small to medium database, i have
added a Clean() and DeletedItems() methods and i have changed the
constructor, now you can pass a file name to the constructor where the
data will be wrote and updated and deleted... if you pass and empty file
name the data will be updated and wrote and deleted only in memory. When
you use a file name in the constructor, many readers and one writer can
proceed concurrently. But when you pass an empty file name to the
constructor, many writers and many readers can proceed concurrently in
memory. and when you read a data item it only use the parallel hashtable
in memory, hardisk is not used.

And when you pass a file name to the constructor and delete or update
the items, those items will be marked deleted from the file, and you can
use DeletedItems() to see how many data items are marked deleted in the
file, and use after that the Clean() method to delete completly those
data items from the file.

How can you store mutiple data inside a database row and map it to a key ?

You can simply use Parallel Variable Filer for that and stream your
mutiple data variants and map them to a key...

As you know ParallelVarFiler is using ParallelHashList (a parallel
hashtable that scales on multicores), so when you are using the
GetKeys() method , and you want to get the data of those keys don't
forget to test if the variant is not empty by using the VarIsEmpty()
function, and when you are using GetStream() you can test if the data
exists by testing the return boolean value from GetStream().

ParallelVarFiler is easy to learn, i have documented all the methods ,
and please read inside ParallelVarFiler.pas about
them.

Now ParallelVarFiler is Fault tolerant to power failures etc. i have
done a simulation of power failures and data file damages and
ParallelVarFiler is recovering from power failures and damages of the
data file ...

As you know ParallelVarFiler is using ParallelHashList (a parallel
hashtable), but as you know ParallelHashList is cache unfriendly since
it's using a hash mechanism etc. and as you know ParallelVarFiler and
ParallelHashlist are memory bound, so since ParallelHashList is cache
unfriendly and it's memory bound so i don't think ParallelVarFiler or
ParallelHahsList will scale very well, to avoid this problem you have to
use something like replication across computers using TCP/IP and using
your database in each computer and load balance between computers, but
even if you are using replication and load balancing this can make the
memory and hardisk truly parallel but this will not avoid the problem
with the network bandwidth limitation. But ParallelVarFiler and
ParallelHashlist are scaling to a certain point and they are fast and
they are still usefull.

Please see the test.pas example inside the zip file to see how i am
using it...

But please read the following:

This software is provided on an "as-is" basis, with no warranties,
express or implied. The entire risk and liability of using it is yours.
Any damages resulting from the use or misuse of this software will be
the responsibility of the user.

please look at the test.pas and test1.pas example inside the zip file to
see how to use ParallelVarFiler...


You can download ParallelVarFiler 1.31 from:

http://pages.videotron.com/aminer/


Here is the methods that have been implemented:

PUBLIC METHODS:

constructor Create(file1:string,size,mrews:integer;casesensitive:boolean);

- Creates a new VarFiler ready to use, with size and with mrews number
of mrewa and casesensitive

for case sensitive keys,the number of
MREWS(multiple-readers-exclusive-writer) must be less or

equal to the Hashtable size and file1 is the the file to write to, if
file1 is an empty string

the data will be wrote only to memory and use SaveToFile() to write to a
file.

destructor Destroy;

- Destroys the ParallelVarFiler object and cleans up.

procedure Clear;

- Deletes all contents.

function Clean:boolean

- Cleans the deleted items from the file.

function DeletedItems:integer

- Returns the number of items marked deleted.

function LoadData:boolean

- Loads the data from the file passed to the constructor.

function Delete(Name : String):boolean;

- Deletes the variable with Name.

function Exists(Name : String) : Boolean;

- Returns True if a variable with Name exists

procedure GetKeys(Strings : TstringList);

- Fills up a TStringList descendant with all the keys.

function Count : Integer;

- Returns the number of variables

function Add(Name : String; Value : Variant ):boolean;

- Adds a new variable , given the Name

function AddStream( Name : String;Stream : TStream ):boolean;

- Adds a new Stream, given the Name

function Update(Name : String; Value : Variant):boolean;

- Updates the new variable , given the Name

function UpdateStream(Name : String; Stream : TStream ):boolean;

- Updates a new Stream , given the Name

function GetStream(Name : String;Stream : TStream):boolean;

- Fills up the stream with data

procedure SaveToStream(Stream : TStream);

- Saves all variables, streams and Graphics to a stream.

procedure SaveToFile(FileName : String);

- Saves all variables, streams and Graphics to a file.

procedure SaveToString(out S : String);

- Saves all variables, streams and Graphics to a string.

procedure SaveToStreamAndDelete(Stream : TStream);

- Saves all variables, streams and Graphics to a stream
and delete after that all the data inside the hashtable
and inside the file.

procedure SaveToFileAndDelete(FileName : String);

- Saves all variables, streams and Graphics to a file.
and delete after that all the data inside the hashtable
and inside the file.

procedure SaveToStringAndDelete(out S : String);

- Saves all variables, streams and Graphics to a string.
and delete after that all the data inside the hashtable
and inside the file.
function LoadFromStream(Stream : TStream):boolean;

- Loads all variables, streams and Graphics from a stream.

function LoadFromFile(FileName : String):boolean;

- Loads all variables, streams and Graphics from a File.

function LoadFromString(S : String):boolean;

- Loads all variables, streams and Graphics from a string.

function Stream2String(Stream1: TStream;var Mystring:string): boolean;

procedure String2Stream(var String2BeSaved:string; Stream1: TStream);

procedure Variant2Stream(var VR:variant; Stream1: TStream);

function Stream2Variant(Stream1: TStream;var VR:variant): boolean;


PUBLIC PROPERTIES:

Items : Variant

- Gets the value (indexed).


Language: FPC Pascal v2.2.0+ and Lazarus / Delphi 7 to 2007:
http://www.freepascal.org/

Operating Systems: Win , Linux and Mac (x86).

Required FPC switches: -O3 -Sd -dFPC -dWin32 -dFreePascal

-Sd for delphi mode....

Required Delphi switches: -DMSWINDOWS -$H+

For Delphi use -DDelphi



Thank you,
Amine Moulay Ramdane.

[aminer]

Hello,


You can be confident with my ParallelVarfFiler , i have done my best to
debug it thouroughly and i think it is now more stable.

Be happy with ParallelVarFiler...


You download ParallelVarFiler from:

http://pages.videotron.com/aminer/