Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Writing and Reading Key & Item To A File

25 views
Skip to first unread message

BeeJ

unread,
May 14, 2013, 7:15:40 PM5/14/13
to
Probably not understanding all of this so please enlighten me.

I would like to write a Dictionary and a Collection to a file and then
later load it back.

What is strange to me is that I can write an array of UDT with Variant
elements and read that back using Put then after doing an Erase do a
Get. I would expect to get an error if the UDT contained an Object.

VB seems to assume that the Dictionary or Collection must contain
Objects whether it does or not so Put and Get are not allowed. In my
case I am not using Objects in anything.

Unfortunately even the PropertyBag will not allow a collection so I
cannot use it to and from a file.

So if not a Dictionary or Collection what might I consider to use as
such that is quickly file written and read. Need fairly quick access
(in memory) to an Item variant (no objects needed) using a string Key.


Mayayana

unread,
May 14, 2013, 8:51:36 PM5/14/13
to
| So if not a Dictionary or Collection what might I consider to use as
| such that is quickly file written and read.

A UDT, which can be of any size and can contain
array elements, numeric, boolean, string, other UDTs.
You could create a variant array and a matching string
array to hold keys matched by index.

Public CustomUDT1 as CustomUDT

Open sPath For Binary Lock Read Write As FF
Get FF, , CustomUDT1
Close FF

Open sPath For Binary Lock Read Write As FF
Put FF, , CustomUDT1
Close FF


ralph

unread,
May 14, 2013, 9:19:03 PM5/14/13
to
Depends on size, frequency, type of keys, etc ... . Lot's of ways.

For example, for several dozen Key-Value pairings with static keys, an
INI-file would work just fine.

You might consider just using a Recordset.

Why does it have to be a Collection?

Write your own custom collection and serialize it out to disk.

...

-ralph

BeeJ

unread,
May 14, 2013, 9:31:35 PM5/14/13
to
It happens that Mayayana formulated :
Yes, I know all of that.
The problem with a UDT is that I cannot quickly find an Item (say
variant) based on a Key (not an index, but a string).

So maybe a linked list or what?

I want in-memory speed and fast file I/O.


BeeJ

unread,
May 14, 2013, 9:39:10 PM5/14/13
to
ralph used his keyboard to write :
I use INI files but saving or getting an Item by Key requires file I/O
for each access.

Does not need to be a collect but a collection (in a class I have
created) does all I want for in-memory stuff and is fast enough but has
no way to fast put or get to a file. Serializing it means a loop for
each. Seems that would be much slower than a UDT array Put and Get.

I am working with rather large data set. Possible maxed out around
100K but I have little control of that.

I have been looking at linked lists but have only found access by an
index (long) and not by a string Key. Any place to look for help
there?

Another thought is to use UDT array and keep it sorted by Key insert
with each insert and then a binary search to access by Key. Darn, I
already built that class and use it extensively ... so I was trying to
shorten the code by using a Collection that is simpler and faster than
my optimized class.


Mayayana

unread,
May 14, 2013, 11:13:28 PM5/14/13
to
| Yes, I know all of that.
| The problem with a UDT is that I cannot quickly find an Item (say
| variant) based on a Key (not an index, but a string).
|
By index I meant matching up indices. In other
words:

Private AItem() As Variant
Private AKey() As String
Private ItemCount as Long, UB As Long

Class init: Redim AItem(100) As Variant
Redim AKey(100) As String
UB = 100

Public Sub AddItem(vItem as Variant, sKey as String)
If ItemCount = UB then
UB = UB + 100
Redim Preserve AItem(UB) As Variant
Redim Preserve AKey(UB) As String
End If
AItem(ItemCount) = vItem
AKey(ItemCount) = sKey
ItemCount = ItemCount + 1
End Sub

Public Function GetItem(sKey As String) As Variant
Dim i as long
For i = 0 to UB
If AKey(i) = sKey then
GetItem = AItem(i)
Exit Function
End If
Next
GetItem = CVar("Null") 'set a standard value for item not found.
End Function

Searching an array is very fast. There are limits, I guess,
if you get into *very* big numbers. But for most purposes
I would think it would work fine.

MM

unread,
May 15, 2013, 2:06:58 AM5/15/13
to
On Tue, 14 May 2013 16:15:40 -0700, BeeJ <spa...@nospam.com> wrote:

One approach I've used many times is disconnected ADO recordsets. You
can create a recordset from scratch in code (no external database
required), and saving it is dead simple. Loading it back in again at
the next session is fast and easy.

Here's an example I just grabbed from an old app:

Public Sub SaveNewsgroupsToRecordset(filename As String)

Dim j As Long

Set rsNewsgroups = New ADODB.Recordset

With rsNewsgroups
Set .ActiveConnection = Nothing
.CursorLocation = adUseClient
.LockType = adLockBatchOptimistic
With .Fields
.Append "Newsgroup", adBSTR, 60
.Append "Total", adInteger, 4
.Append "FirstArticle", adInteger, 4
.Append "LastArticle", adInteger, 4
End With
.Open

For j = 1 To UBound(mNewsGroups) 'NB: Element 0 wasted
.AddNew
.Fields("Newsgroup") = mNewsGroups(j).GroupName

If mNewsGroups(j).FirstArticle = 0 Then
.Fields("Total") = 0
Else
.Fields("Total") = (mNewsGroups(j).LastArticle -
mNewsGroups(j).FirstArticle) + 1
End If

.Fields("FirstArticle") = mNewsGroups(j).FirstArticle
.Fields("LastArticle") = mNewsGroups(j).LastArticle
.Update
Next

.Save filename, adPersistADTG
.Close
End With

Set rsNewsgroups = Nothing

End Sub

MM

ObiWan

unread,
May 15, 2013, 2:58:03 AM5/15/13
to

> The problem with a UDT is that I cannot quickly find an Item (say
> variant) based on a Key (not an index, but a string).
>
> So maybe a linked list or what?
>
> I want in-memory speed and fast file I/O.

Why don't you use an array of UDT *and* a dictionary ? When inserting
element into the UDT array you'll also add an entry to the dictionary,
indexed using the UDT element key and containing the array index, then,
when you'll need to retrieve a given UDT element you'll just need to
lookup the key in the dictionary, obtain the index and there you go

Sure, there are other solutions to your problem, but the above is quite
simple and straightforward to implement using vanilla VB objects


ObiWan

unread,
May 15, 2013, 3:05:46 AM5/15/13
to

> Does not need to be a collect but a collection (in a class I have
> created) does all I want for in-memory stuff and is fast enough but
[...]
> Another thought is to use UDT array and keep it sorted by Key insert
> with each insert and then a binary search to access by Key. Darn, I
[...]

why don't you mix the two; you may create a collection class, that is a
class which internally holds a collection of classes, the latter will
expose the get/let/set properties you need (match them with the
elements you'd use for your UDT); not just that, the collection class
may also expose "serialize" and "deserialize" methods, those will
basically do the following

* when serializing, the method will invoke each element (class)
"serialize" method, the latter will return a "data blob" representing
the element data (it may be an array, an XML blob, a serialized
propertybag or whatever); the collection (master) class will then
handle the data as desired (add to a buffer, write to a file...)

* when deserializing, the method will fetch a datablock related to a
given element (class), will create a new istance of the class and
then call the class "deserialize" method passing it the datablock,
the class will pick the data, decode it and fill its values

this way, not only you won't need to deal with data persistence, but
you'll also be able to seek a given block of data using either an index
or a key or even to use "for each" to go through the whole dataset

ObiWan

unread,
May 15, 2013, 6:18:49 AM5/15/13
to

> why don't you mix the two; you may create a collection class, that is
> a class which internally holds a collection of classes, the latter
> will expose the get/let/set properties you need (match them with the

If you want to quick try it, fire up your VB6 IDE and create a new,
empty project, next go to the "add-ins" menu and open the "class
builder utility", right click on the left pane node and select "new"
and "class", name the new class "CDataItem" and confirm, you'll see the
new class in the class builder, select the class and right click on the
"properties tab" in the right pane, add a string property and call it
"ItemKey", then, go on adding whatever properties you want (you may
match your UDT if you want); once you have all your properties, select
the "methods" tab (right pane) and add two methods, that is a
"Serialize" one w/o parameters and returning a variant and a
"Deserialize" one accepting a single variant parameter and returning a
boolean value

Now it's time to create the collection; click on the left hand pane
root node and select "new" and "collection", call the collection
"CDataItems", leave the "based on" empty and select the "CDataItem"
entry in the "collection of" list, confirm and you'll have your
collection class, now add to it two methods, again "Serialize" and
"Deserialize" matching the ones you added to the "CDataItem" class

At this point, just go to the file menu, select "update project" and
close the class builder; now your project will contain two classes, the
"CDataItem" used to store the single items and the "CDataItems" one
which contains a collection of all your items; the next steps will be
changing the "Add" method of "CDataItems" to use the "ItemKey" as the
collection key (easy) and adding code inside the serialize/deserialize
methods of both the CDataItem and CDataItems; the first will contain
code to pick the various values of the item and turn them into (say) a
string or stuff them inside a propertybag and returning it (the
opposite for the deserialize, pick the "blurb", extract the values and
assign them to the internal values), the second (CDataItems) will
either loop through the CDataItem collection, call the "serialize"
method for each element and add the returned "blurb" to some kind of
buffer/array/propertybag/... or will pick the buffer containing the
serialized data, loop through it and for each element create a new
instance of CDataItem add it to the collection and call its deserialize
method to populate it

HTH

ObiWan

unread,
May 15, 2013, 6:27:24 AM5/15/13
to
> the first will contain code to pick the various values of the item
> and turn them into (say) a string or stuff them inside a propertybag
> and returning it (the opposite for the deserialize, pick the "blurb",

an idea may be having a "master" propertybag returned by the
"CDataItems" class and containing all the serialized data, basically
your PB will contain something like

ItemCount long, the number of "CDataItems" stored

ItemData1...n binary, carries the PB of the single "CDataItem"

the above will ease serializing/deserializing the data and you'll be
able to save and restore your collection using really simple code



BeeJ

unread,
May 15, 2013, 3:47:59 PM5/15/13
to
Thanks for all the ideas. I will probably try most of them that will
support what I need.

Regarding using the dictionary or collection in any way I just do not
see how to get that to a file and back.

I am now working with a doubly-linked balanced linked list that uses a
UDT for key and value (and nodes). This I can write and read in one
kerchunk although it is suprisingly slow doing so (100,000 Key-Item
pairs).

Finding the Item from the Key is fairly fast. It seems (right now
until I dig further) to take only around 16 iterations to find the
Item(Variant) from the Key (string). I am guessing that this is normal
since a Binary Search usually takes a max of 8 iterations. As this is
a doubley balanced linked list then 2 x. Maybe? Anyway it seems OK.

Key or Item by index is lightening fast.

I am doing some speed tests on 100,000 Key Item pairs.

This class incarnation has:
Where Key is string and Item is a Variant (non-object for the UDT save
and load to work).
Add Key Item
Index = Find Key
Item = FindItem(Index)
Key = FindKey(Index)
Remove Index
Clear
Save Filename
Load FileName
Count


All of this I am comparing against the List Class i wrote that uses
binary search to add Key Item pairs so that searching for the Item from
the Key will be quick by again using binary search.


ObiWan

unread,
May 16, 2013, 10:51:46 AM5/16/13
to

> Regarding using the dictionary or collection in any way I just do not
> see how to get that to a file and back.

which basically means that you didn't read/understand what was
suggested; I suggested the "serialize" methods and the use of property
bags, others the use of disconnected recordset... and then some more
and, sincerely, your "do not see how" makes you appear as totally dumb

> I am now working with a doubly-linked balanced

Pardon me, but at this point, I don't give a darn f*ck about what
you're working with, go on working with whatever you want and avoid
waving here; there's no need for someone which tries using this group
as his own, personal "help file"


CoderX

unread,
May 16, 2013, 6:16:52 PM5/16/13
to

"ObiWan" <alb.20.t...@spamgourmet.com> wrote in message
news:20130516165...@deathstar.mil...
>
> which basically means that you didn't read/understand what was
> suggested; I suggested the "serialize" methods and the use of property
> bags, others the use of disconnected recordset... and then some more
> and, sincerely, your "do not see how" makes you appear as totally dumb
>
> Pardon me, but at this point, I don't give a darn f*ck about what
> you're working with, go on working with whatever you want and avoid
> waving here; there's no need for someone which tries using this group
> as his own, personal "help file"
>

You're just now putting this together?


ObiWan

unread,
May 17, 2013, 2:51:36 AM5/17/13
to

> > which basically means that you didn't read/understand what was
> > suggested; I suggested the "serialize" methods and the use of
> > property bags, others the use of disconnected recordset... and then
> > some more and, sincerely, your "do not see how" makes you appear as
> > totally dumb
> >
> > Pardon me, but at this point, I don't give a darn f*ck about what
> > you're working with, go on working with whatever you want and avoid
> > waving here; there's no need for someone which tries using this
> > group as his own, personal "help file"

> You're just now putting this together?

no, but I think people needs some chance to grow, then, if that
repeatedly fails ... well

MikeB

unread,
May 17, 2013, 9:43:08 AM5/17/13
to

"ObiWan" <alb.20.t...@spamgourmet.com> wrote in message
news:20130517085...@deathstar.mil...
I'll bet I haven't asked for help in a NG two times in ten years.
I have used the NGs as a resource as I am sure a vast majority have.

Given the unenumerable amount of information at ones fingertips, it seem
illogical to not find within a reasonable degree of exactness, the answer to
whatever niggling little problem of the day is bothering you.

I don't think asking for help here first is a first line of defense, but as
a last resort and further, when the OP persists with tedious follow ups,
till someone regurgitates source code to silence them, doesn't help them
grow in the least amount.

>


GS

unread,
May 17, 2013, 6:18:05 PM5/17/13
to
Well said!

--
Garry

Free usenet access at http://www.eternal-september.org
Classic VB Users Regroup!
comp.lang.basic.visual.misc
microsoft.public.vb.general.discussion


MM

unread,
May 18, 2013, 3:56:24 AM5/18/13
to
On Fri, 17 May 2013 18:18:05 -0400, GS <g...@somewhere.net> wrote:

>Well said!

I think you're all talking bollocks. Wha a superior bunch you must
think you are!

What is a discussion group for if not to discuss things?

You're all far too up yourselves, in my view, especially that ObiWan
fellah. I though his response to Beej was unconscionably rude. He
could have just STFU, couldn't he? Why come in with the heavies on a
bloke who's just asking a question, jeesh.

By the way, anyone know how to use a text box control?

MM

se

unread,
May 18, 2013, 4:19:30 AM5/18/13
to

"MM" <kyli...@yahoo.co.uk> skrev i meddelelsen
news:mncep8p6f3ausocs6...@4ax.com...
Fully agree with you. I'll go further. The reason they are here is for
showing
up themself - and not as helpers. In fact a behaviour caused by dumbness.
Those two persons should rather f*ck off.

/ se

Larry Serflaten

unread,
May 18, 2013, 11:38:34 AM5/18/13
to
BeeJ wrote:

> Finding the Item from the Key is fairly fast. It seems (right now
> until I dig further) to take only around 16 iterations to find the
> Item(Variant) from the Key (string). I am guessing that this is normal
> since a Binary Search usually takes a max of 8 iterations. As this is
> a doubley balanced linked list then 2 x.

I am having a hard time rationalizing using a Variant when you know you want
to move to and from a file. The file is just so many bytes on a disk, it
could translate well to so many bytes in memory.

What are you storing in the item (Variant) that would not fit easily in
a string (for example) ???

LFS

BeeJ

unread,
May 18, 2013, 2:48:38 PM5/18/13
to
Good question and I have been wondering about that myself.
I am set up for a variant so I can soon try to see if an array or other
'simple" objects can be stored. I probably define simple as not
controls. I will try an array and a class to see if that will work
with the save and load. I am sure it will be OK otherwise but my big
want and need is to do an easy and quick Save and Load.

I have been working on another project using activeX exe so I can pass
stuff off to an array of activex exes to maximize CPU processing. so
far so good. later today I will try the Variant with array and class.

I have been trying to find out about serialization but so far nothing I
read seems appropriate. Any good reading sites?


Larry Serflaten

unread,
May 18, 2013, 5:29:36 PM5/18/13
to
BeeJ wrote:

> > What are you storing in the item (Variant) that would not fit easily in
> > a string (for example) ???


> Good question and I have been wondering about that myself.
> I am set up for a variant so I can soon try to see if an array or other
> 'simple" objects can be stored. I probably define simple as not
> controls. I will try an array and a class to see if that will work
> with the save and load. I am sure it will be OK otherwise but my big
> want and need is to do an easy and quick Save and Load.


As has already been mentioned, an array of UDT's is a nice fit for quick
and simple save and load operations.


> I have been working on another project using activeX exe so I can pass
> stuff off to an array of activex exes to maximize CPU processing. so
> far so good. later today I will try the Variant with array and class.


I would suggest that transposing (serializing) an array or class to a
byte stream and back would cause a minimal delay compared to the initial
look-up process to find the item of interest. In other words, you could
store an array or class as a delimited string in the UDT and the time it
takes to convert a string back to the array or class would not be that
long. Of course situations vary, frequency of look-ups would be a determining
factor, but for low look-up rates, the few milliseconds used to do the
conversion may not be as significant as the time taken in finding the
correct item. You might use the last character of a key (or a separate
field in a UDT) to indicate the type of data held in the item element.
(Integer, Decimal, String, Integer Array, Decimal Array, String Array ...)


> I have been trying to find out about serialization but so far nothing I
> read seems appropriate. Any good reading sites?

I don't have any good reads off the top of my head, but it is not that deep
of a subject. Serialization is just the process used to convert what you have
to a stream of bytes so that it can be saved to disk or marshaled across
processes and/or networks. XML is often used when handling objects, but
nothing is stopping you from devising your own methods.

The main consideration is what your intentions are. Do you want an optimized
process for a specific application, or are you looking to make a more
generalized package for drop in use in other programs?


LFS

BeeJ

unread,
May 18, 2013, 6:42:13 PM5/18/13
to
Larry Serflaten has brought this to us :
> BeeJ wrote:
>

>
> I would suggest that transposing (serializing) an array or class to a
> byte stream and back would cause a minimal delay compared to the initial
> look-up process to find the item of interest. In other words, you could
> store an array or class as a delimited string in the UDT and the time it
> takes to convert a string back to the array or class would not be that
> long. Of course situations vary, frequency of look-ups would be a determining
> factor, but for low look-up rates, the few milliseconds used to do the
> conversion may not be as significant as the time taken in finding the
> correct item. You might use the last character of a key (or a separate
> field in a UDT) to indicate the type of data held in the item element.
> (Integer, Decimal, String, Integer Array, Decimal Array, String Array ...)
>
>
>> I have been trying to find out about serialization but so far nothing I
>> read seems appropriate. Any good reading sites?
>
> I don't have any good reads off the top of my head, but it is not that deep
> of a subject. Serialization is just the process used to convert what you
> have to a stream of bytes so that it can be saved to disk or marshaled across
> processes and/or networks. XML is often used when handling objects, but
> nothing is stopping you from devising your own methods.
>
> The main consideration is what your intentions are. Do you want an optimized
> process for a specific application, or are you looking to make a more
> generalized package for drop in use in other programs?
>
>
> LFS

What I have seen so far is to do something like the following. The
examples I have seen are much more complex and create a very large data
item string.

Say from a data class, the simplest seems to me to be:

sSer = cstr(class.dDateNow) & sDelim & cstr(class.lngWord) ...

This is not a general case but I see sample code for a data class that
is general and I can greatly simplify; however, a lot of processing is
involved for each item.


Larry Serflaten

unread,
May 19, 2013, 10:49:02 AM5/19/13
to
BeeJ wrote:

> Say from a data class, the simplest seems to me to be:
>
> sSer = cstr(class.dDateNow) & sDelim & cstr(class.lngWord) ...
>
> This is not a general case but I see sample code for a data class that
> is general and I can greatly simplify; however, a lot of processing is
> involved for each item.

It seems you are confusing the serializing of a bunch of properties with
serializing a specific set of formatted data. You started the thread looking
for a value pair (key/item) solution, but are now messing with arbitrary
property types (dates, longs, etc.)

Have a look (try out) this example. It handles the different primitive types
in a UDT fashion.

https://groups.google.com/d/msg/microsoft.public.vb.general.discussion/Ix0RixyaFNg/JBvi9EbjMd8J

Note there that copy process involves passing a string from one class to
another. That same string could be sent to a file. So there you have
serialization without all the CStr(), and what not you wanted to avoid.

But you didn't respond to the question, are you looking to optimize a
key/item process for a specific app, or are you trying to make a more
generalized version for use in any program? The decision there helps to
define what can and cannot be 'safely' assumed....

LFS

BeeJ

unread,
May 19, 2013, 3:29:04 PM5/19/13
to
Larry Serflaten was thinking very hard :
A general solution is what I want.
The Key/Item pair was a string Key and a Variant Item so the Item could
hold stuff. Stuff might include a Class (data) or primitives or ... but
not complex objects. But it turns out the Save and Load using the
complex UDT gets to be a problem.

Very interesting code you presented.
I never knew that LSET UDT = UDT was possible in VB. A union?

A more general case of your code would be to handle variable length
strings. But there is no easy way that I know of to do that.

e.g. my UDT might be
Public Type MyUDT
Filename as string (non *)
FileSize as variant
FileAttr as long
End Type.

I guess I could make the Filename as string* huge-ish (along with the
appropriate type Data element for a size match.

I have more experimenting to do.


Henning

unread,
May 19, 2013, 5:32:15 PM5/19/13
to

"BeeJ" <nos...@spamnot.com> skrev i meddelandet
news:knb8uj$aci$1...@dont-email.me...
I guess Len() is out of question?

/Henning



Larry Serflaten

unread,
May 19, 2013, 7:45:44 PM5/19/13
to
BeeJ wrote:

> A general solution is what I want.

Which usually means you'll be adding code to 'cover all the bases'.


> The Key/Item pair was a string Key and a Variant Item so the Item could
> hold stuff. Stuff might include a Class (data) or primitives or ... but
> not complex objects. But it turns out the Save and Load using the
> complex UDT gets to be a problem.

Have you thought about using 2 files, one for keys and the other for items?
I haven't tried it recently, but Put and Get may work with an array of
Variants (items) as it would for an array of Strings (keys)....


> Very interesting code you presented.
> I never knew that LSET UDT = UDT was possible in VB. A union?

More like a VB hack to use the CopyMemory (API) routine. The data from one
is copied into the memory of the other. A union is the case where one set
of data is in memory but you are using 2 or more variable types to access it.

LFS

ObiWan

unread,
May 20, 2013, 5:08:19 AM5/20/13
to

> Given the unenumerable amount of information at ones fingertips, it
> seem illogical to not find within a reasonable degree of exactness,
> the answer to whatever niggling little problem of the day is
> bothering you.

yes, but then it also depends from the degree of experience of the folk
asking and from the level of knowledge; for example a solution based on
a bunch of API calls may be "unsuitable" for someone which only has a
very basic knowledge of the VB language

> I don't think asking for help here first is a first line of defense,
> but as a last resort and further, when the OP persists with tedious
> follow ups, till someone regurgitates source code to silence them,
> doesn't help them grow in the least amount.

Agreed, the idea is that one should first search the 'net, helpfiles,
documents and so on and, if still in doubt (or in case he still needs
some clarifications), he may post on the groups telling what he found
and tried (detailing the whole thing as much as possible) and then,
asking for some very "targeted" kind of help

But then, if one keeps asking and asking and continues to apparently
ignore suggestions and replicate the same kind of mistakes over and
over... well :P

0 new messages