Indexes vs xref files - reliability of indexes
eppick77
various platforms. Do you use the indexes or do you keep your data in
xref files?
How reliable are the indexes on the various platforms?
Specifically how are they on PS/RD/TL systems? Windows vs Linux etc.
I was have a discussion recently and it was suggested that it is
better to use xref than the indexes on a PS platform.
Thanks
Eugene
Peter McMurray
I use a third party B-Tree at the moment as I have had issues with B-tree
implementations on a couple of MV platforms. However I think that you might
get the best of both worlds if you write out your required complex key as
the first attribute of a simple numeric keyed record and use the system
index on that first attribute as most of them should surely work on a single
attribute.
Peter McMurray
"eppick77" <eppi...@yahoo.com> wrote in message
news:1ce38efd-ab7d-48e1...@m26g2000yqb.googlegroups.com...
Martin Phillips
Since you are a QM user, here is the QM situation....
We guarantee that the index will be updated at the same time as a
write/delete regardless of how this write/delete is initiated
(application, editor, command line, QMClient, etc). The only situation
in which an index may be incorrect is if there is a system failure
that results in the data file update being written to disk but the
index update never escaping from the o/s disk cache or vice versa.
Since you are also using the EasyCo's MFT, the updates are nearly
instantaneous so the risk of error is very low.
Release 2.10-0 (next week) introduces some additional options
regarding when we sync a file. These could help non-MFT users avoid
the rare index woes but the best solution is to have a UPS so that the
system doesn't go down through power failures.
Martin Phillips, Ladybridge Systems.
mdsi2000
wrote:
Eugene,
I use system's INDEX. I don't have to rewrite SELECT/ LIST/SORT
statements.
I have never had a problem with missing items or at least no user has
ever complained about missing item. It' a great time save.
-Peter G.
eppick77
Peter,
I am very happy with the indexes on QM. My big concern was with PS/RD/
TL. I was also curious how people felt about the ones on U2.
So, the big question for me is whether the index issue on PS/RD/TL has
been resolved.
Eugene
Peter McMurray
<snip>
Peter,
I am very happy with the indexes on QM. My big concern was with PS/RD/
TL. I was also curious how people felt about the ones on U2.
So, the big question for me is whether the index issue on PS/RD/TL has
been resolved.
Eugene
Since I have never had an official response to the issues I raised many
moons ago I think that I can safely say the answer is no.
There appear to be significant differences vetween the FSI (the best idea in
many years) and the dratted VME so beloved of Unix users. I have neither
the time nor the inclination to spend even more effort searching for a
solution so I looked for an effective bypass.
By the way I submitted FSI and VME account saves to demonstrate my points
and I know that Ross Ferriss is the "go to" man in this regard.
Peter McMurray
Tony Gravagno
semi-surprised that there was no final conclusion through them from
TL. Did anyone ever get back to you with a "no you can't do this"?
Was there a confirmation of a bug? Or in your opinion was the issue
dropped?
On one hand I'm content to let the cards fall where they may if TL is
offered an opportunity to prove their indexes work and they decline or
fall short.
On the other hand I just want to make sure this has been played
through to it's proper conclusion.
TL has an automated process where an automated email is sent out for
unresolved issues more than a couple weeks old. So my questions are:
1) Were the support calls closed?
2) Were action items created?
3) Were you sent emails and then never responded?
4) Have you followed-up with anyone to indicate that you'd like a
resolution to the issues you presented?
5) If you don't know what the TL response was, have you done similar
follow-through with TData?
Again, I don't care which way this goes but let's be fair and ensure
we all have the proper information before coming to any conclusions.
T
Peter McMurray
TData have a very good support follow up mechanism and have always been very
helpful.
You are quite correct I have not chased through for a couple of reasons.
One too busy and two I have found from past experience that banging one's
head against a brick wall is counter productive.
I really had hoped to see some announcement, joy, whatever from the
conference but as far as I can tell everyone went back to sleep again.
By the way I first spoke to PS about B-trees when they were first released
and I know the chaps I was speaking to understood what I was saying because
one of them came from the Microstar team. I first used B-trees circa 1982
on MicroMax/Star Aus/US. (Remember when they used to make decent gear in
the US. Lear Siegler went to Mexico and it has been down hill ever since.)
But the same issues are still there, in fact with VME versus FSI they seem
to have got worse. Sad Movies eh! :-)
Happy New Year from the centre of The Universe (oops no pun intended)
Peter McMurray
"Tony Gravagno" <address.i...@removethis.com.invalid> wrote in
message news:pdrpj5pspg9lfsblv...@4ax.com...
Tony Gravagno
>TData have a very good support follow up mechanism and have always been very
>helpful.
I have full confidence in their intent to follow-though but so far it
seems you still don't really know what the status is on your query.
>You are quite correct I have not chased through for a couple of reasons.
>One too busy and two I have found from past experience that banging one's
>head against a brick wall is counter productive.
Well, like I said, protocols are in place so I don't think TL dropped
their side of it. It's quite rare that email follow-ups to open items
are not automatically sent out for follow-up. So I'm curious to know
what they and/or TData had to say about your indexes not working.
Personally I don't trust indexes in any of the MV products (and I'm
thinking it was a statement like that to Eugene which prompted this
thread). They all work differently across platforms, sometimes don't
work consistently or across releases, and from one week to the next
there is almost always a note in one MV forum or another with someone
citing about an index issue. That is where I'd prefer not to bang my
head against a wall as I work with almost every MV platform available.
Anyway, it seems to me the question about indexes with D3 is still
open, and the people who have dropped communications need to pick up
the ball again, otherwise we're once again dancing around a bunch of
unsubstantiated hearsay.
>I really had hoped to see some announcement, joy, whatever from the
>conference but as far as I can tell everyone went back to sleep again.
I wanted to see what they'd do to follow-up their conference as well.
If they were waiting for public review to get people excited about
their direction, they miscalculated. They need to do their own
follow-up marketing.
The conference had a few highlights but I personally had a dark moment
that caused me to rethink this whole market, or at least my
perspective of the D3 developer base. This had nothing to do with
D3v9 or their other technical offerings. If I can find some time I
might blog about some of this one of these days.
Tony Gravagno
Nebula Research and Development
TG@ remove.pleaseNebula-RnD.com
Nebula R&D sells mv.NET and other Pick/MultiValue products
worldwide, and provides related development services
remove.pleaseNebula-RnD.com/blog
Visit PickWiki.com! Contribute!
http://Twitter.com/TonyGravagno
Peter McMurray
It seems that we both mistrust indices on several platforms which is most
unfortunate but based on sad experience.
I am disappointed to see that you had a dark moment at what should have been
an uplifting experience. I look forward to the blog albeit with
trepidation.
Peter McMurray
"Tony Gravagno" <address.i...@removethis.com.invalid> wrote in
Scott Ballinger
I have used the native indexes on both D3 and Universe [Linux]
extensively for years. Both are highly reliable; the last index-
related error I had was on UV [over a year ago] and was probably
related to not having exclusive hold of the file when the index was
built. I can't remember the last index error I had on a D3 system.
The native indexes require zero maintenance and are super fast. As I
update the various legacy system I work on I often replace the xref
files with indexes.
Both D3 and UV have unique functions for accessing the indexes through
basic. In order to simplify things I use a standard subroutine call to
retrieve indexed data. Here is my D3 version:
(used like result = "select filename with fieldname = thing]")
get.btree.sub
001 subroutine get.btree.sub(filename,fieldname,thing,result)
002 * 2004 asb
003 * Get result id(s) from the btree index fieldname for filename.
004 * Result is mv if more than 1 id found for thing.
005 * If thing ends with ] then return all starting with thing.
006 * If thing dosn't end with ] then return all exactly matching
thing.
007 * If thing requires an iconv (e.g. a date) then iconv before
passing it.
008
009 * Prepend an "x" to strings to make sure we compare strings not
numbers,
010 * e.g. bank-routing-number dot account-number may get interpreted
as a
011 * large number with a lot of decimal places, and precision
limitation
012 * would erroneously return 123456789.1234567891000 =
123456789.123456789
013 * note least significant digits ignored ^^^^
014
015 begin case
016 case fieldname matches "'a'0n"
017 ac = fieldname
018 case fieldname matches "0n"
019 ac = "a":fieldname
020 case 1
021 open "dict",filename to dict.filevar else
022 sleep 1
023 open "dict",filename to dict.filevar else
024 stop 201,"dict ":filename
025 end
026 end
027 read rec from dict.filevar,fieldname then
028 if rec<8>[1,1] eq "a" then
029 ac = rec<8>
030 end else
031 ac = "a":rec<2>
032 end
033 end else
034 print "error! can't read item ":fieldname:" in dict
":filename
035 stop
036 end
037 end case
038
039 root filename,ac to fileroot else
040 sleep 1
041 root filename,ac to fileroot else
042 print "error! no index defined for
":filename:",":ac:" (":fieldname:")"
043 stop
044 end
045 end
046
047 lent = len(thing)
048 if thing[lent,1] eq "]" then
049 wildcard = 1
050 lent -= 1
051 thing = thing[1,lent]
052 end else
053 wildcard = 0
054 end
055
056 search = thing
057 result = ""
058 id = ""
059 done = 0
060 xthing = "x":thing ;* make sure we compare strings not numbers
061
062 loop
063 key("x",fileroot,search,id) then
064 if wildcard then
065 if "x":(search[1,lent]) ne xthing then done = 1
066 end else
067 if "x":search ne xthing then done = 1
068 end
069 end else
070 done = 1
071 end
072 until done do
073 result<-1> = id
074 repeat
075
076 convert @am to @vm in result
077
078 return
I also have a very handy tool named 'bselect' that uses a select list
to look up indexed values in a file, e.g.
select customers with st = "NY"
bselect invoices customer <-- i.e. customer is an indexed field of
customer ids on the invoices file
bselect
001 * select records with indexed field matching a select list
002 * 12-13-04 asb
003 * usage: bselect target-file indexed-field
004
005 tot = system(11)
006 if tot else stop "this process requires an active select list"
007
008 prompt ""
009
010 tclread buf
011 buf = trim(buf)
012 targetfile = ""
013 fieldname = ""
014
015 pos = index(buf,"bselect",1)
016 buf = buf[pos,99]
017 targetfile = oconv(buf,"g1 1")
018 fieldname = oconv(buf,"g2 1")
019
020 if targetfile eq "" then
021 print "target file to select against: ":
022 input targetfile
023 call input.test(targetfile)
024 if targetfile eq "" or targetfile eq "quit" then stop
025 end
026 open targetfile else stop 201,targetfile
027
028 if fieldname eq "" then
029 print "indexed field on ":targetfile:": ":
030 input fieldname
031 call input.test(fieldname)
032 if fieldname eq "" or fieldname eq "quit" then stop
033 end
034
035 c = 0 ;* original list ids processed
036 c1 = 0 ;* original list ids found in target file via
fieldname btree
037 c2 = 0 ;* new ids to target file returned
038 t = time()
039 d = date()
040 a0 = @(0)
041 cl = @(-4)
042
043 dim list(10)
044 max = 10
045 n = 1
046 list(1) = ""
047
048 10 *
049
050 readnext id else go 999
051 call get.btree.sub(targetfile,fieldname,id,result)
052 if result eq "" then
053 ** print a0:id:" not found.":cl
054 end else
055 c1 +=1
056 convert@vm to @am in result
057 c2 += dcount(result,@am)
058 list(n)<-1> = result
059 if len(list(n)) gt 10000 then
060 * too many items in list and it starts getting slowwwww...
061 n +=1
062 if n gt max then
063 max += 10
064 dim list(max)
065 end
066 list(n) = ""
067 end
068 end
069
070 c += 1
071 if mod(c,100) eq 0 then
072 call counter.sub(c,t,d,tot,txt)
073 print a0:c:" ":c1:" ":c2:txt:cl:
074 end
075
076 go 10
077
078 999 *
079
080 call final.count.ids(c,t,d)
081 print c1:" ids found in '":fieldname:"' field on '":targetfile:"'
file."
082 print c2:" '":targetfile:"' records selected."
083 **print n:" 10k blocks used to make select list"
084
085 if c2 eq 0 then
086 stop 401
087 end else
088 open "pointer-file" else stop 201,"pointer-file"
089 lid = "bselect":system(22)
090 write list on lid
091 chain "get-list ":lid
092 end
093
In short, I use the native D3 and UV indexes as much as possible. In
the Linux environment, these indexes are as reliable as any other
files on the system. They are an inherent part of the D3 and UV
database environment functionality. The idea that indexes should be
avoided because they are somehow "unreliable" is yet another
stereotype of the MV environment that many in this forum seem
delighted to perpetuate.
/Scott Ballinger
Pareto Corporation
Edmonds WA USA
206 713 6006
Peter McMurray
I believe that you are not giving a fair answer and have not done much
testing.
The example you give is simplistic and I made the point that any of them
should do that with ease.
Your methodology is excellent and a welcome confirmation of my feelings.
The fact is that some of us have been using more complex indices for over 25
years and these are not handled well.
Again that is why I suggested using a simple approach. A single fixed
length ASCII string combining the variables that you want is a lot safer
than relying on a plethora of dictionaries that may/do work differently in
the VME to the FSI. In Universe answering a customer query as to why their
sales analysis did not add up to total sales - reindex problem solved, but
not thrilling.
Peter McMurray
"Scott Ballinger" <scott.b...@gmail.com> wrote in message
news:e321e73b-f89b-4ac5...@r24g2000yqd.googlegroups.com...
Scott Ballinger
My answer was based on my experience, which is that the native indexes
in D3 and Universe [in Linux} have been highly reliable- no more
subject to corruption than any other files in the D3 and Universe
environment. How is that not a "fair" answer? Wasn't that the question
posed by the OP?
As to the "not much testing" complaint, how could you possibly know?
Your presumption is 100% wrong on this point: I recently upgraded a D3
customer with about 300GB of pick data to a new server using EasyCo's
MFT drives [which work GREAT by the way!]. As as part of that upgrade
we did a lot of system stress testing, writing and deleting 100's of
millions of records to indexed files with zero index issues.
As part of my reply, I included two programs that facilitate the use
of indexes. The "example" that you refer to was demonstrate the use of
those programs. You seem to have misunderstood the context of the
"example;" there was no attempt to make any point beyond that.
As I stated before, these days I often find myself replacing xref
files with indexes as part of on-going system maintenance and
improvements. Why? Because they are more reliable than the xref files
they replace. Virtually very xref system I have ever implemented
includes by necessity a utility to rebuild the xref file. Inevitably
[probably more often than not due to poor transaction bracketing] I
end up needing to rebuild those xref files when the primary data file
and the xref file get out of sync. Some legacy systems I've inherited
rebuild critical xref files every night. My experience has been that
converting xref files to indexes increases reliability and reduces
maintenance.
Your statement that "complex indices... are not handled well." also
does not match my experiences. I have found that complex indices are
no more problematic than simple ones, and actually quite the opposite.
It seems to me that the more complex the indexing requirement, the
more opportunities there are to screw it up. Native indexes in D3 and
Universe are a "write once, automatically used everywhere" solution.
They are more reliable because the index logic is implemented once, in
one place, forever, and does not have to be incorporated in any other
(future) programs that might create, modify, delete, copy, or move the
indexed data.
Indexes are system tools. I use them because they make development and
system administration easier.
Oh, and "the fact is" I too have have been doing this for a very long
time (since 1982).
/Scott Ballinger
Pareto Corporation
Edmonds WA USA
206 713 6006
On Jan 2, 12:36 pm, "Peter McMurray" <excalibu...@bigpond.com> wrote:
> Hi Scott
> I believe that you are not giving a fair answer and have not done much
> testing.
> The example you give is simplistic and I made the point that any of them
> should do that with ease.
> Your methodology is excellent and a welcome confirmation of my feelings.
> The fact is that some of us have been using more complex indices for over 25
> years and these are not handled well.
> Again that is why I suggested using a simple approach. A single fixed
> length ASCII string combining the variables that you want is a lot safer
> than relying on a plethora of dictionaries that may/do work differently in
> the VME to the FSI. In Universe answering a customer query as to why their
> sales analysis did not add up to total sales - reindex problem solved, but
> not thrilling.
> Peter McMurray
>
> "Scott Ballinger" <scott.ballin...@gmail.com> wrote in message
Ross Ferris
It doesn't take much to see where the "problem" lies.
You stated very clearly that your were talking about LINUX hosted
systems (which I assume is the case with the client with 300Gb data!),
whereas Peter is talking about his experience with D3/NT (I think my
feeling are reasonably well understood on THIS subject!)
THAT single FACT would, I believe, explain the discrepancy --> I
believe that the you are both telling the truth!
You don't have any issues with indices under D3/Linux.
Peter has issues with indices under D3/NT (and more specifically I'm
guessing the FSI).
I can confirm both are correct :-)
Happy 2010
Mike Preece
So how do they work?
I am guessing that when you create an index some control data - like
maybe the "file control block" - is updated to record the fact that
this index is based on data in these attributes in this file. Maybe if
we create an index based on data from multiple files (an index based
on a dictionary item containing file translates or a subroutine call
in which other files are referenced) the processing involved in the
create-index is clever enough to update multiple "file control
block"s. That would be good. Difficult - but good. Probably doesn't
happen. If I'm right then there will inevitably be issues with any
index based on data from more than one file.
Then whenever a file update takes place this list of attributes on
which indexes are based is checked to see which indexes need to be
updated.
That is all speculation. I look forward to being educated.
Jeff Caspari
Why wouldn't they?
Perhaps CDP has been so quiet because people are tired of posting a question
or answer and then having to defend themselves for incorrectly asking or
answering. It's almost expected for every post now.
Jeff
eppick77
Yes, Tony. It was our conversation that sparked my post.
I had a very good experience with indexes on AP. Then D3 came along
and my whole experience with D3/NT was frustrating with the FSI and
VME giving different results.
I have enjoyed working with D3/Linux and felt pretty comfortable with
the indexes there.
I use indexes extensively on QM and feel very comfortable about them.
Eugene
Dan McGrath
<address.i...@removethis.com.invalid> wrote:
>their side of it. It's quite rare that email follow-ups to open items
>are not automatically sent out for follow-up. So I'm curious to know
There is no such word as "rare".
- Dan
--
Daniel G. McGrath
Binghamton, New York
e-mail: dmcg6174[AT]gmail[DOT]com
Brian Speirs
> On Fri, 01 Jan 2010 13:32:42 -0800, Tony Gravagno
> <address.i...@removethis.com.invalid> wrote:
>
>> their side of it. It's quite rare that email follow-ups to open items
>> are not automatically sent out for follow-up. So I'm curious to know
>
> There is no such word as "rare".
>
> - Dan
So, how DO you like your steak done?
I note that you've cross-posted to alt.usage.english. It must really be
an alternative form of English where rare doesn't exist. Rarely do we
such an assertion!
Brian
Peter Duncanson (BrE)
wrote:
Dan McGrath has a personal problem with the word he objects to. It seems
to have an effect on him like the sound of fingernails on a blackboard,
only worse. Dan has a highly literal and logical mind. The word he
objects to is really two or three words of completely different origin
that have the same spelling and now appear to be a single word with two
or three meanings. This appears to offend his inbuilt sense of logic. He
seems to find the existence of the word intolerable. I do not envy him.
--
Peter Duncanson, UK
(in alt.usage.english)
frosty
> <address.i...@removethis.com.invalid> wrote:
>
>>their side of it. It's quite rare that email follow-ups to open items
>>are not automatically sent out for follow-up. So I'm curious to know
"Dan McGrath" <inv...@invalid.invalid> wrote in message
news:tc07k5tnjntukad12...@4ax.com...
>
> There is no such word as "rare".
But there is such a word as "*PLONK*"!
--
frosty