Big speed difference between EXCLUSIVE and SHARED

[Enrico Maria Giordano]

Dear friends, I found a big speed difference in skipping and in creating
indexes between DBFs open in EXCLUSIVE mode and in SHARED mode:

EXCLUSIVE:

Skip: 22 secs
Create index: 0.26 secs
Skip with index: 0.25

SHARED:

Skip: 128 secs
Create index: 14 secs
Skip with index: 125 secs

Is it expected or is there a trick I can use to speed up operations in
shared mode?

Thank you in advance.

EMG

--
EMAG Software Homepage: http://www.emagsoftware.it
The EMG's ZX-Spectrum Page: http://www.emagsoftware.it/spectrum
The Best of Spectrum Games: http://www.emagsoftware.it/tbosg
The EMG Music page: http://www.emagsoftware.it/emgmusic

[Massimo Belgrano]

Creating index in an activity that you can do in esclusive
so imo it is not a good example about differences between index and shared
i typically use exclusive and create index
so the trip is use esclusive if is possible

i create index in excliusive and in server side
netio will be a good way for do it

can you post the index expression that give this differences?
with time if executed on the server and on client?
2010/7/23 Enrico Maria Giordano <e.m.gi...@emagsoftware.it>:

--
Massimo Belgrano

[druzus]

On 23 Lip, 11:05, "Enrico Maria Giordano"
<e.m.giord...@emagsoftware.it> wrote:

Hi,

> Dear friends, I found a big speed difference in skipping and in creating
> indexes between DBFs open in EXCLUSIVE mode and in SHARED mode:

I guess you made tests on remote drive and your results confirms
standard
MS-Windows network redirector behavior but has nothing to Harbour.
Of course it's also possible that with some additional conditions it's
possible
to replicate such results even locally anyhow the reasons will be the
same.

> EXCLUSIVE:
> Skip: 22 secs
> Create index: 0.26 secs
> Skip with index: 0.25

The file is open in exclusive mode so in the 1-st
pass data is read by windows network client in
some blocks f.e. 1024 bytes length (I do not know
the exact buffer size) so few records are read in
one network read request and then retrieve from
cache (your shared results suggests that ~6 records
so you can check the buffer size using record size
in your DBF). Then read data is stored in local cache.
It's safe because file is open in exclusive mode so
other stations cannot change it.
In all other operations network client access data from
this local cache so the performance is not reduced by
network overhead. Also write operations when index is
created are buffered and the index data is send to the
server asynchronously on some spare time or just before
file is closed.

> SHARED:
> Skip: 128 secs
> Create index: 14 secs
> Skip with index: 125 secs

File is open in shared mode and oprotunistic locks
are disabled so all operations generate real network
read request and network redirector cannot buffer file
context on the local side.
The indexing is faster then skipping though it also
needs to scan whole DBF file because in native RDDs
I implemented read ahead buffers so more then one
record is read in single read operation. It's CL5.3
compatible feature which can be disabled by
SET STRCITREAD ON

> Is it expected or is there a trick I can use to speed up operations in
> shared mode?

Forget about MS-Windows and switch to some serious operating
system and execute your programs remotely ;-).

best regards,
Przemek

ps. I'm OFF-LINE and I have very irregular inet access so I cannot
answer for any e-mails send to me. I'll be ON-LINE probably in a
month.

[Enrico Maria Giordano]

-----Messaggio Originale-----
Da: "francesco perillo" <fper...@gmail.com>
A: <harbou...@googlegroups.com>
Data invio: venerd� 23 luglio 2010 11.25
Oggetto: Re: [harbour] Big speed difference between EXCLUSIVE and SHARED

> in Exclusive you are the ONLY user, so you can use cache/read-ahead
> tecniques, like reading 100 records with one disk read.
>
> in Shared mode you NEED to read one record at time, so 100 records,
> 100 disk reads (it is simplified example)

Ok, but is there any workaround?

[Enrico Maria Giordano]

-----Messaggio Originale-----
Da: "Massimo Belgrano" <mbel...@deltain.it>
A: <harbou...@googlegroups.com>
Data invio: venerd� 23 luglio 2010 11.50

Oggetto: Re: [harbour] Big speed difference between EXCLUSIVE and SHARED

> can you post the index expression that give this differences?

The slowness is without index too, just skipping: 22 vs. 128 secs.

[Enrico Maria Giordano]

-----Messaggio Originale-----
Da: "druzus" <dru...@acn.waw.pl>
A: "Harbour Developers" <harbou...@googlegroups.com>
Data invio: venerd� 23 luglio 2010 12.55
Oggetto: [harbour] Re: Big speed difference between EXCLUSIVE and SHARED

> > Is it expected or is there a trick I can use to speed up operations in
> > shared mode?
>
> Forget about MS-Windows and switch to some serious operating
> system and execute your programs remotely ;-).

I meant, a "feasible" workaround. :-)

[Tamas TEVESZ]

On Fri, 23 Jul 2010, Enrico Maria Giordano wrote:

> > > Is it expected or is there a trick I can use to speed up operations in
> > > shared mode?
> >
> > Forget about MS-Windows and switch to some serious operating
> > system and execute your programs remotely ;-).
>
> I meant, a "feasible" workaround. :-)

a client-server scenario like ads, or running your app remotely. other
than that, no.

the moment any sort of network file system comes to play, performance
gets scared and hides in the basement.

--
[-]

mkdir /nonexistent

[Qatan]

Hello,

I'd suggest NETIO but didn't test it yet although very interested in
using it.
I'm not sure how would be the test results using it but a comparation
would be very nice at this point.

Qatan

[Enrico Maria Giordano]

-----Messaggio Originale-----
Da: "Tamas TEVESZ" <i...@extreme.hu>
A: <harbou...@googlegroups.com>
Data invio: venerd� 23 luglio 2010 18.47
Oggetto: Re: [harbour] Re: Big speed difference between EXCLUSIVE and SHARED

> a client-server scenario like ads, or running your app remotely. other
> than that, no.
>
> the moment any sort of network file system comes to play, performance
> gets scared and hides in the basement.

Ok, thank you. :-(

[Enrico Maria Giordano]

-----Messaggio Originale-----
Da: "Massimo Belgrano" <mbel...@deltain.it>

A: <harbou...@googlegroups.com>
Data invio: venerd� 23 luglio 2010 19.09
Oggetto: Re: [harbour] Re: Big speed difference between EXCLUSIVE and SHARED

> Without an altruistic motivation, scientists cannot distinguish between
> beneficial technologies and the merely expedient.

???

[Enrico Maria Giordano]

-----Messaggio Originale-----
Da: "Qatan" <wanst...@gmail.com>
A: <harbou...@googlegroups.com>
Data invio: venerd� 23 luglio 2010 19.50
Oggetto: Re: [harbour] Re: Big speed difference between EXCLUSIVE and SHARED

> Hello,
>
> I'd suggest NETIO but didn't test it yet although very interested in
> using it.
> I'm not sure how would be the test results using it but a comparation
> would be very nice at this point.

Thank you.

[Massimo Belgrano]

Only a good assertion and a common word :expedient.

2010/7/23 Enrico Maria Giordano <e.m.gi...@emagsoftware.it>

-----Messaggio Originale----- Da: "Massimo Belgrano" <mbel...@deltain.it>

A: <harbou...@googlegroups.com>
Data invio: venerdì 23 luglio 2010 19.09

Oggetto: Re: [harbour] Re: Big speed difference between EXCLUSIVE and SHARED

Without an altruistic motivation, scientists cannot distinguish between
beneficial technologies and the merely expedient.

???

EMG

--
Massimo Belgrano

[ToninhoFWi]

> I meant, a "feasible" workaround. :-)

Enrico. I had the same situation like your in some months ago. I have a
table with 43000 records and about 100 fields and common
while/replace/skip/enddo process needed about 4 minutes to be complete. I
replace it by a sql script and the time now is: (believe me) 8 seconds using
ADS.

I sugest you try netio or use ADS with server side process.

Regards,

Toninho.

__________________________________________________
Fale com seus amigos de gra�a com o novo Yahoo! Messenger
http://br.messenger.yahoo.com/

[Massimo Belgrano]

Maj be very usefull see  part of source code first and after

Do Use ads via  sql?

2010/7/24 ToninhoFWi <tonin...@yahoo.com.br>

I meant, a "feasible" workaround. :-)

Enrico. I had the same situation like your in some months ago. I have a table with 43000 records and about 100 fields and common while/replace/skip/enddo process needed about 4 minutes to be complete. I replace it by a sql script and the time now is: (believe me) 8 seconds using ADS.

I sugest you try netio or use ADS with server side process.

Regards,

Toninho.

__________________________________________________
Fale com seus amigos  de graça com o novo Yahoo! Messenger http://br.messenger.yahoo.com/

--
Massimo Belgrano

[ToninhoFWi]

Hi Massimo.

 

ADS have full support for SQL, please see ADS DOCs:

 

http://devzone.advantagedatabase.com/dz/webhelp/Advantage10/master_advantage_sql_engine.htm

 

 

Regards,

 

Toninho.

[hmpaquito]

Enrico,

Perhaps, sometimes, you can to copy your .dbf to new.dbf and open
new.dbf in exclusive mode. Perhaps for to copy you can to use MemIO
protocol for copy to ram memory (more faster).

Regards


On 23 jul, 11:05, "Enrico Maria Giordano"

[hmpaquito]

To copy with COPY FILE or command system COPY. Can to be very quick.

[Enrico Maria Giordano]

-----Messaggio Originale-----
Da: "hmpaquito" <hmpa...@gmail.com>
A: "Harbour Developers" <harbou...@googlegroups.com>
Data invio: sabato 24 luglio 2010 9.10
Oggetto: [harbour] Re: Big speed difference between EXCLUSIVE and SHARED

Thanks to all for your suggestions.

[hmpaquito]

Enrico,

Now, I need speed on dbfs. I try on local disk. (my pc disk)

1st. Dbf in exclusive mode. More or less equal speed.
2st. BmDbfcdx from xHarbour is a bit more speed than DbfCdx
3st. Worderfull to create ramdisk unit. It's super fast.

Standard time on init reboot is 320 seconds... with ramdisk is ¡ 7
seconds !


My tests to include a reboot for each test for avoid pc cache.
For 3st test I have empty (I think) after to copy to ramdisk dbf...
how ? Copying something big files in something directory.

Regards

On 24 jul, 10:48, "Enrico Maria Giordano"
<e.m.giord...@emagsoftware.it> wrote:
> -----Messaggio Originale-----
> Da: "hmpaquito" <hmpaqu...@gmail.com>

> A: "Harbour Developers" <harbou...@googlegroups.com>
> Data invio: sabato 24 luglio 2010 9.10

> Oggetto: [harbour] Re:Bigspeeddifference between EXCLUSIVE and SHARED