LetoDbf speed issues on windows 2008r2/2012r2 servers

[Miro Della Bitta]

Hi community, what could be the reason? oplocks disabled on both installations, fair share disabled on win 2012.

The harbour soft i did is on production since last october without problem, but when migrated to a cloud server to improve speed, it didn't improve anything, a process that usually takes 26 secs. is now in 3mins.

Any hint appreciated!

Greetings

Miro

[Alexandre Cavalcante Alencar]

Hi Miro,

With the Windows Server 2012, the FairShare Scheduling (CPU, Disk, Network) was enabled by default. Tune it according to your needs and check if it solves your problems

Local Computer Policy / Computer Configuration / Administrative Templates / Windows Components / Remote Desktop Services / Remote Desktop Session Host / Connections / Turn off Fair Share CPU Scheduling

Registry Key (1 - enabled, 0 - disabled)

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\SessionManager\DFSS\EnableDFSS

Registry Key (1 - enabled, 0 - disabled)

HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Quota System

Disk Registry Key (1 - enabled, 0 - disabled)

HKLM\SYSTEM\CurrentControlSet\Services\TSFairShare\Disk\EnableFairShare

Network Registry Key (1 - enabled, 0 - disabled)

HKLM\SYSTEM\CurrentControlSet\Services\TSFairShare\NetFS\EnableFairShare

Regards

[Miro]

I appreciate your help, but already did all these mods with no result.

i´m now working on win 2008, oplocks disabled, firewall disabled, remote access disabled, identical low speed.

Thanks again!

--
--
You received this message because you are subscribed to the Google
Groups "Harbour Users" group.
Unsubscribe: harbour-users+unsubscribe@googlegroups.com
Web: http://groups.google.com/group/harbour-users

---
You received this message because you are subscribed to a topic in the Google Groups "Harbour Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/harbour-users/GzUFxNSDTsE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to harbour-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[elch]

Hi Miro,

i´m now working on win 2008, oplocks disabled, firewall disabled, remote access disabled, identical low speed.

OPlocks settings relate to SMB protocol (Samba), and in *no* kind to LetoDBf.

---

Where are the databases located at the server ?,
aka accessed from a local hard-drive or 'over network' ?

Maybe show the "letodb.ini" config ...

best regards

Rolf

[Miro]

Hi Rolf, the databases are located at C:\Bases and accessed over network, with no shares, pure client-server and here is my letodb.ini

Thanks for your support!

Server = 
IP = 
Port = 2812
DataPath = C:\Bases
;LogPath = /tmp
Default_Driver = CDX
;Lock_Scheme = 6
;Memo_Type = FPT
Share_Tables = 0
No_Save_WA = 1
Lower_Path = 0
EnableFileFunc = 1
EnableAnyExt = 1
Allow_UDF = 1
Pass_for_Login = 0
Pass_for_Manage = 0
Pass_for_Data = 0
;Pass_File = "leto_users"
Cache_Records = 21
;Max_Vars_Number = 1000
;Max_Var_Size = 67108864
;Tables_Max  = 999
;Users_Max = 99
Debug = 1
Optimize = 1
;AutOrder = 0
;ForceOpt = 0
;TimeOut = 360
;Zombie_Check = 0
;Server_User = advantage
;Server_UID = 1000
;Server_GID = 4
;BC_Services = letodb;
;BC_Interface = eth2
;BC_Port = 2812
DataBase = C:\Bases
Backup = C:\Bases\backup
Mask = *.dbf,*.dbt,*.ntx
Lock = 1
Seconds = 30
Wait = 1
ArcCmd = tar -cvzf /Bases/backup/leto.tar.gz /Bases/backup/*

--

[Miro]

And letodbf running as a service

[elch]

Hi Miro,

with 'accessing over network' i had something in mind,
that "C:\Bases" is not local at server, but a 'mapped network drive'
-- but 'C:\...' sounds very likely the local hard-drive in server.

Once again about 'oplocks':

it doesn't ! touch LetoDB]f] as it uses pure TCP-IP sockets to the clients,

without any protocol like SMB above.

It may matter when the DataPath points to a 'mapped network drives' using SMB ...

( you know this length award-worthy thread about Samba concurrency ;-)

---
letodb.ini looks quite well -- nothing in there, which would cause slowness.
Running as Win-service only makes a difference *how* the server is started.

Two additional remarks:

IP = 

this is equal to an out-commented (';') IP:
then you can connect to LetoDBf over any of multiple network cards in server,
also the 'virtual' loopback (lo) is then possible -- fine!

DataBase = C:\Bases

this is something obsolete option left over from LetoDB

-- obsolete because client application can set the RDD (NTX,CDX,..)

aside the given server default with Leto_DbDriver() 

---
A slow running server is somehow better than *not* running

-- we have to find out why!
Sure i made some recent changes:

if i long enough look at the source,

i get an idea how to let a part look more like a 'poem', or find an obsolete <if> ...

---
Up now i suppose <something> in your environment:

what may be different against before ?
did it happen after an LetoDBf update ?

is the deceleration in general or only for specific operations ?

best regards

Rolf

[Nenad Batoćanin]

Maybe slowdown has nothing to do with LetoDbf. Try a program that works directly with DBF tables on the server.

Did you check: 

System/Advanced system settings/Performance/Data Execution Prevention

and

Processor scheduling/Adjust for best performance of (try different settings)

Also check Windows Defender or other AV program: if they scan dbf/ntx files, everything will be _very_ slow. 

Regards, NB

[Miro]

Thanks, Rolf for your detailed response.

Yes c:\ stands for the hard disk.

Old setting: pentium 4 with 2gb memory, windows 7, 10 mb/s upload internet connection

New setting: Virtual cloud server, 4gb memory, dual processor, 270! mb/s upload internet connection

Same software, same dbf's and same letodbf.

Time spent on the old setting to open and traverse 7 dbf's: 26 secs 

Time spent on the new setting to open and traverse same 7 dbf's: 205 secs

The only change between the 2 situations is the real pc for a virtual pc with higher performance.

I´m sure it's not letodbf, but windows environment

Thanks again

--

[elch]

Hi Miro,

I´m sure it's not letodbf, but windows environment

Let me guess: 'something' with network
( as in general the overall limiting factor )

In the 'tests' directory of LetoDBf is a 'test_mem.prg',
switch into that directory ( the 'hbmk.hbm' there automatize some things ):
hbmk2 test_mem
then: ( xxx == IP or DNS-name )
test_mem xxx

It needs a while!, if ready done it waits [ for two ENTER to end ].
By default it use a DBF in RAM [ mem:test1.dbf ]
-- this way we can exclude HD problems, aka to access a DBF at a drive.

At the bottom: maximum request-rate ?

best regards

Rolf

[elch]

addendum, Miro

test_mem xxx

At the bottom: maximum request-rate ?

 

aim should be to narrow the area where to search further:
if OS hinder the execution of LetoDBf, if something hinder the DBF access at drive or if its the network.

Max request rate is a 'ping-pong' (request-answer), independent of DBF access,
this marks the network limit for other actions.
Time to create an index is one single request, aka shows pure LetoDBf performance,
other values a mixture of both ...

If you like, you can mark&copy the lower half of screen, starting with:
please wait ...

best regards

Rolf

[Miro]

Hi Nenad, no AV, no DEP, same behaviour. Surely is windows the culprit, but where?

Thank you for the help!

--

[Miro]

Hi Rolf, 

test_mem got stuck several times where the screen shows GoTop, seek... 

One of these times i entered the win 2012 server (trying again on 2012 after 2008 wreckage;) Resource Monitor, while test_mem was running, attached screen capture01.png.

The 2012 monitor shows aprox. 1800 b/s and after completing "skipping top to bottom" process, it drops to 65 b/s, while, simultaneously, test-mem doesn't respond, hanged. (capture02.png)

Network problems, no disk usage, as you said, low memory usage neither.

No firewall at the cloud server control panel, no windows firewall.

Thxs & Regards!

[Francesco Perillo]

Is this a cloud server? I mean, on the internet? 

If yes, do the server has throttling or is it rate-limited?

Just for a test, can you install a linux server?

Unsubscribe: harbour-user...@googlegroups.com

Web: http://groups.google.com/group/harbour-users

---
You received this message because you are subscribed to a topic in the Google Groups "Harbour Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/harbour-users/GzUFxNSDTsE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to harbour-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
You received this message because you are subscribed to the Google
Groups "Harbour Users" group.

Unsubscribe: harbour-user...@googlegroups.com
Web: http://groups.google.com/group/harbour-users

---
You received this message because you are subscribed to the Google Groups "Harbour Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.

[Miro]

Yes Francesco, is a cloud server only remotely accesible. We hire it because of the high speed access, 270 mb/s. If you look at the screen capture, you'll see that the remote access through i am watching runs at +- 4.800 b/s while leto runs at 65 b/s.

Next step in my job is to learn linux in order to install a linux server w/samba, letodbf and dbf´s, and throw the f****ing windows through the window.

Thanks a lot

Unsubscribe: harbour-users+unsubscribe@googlegroups.com

Web: http://groups.google.com/group/harbour-users

---
You received this message because you are subscribed to a topic in the Google Groups "Harbour Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/harbour-users/GzUFxNSDTsE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to harbour-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
You received this message because you are subscribed to the Google
Groups "Harbour Users" group.

Unsubscribe: harbour-users+unsubscribe@googlegroups.com
Web: http://groups.google.com/group/harbour-users

---

You received this message because you are subscribed to the Google Groups "Harbour Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to harbour-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
You received this message because you are subscribed to the Google
Groups "Harbour Users" group.

Unsubscribe: harbour-users+unsubscribe@googlegroups.com

Web: http://groups.google.com/group/harbour-users

---
You received this message because you are subscribed to a topic in the Google Groups "Harbour Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/harbour-users/GzUFxNSDTsE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to harbour-users+unsubscribe@googlegroups.com.

[Francesco Perillo]

You may try using this tool to understand what it is happening:
http://blogs.technet.com/b/robertsmith/archive/2012/02/07/analyzing-storage-performance-using-the-windows-performance-toolkit.aspx

http://blogs.technet.com/b/yongrhee/archive/2012/11/23/wpr-xperf-capture-high-cpu-disk-i-o-file-registry-networking-private-bytes-virtual-bytes-paged-pool-nonpaged-pool-and-or-application-slowness.aspx

[Miro]

Thanks, Francesco, i downgraded the server to win 7, and the same issue is happening, receiving speed too low (60 b/s).

I´ll try with these tools and comment, thanks again!

[elch]

Hi Miro,

...I´ll try with these tools and comment, thanks again!

with 'cloud-server' you mean,
there is no hardware at your place as <server>, but 'it' is accessed over Internet ?

And before that, you had a 'real' server at your place with the DBFs on it's hard-drive ?

This case no wonder -- benchmark does not 'hang', it will need many hours :-)

----
This 'crazy' benchmark test_mem produces an average flow rate of 4 M-Byte/s,
with single peeks up to 7 M-Byte/s -- then it is done in a minute.

That translates roughly to average 40 M-BIT,
when i calculate for easiness factor 10 for BYTE <> BIT in *Intra*-net !

This value is the sum of transmitted bytes,
where these are transmitted in *variable sized packages*

[ plus some overhead per *package* ]

Your rate of 270(*) M-BIT refers to *maximum filled* packages,
you may reach while downloading an big image.

(*) is there a clause: 'UP TO' 270 -- or really: guaranteed 270 UP- and DOWN- load ?

Please have a look for an older note, it may help to explain:
https://groups.google.com/d/msg/harbour-users/q_ZqmOB6Sns/HFVZJlcMBwAJ

Then we start talking about the <core problem>:
*latency* difference of Intra-net versus Inter-net.
This are <fractions> of a millisecond ( 1/1000 ) for 1 G-BIT *Intra-net*
versus <X - XX> milliseconds for *Internet*
A common on-board tool to view the latency/ 'round trip time' is <ping>,
and this reflects also the higher <packet loss> rate in Internet.

This difference of INTRA- versus INTER-net is such significant,
that i can leave out of view the obvious <port-mapper>, an we may have even no influence possibility )

[ captura01.png : local 2812 -- remote 4518 ]

As you experienced:

this difference is independent of the any OS at server in the s***! 'cloud' 

----

Have a look into test_mem.prg, search for: leto_Ping().

It is enclosed in a DO WHILE loop, around the loop the time is measured.

Make yourself a snippet, with only a Leto_Connect() to server,

and then use this 'block' around Leto_PING(), starting with:

IF RDDSETDEFAULT() == "LETO"

...

ENDIF

-- *but* lower 150K iterations to only 1000 loops.

Increase that value if too low, test should last just a few seconds.

I like to know the request rate / s.

best regards

Rolf

[Miro]

Hi Rolf, appreciate your help. I´ll take a try with a dedicated server with real windows 7 32bits next monday.

>with 'cloud-server' you mean,
>there is no hardware at your place as <server>, but 'it' is accessed over Internet ?

>And before that, you had a 'real' server at your place with the DBFs on it's hard-drive ?

Exactly, we have a real server connected (still in production) trough an Inet conn. of 10 Mb/s (max) and when we tried (new scenario) with a cloud server Hyper-V win 2012 and 270 Mb/s (max) speed, we got very poor results, an increase of 10 X in waiting times.

Then we start downgrading window$, first we tried with win 2008 server, identical results, then cancelled the account and we hired a dedicated server at iweb, installing a win 7 64bits, 90 Mb/s connection, same results, and now moving into a win 7 32bits.

In order to clarify, the software doesn´t run on the server, it only acts as a file server. I'm opening the dbf's with these command:

USE &Noarchi ALIAS &cNetAlias SHARED NEW VIA 'LETO'

where cNetAlias is an alias made of first 3 characters of dbf's name plus a number from 01 to 99 (no duplicated alias)

If I understood letodbf documentation, leto itself opens the db's in exclusive mode, or shall I do it, instead?

>Have a look into test_mem.prg, search for: leto_Ping().

>It is enclosed in a DO WHILE loop, around the loop the time is measured.

>I like to know the request rate / s.

Modifying now, and testing soon...

For Francesco, I installed the windows performance toolkit, but it never run. I´m tired of MS Bullshit, but must struggle against it. I rather prefer to stand for simple tools like ping, as Rolf suggested.

Thanks for your help!!!

(Sorry for my not fluent english, still i understand it better than i can write it)

 

--

[elch]

Hi Miro,

If I understood letodbf documentation, leto itself opens the db's in exclusive mode, or shall I do it, instead?

 

No !! - stop ! -- all is fine.

Network ! seem to be the problem -- please show me leto_ping() request rate.

[ LetoDBf have 4 server modes, and in one mode it does so.
But keep this box closed, it is *not* the cause of the problem. ]

-----

*INTRA*-net ( my server behind me ): lower as 1 millisecond

F:\>ping 192.168.2.47
Pinging 192.168.2.47 with 32 bytes of data:

Reply from 192.168.2.47: bytes=32 time<1ms TTL=64

...

Compare with:

ping your-server.com

--

[ Leto_Ping() is better -- it proofs that LetoDBf is working ]

best regards

Rolf

[Miro]

Hi Rolf, formatting the server... it will continue...

Regards 

--

[elch]

Hi Miro,

attached you: test_ping.prg
copy it into LetoDBf "tests" directory,
then compile it there: hbmk2 test_ping

Usage is same as test_mem, aka: xxx == IP or DNS-name
test_ping xxx

It is auto-adjusting, aka it measure time for 100x Leto_Ping(),
to determine how many needed to run the test for ~ 10 second.

Please no screenshot, i just want to know the value: xxxx / s.

This will give us a first estimate ...

best regards

Rolf

[Miro]

Thanks for the job you do, Rolf. I´ve already done the modifications suggested and run it over the production connection (older setting), it gave 32ms ( which seems to be wonderfull) but we don´t have the virtual sever hired any more, we were on a 1st. month free usage and finished last weekend.

So i cannot test ping anymore, but, instead we're wondering about a linux letodbf server with windows clients, if you could guide us somehow, it will be very appreciated. I mean, some questions about letodbf and linux, It's necessary to install samba server if there are no needs of share folder in my setup?

Probably it will be on a Centos 7, unless you recommend another distro.

Thanks again!

--

[elch]

Hi Miro,

... (older setting), it gave 32ms ( which seems to be wonderfull) but we don´t have the virtual sever hired any more

the "test_ping" is for use with any LetoDBf server, equal if Intra- or Inter- net.
So can i please know what values it shows with the actually used LetoDBf server ?

---
32 ms is 'out of discussion' --> 32 requests / s --> not use-able !
That was for sure *no* 270 M-Bit Internet connection.

For Intra-net ( server at home ) i talk about 3000 - 6000 requests / s,

pure Linux, aka both server and client, goes up to 10000 ...

 

It's necessary to install samba server if there are no needs of share folder in my setup?

 

LetoDBf needs ... NOTHING !

OK, the network must be configured to be reachable.
( Quite in opposite, using LetoDBf concurrently with Samba makes work and some 'pain'. )

 

Probably it will be on a Centos 7, unless you recommend another distro.

 

I would take a stable Debian -- as Software package management is ".deb" likely Ubuntu.

These both have ~ 70% market share, so expect easy to find help and many tools.

CentOS is a 'fork' of (also) military supplier RedHat  - a question of personal taste ;-),

and with a 'rpm' package management. I have very less experience with it.

best regards

Rolf

[Miro]

>>So can i please know what values it shows with the actually used LetoDBf server ?

This are the values of the actual Letodbf server just in production, with a no-ip ddns connection

 32  requests / s (Wan value) running test from different city locations

508 requests / s (Lan Value) running test from a local pc

920 requests / s (Local value) running test from the server itself

Each test was ran as: test_ping nnvtecnica.ddns.net

Very far from your values, as you can see.

Clarifying, we never ran test_ping against the 270Mb / s server (and impossible by now), that will be coming soon, probably next week.

Best Regards and agree with your taste about CentOS ;)

Miro

 

--

[elch]

Hi Miro,

my justifiable opinion:
data in the cloud is no more your data -- it belongs others.
Server out the cloud is ultimate MCA [ can't throw it out the window 8-) ]

---
Correct me if i'm wrong:
270 M-Bit is the Internet-capability at the cloud server itself!
Not changed provider & contract at your side, you will still have a 10 M-Bit Internet connection to it.

---
This 'request-rate' translates directly into performance,
as each DBF related action [ DbGoto, Dbseek, ... ] is a request to server, then waiting for the answer from server.
So 3000 versus 500 means: mentioned 26 s from initial post --> 5-6 s.

Hardware requirement:

each connection to server is a thread, so LetoDBf server likes multiple CPU cores.

I would suggest at very least two CPU cores, 4 is better.

Computing power requirement for each core is moderate.

And as multiple times said:
*network* is the limiting factor, so 1G-Bit cards should be taken for granted.
Often overlooked is a 'good' switch.

----
Below the line:
i see there very much potential in upgrading your hardware server at home,
plus looking for a provider offering significant higher Internet speed ( if anyhow possible ! )

best regards
Rolf

[Miro]

Hi Rolf, Francesco & others

>> Correct me if i'm wrong:
>> 270 M-Bit is the Internet-capability at the cloud server itself!
>> Not changed provider & contract at your side, you will still have a 10 M-Bit Internet connection to it.

Yes, true story.

Sorry for the delay, i was moving the whole thing to linux ( and learning a new OS), recompiling harbour and letodbf on debian 8 and trying to run it on a linux cloud server. It runs as a daemon, but throwing letodbf error 1001 (file not found, dos error 2) when trying to access any dbf. Such dbfs were build on a windows pc and were granted with r/w/x permissions. 

How can i solve this issue?

Sorry again for the thread disruption, but some times are overwhelming.

Best Regards

--

[Francesco Perillo]

Linux filenames are case sensitive for example...

Unsubscribe: harbour-user...@googlegroups.com

Web: http://groups.google.com/group/harbour-users

---
You received this message because you are subscribed to a topic in the Google Groups "Harbour Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/harbour-users/GzUFxNSDTsE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to harbour-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
You received this message because you are subscribed to the Google
Groups "Harbour Users" group.

Unsubscribe: harbour-user...@googlegroups.com
Web: http://groups.google.com/group/harbour-users

---

You received this message because you are subscribed to the Google Groups "Harbour Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.

[Miro]

Good point! 

Question: Is it mandatory adding ".DBF" to the file name?

Thanks

Unsubscribe: harbour-users+unsubscribe@googlegroups.com

Web: http://groups.google.com/group/harbour-users

---
You received this message because you are subscribed to a topic in the Google Groups "Harbour Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/harbour-users/GzUFxNSDTsE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to harbour-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
You received this message because you are subscribed to the Google
Groups "Harbour Users" group.

Unsubscribe: harbour-users+unsubscribe@googlegroups.com
Web: http://groups.google.com/group/harbour-users

---

You received this message because you are subscribed to the Google Groups "Harbour Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to harbour-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
You received this message because you are subscribed to the Google
Groups "Harbour Users" group.

Unsubscribe: harbour-users+unsubscribe@googlegroups.com

Web: http://groups.google.com/group/harbour-users

---
You received this message because you are subscribed to a topic in the Google Groups "Harbour Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/harbour-users/GzUFxNSDTsE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to harbour-users+unsubscribe@googlegroups.com.

[Miro]

Yes it´s mandatory, and indexes extensions are lowercase (.cdx)

It´s working now, finally. 

Thank´s everyone, specially to Rolf

[elch]

Hi!!, Miro

sorry for delay, overlooked the post as was 'locked' with an update ;-)

---

Yes it´s mandatory, and indexes extensions are lowercase (.cdx)

..

Good point! 

Question: Is it mandatory adding ".DBF" to the file name?

 

No !

In DOS-age all filenames were UPPERcase,

in Unix [Linux] environment filenames are commonly lowercase,

as all default file extensions used by Harbour in Linux are lowercase.

* This is the cause, Harbour doesn't found the .DBFs if extension is left away *

Best 'instant' workaround, in short:

rename all files to lowercase [ google for the command ]

*plus* use letodb.ini config option: <Lower_Path> [ see ReadMe.txt ]

On the long run, maybe rename filenames in source code to lowercase.

This would 'fit' to *ix OS' and they will be found in Windows ...

-----

Next point would be, <who> handles the database files.

This is the 'user' <who> started the server executable.

If this is done by the <root> user in a startup-script,

which maybe is not adviced to do that, but better a privileged user,

we can set it with <Server_User> -or- <Server_UID> + <Server_GID>.

best regards

Rolf

[elch]

Hi, Miro

>> Correct me if i'm wrong:

>> 270 M-Bit is the Internet-capability at the cloud server itself!
>> Not changed provider & contract at your side, you will still have a 10 M-Bit Internet connection to it.

Yes, true story.

this will be a topic: LetoDbf speed issues on .... a 10 M-BIT connection.

LetoDBf follows the aim: use your source 'as is', just link the RDD -- and 'feel good' ... ;-)

[ In 'best case', by using a 'rddleto.ini' along your application,

 it needs not even to add a line with Leto_Connect() ... ]

BUT for such low bandwidth connection, we should [ and can ! ] 'tweak' the application.

General aim is to reduce network traffic as very most as possible,

and the TCP <packages> to be 'well filled' up to max package size.

I just uploaded a longer prepared update,

there is in "Readme.txt" a new chapter:

11 Trouble-Shooting, and therein some tips for low-bandwidth.

Best to update for some new features ;-)

----

If not already done,

build the server monitor console:'utils/manager/console.hbp'.

With that we can temporary! change server report (debug) level,

with ALT-M[enu] -> 'D'ebug level.

[ IMO more 'easy' alternative as to add in source: RDDInfo( RDDI_DEBUGLEVEL, nValue ) ]

[ Note: debug-level is guilty for all user, aka all active connections will start to babble like crazy ! ]

So start server monitor,

then the application and run it up to the point where a 'long lasting task' starts,

then change debug-level to <15> and let the application do its work,

then reset ! debug-level back to <1>

So you get a 'letodbf_01.log' -- rename it to different name,

as the next connection with "ID 1" will overwrite it.

Look into: that is going on as traffic between client and server.

Example:

# many! consecutive ! "<< skip" in less than a second --> skipbuffer increase

[ where each "<< skip" returns as many! records as skipbuffer is set to ]

# Transactions may be a topic, if data is consecutive updated

You can see also the size of the *send* answer:

a well full-filled TCP-package is about 1500 bytes.

I.e. questions to SKIP or the answers to a GOTO will ever be short ones,

wasting network capability ...

You may have some further questions, welcome ! to ask.

Optimizing LetoDBf usage 'depends' on specific situation,

so do NOT post here these large ! server logs at whole.

best regards

Rolf

[Mario H. Sabado]

Many thanks Rolf for your recent updates!

Best regards,
Mario

--

--
You received this message because you are subscribed to the Google
Groups "Harbour Users" group.

Unsubscribe: harbour-user...@googlegroups.com
Web: http://groups.google.com/group/harbour-users

---

You received this message because you are subscribed to the Google Groups "Harbour Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to harbour-user...@googlegroups.com.

[Miro]

Many thanks Rolf, glad to read you again!

Discovered that was making leto server for linux with hb3.2 and rddleto.lib for windows client with hb3.0 (my stupid fault)

Trying to update rddleto.lib to hb3.2 and bcc6.5 with no success:

hbmk2 rddleto.hbp -trace -info

hbmk2: Auto-detected platform: win

hbmk2: Auto-detected C compiler: bcc

hbmk2: Usando Harbour: C:\hb32\bin C:\hb32\include C:\hb32\lib\win\bcc

       C:\hb32\bin C:\hb32\contrib C:\hb32\addons

hbmk2: Usando compilador C: C:\Borland\BCC65\Bin\bcc32.exe

hbmk2: Harbour: Compilando módulos...

hbmk2: Harbour compiler command (built-in):

(C:\hb32\bin\harbour.exe) -n1 source\client\rddsys.prg -n -w3 -q0 -es2 -gc3 -oli

b\.hbmk\win\bcc\ -iC:\Borland\BCC65\Include -iC:\Borland\BCC65\Include\dinkumwar

e -iC:\Borland\BCC65\Include\windows\crtl -iC:\Borland\BCC65\Include\windows\rtl

 -iC:\Borland\BCC65\Include\windows\sdk -iC:\hb32\include -iinclude -undef:.ARCH

. -D__PLATFORM__WINDOWS -D__LITTLE_ENDIAN__ -D__ARCH32BIT__

hbmk2: Compilando...

hbmk2: Comando de compilador C/C++:

bcc32.exe -c -q -CP437 -d -O2 -OS -Ov -Oc -Oi -6  -DUSE_PMURHASH=1 -tWM -w -Q -w

-sig- -nlib\.hbmk\win\bcc -IC:\Borland\BCC65\Include -IC:\Borland\BCC65\Include\

dinkumware -IC:\Borland\BCC65\Include\windows\crtl -IC:\Borland\BCC65\Include\wi

ndows\rtl -IC:\Borland\BCC65\Include\windows\sdk -IC:\hb32\include -Iinclude lib

\.hbmk\win\bcc\rddsys.c source\client\letocl.c source\client\leto1.c source\clie

nt\letomgmn.c source\common\blowfish.c source\common\common_c.c source\common\PM

urHash.c

lib\.hbmk\win\bcc\rddsys.c:

source\client\letocl.c:

source\client\leto1.c:

source\client\letomgmn.c:

source\common\blowfish.c:

source\common\common_c.c:

source\common\PMurHash.c:

hbmk2: Creando librería estática... lib\rddleto.lib

hbmk2: Comando LIB:

tlib.exe @C:\DOCUME~1\Miro\CONFIG~1\Temp\1hql74.lnk

hbmk2: Archivo de órdenes de biblioteca:

/P128   lib\rddleto.lib -+ lib\.hbmk\win\bcc\rddsys.obj -+ lib\.hbmk\win\bcc\let

ocl.obj -+ lib\.hbmk\win\bcc\leto1.obj -+ lib\.hbmk\win\bcc\letomgmn.obj -+ lib\

.hbmk\win\bcc\blowfish.obj -+ lib\.hbmk\win\bcc\common_c.obj -+ lib\.hbmk\win\bc

c\PMurHash.obj

TLIB 6.3 Copyright (c) 1987-2011 Embarcadero Technologies, Inc.

/P128   lib\rddleto.lib -+ lib\.hbmk\win\bcc\rddsys.obj -+ lib\.hbmk\win\bcc\let

ocl.obj -+ lib\.hbmk\win\bcc\leto1.obj -+ lib\.hbmk\win\bcc\letomgmn.obj -+ lib\

.hbmk\win\bcc\blowfish.obj -+ lib\.hbmk\win\bcc\common_c.obj -+ lib\.hbmk\win\bc

c\PMurHash.objWarning: 'rddsys' not found in library

Warning: 'letocl' not found in library

Warning: 'leto1' not found in library

Warning: 'letomgmn' not found in library

Warning: 'blowfish' not found in library

Warning: 'common_c' not found in library

Warning: 'PMurHash' not found in library

But rddsys.obj, letocl.obj, etc, do exist in folder ..\lib\.hbmk\win\bcc

Any clue, please?

Best Regards!

Miro

Unsubscribe: harbour-users+unsubscribe@googlegroups.com
Web: http://groups.google.com/group/harbour-users

---

You received this message because you are subscribed to the Google Groups "Harbour Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to harbour-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
You received this message because you are subscribed to the Google
Groups "Harbour Users" group.

Unsubscribe: harbour-users+unsubscribe@googlegroups.com
Web: http://groups.google.com/group/harbour-users

---

You received this message because you are subscribed to a topic in the Google Groups "Harbour Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/harbour-users/GzUFxNSDTsE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to harbour-users+unsubscribe@googlegroups.com.

[elch]

Hi Miro,

...

TLIB 6.3 Copyright (c) 1987-2011 Embarcadero Technologies, Inc.

Warning: 'letocl' not found in library

....

But rddsys.obj, letocl.obj, etc, do exist in folder ..\lib\.hbmk\win\bcc

That is old ! Borland C++ and their 'tlib.exe', it throw a warning':
...

Warning: 'letocl' not found in library

...

Where it is not even worth a 'note' about,
when 'tlib' have to add an 'object' to a 'library', not in the lib before.
OH!, something new to add ...

---
All looks well so far, just ignore that silly! <warning> -- continue !!

best regards
Rolf

[elch]

Hi Miro,

...  rddleto.lib to hb3.2 and bcc6.5 ...

I got a serious warning for BCC 5.5! about 'LZ4':
LZ4 is a 3rd party compression library included in LetoDBf.

As result i have disabled for *all BCC* versions to build with LZ4.
Then BCC will use instead 'classic' but much slower ZLIB compression.

This will get a problem, if the server is build with a different compiler.
Because *all other C-compiler* will build LetoDBf *with* LZ4 compression.
Both sides -- the server and the LetoDBf client library -- must use *same* compression !

Two possible ways to go:
#1: disable LZ4 for the server
#2: enable LZ4 for client with BCC 6.5

#2: would be nice,

BUT we must check together, if BCC 6.5! can build LZ4 -- never tested.
In rddleto[addon].hbp, second line, we remove the restriction for BCC:
{!bcc}-env:__LZ4=yes
-->
-env:__LZ4=yes

* I need to see what you get as feedback, when building the client lib *

I expect one or more warnings to appear -- show them ...

#1: is ever possible, you can use immediate:
the second line in "letodb.hbp":
{!bcc}-env:__LZ4=yes
This "{!bcc}" means not! BCC, but all other -- so comment the line out: -->
#{!bcc}-env:__LZ4=yes
then no compiler will build with LZ4

best regards

Rolf

[Miro]

Hi Rolf!

Sorry, you asked for it;)

C:\LetoDBf-master>hbmk2 rddleto -rebuild -trace -info

hbmk2: Auto-detected platform: win

hbmk2: Auto-detected C compiler: bcc

hbmk2: Using Harbour: C:\minigui\harbour\bin C:\minigui\harbour\include

       C:\minigui\harbour\lib C:\minigui\harbour\bin C:\minigui\harbour\contrib

       C:\minigui\harbour\addons

hbmk2: Using C compiler: C:\Borland\BCC65\Bin\bcc32.exe

hbmk2: Compiling Harbour sources...

hbmk2: Harbour compiler command (built-in):

(C:\minigui\harbour\bin\harbour.exe) -n1 source\client\rddsys.prg -n -w3 -q0 -es

2 -gc3 -olib\.hbmk\win\bcc\ -iC:\Borland\BCC65\Include -iC:\Borland\BCC65\Includ

e\dinkumware -iC:\Borland\BCC65\Include\windows\crtl -iC:\Borland\BCC65\Include\

windows\rtl -iC:\Borland\BCC65\Include\windows\sdk -iC:\minigui\harbour\include

-iinclude -isource\3rd\lz4\lib

hbmk2: Compiling...

hbmk2: C/C++ compiler command:

bcc32.exe -c -q -CP437 -d -O2 -OS -Ov -Oc -Oi -6  -DUSE_PMURHASH=1 -DUSE_LZ4=1 -

tWM -w -Q -w-sig- -nlib\.hbmk\win\bcc -IC:\Borland\BCC65\Include -IC:\Borland\BC

C65\Include\dinkumware -IC:\Borland\BCC65\Include\windows\crtl -IC:\Borland\BCC6

5\Include\windows\rtl -IC:\Borland\BCC65\Include\windows\sdk -IC:\minigui\harbou

r\include -Iinclude -Isource\3rd\lz4\lib lib\.hbmk\win\bcc\rddsys.c source\clien

t\letocl.c source\client\leto1.c source\client\letomgmn.c source\common\blowfish

.c source\common\common_c.c source\3rd\lz4\lib\lz4.c source\common\lz4net.c sour

ce\common\PMurHash.c

lib\.hbmk\win\bcc\rddsys.c:

source\client\letocl.c:

WARNING: You need to implement LZ4_DEPRECATED for this compiler

source\client\leto1.c:

WARNING: You need to implement LZ4_DEPRECATED for this compiler

source\client\letomgmn.c:

WARNING: You need to implement LZ4_DEPRECATED for this compiler

source\common\blowfish.c:

WARNING: You need to implement LZ4_DEPRECATED for this compiler

source\common\common_c.c:

source\3rd\lz4\lib\lz4.c:

WARNING: You need to implement LZ4_DEPRECATED for this compiler

Warning W8072 source\3rd\lz4\lib\lz4.c 270: Suspicious pointer arithmetic in fun

ction LZ4_NbCommonBytes

Warning W8055 source\3rd\lz4\lib\lz4.c 299: Possible overflow in shift operation

 in function LZ4_NbCommonBytes

Warning W8055 source\3rd\lz4\lib\lz4.c 299: Possible overflow in shift operation

 in function LZ4_NbCommonBytes

source\common\lz4net.c:

WARNING: You need to implement LZ4_DEPRECATED for this compiler

source\common\PMurHash.c:

hbmk2: Creating static library... lib\rddleto.lib

hbmk2: Lib command:

tlib.exe @C:\DOCUME~1\Miro\CONFIG~1\Temp\e7kvr8.lnk

hbmk2: Lib script:

/P64   lib\rddleto.lib -+ lib\.hbmk\win\bcc\rddsys.obj -+ lib\.hbmk\win\bcc\leto

cl.obj -+ lib\.hbmk\win\bcc\leto1.obj -+ lib\.hbmk\win\bcc\letomgmn.obj -+ lib\.

hbmk\win\bcc\blowfish.obj -+ lib\.hbmk\win\bcc\common_c.obj -+ lib\.hbmk\win\bcc

\lz4.obj -+ lib\.hbmk\win\bcc\lz4net.obj -+ lib\.hbmk\win\bcc\PMurHash.obj

TLIB 6.3 Copyright (c) 1987-2011 Embarcadero Technologies, Inc.

/P64   lib\rddleto.lib -+ lib\.hbmk\win\bcc\rddsys.obj -+ lib\.hbmk\win\bcc\leto

cl.obj -+ lib\.hbmk\win\bcc\leto1.obj -+ lib\.hbmk\win\bcc\letomgmn.obj -+ lib\.

hbmk\win\bcc\blowfish.obj -+ lib\.hbmk\win\bcc\common_c.obj -+ lib\.hbmk\win\bcc

\lz4.obj -+ lib\.hbmk\win\bcc\lz4net.obj -+ lib\.hbmk\win\bcc\PMurHash.objWarnin

g: 'rddsys' not found in library

Warning: 'letocl' not found in library

Warning: 'leto1' not found in library

Warning: 'letomgmn' not found in library

Warning: 'blowfish' not found in library

Warning: 'common_c' not found in library

Warning: 'lz4' not found in library

Warning: 'lz4net' not found in library

Warning: 'PMurHash' not found in library

[elch]

Hi Miro,

We get the explained 'warnings' of TLIB,
some harmless (LetoDBf don't need) warnings about LZ4_DEPRECATED,

but these made me headaches already with BCC 5.5:

...

source\3rd\lz4\lib\lz4.c:

WARNING: You need to implement LZ4_DEPRECATED for this compiler

Warning W8072 source\3rd\lz4\lib\lz4.c 270: Suspicious pointer arithmetic in fun

ction LZ4_NbCommonBytes

Warning W8055 source\3rd\lz4\lib\lz4.c 299: Possible overflow in shift operation

 in function LZ4_NbCommonBytes

Warning W8055 source\3rd\lz4\lib\lz4.c 299: Possible overflow in shift operation

 in function LZ4_NbCommonBytes

Its about SHIFTing in 64 Bit values,
and the warnings imply that BCC doesn't understand the source correct,

and/or possible will have problems handling that.

--> #1 of last post

rddleto.hbp: [ no LZ4 for BCC ]
{!bcc}-env:__LZ4=yes

letodb.hbp: [ no LZ4 for no compiler for the server ]
#{!bcc}-env:__LZ4=yes

to get both sides in sync.

[ Or to use another C-compiler for Windows .....

but for the moment we can live with above ]

best regards

Rolf

[Miro]

Hi Rolf 

Victory! Applying --> #1 of last post solution, it compiled without LZ4 warnings and is running very! fast when connecting to windows 7 letodb server. 

I've no problem on using another compiler enabling LZ4 compression, if you tell me which one.

Thanks again!!!

Best Regards

--

[elch]

Hi Miro,

I've no problem on using another compiler enabling LZ4 compression, if you tell me which one.

in the binary Harbour package at SF:
https://sourceforge.net/projects/harbour-project/files/
--> binaries-windows --> nightly --> .7z
you will find MinGW 5.3 in: comp/mingw/bin

This directory (and the 'bin' from Harbour package ) you should add to your OS search path for executables.

Refresh also LetoDbf again, just got an update ...
[ it overwrites also the changes we did for BCC ( the '#' in letodb.hbp ) ]


###
back to the real topic of this thread: performance on 10 MBit

> ... a process that usually takes 26 secs ...
sounds 'tweak-able' to me -- lets try to bring it down to less 5 secs .. 8-)

# does this process change data or only read records ?
# average record count in DBF tables: 10K, 100K or millions ?
# SET FILTER conditions used ?

best regards
Rolf

[Miro]

Hi Rolf, when issuing a Leto_ping from test_mem_mod with 100 loops to a "homemade" win 7 32bits server i´m getting 31.54 requests/sec, on the other side, issuing the same to a win 2008 64 bits CLOUD server gives 3.67 req/sec.

That's a mistery for me. Using same rddleto and same letodbf server versions, from the same PC.

Regarding to my soft:

# does this process change data or only read records ?

Read only  

# average record count in DBF tables: 10K, 100K or millions ?

Shaming 200 records

# SET FILTER conditions used ?

No Set Filter

Average time for getting 200 records:

From win7 server: 5 sec

From cloud server: 54 sec

I wonder if cloud servers virtualization affects letodbf.

Now trying to compile with mingw...it will continue...

Best Regards and thanks!!