Oplocks

[Ash]

Hello,

My application is a multiuser one. Should I be concerned about oplocks?

Many thanks for your comments.

Regards

Ash

[Klas Engwall]

Hi Ash,

> My application is a multiuser one. Should I be concerned about oplocks?
>
> Many thanks for your comments.

Yes, definitely, when networking over SMB (= a Windows server or a
Windows workstation acting as a server or a Linux server (or NAS disk)
with Samba. And this applies to all filesharing database systems like
Clipper, Harbour, DataFlex etc etc.

Search the newsgroup archive for the oplocks keyword and you will find
many discussions about it. And hwile you are at it, also study what has
been said about cachedopenlimit, SMB2 and possibly SMB3 too.

Reg files for fixing workstations and servers can be found here:
http://www.witzendcs.co.uk/nt_networking.html

Regards,
Klas

[Agustianes US]

Hi Ash,

If you are using Linux Samba server then you should disable oplocks completely for database related files. In mulituser environment oplocks could ruin your database files.

For example, I have this line in my Samba share definition:

veto oplock files = /*.dbf/*.DBF/*.cdx/*.CDX/*.fpt/*.FPT/

Read more at https://www.samba.org/samba/docs/man/Samba3-HOWTO/locking.html

Hope that helps.

Regards,

Anes

--
--
You received this message because you are subscribed to the Google
Groups "Harbour Users" group.
Unsubscribe: harbour-users+unsubscribe@googlegroups.com
Web: http://groups.google.com/group/harbour-users

--- You received this message because you are subscribed to the Google Groups "Harbour Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to harbour-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ash]

Hello Klas, anes,

I have been using Ubuntu based file servers for many years. It must have been due to good luck that my customers didn't experience damaged tables and/or index files. Change came as one of my customers asked for his server to be upgraded to RAID1 configuration. As software RAID was not an option, too complicated for me to setup and manage, I found Synology DS713+. The performance of this little box is better than that of an Ubuntu box - both with RAID1 configuration and oplocks enabled. I have disabled oplocks in DS713+ for slower performance but ended up with a more reliable server. It has been in use for about a month with over fifteen users sharing data via a Harbour based application.

Many thanks for your guidance.

Regards.

Ash

[Ash]

Hello,

Would I be correct in thinking that as long as one commits updates to rows immediately and that there is no file level processing (create, update, delete) in an application, oplocks can be left enabled? I am asking because with oplocks disabled, my application slows down by a factor of three. I am using Windows 7 Pro for all workstations sharing data on DS713+ NAS in a gigabit network. 

Many thanks for your guidance.

Regards.

Ash

[Klas Engwall]

Hi Ash,

> Would I be correct in thinking that as long as one commits updates to
> rows immediately and that there is no file level processing (create,
> update, delete) in an application, oplocks can be left enabled?

No

Regards,
Klas

[Alex Strickland]

Hi

>
> Would I be correct in thinking that as long as one commits updates to
> rows immediately and that there is no file level processing (create,
> update, delete) in an application, oplocks can be left enabled? I am
> asking because with oplocks disabled, my application slows down by a
> factor of three. I am using Windows 7 Pro for all workstations sharing
> data on DS713+ NAS in a gigabit network.
>

What Klas said.

Regards
Alex

[Daniele Campagna]

I think the question is not to be dismissed quickly.
If the network is 100+Mb/s, no interaction is active with Linux
executables and PCs have no issues (they don't crash, freeze etc. on a
regular basis) oplocks can be left enabled. They become a problem only
in certain circumstances, if I read correctly the technical
documentation from Microsoft and Samba (remote access, slow network,
suddenly frozen PCs etc.)
In a network of Samba and only Windows clients I never had to tinker
with registry, and I experienced only random index corruption. And I
ask, in turn, anybody with oplocks disbled can say he never had index
issues?
This is for me a very fundamental question.
My 2 cents
Dan

[Klas Engwall]

Hi Daniele,

> I think the question is not to be dismissed quickly.
> If the network is 100+Mb/s, no interaction is active with Linux
> executables and PCs have no issues (they don't crash, freeze etc. on a
> regular basis) oplocks can be left enabled. They become a problem only
> in certain circumstances, if I read correctly the technical
> documentation from Microsoft and Samba (remote access, slow network,
> suddenly frozen PCs etc.)

What does Linux executables have to do with it? The network redirector
doesn't care who is requesting the files to be opened. It is just a
request from a workstation somewhere over the SMB protocol, not from any
special kind of executable. And "crash" and "freeze" is what you may see
as a result of oplocks being enabled, iow the oplocks create the issues.

As I have always said in response to oplock related questions for more
than fifteen years, you should really search the web for discussions
about opportunistic locking. Microsoft dismisses shared file access as
"old database systems" (or similar wording) and focuses on the benefits
of oplocks when opening files exclusively, for example in the case of
Word or Excel. Better to instead read what people behind other
filesharing database systems say if you do not believe what seasoned
Clipper/Harbour users have to say about it. Those running DataFlex, to
just mention one example.

> In a network of Samba and only Windows clients I never had to tinker
> with registry, and I experienced only random index corruption. And I
> ask, in turn, anybody with oplocks disbled can say he never had index
> issues?

I have always disabled oplocks since my first experiences with Windows
NT4 servers in the nineties. And I *never* have *any* index corruption
or any other data corruption. Some people talk about reindexing weekly
or nightly or whatever. I never reindex except after changing the dbf
file structure or similar maintenance.

Random corruption is what those who have oplocks enabled experience, you
are absolutely right about that.

> This is for me a very fundamental question.
> My 2 cents

Sorry, but I have to say this: What I do not understand is how this can
still, after all these years, pop up and be questioned.

Regards,
Klas

[Francesco Perillo]

I have this samba configuration. My samba is old... really old... so the syntax may have been changed a bit...

This server was installed on 2008, it is a openSuse 11. I started with suse 8.x.... and I'm sure I was already using a samba based server in 1999..

I don't know if these settings are the best but in all these years I experienced almost no index corruption and the very few I had, some were due to blackouts...

Comments from experts welcome.

# locking - default yes
# If yes, turns on byte-range locks.
        locking=yes

# strict locking - default no
# If yes, denies access to an entire file if a byte-range lock exists in it.
        strict locking = yes

# oplocks - default no
# If yes, turns on local caching of files on the client for this share.
# Was suggested to be yes, but timeout problems arises
        oplocks=no

# blocking locks - default yes
# Allows lock requestor to wait for the lock to be granted.
        blocking locks=yes
# disable oplocks !!!!
# level2 oplocks - default yes
# If yes, allows oplocks to downgrade to read-only.
        level2 oplocks = no

[Daniele Campagna]

Il 02/02/2015 23:00, Klas Engwall ha scritto:
> Hi Daniele,
>
>> I think the question is not to be dismissed quickly.
>> If the network is 100+Mb/s, no interaction is active with Linux
>> executables and PCs have no issues (they don't crash, freeze etc. on a
>> regular basis) oplocks can be left enabled. They become a problem only
>> in certain circumstances, if I read correctly the technical
>> documentation from Microsoft and Samba (remote access, slow network,
>> suddenly frozen PCs etc.)
>
> What does Linux executables have to do with it? The network redirector
> doesn't care who is requesting the files to be opened. It is just a
> request from a workstation somewhere over the SMB protocol, not from
> any special kind of executable. And "crash" and "freeze" is what you
> may see as a result of oplocks being enabled, iow the oplocks create
> the issues.
>

I read about the option (in Samba configuration file) KERNEL OPLOCKS.
It seems that Linux native programs ignore by default the Windows lock
scheme. Since this is my case (an executable under Linux - a cgi written
in Harbour- interacting with Windows workstations on the same .dbf's), I
am trying to understand if (x)Harbour native programs are to be
considered "Linux native programs". I am supposing yes.
So the "disclaimer". :-)

> As I have always said in response to oplock related questions for more
> than fifteen years, you should really search the web for discussions
> about opportunistic locking. Microsoft dismisses shared file access as
> "old database systems" (or similar wording) and focuses on the
> benefits of oplocks when opening files exclusively, for example in the
> case of Word or Excel. Better to instead read what people behind other
> filesharing database systems say if you do not believe what seasoned
> Clipper/Harbour users have to say about it. Those running DataFlex, to
> just mention one example.
>

Yes, I have read the instructions provided by Dataflex
administrators/developers, but still am not sure the point has been
clearly made. They all talk about wide area networks and slow data
access. I understand that in this scenario prolems can arise.

>> In a network of Samba and only Windows clients I never had to tinker
>> with registry, and I experienced only random index corruption. And I
>> ask, in turn, anybody with oplocks disbled can say he never had index
>> issues?
>
> I have always disabled oplocks since my first experiences with Windows
> NT4 servers in the nineties. And I *never* have *any* index corruption
> or any other data corruption. Some people talk about reindexing weekly
> or nightly or whatever. I never reindex except after changing the dbf
> file structure or similar maintenance.
>

This is interesting. On the other side (about index corruption) we have
people that had problems - apparently - in a non-multiuser environment,
particularly with the pack() function, and it seems that this has
nothing to do with oplocks.
Have we any evidence that oplocks in a perfectly-running network
environment with well-configured clients create issues?
This seems to be a key question. Before releasing networked programs I
made a lot of testing, but in my tests I never had a problem, and no
oplocks-strategy was adopted.

> Sorry, but I have to say this: What I do not understand is how this
> can still, after all these years, pop up and be questioned.
>

Because there's nothing wrong "per se" in the oplocks strategy, a
station caches a chunk of a database, and when instructed by the network
server, flushes data and/or deletes the cache and re-reads data...
problems arise only - per Microsoft admission - on network/station
errors/hangs. If all works correctly, it seems that no errors should
arise and a great speed improvement would be granted.

This speed improvement is very important in a networked-day-to-day
working environment. I am not talking about batch programs that do
something in the night, or programs that run unassisted... I am talking
about users that open databases, browse, search, reindex etc. and call
me if a minimal slowness is detected... :-) If disabling oplocks would
slow the working environment by a factor 3, then suddenly I would be
better off 5000 km away from here... and with no known address... :-)
This is the reason of my... reasoning.

Dan

[Klas Engwall]

Hi Daniele,

I have noticed that you had the same discussion with Gerald Drouillard
in the comp.lang.xharbour newsgroup three weeks ago and that he told you
the same things I have said. I see no point in trying to argue this
further. I am not going to convince you, no matter what I say.

Just two more comments before closing:
1) Forget about "Linux native programs" and consider instead *any* two
program instances accessing the same file. That oplocks should be
disabled has been true for Windows only networks since the beinning of
NT, and it is true for Samba networks too.
2) Packing was always a bad idea, and corruption caused by pack is an
entiry different problem than the other one. You looked up the sources
yourself and saw that it re-writes records in the same file. That is not
safe, and it has always been discouraged. Copy to new_file for .not.
deleted() instead. They suggested that in the xhb newsgroup too. The
original pack was written when people had floppy-based systems or 10MB
harddisks, so copying to a new file would easily result in "disk full"
errors. Pack still works the same way, but disks are large and cheap now.

Regards,
Klas

[Klas Engwall]

Hi Francesco,

OK in general, but I am not so sure about the "strict locking" setting.
It seems to prohibit file sharing entirely while any lock is active. But
I am not a Linux admin, so ...

Here is an example (below the OS_NetRegOk() code) of smb.conf, from a
source I trust, that you compare yours with.

https://groups.google.com/d/msg/comp.lang.xharbour/yB7SXI0Zn10/kF1LM7c1TwwJ

Regards,
Klas

[Daniele Campagna]

Il 03/02/2015 03:26, Klas Engwall ha scritto:
> Hi Daniele,
>
> I have noticed that you had the same discussion with Gerald Drouillard
> in the comp.lang.xharbour newsgroup three weeks ago and that he told
> you the same things I have said. I see no point in trying to argue
> this further. I am not going to convince you, no matter what I say.
>

I am not willing too to have discussions pointlessly. And I am not
arguing just for the sake of it. I am trying to understand what went
wrong with my corrupted databases. As I said, everybody says that
oplocks are responsible, and I could believe it, if not for the fact
that I never disabled them and never had problems in maaany years. I
ended up thinking that was Linux kernel v. 3 and/or Samba v.4 that
messed up things. Searching info about these 2 topics brings a lot of
interesting stuff, enough anyway for me to stick with a kernel 2.6 and
Samba 3. F.e:
http://www.databasesoup.com/2014/09/why-you-need-to-avoid-linux-kernel-32.html

The previous exchange in xHarbour NG was about the fact that I was not
aware this oplocks strategy applies to Samba too, and the pack command.

> Just two more comments before closing:
> 1) Forget about "Linux native programs" and consider instead *any* two
> program instances accessing the same file. That oplocks should be
> disabled has been true for Windows only networks since the beinning of
> NT, and it is true for Samba networks too.

The problem is that I tried to disable oplocks but the performance of
some database operations degraded of a factor 10.

BTW please can you explain why you dismiss the possibility that kernel
oplocks could have something to do with lock problems?

Thank you for your time,
Dan

[Francesco Perillo]

On Thu, Feb 5, 2015 at 5:04 PM, Daniele Campagna <cyber...@tiscalinet.it> wrote:

The problem is that I tried to disable oplocks but the performance of some database operations degraded of a factor 10.

Without oplocks you are saying to the operating system to not do any caching of the data and to read/write them to the disk each time. And this is the way that DBFs should be handled.

It is normal that it makes the program go slower... but it is the safe way to go.

If it is toooo slooooow you have to modify the program adding indexes and changing queries, or you can move some data processing to the server (RPC calls in NETIO or LetoDB) or switch to a SQL backend (ADS, mysql, postgres, oracle...)

 

[Daniele Campagna]

Eh, but to change technology to speed up a single operation seems a bit impractical.
Luckily I had a glance at the (very old) sources and I am optimizing them (plenty of room to do it).
I never noticed how much the "actual" (15 years old) code is inefficient, thanks to oplocks... :-)
Anyway another program -more recent- after disabling oplocks showed no slowness at all.
It seems at least that in the last 15 years my programming skills have improved a bit...

Dan

[elch]

Hi Dan,

 

everybody says that oplocks are responsible, and I could believe it, if not for the fact
that I never disabled them and never had problems in maaany years

 

http://www.databasesoup.com/2014/09/why-you-need-to-avoid-linux-kernel-32.html

graphs in link shows a comparison between Kernel 3.2 versus 3.13

( aka a comparison between Ubuntu 12.04 and 14.04 ) -- still not havn't tested 14.04 ..

According my (maybe wrong) researches, the defaults for the OPLOCKs settings changed in some distros,

especially! at client side -- longer ago i adviced to use: cache=none as mount option for cifs at client.

Agree to Klas, that the option: strict locking = yes

really sounds like overkill.

Here is a good document from original Samba doc (server was just un-available, google cache works .. ):

https://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/locking.html

like to cite from there:

"Opportunistic locking is actually an improper name for this feature. The true benefit of this feature is CLIENT-SIDE DATA CACHING .."

and:

"Oplocks is by default set to <on> by Samba on all configured shares"

and:

"Kernel Oplocks are essentially a method that allows the Linux kernel to co-exist with Samba's oplocked files "

which IMHO don't apply to you ...

--

But as long as you don't like to see the logical problem of a WRITE cache at client side as a generally origin problem for shared databases

-- what shell we write else ?

How much RAM have the server ?

May i advice to add more, because Linux server is ever caching it's files there -- and such is even faster than a SSD.

best regards

Rolf

[Francesco Perillo]

On Thu, Feb 5, 2015 at 6:54 PM, Daniele Campagna <cyber...@tiscalinet.it> wrote:

Eh, but to change technology to speed up a single operation seems a bit impractical.

Of course. But that are the possibilities available to us. NETIO should be really simple to implement, and as a nice side-effect, it will allow you to hide DBFs files... I'm very afraid of the ransomware malware to encrypt the dbf... if you remove them from the file system it on't encrypt them.

 

Luckily I had a glance at the (very old) sources and I am optimizing them (plenty of room to do it).

I don't know if it was necessary with old clipper versions, if it solved a problem, or what, but I found in old (and new code)
SET ORDER TO 2

GO TOP

SEEK value

 

I never noticed how much the "actual" (15 years old) code is inefficient, thanks to oplocks... :-)
Anyway another program -more recent- after disabling oplocks showed no slowness at all.
It seems at least that in the last 15 years my programming skills have improved a bit...

:-) As I see, there is a lot of code porting going on in these days...

A couple of days ago I was wondering how many original clipper programs are still in use in Italy and in the world. Quite daily I get news of a clipper program still running somewhere...

[Klas Engwall]

Hi Daniele,

>> 1) Forget about "Linux native programs" and consider instead *any* two
>> program instances accessing the same file. That oplocks should be
>> disabled has been true for Windows only networks since the beinning of
>> NT, and it is true for Samba networks too.
> The problem is that I tried to disable oplocks but the performance of
> some database operations degraded of a factor 10.
>
> BTW please can you explain why you dismiss the possibility that kernel
> oplocks could have something to do with lock problems?

Then let me rephrase it: Forget all the special cases that you have
mentioned and just consider *any* two program instances accessing the
same file. "Linux native programs" are included in "any" just like
Windows programs, local programs, networked programs ...

It is not a question of one kind of programs being more guilty than
another kind of programs, even if you could show us statistics that one
kind is in fact involved more often than another when corruption occurs.
It is simply a question of the way that opportunistick locking works, as
soon as oplocks are enabled there is local caching on the workstations,
and those caches have to be negotiated, terminated and flushed. That is
when things tend to go wrong.

Oplocks are for exclusive use of files, not for shared use.

Regards,
Klas

[Daniele Campagna]

Il 05/02/2015 18:58, elch ha scritto:

According my (maybe wrong) researches, the defaults for the OPLOCKs settings changed in some distros,

especially! at client side -- longer ago i adviced to use: cache=none as mount option for cifs at client.

Thanks for answering, elch. I don't have any Linux client, anyway: the network consists in Windows PC and a Linux server. The Linux server runs a Harbour CGI interacting with dbfs that are in turn shared with the Windows workstations, running a Window program that accesses the dbfs.

This is the reason for I was considering the option kernel oplocks, allegedly aimed to allow for the co-existence of Linux native programs and Windows programs accessing the same files (on a ext2fs Samba-shared directory). Is the CGI a Linux Native program? Of course it is. But it seems that the entire question is irrelevant. Ok.

and:

"Kernel Oplocks are essentially a method that allows the Linux kernel to co-exist with Samba's oplocked files "

which IMHO don't apply to you ...

...

But as long as you don't like to see the logical problem of a WRITE cache at client side as a generally origin problem for shared databases

-- what shell we write else ?

What's wrong "per se" in a cache mechanism? Use of caching is ubiquitous. Processors use cache, and branch prediction, too. When a prediction is wrong, the data fetched are discarded and nothing happens, apart a re-fetching of data. Hard disks use cache, the use of a caching mechanism is fundamental to achieve better performances in every aspect of computation. Servers use cache. Your browser caches data and often is able to display big images so fast just because it has already them in the cache... and so on.

I *do* see the problem of writing the cache when it is stored "remotely", so to say: it is strictly tied to the proper functioning of the medium that connects the two entities and the software layer that manages it all. Time ago, with networks transmitting and receiving data at 1Mb/sec (10Net etc.) and PCs running DOS 8-bit with scarce computational power and subject to crash, maybe the problem was more important. But now? we have powerful workstation running multi-threaded and multi-tasking OSes, and crashes are rare. The network itself grants 100 Mb/sec transfer rate. In this scenario I was wondering if some of the issues of the past are still actual or are just old stuff to dismiss, considering that I have no remote users apart the ones using the CGI, but this is a local access in reality. That's all.

How much RAM have the server ?

May i advice to add more, because Linux server is ever caching it's files there -- and such is even faster than a SSD.

Servers have 2-4 Mb RAM and the same as swap partition... Enough, the system never uses even the half of that.
The data corruption I experienced was caused by something lying between the Kernel 3 and the Samba 4 implementations / configurations. It is my fault not to have made deep tests before upgrading, but you know... never happened before, dating back from Linux Slackware 7 with Samba 2.2.... so I was overconfident.

Regards,
Dan

[Daniele Campagna]

Il 06/02/2015 02:37, Klas Engwall ha scritto:
>
>> BTW please can you explain why you dismiss the possibility that kernel
>> oplocks could have something to do with lock problems?
>
> Then let me rephrase it: Forget all the special cases that you have
> mentioned and just consider *any* two program instances accessing the
> same file. "Linux native programs" are included in "any" just like
> Windows programs, local programs, networked programs ...
>
> It is not a question of one kind of programs being more guilty than
> another kind of programs, even if you could show us statistics that
> one kind is in fact involved more often than another when corruption
> occurs. It is simply a question of the way that opportunistick locking
> works, as soon as oplocks are enabled there is local caching on the
> workstations, and those caches have to be negotiated, terminated and
> flushed. That is when things tend to go wrong.
>
> Oplocks are for exclusive use of files, not for shared use.

Ok Klas, thanks for the answer.
I clarified in another message why I was asking such questions on an old
topic that has already been treated.

Dan

[Francesco Perillo]

Samba 2 didn't Support the new Microsoft protocol so the client reverted to the old one and there were no problems.

Cache coherency is the Main problem of caches. Windows needs to coordinate the cache in beach workstation and the server: when someone updates a record all the othdr workstations should get a message of cache invalid. Do the receive it? Can they handle it? In a correct way?

[Daniele Campagna]

Il 10/02/2015 02:35, Francesco Perillo ha scritto:

Samba 2 didn't Support the new Microsoft protocol so the client reverted to the old one and there were no problems.

SMB 1.0 vs SMB 2, implemented in Samba 3.5/3.6.

Cache coherency is the Main problem of caches. Windows needs to coordinate the cache in beach workstation and the server: when someone updates a record all the othdr workstations should get a message of cache invalid. Do the receive it? Can they handle it? In a correct way?

A detailed explanation here:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365433(v=vs.85).aspx
However the oplocks strategy changes at every new release of Windows, according to Wikipedia. Brrr!...

[Zeljko]

Daniele, you should adhere to Klas Engwall's advice. I have a large Windows network and whenever I turned oplocks on on the server (be it NT4 or Windows server 2012) bad things happened. The indexes (and even dbf files) got corrupted.As soon as oplocks were turned off, there were no corruptions anymore.

[Randy]

Hi Klas (and all),

I noticed in http://www.witzendcs.co.uk/nt_networking.html that SMB2 is being disabled (since you cannot disable oplocks in SMB2). However, in http://www.dataaccess.com/whitepapers/opportunlockingreadcaching.html it recommends that, after you disable SMB2, you re-enable SMB1 while still disabling oplocks with the settings - I believe this is recommended so you still get the other performance benefits of SMB1.

Do you recommend re-enabling SMB1 while disabling oplocks after SMB2 is disabled?

Thanks,
Randy.

On Thursday, January 22, 2015 at 11:49:23 AM UTC-5, Klas Engwall wrote:

[Randy]

...also, https://support.microsoft.com/en-us/kb/2696547 states that you must use the PowerShell cmdlet to disable SMB2/3 on Windows 8 and Windows Server 2012 instead of using the registry.

[Klas Engwall]

Hi Randy,

> ...also, https://support.microsoft.com/en-us/kb/2696547 states that you

> must use the PowerShell cmdlet to disable SMB2/3 onWindows 8 and Windows

> Server 2012 instead of using the registry.
>
> On Wednesday, August 3, 2016 at 2:54:20 PM UTC-4, Randy wrote:
>
> Hi Klas (and all),
>
> I noticed in http://www.witzendcs.co.uk/nt_networking.html
> <http://www.witzendcs.co.uk/nt_networking.html> that SMB2 is being
> disabled (since you cannot disable oplocks in SMB2). However, in

> http://www.dataaccess.com/whitepapers/opportunlockingreadcaching.html <http://www.dataaccess.com/whitepapers/opportunlockingreadcaching.html>

> it recommends that, after you disable SMB2, you re-enable SMB1 while
> still disabling oplocks with the settings - I believe this is
> recommended so you still get the other performance benefits of SMB1.
>
> Do you recommend re-enabling SMB1 while disabling oplocks after SMB2
> is disabled?

It has been a few years since I used a "real" Windows (TM) server, the
networks I am involved in are all Samba based nowadays.

But, in general, following what the Dataflex folks say about the latest
Windows versions is a good idea. They deal with the exact same problems
in the SMB environment as Clipper/Harbour users do.

About PowerShell, does the KB really say that? All I can see is "use
Windows PowerShell or Registry Editor" regarding the server. It does not
specifically mentioon the registry editor (either way) on the client
side, but AFAIK it should still work.

Regards,
Klas