Memory Leaks

[dhaxton]

Well, we're just finishing up a major app using ExtPascal, on Debian
with Apache and FastCGI. Works wonderfully - until it's run for a few
hours. Then it crashes, out of memory. Further investigation revealed
that a TExtDirectPollingEvent was bumping memory by ~ 100k every time
it was invoked - and it just builds a string. Puzzling .... but on a
hunch I tried an experiment. I commented out everything in the
handler, and lo! it still bumped memory usage on every invocation.
Digging deeper, we found that *any* event had this behavior, from
button clicks to windows opening and closing. I'll keep digging
further Monday, and can post sample code then (nothing too awfully
different from the BorderLayout sample, frankly), but at this point it
looks like there's something seriously amiss... would appreciate any
ideas at all!

Thanks much,
Dave H.

[Marcos Douglas]

If post a sample code, as you did say, I think will make it easier to
solve this problem.
Tell us more about your app. Is always good to know about some
application made with ExtPascal running in production. ;-)

IMHO, we have to change the type of engine that free the memory in
ExtPacal apps, unfortunately.

Marcos Douglas

[dhaxton]

Hi Marcos:

On Jan 21, 9:59 am, Marcos Douglas <m...@delfire.net> wrote:

> If post a sample code, as you did say, I think will make it easier to
> solve this problem.

I hope so - I'm going to try and find some simple process that
replicates the issue and will hopefully be able to post it Monday
morning before noon (US Eastern time). We found this problem late
Friday by using top and observing the memory usage - I made an attempt
to intall heaptrc but got no errors reported (assuming I was looking
in the right place) I may not have taken into consideration the multi
threaded nature of the process to use heaptrc correctly. That's a
subject for my further investigation. Do you have any suggestions
about tools I could possibly use to track this thing down?

> Tell us more about your app. Is always good to know about some
> application made with ExtPascal running in production.  ;-)

Well, I don't know how much at liberty I am to say completely -
suffice it at this time to say that it's the web interface to a device
that used to configure and control equipment used extensively in the
broadcast TV, cable TV and satellite TV industries, with add on
modules for IPTV. Most of the rest of the system is in Python, C and C+
+, but ExtPascal was chosen for a replacement web interface after an
attempt to use a tool called 'pyjamas' initially. ExtPascal allows us
to use familiar tools and code for both server and browser side at the
same time. Since none of use have much experience as web developers,
this has proven to be a godsend, and we have got the system up and
running in less than three months start to finish (and it's a
complicated system, let me tell you!). We've also taken advantage of
ExtPascalTools for visual development.

Just for into, we're using Lazarus 0.9.30 with (IIRC) FPC 2.4.x on
Ubuntu for our development machines - the target is Debian based as
well.

Thanks again for the quick reply, and I'll get back on this first
thing in the morning!

Be well,
Dave H.

[bdgr...@mailbolt.com]

At 10:24 AM 1/22/2012, you wrote:
>Hi Marcos:
>
>On Jan 21, 9:59 am, Marcos Douglas <m...@delfire.net> wrote:
>
> > If post a sample code, as you did say, I think will make it easier to
> > solve this problem.
>
>I hope so - I'm going to try and find some simple process that
>replicates the issue and will hopefully be able to post it Monday
>morning before noon (US Eastern time). We found this problem late
>Friday by using top and observing the memory usage - I made an attempt
>to intall heaptrc but got no errors reported (assuming I was looking
>in the right place) I may not have taken into consideration the multi
>threaded nature of the process to use heaptrc correctly. That's a
>subject for my further investigation. Do you have any suggestions
>about tools I could possibly use to track this thing down?

You may want to try EurekaLog. It has an option to track memory leaks.

Mike

[Marcos Douglas]

On Sun, Jan 22, 2012 at 2:24 PM, dhaxton <da...@haxton.org> wrote:
> Hi Marcos:
>
> On Jan 21, 9:59 am, Marcos Douglas <m...@delfire.net> wrote:
>
>> If post a sample code, as you did say, I think will make it easier to
>> solve this problem.
>
> I hope so - I'm going to try and find some simple process that
> replicates the issue and will hopefully be able to post it Monday
> morning before noon (US Eastern time). We found this problem late
> Friday by using top and observing the memory usage - I made an attempt
> to intall heaptrc but got no errors reported (assuming I was looking
> in the right place) I may not have taken into consideration the multi
> threaded nature of the process to use heaptrc correctly. That's a

> subject for my further investigation[...]

I don't know if heaptrc do not works very well with threads, but I
guess that have no differences... but I'm not sure.
Try to use heaptrc using a CGI app, not a FastCGI. See yours results.
I think the ExtPascal engine do not releases the objects in a good
time. If you have many requests in a little time, you will see the
memory rising like a rocket!

> [...]Do you have any suggestions

> about tools I could possibly use to track this thing down?

No, I'm sorry. I just know heaptrc. =(

>> Tell us more about your app. Is always good to know about some
>> application made with ExtPascal running in production.  ;-)
>
> Well, I don't know how much at liberty I am to say completely -
> suffice it at this time to say that it's the web interface to a device
> that used to configure and control equipment used extensively in the
> broadcast TV, cable TV and satellite TV industries, with add on
> modules for IPTV. Most of the rest of the system is in Python, C and C+
> +, but ExtPascal was chosen for a replacement web interface after an
> attempt to use a tool called 'pyjamas' initially.

You did use a web-framework and a very famous language to web apps,
but chosen ExtPascal... very interesting!

> [...]ExtPascal allows us

> to use familiar tools and code for both server and browser side at the
> same time.

That's my idea too.

> Since none of use have much experience as web developers,
> this has proven to be a godsend, and we have got the system up and
> running in less than three months start to finish (and it's a
> complicated system, let me tell you!). We've also taken advantage of
> ExtPascalTools for visual development.

Three months; no experience as web developers; a complicated system.
Well, that sounds good!

> Just for into, we're using Lazarus 0.9.30 with (IIRC) FPC 2.4.x on
> Ubuntu for our development machines - the target is Debian based as
> well.

I'm using Lazarus from /trunk and FPC 2.6.1 from /fixes_2_6.

> Thanks again for the quick reply, and I'll get back on this first
> thing in the morning!

I want to use ExtPascal in a real app, in the future, then your
informations are very important for me. We help each other. ;)
I will wait your mail...

Marcos Douglas

[dhaxton]

Here's the sample code as promised, for what it's worth:
//begin unit1.pas
unit Unit1;

interface

uses
SysUtils, Classes,
{$IFDEF UseRuntime}
Ext, ExtPascal, ExtPascalUtils, ExtForm,
ExtData, ExtGrid, ExtUtil, ExtAir, ExtDd,
ExtLayout, ExtMenu, ExtDirect, ExtState, ExtTree,
ExtUxForm;

type
{$M+}
TExtPanel_Tab = TExtPanel;
TExtFormTextField_Grid = TExtFormTextField;
TExtFormNumberField_Grid = TExtFormNumberField;
TExtFormDateField_Grid = TExtFormDateField;
TExtFormTimeField_Grid = TExtFormTimeField;
TExtFormCheckbox_Grid = TExtFormCheckbox;
TExtFormComboBox_Grid = TExtFormComboBox;
{$M-}

{$ELSE}
ExtP_Design_Ctrls;
{$ENDIF}

type

{ TExtWindow1 }

TExtWindow1 = class(TExtWindow)
ExtButton1: TExtButton;
ExtButton2: TExtButton;
ExtFormLabel1: TExtFormLabel;
ExtFormTextField1: TExtFormTextField;
procedure ExtButton1Click;
procedure ExtButton2Click;
private
public
tdp: TExtDirectPollingProvider;
constructor Create;
procedure Show;
published
procedure FillData;
end;

implementation

uses
AppThread;

procedure TExtWindow1.ExtButton1Click;
begin
tdp.Disconnect;
end;

procedure TExtWindow1.ExtButton2Click;
begin
Ajax(FillData);
end;

constructor TExtWindow1.Create;
begin
inherited;

Id := 'ExtWindow1';
Height := 334;
Width := 408;
OnEsc := JSFunction('');
Title := 'MemTest1';
Layout := lyAbsolute;
Closable := false;


ExtFormTextField1 := TExtFormTextField.Create;
with ExtFormTextField1.AddTo(Items) do
begin
Id := 'ExtWindow1_ExtFormTextField1';
X := 32;
Height := 23;
Y := 64;
Width := 336;
Value := {$IFNDEF MSWINDOWS}AnsiToUTF8{$ENDIF}
('ExtFormTextField1');
end;

ExtFormLabel1 := TExtFormLabel.Create;
with ExtFormLabel1.AddTo(Items) do
begin
Id := 'ExtWindow1_ExtFormLabel1';
X := 0;
Height := 18;
Y := 283;
Width := 99;
AutoWidth := True;
Text := 'ExtFormLabel1';
end;

ExtButton1 := TExtButton.Create;
with ExtButton1.AddTo(Items) do
begin
Id := 'ExtWindow1_ExtButton1';
X := 32;
Height := 25;
Y := 184;
Width := 139;
Text := 'Disconnect Timer);
On('click', Ajax(ExtButton1Click));
TooltipType := 'title';
end;

ExtButton2 := TExtButton.Create;
with ExtButton2.AddTo(Items) do
begin
Id := 'ExtWindow1_ExtButton2';
X := 200;
Height := 25;
Y := 184;
Width := 131;
Text := 'Manual Update);
On('click', Ajax(ExtButton2Click));
TooltipType := 'title';
end;

tdp := TExtDirectPollingProvider.Create;
with tdp do begin
TypeJS := 'polling';
URLFunction := Ajax(FillData);
Interval := 1000;
end;
Ext.ExtDirect.AddProvider(tdp);
end;

procedure TExtWindow1.Show;
begin
inherited Show;
end;

procedure TExtWindow1.FillData;
var
tmpStr: String;
begin
tmpStr :='<p><b>System:</b> '+ CurrentThread.SystemDescription +
'&#160;&#160;' + CurrentThread.SystemVersion +
'&#160;&#160;&#160;&#160;&#160;&#160;<b>User:</b> ' +
CurrentThread.User + '</p>';
ExtFormLabel1.SetText(tmpStr, false);
tmpStr := CurrentThread.SnarfDescriptions;
ExtFormTextField1.SetValue(tmpStr);
end;

end.


//begin appthread.pas
unit AppThread;

interface

uses
ExtPascal,
ExtPascalUtils,
unit1;

type
TAppThread = class(TExtThread)
public
ExtWindow1 : TExtWindow1;
SystemDescription : String;
User : String;
SystemVersion: String;
constructor Create(NewSocket : integer); override;
function SnarfDescriptions: String;
published
procedure Home; override;
end;

function CurrentThread : TAppThread;


implementation

uses
{$IFNDEF WebServer}
FCGIApp,
{$ELSE}
IdExtHTTPServer,
{$ENDIF}
sysutils;

function CurrentThread : TAppThread;
begin
Result := TAppThread(CurrentFCGIThread);
end;


procedure TAppThread.Home;
begin
ExtWindow1 := TExtWindow1.Create;
ExtWindow1.Show;
end;

constructor TAppThread.Create(NewSocket : integer);
begin
SystemDescription := 'MemTest';
SystemVersion := '1.0';
User := 'Jack B. Nimble';
inherited Create(NewSocket);
end;

function TAppThread.SnarfDescriptions: String;
begin
Result := FormatDateTime('dd mmm yyyy hh:nn:ss',Now);
end;

end.
// end sample

I built this using the ExtPascalToolKit - it's simple and pretty ugly
but it shows the issue clearly. It sets a timer and really eats memory
up by just displaying a clock. I put a couple of buttons on so you can
disconnect the timer and eat the RAM manually if you'd like ...;>)

The really interesting this about this is that the ExtPascalSamples
that ship with the product exhibit the same symptoms of memory
leakage! - just run the app, open several tabs, close them, whatever,
the memory goes up but doesn't seem to come down, even after the last
thread is closed.

The other curious thing I ran into today was attempting to use
valgrind. Heaptrc was useless, more or less. (and EurkaLog is Windows
only - can't use it here, but thanks for the suggestion). Valgrind
reported some leakage - but only about 200k after program termination.
With the -gv option specified the memory eating seemed to slow down
considerably - not stop, mind you, but slowed by like 75%.

I'm not sure anymore if I'm seeing a leak or some asynch thing that
simply not garbage collecting fast enough ... and I don't understand
what's allocating such huge chunks of memory at once - just one click
in the program listed above call eat 600k!

BTW, I'm looking at memory usage using top and the Ubuntu system
monitor app.

Hints and help appreciated!!!

Be well,
Dave H.

[Marcos Douglas]

Hi Dave,

I think Wanderlan, the autor of ExtPascal, can help you more faster than me.
I knew it about the release memory problems, but I thought it was
happening only in sophisticated applications -- I didn't imagine that
ExtPascalSamples have the same problems.

If you can, let's wait what he can tell us, OK?

Marcos Douglas

[Marcos Douglas]

Well, it's very strange Wanderlan has not responded yet. =(

Dave,
Have you got anything else about the problem?

Marcos Douglas

[Wanderlan Santos dos Anjos]

Hi Marcos,

I'm very busy now...
On Saturday I will see.

Marcos Douglas

--
You received this message because you are subscribed to the Google Groups "ExtPascal" group.
To post to this group, send email to extp...@googlegroups.com.
To unsubscribe from this group, send email to extpascal+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/extpascal?hl=en.

--
Att,

Wanderlan Santos dos Anjos
(061) (r)3345-5496; (c)9133-3804, (t)3038-1906

[dhaxton]

Well, I've been digging in since Monday - I had been using the
download package, but got into SVN and retrieved build 751 because
we're still using ExtJS 3.4 - and the newest revision out of SVN
didn't seem to build anyway ;>( 751 seems pretty

I figured out that each request triggers a new thread, and each thread
creates a new session. My understanding at this point is that these
should be destroyed when the idle time is exceeded and no sessions are
active, but this doesn't seem to be happening.

I'm still digging and will post my results tomorrow evening - we
really want to use this system, but cannot tolerate every request
allocating > 1MB on the server...

Be well,
Dave H.

[Daithi Haxton]

On Jan 24, 2012, at 5:54 AM, Marcos Douglas wrote:
> Hi Dave,
>
> I think Wanderlan, the autor of ExtPascal, can help you more faster
> than me.
> I knew it about the release memory problems, but I thought it was
> happening only in sophisticated applications -- I didn't imagine that
> ExtPascalSamples have the same problems.
>
> If you can, let's wait what he can tell us, OK?
>
> Marcos Douglas

Found a big portion of it : put cmem in the uses clause of
FCGIApp.pas, just after cthreads. Most issues disappear after that -
am still tracking down some residuals - will keep the group posted!

Be well,
Dave H.

[Wanderlan Santos dos Anjos]

Hi Dave,

Please, try to insert this in ExtPascal.pas:

881 880 // Frees this list and all objects linked in it 882 - destructor TExtObjectList.Destroy; begin 881 + destructor TExtObjectList.Destroy; 882 + var 883 + I : integer; 884 + begin 885 + for I := high(FObjects) downto 0 do 886 + try 887 + FObjects[I].Free 888 + except end; 883 889 SetLength(FObjects, 0); 884 890 inherited; 885 891 end;

Att,

Wanderlan

--
You received this message because you are subscribed to the Google Groups "ExtPascal" group.
To post to this group, send email to extp...@googlegroups.com.

To unsubscribe from this group, send email to extpascal+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/extpascal?hl=en.

[Daithi Haxton]

Thanks, Wanderlan, will give it a shot on Monday and let you know the
results. I've not got the time to look today, but from what I remember
of the code your patch makes sense and should clear up the residual
leaks we've got in short order.

BTW, let me take this opportunity to thank you for creating ExtPascal
- it's nothing short of brilliant in it's conception, and is amazingly
efficient at creating complex web designs in short periods of time.

Be well,
Dave H.

[Marcos Douglas]

Well, this code could fix the memory leak but, IMHO, I still think we should take a look at "Garbage Collector".

This code hide any exceptions that could exist using try-except-end.

Marcos Douglas

[dhaxton]

Hi Wanderlan:
Didn't seem to help - memory usage still increases by 4-8k per request
and never seems to go down on free. Just tried it with my little test
program - haven't tried it with the ExtSamples yet. Will keep
investigating and report here with any new findings ... thanks for
your help!
Be well,
Dave H.

[Wanderlan Santos dos Anjos]

Hello Dave,

More one another try:

In FCGIApp.pas:

Change line about 244:

 {$IFDEF MSWINDOWS}inherited;{$ENDIF} // Collateral effect of Unix RTL FPC bug

by
 inherited;

and 
remove line about 629:


{$IFNDEF MSWINDOWS} EndThread(0); {$ENDIF}

Att
Wanderlan

Be well,
Dave H.

--
You received this message because you are subscribed to the Google Groups "ExtPascal" group.
To post to this group, send email to extp...@googlegroups.com.

To unsubscribe from this group, send email to extpascal+...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/extpascal?hl=en.

[Marcos Douglas]

On Tue, Jan 31, 2012 at 8:17 AM, Wanderlan Santos dos Anjos
<wanderl...@gmail.com> wrote:
>
> Hello Dave,
>
> More one another try:
>
> In FCGIApp.pas:
>
> Change line about 244:
>
> {$IFDEF MSWINDOWS}inherited;{$ENDIF} // Collateral effect of Unix RTL FPC bug
>
> by
> inherited;
>
> and
> remove line about 629:
>
>
> {$IFNDEF MSWINDOWS} EndThread(0); {$ENDIF}

Wanderlan,

These codes should be merged in 0.9.8 in a branch, don't you think?
Could you give me access to create the branch 0.9.8? Maybe you could
create but give me access...

Marcos Douglas

[Wanderlan Santos dos Anjos]

These codes should be merged in 0.9.8 in a branch, don't you think?

Yes

Could you give me access to create the branch 0.9.8? Maybe you could
create but give me access...

Once Dave gives us the results I will give more powers.

Marcos Douglas

--
You received this message because you are subscribed to the Google Groups "ExtPascal" group.
To post to this group, send email to extp...@googlegroups.com.
To unsubscribe from this group, send email to extpascal+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/extpascal?hl=en.

[Marcos Douglas]

On Tue, Jan 31, 2012 at 11:58 AM, Wanderlan Santos dos Anjos
<wanderl...@gmail.com> wrote:
>
>> These codes should be merged in 0.9.8 in a branch, don't you think?
>
> Yes
>>
>> Could you give me access to create the branch 0.9.8? Maybe you could
>> create but give me access...
>>
> Once Dave gives us the results I will give more powers.

=)

But, IMHO, we need a branch to merge patches. Only patches... new
features only in trunk.
So, we need 3 things:
1- Create a branch 0.9.8
2- Create a Tag 0.9.8 (the stable version that exists to download)
3- My new powers! ;-)

Marcos Douglas

[dhaxton]

Hi Wanderlan:
Sorry about the delay getting back to you but I had some personal
business I had to attend to yesterday. Anyway, there's no real change
from the mods. However, I did get what I think is a pretty good map of
what's happening from valgrind - most of it seems to be in the
initialization of objects, and some seems to be connected to
threading. Is there somewhere I can upload/send the valgrind output
file so you could look it over?

Thanks again for all your assistance and
Be well,
Dave H.

[Marcos Douglas]

You can upload in the list.

Your program is already running in production?

Marcos Douglas

[dhaxton]

Yeah, we're in production. I got the leakage to a level that's "under control", and we've set the server to auto-die after a timeout and then relaunch, so we shouldn't have too much problems. Got five weeks to find a real fix and push an update....

Attached are two valgrind logs. The one marked "release' is from mu sample program compiled with the release version of extPas, the one that's just valgrind.output is from SVN version 751. Both are gzipped.

Both instances were stopped using ctrl-c - the server timeout feature doesn't seem to work in the 751 release, and killing the process by issuing a shutdown actually increases the leakage substantially.

Most of the issues in the release version (and quite a few in the newer one) seem to have to do with the AFterHandleEvent procedure in the thread. Curiously, they seem to center around functions that are manipulating ANSIstrings. Other leaks seem directly related to threads not being freed.

I've gone over this stuff for better than a week with a pretty fine toothed comb, and am starting to wonder how much of the issue I'm seeing is with the Linux threading libs (pthreads) or even the way FPC handles AnsiStrings with the cmemory manager in place.

I've got to spend a couple of weeks on another project but will be back to this after that's cleaned up. ExtPascal is a wonderful project, with huge potential. I'm sure this issue is resolvable - one way or another.

Be well,
Dave H.

[nandod]

Hello Dave,

I too have been experiencing leaks of various kinds (on Windows); please see this thread:

https://groups.google.com/forum/#!topic/extpascal/I57JOOyQW84 

It might give you some pointers or ring a bell.

HTH

-- 

Nando

[dhaxton]

Hi Nando:
Thanks for the heads up - I read the thread in question earlier, but
as we are using the release version of ExtPascal and on Linux as
opposed to Windows, much of the discussion was not relevant. It was
useful nonetheless, however, if only because we'll be moving our
version up as we move to ExtJS4.
The thing that helped us the most was the addition of cmem to our uses
clause in FCGIapp - that cut the leaking down to manageable (or at
least tolerable) levels. Won't do you any good on Windows, however.
And since we're running on an embedded device and not the public
internet, we can take advantage of the ability to kill and restart the
server periodically, which is a crude hack, but it works and we've got
a schedule to keep ...;>)

I'll get back to finding leaks in a week or so and will be sure to
keep you (and the rest of the group) posted.

Thanks and
Be well,
Dave H.

[dhaxton]

Just a reply to the group about the resolution of this problem...

It seems as though adding 'cmem' to the uses clause of FCGIApp.pas
eliminated most of the "leaks" we discovered that were attributable to
ExtPascal. A better way to achieve the same effect is to compile the
project with support for valgrind, which adds the cmem unit at the
project level. We left no other changes to the code in place.

That being said, the app still "leaks" ... about 4k per session.
However, once this number reaches some size (seems to vary machine to
machine) it stops growing. In no case has it effected machine
performance. We are unsure if we are seeing an artifact of the tools
we're using to scan for leaks, or some small actual leak generated by
ptherds. In any case, we relase on Friday, so we're pretty confident.
Famous last words ...;>)

Will keep an eye on this and post back here if we come up with
anything else.

And thanks, Wanderlan, for building a great project!

Be well,
Dave H

[Wanderlan Santos dos Anjos]

Hello Dave,

 

That being said, the app still "leaks" ... about 4k per session.
However, once this number reaches some size (seems to vary machine to
machine) it stops growing.

What size is it, approximately?
How many sessions are opened?
 

In no case has it effected machine
performance. We are unsure if we are seeing an artifact of the tools
we're using to scan for leaks, or some small actual leak generated by
ptherds. In any case, we relase on Friday, so we're pretty confident.
Famous last words ...;>)

My congrats! :)
 

Will keep an eye on this and post back here if we come up with
anything else.

And thanks, Wanderlan, for building a great project!

Thanks!

But, is there any chance you publish a link to a demo of this application?
That would be very good for ExtPascal ...

TIA,

Wanderlan Santos dos Anjos

[dhaxton]

On Thursday, March 22, 2012 10:38:30 AM UTC-4, Wanderlan wrote:

What size is it, approximately?
How many sessions are opened?

   "Leak" size seems to run around 4k per session - operative word is "seems"... we can run out of sessions rather quickly. That's more of a problem at this point than the memory. We have a configurable 5 second timer running on one screen that eats sessions for lunch. But GC always seems to catch up, and so far we've not hit a show stopper limit. We've set MaxConns to 5k just in case we have multiple timers (on multiple machines) running ... 

 

But, is there any chance you publish a link to a demo of this application?
That would be very good for ExtPascal ...

   Would love to, but as it's running in an embedded box as our configuration GUI, it probably won't be possible (too many support files/programs). However, our sales people have indicated that they'd like a demo running on the website someday - and since Lazarus gives us the opportunity to run on Windows (IIS)  as well as Linux (Apache) that's possible. But there'd have to be a lot of changes to the code to get it to run without the background stuff... <sigh> maybe someday.

Thanks again for all the support and

Be well,
Dave H

[Wanderlan Santos dos Anjos]

Hello Dave,

Some screenshots would be enough...

-Wanderlan-

--

You received this message because you are subscribed to the Google Groups "ExtPascal" group.

To view this discussion on the web visit https://groups.google.com/d/msg/extpascal/-/gM8C_A45Fp8J.

To post to this group, send email to extp...@googlegroups.com.
To unsubscribe from this group, send email to extpascal+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/extpascal?hl=en.