[Rocks-Discuss] sge ulimit

[Joerg Hohwieler]

I have some strange problems with ulimit and SGE.

When I use normal ssh session I have no limit:
[testuser@frontend ~]$ ssh compute-0-3
[testuser@compute-0-3 ~]$ ulimit -l
unlimited

But when I use SGE, e.g. qlogin I have a limit:
[testuser@frontend ~]$ qlogin
[testuser@compute-0-3 ~]$ ulimit -l
32

I configured /etc/security/limits.confs on all machines:
...
* soft memlock unlimited
* hard memlock unlimited
...

How can I get unlimited locked memory in SGE sessions?

Best regards
Joerg

[Joerg Hohwieler]

Solved ...

Also need to add 'ulimit -l unlimted' to /etc/init.d/sgeexecd.frontend on
each compute node.

What would be the best way to do this automatically?

Joerg

[Sudarshan Wadkar]

you can configure extend-compute.xml file in
/export/rocks/install/site-profiles/5.3/nodes/ and put something like

<post>
<!-- Insert your post installation script here. -->
echo "ulimit -l unlimited" >> /etc/init.d/sgeexecd.frontend
</post>

then the usual set of commands

$ cd /export/rocks/install
$ rocks create distro

this creates your distro and then reinstall compute nodes

$ rocks run host "reboot" (or use shoot-node)

The following method may or may not work --
you can also create a copy of /etc/init.d/sgeexecd.frontend on your frontend
(say /root/config_files/sgeexecd.frontend) edit it accordingly and then use
411 service to distribute that file (so that it replaces
/etc/init.d/sgeexecd.frontend on compute nodes). To read more about 411
service, refer to rocks user documentation.

Hope this helps.

~$udhi
"Success is getting what you want. Happiness is wanting what you get."
- Dale Carnegie
"There is only one success - to be able to spend your life in your own way."
- Christopher Morley

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20100115/409f5860/attachment.html

[Joerg Hohwieler]

On Fri, 15 Jan 2010 - 18:05, Sudarshan Wadkar wrote:
> you can configure extend-compute.xml file in
> /export/rocks/install/site-profiles/5.3/nodes/ and put something like
>
> <post>
> <!-- Insert your post installation script here. -->
> echo "ulimit -l unlimited" >> /etc/init.d/sgeexecd.frontend
> </post>

I tried this above and I also tried this in extend-compute.xml:
<post>
<file name="/etc/init.d/sgeexecd.frontend" mode="append">
ulimit -l unlimited
</file>
</post>

But with both methods after the installation the file only consists of
this line:
ulimit -l unlimited

What's wrong ?

Best regards
Joerg

[Sudarshan Wadkar]

according to following link it should work.
https://wiki.rocksclusters.org/wiki/index.php/FAQ#Q:_Is_there_a_guide_for_configuring_extend-compute.xml_file_to_customize_the_compute_nodes.3F

@Greg : are we missing something here??

I am trying something similar here. I want to export certain variables, and
I just edited extend-compute.xml
<post>
export myappsdir="/state/partition1/sudhi/apps"
</post>
I am rebuilding compute nodes, I will get back to you with my results.

~$udhi
"Success is getting what you want. Happiness is wanting what you get."
- Dale Carnegie
"There is only one success - to be able to spend your life in your own way."
- Christopher Morley

URL: https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/attachments/20100116/841a5327/attachment.html

[Greg Bruno]

On Fri, Jan 15, 2010 at 10:11 AM, Joerg Hohwieler <jo...@hohwie.com> wrote:
> On Fri, 15 Jan 2010 - 18:05, Sudarshan Wadkar wrote:
>> you can configure extend-compute.xml file in
>> /export/rocks/install/site-profiles/5.3/nodes/ and put something like
>>
>> <post>
>> <!-- Insert your post installation script here. -->
>> echo "ulimit -l unlimited" >> /etc/init.d/sgeexecd.frontend
>> </post>
>
> I tried this above and I also tried this in extend-compute.xml:
> <post>
> <file name="/etc/init.d/sgeexecd.frontend" mode="append">
> ulimit -l unlimited
> </file>
> </post>
>
>
> But with both methods after the installation the file only consists of
> this line:
> ulimit -l unlimited
>
>
> What's wrong ?

the file /etc/init.d/sgeexecd.frontend gets built on the first boot
after the compute node is installed. this is done with the file:

/etc/rc.d/rocksconfig.d/post-91-sge-execd

you won't see that file after a boot, because the last line of the
script removes itself after it runs -- that is, this script is setup
to only run once.

so, what you need to do is create a script that runs after post-91-sge-execd.

in your post section, add:

<post>

<file name="/etc/rc.d/rocksconfig.d/post-92-sge-fix" perms="0755">
#!/bin/bash

echo "ulimit -l unlimited" >> /etc/init.d/sgeexecd.frontend

rm -f /etc/rc.d/rocksconfig.d/post-92-sge-fix
</file>

</post>

then rebuild the distro and reinstall a compute node.

- gb

[Joerg Hohwieler]

Adding this line to extend-compute.xml seems to work:
sed -i '2i ulimit -l unlimited' /opt/gridengine/default/common/sgeexecd

I guess /etc/init.d/sgeexecd.frontend is created on the first start and
this would be skipped if the file already exists by echoing it.

Joerg

[Joerg Hohwieler]

thanks, that works perfect.

Best regards
Joerg

[Mike Hanby]

In SGE 6.2 and later it's a lot easier to enable this via the scheduler:

Modify the configuration using this command and add the following to 'execd_params'
qconf -mconf

execd_params H_MEMORYLOCKED=infinity

Mike