this is interesting behaviour indeed, and affects *ALL* threaded
applications...
$ ulimit -s 1048576
$ /opt/lcg/bin/lfc-mkdir /grid/pvier/janjust/test
t2146736864:p10274: Fatal error: [Thread System] GLOBUSTHREAD:
pthread_create() failed
[Thread System] insufficient memory (ENOMEM)Aborted
$ ulimit -s 40000
$ /opt/lcg/bin/lfc-mkdir /grid/pvier/janjust/test
< command runs fine >
the system default is to have an ulimited stack size so that case is
apparently treated differently...
Alvaro,
this means that this is *NOT* an lfc bug , unless people agree that the
'lfc-mkdir' command should not be pthreaded-based ...
cheers,
JJK / Jan Just Keijser
Nikhef Amsterdam
Andreas Haupt wrote:
> Hi Alvaro,
>
> you might try to limit h_stack to 10M for all jobs (or as a queue
> limit). SGE (imho wrongly - I think, it's a bug...) sets the stack size
> limit to the same value as the virtual memory limit (h_vmem). All jobs
> using threads (python, java, ...) show exactly your mentioned problem
> with those shell limits.
>
> Cheers,
> Andreas
>
> On Fri, 2008-10-10 at 09:20 +0200, Alvaro Simon Garcia wrote:
>
>> Hi Dennis
>>
>> Our SGE batch system is configured to limit user memory, It's set to
>> 1GB, but as you said it seems that lfc-mkdir needs more... We will open
>> a bug regarding this.
>>
>> Cheers and thanks
>> Alvaro
>>
>>> Alvaro Simon Garcia schreef:
>>>
>>>> Hi
>>>>
>>>> We have found some diferences...
>>>>
>>>> on WN
>>>> #ulimit -a
>>>>
>>>> virtual memory (kbytes, -v) unlimited
>>>>
>>>> with batch system:
>>>> #ulimit -a
>>>> virtual memory (kbytes, -v) 1048576
>>>>
>>>> our strace output:
>>>>
>>> [...]
>>>
>>>> mmap2(NULL, 1073745920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
>>>> -1, 0) = -1 ENOMEM (Cannot allocate memory)
>>>>
>>> [...]
>>>
>>> You've got all the evidence right there. Set ulimit of virtual memory to
>>> the amount used by the batch system, this should result in the same error.
>>>
>>> I don't know who is responsible for trying to mmap over a GB of memory,
>>> but I would consider that a bug ;-)
>>>
>>> Cheers,
>>>
>>> Dennis van Dok
>>>
|