Received: by 10.50.182.132 with SMTP id ee4mr6102885igc.5.1350933934337;
Mon, 22 Oct 2012 12:25:34 -0700 (PDT)
X-BeenThere: pylearn-dev@googlegroups.com
Received: by 10.50.41.132 with SMTP id f4ls10296527igl.1.gmail; Mon, 22 Oct
2012 12:25:33 -0700 (PDT)
Received: by 10.42.172.68 with SMTP id m4mr5411012icz.1.1350933933717;
Mon, 22 Oct 2012 12:25:33 -0700 (PDT)
Received: by 10.42.172.68 with SMTP id m4mr5411009icz.1.1350933933692;
Mon, 22 Oct 2012 12:25:33 -0700 (PDT)
Return-Path: <goodfellow....@gmail.com>
Received: from mail-ie0-f170.google.com (mail-ie0-f170.google.com [209.85.223.170])
by gmr-mx.google.com with ESMTPS id dx8si1076246igc.1.2012.10.22.12.25.33
(version=TLSv1/SSLv3 cipher=OTHER);
Mon, 22 Oct 2012 12:25:33 -0700 (PDT)
Received-SPF: pass (google.com: domain of goodfellow....@gmail.com designates 209.85.223.170 as permitted sender) client-ip=209.85.223.170;
Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of goodfellow....@gmail.com designates 209.85.223.170 as permitted sender) smtp.mail=goodfellow....@gmail.com; dkim=pass header...@gmail.com
Received: by mail-ie0-f170.google.com with SMTP id c12so4959823ieb.29
for <pylearn-dev@googlegroups.com>; Mon, 22 Oct 2012 12:25:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=mime-version:in-reply-to:references:date:message-id:subject:from:to
:content-type;
bh=BZjlPND6y4kQPPsjKyDLH8Nutx+x+q2MoRl9NdcWRsE=;
b=zOdVGdxrFUIweRUxP6SgmCvKr9wjIyEpxIb1i3ZCNvqE0ycSVxMHSNBWBfQrgYCKQl
hrla5kQOFYyPvw0GCtyU3TO7wZNVntmeXE6aw7FzBnrfT0INCcn+0wJQ+VAX2/1hSNxS
IYgTlBF6qKZUcu0XgEdklwhEcaj1s8Y84nNRe7kqlutSKVEKUdn5varGeAiOCq9sYIXk
hVzZyL0jK6a+Gu2p5efFWCwVjXWFAOGMXJ6JPZudVIpXNDQH924/wf92qLxU0A0cE9C9
UyZTFOfNAbIayBumRzRuOmu5b5gDALpi5VxjgbN1FV7yi5zxG8RWvLB4/h8P4tNNz92F
gDag==
MIME-Version: 1.0
Received: by 10.50.12.138 with SMTP id y10mr17273452igb.58.1350933933540; Mon,
22 Oct 2012 12:25:33 -0700 (PDT)
Received: by 10.43.17.134 with HTTP; Mon, 22 Oct 2012 12:25:33 -0700 (PDT)
In-Reply-To: <CADKKbtjNDgK2L5SQNg8nJKPxzMjdJaQby5dr73D6hfNyED7...@mail.gmail.com>
References: <CAPXn81rV3sshYwvS4m_J6zo2iP7dNxBEpfjJW2yRwN=W9zD...@mail.gmail.com>
<CALc6Xo66TxM39TSZB0M9BuVevKz3vihYqEXgWtCCNp2vx2K...@mail.gmail.com>
<CAPXn81pxsCdu6HW1NfGczubwgggz6O--_FCDPNzCG=MRqm=...@mail.gmail.com>
<CALc6Xo62xNXwVris-whh_CzJRnzU7H18kk-ceFPS_zexR1a...@mail.gmail.com>
<CALc6Xo6V53Ej2CjhStrBq-7wnvC_1-C=HK5fHO_2sb-4Gaq...@mail.gmail.com>
<CADKKbtii69aVWEs=zwhGhdNwAAqofyF=JFjD8LqF5tsxTEa...@mail.gmail.com>
<CAPXn81p+kg6quRye4n+XS3CzB7EFcyOfseqfQ3=_ogH0E55...@mail.gmail.com>
<CADKKbtjNDgK2L5SQNg8nJKPxzMjdJaQby5dr73D6hfNyED7...@mail.gmail.com>
Date: Mon, 22 Oct 2012 15:25:33 -0400
Message-ID: <CAPXn81pApXdNR+ffFDRAJnKiyvBATjyMXxH5uoX9=W86gLw...@mail.gmail.com>
Subject: Re: [pylearn-dev] Printed output on clusters
From: Ian Goodfellow <goodfellow....@gmail.com>
To: pylearn-dev@googlegroups.com
Content-Type: multipart/alternative; boundary=14dae93405173946aa04ccaad13d
--14dae93405173946aa04ccaad13d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Yeah, I'll try it next time.
On Mon, Oct 22, 2012 at 2:46 PM, Fr=E9d=E9ric Bastien <no...@nouiz.org> wro=
te:
> ok,
>
> if you have the problem again, can you tell me if "jobman cmdline" fix
> the problem?
>
> Fred
>
> On Mon, Oct 22, 2012 at 1:57 PM, Ian Goodfellow
> <goodfellow....@gmail.com> wrote:
> >
> >
> > On Mon, Oct 22, 2012 at 1:27 PM, Fr=E9d=E9ric Bastien <no...@nouiz.org>
> wrote:
> >>
> >> Hi,
> >>
> >> The easy hack would be to change sys.stdin to a file descriptop that
> >> you open in writing.
> >>
> >> Logger could be good, but that should be used only for what in inside
> >> pylearn2. This don't fix the problem of the stuff the user print
> >> itself.
> >>
> >> jobman already redirect the stdout/stderr correctly to a file. You do
> >> not need a database to use it. You can use "jobman cmdline" to just do
> >> the rediction. "jobman cmdline --help" for detail. This will also add
> >> some debug information that we gatter automatically, like the 'job
> >> scheduler' job id, the host name, etc. I think it is better to collect
> >> automatically some debug info in only one system if possible. We can
> >> discuss where.
> >>
> >> Can you tell me what you did finally?
> >
> >
> > In this most recent case, I told the admins about some of the problems =
I
> was
> > having,
> > and they fixed something related to the "username server".
> >
> >>
> >>
> >> Also, on witch cluster you had the problem?
> >
> >
> > briaree. Pretty much any time I have a problem on briaree, the logs nev=
er
> > appear,
> > or are missing a lot of output.
> >
> >>
> >>
> >> Fred
> >>
> >> On Sun, Oct 21, 2012 at 2:02 PM, David Warde-Farley
> >> <d.warde.far...@gmail.com> wrote:
> >> > On Sun, Oct 21, 2012 at 1:54 PM, David Warde-Farley
> >> > <d.warde.far...@gmail.com> wrote:
> >> >> On Sun, Oct 21, 2012 at 10:48 AM, Ian Goodfellow
> >> >> <goodfellow....@gmail.com> wrote:
> >> >>> If the client is also using the logging module, would we overwrite
> >> >>> their
> >> >>> settings?
> >> >>> I seem to remember there being a discussion of something like that
> >> >>> happening
> >> >>> on one of the theano mailing lists.
> >> >>
> >> >> I think that can only happen if we modify the "global logger", whic=
h
> >> >> we may want to but we needn't necessarily. I'll have a look at the
> >> >> mailing list to see if I can find the issue.
> >> >
> >> > So here's the issue: the logging module has a concept of a "log
> >> > level", so messages are classified in a severity ordering. INFO is
> >> > higher priority than DEBUG, WARNING is higher than INFO, ERROR is
> >> > higher than WARNING, and CRITICAL is higher than all. Then you tell
> >> > the logger the minimum level you want to see.
> >> >
> >> > What had happened in Theano was that someone had modified the global
> >> > logger state from minimum level NOTSET to minimum level DEBUG, so a
> >> > bunch of third party modules also started spewing debug messages. Th=
is
> >> > is pretty easily avoided.
> >
> >
>
--14dae93405173946aa04ccaad13d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Yeah, I'll try it next time.<br><br><div class=3D"gmail_quote">On Mon, =
Oct 22, 2012 at 2:46 PM, Fr=E9d=E9ric Bastien <span dir=3D"ltr"><<a href=
=3D"mailto:no...@nouiz.org" target=3D"_blank">no...@nouiz.org</a>></span=
> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">ok,<br>
<br>
if you have the problem again, can you tell me if "jobman cmdline"=
; fix<br>
the problem?<br>
<br>
Fred<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br>
On Mon, Oct 22, 2012 at 1:57 PM, Ian Goodfellow<br>
<<a href=3D"mailto:goodfellow....@gmail.com">goodfellow....@gmail.com</a=
>> wrote:<br>
><br>
><br>
> On Mon, Oct 22, 2012 at 1:27 PM, Fr=E9d=E9ric Bastien <<a href=3D"m=
ailto:no...@nouiz.org">no...@nouiz.org</a>> wrote:<br>
>><br>
>> Hi,<br>
>><br>
>> The easy hack would be to change sys.stdin to a file descriptop th=
at<br>
>> you open in writing.<br>
>><br>
>> Logger could be good, but that should be used only for what in ins=
ide<br>
>> pylearn2. This don't fix the problem of the stuff the user pri=
nt<br>
>> itself.<br>
>><br>
>> jobman already redirect the stdout/stderr correctly to a file. You=
do<br>
>> not need a database to use it. You can use "jobman cmdline&qu=
ot; to just do<br>
>> the rediction. "jobman cmdline --help" for detail. This =
will also add<br>
>> some debug information that we gatter automatically, like the '=
;job<br>
>> scheduler' job id, the host name, etc. I think it is better to=
collect<br>
>> automatically some debug info in only one system if possible. We c=
an<br>
>> discuss where.<br>
>><br>
>> Can you tell me what you did finally?<br>
><br>
><br>
> In this most recent case, I told the admins about some of the problems=
I was<br>
> having,<br>
> and they fixed something related to the "username server".<b=
r>
><br>
>><br>
>><br>
>> Also, on witch cluster you had the problem?<br>
><br>
><br>
> briaree. Pretty much any time I have a problem on briaree, the logs ne=
ver<br>
> appear,<br>
> or are missing a lot of output.<br>
><br>
>><br>
>><br>
>> Fred<br>
>><br>
>> On Sun, Oct 21, 2012 at 2:02 PM, David Warde-Farley<br>
>> <<a href=3D"mailto:d.warde.far...@gmail.com">d.warde.farley@gma=
il.com</a>> wrote:<br>
>> > On Sun, Oct 21, 2012 at 1:54 PM, David Warde-Farley<br>
>> > <<a href=3D"mailto:d.warde.far...@gmail.com">d.warde.farle=
y...@gmail.com</a>> wrote:<br>
>> >> On Sun, Oct 21, 2012 at 10:48 AM, Ian Goodfellow<br>
>> >> <<a href=3D"mailto:goodfellow....@gmail.com">goodfello=
w....@gmail.com</a>> wrote:<br>
>> >>> If the client is also using the logging module, would=
we overwrite<br>
>> >>> their<br>
>> >>> settings?<br>
>> >>> I seem to remember there being a discussion of someth=
ing like that<br>
>> >>> happening<br>
>> >>> on one of the theano mailing lists.<br>
>> >><br>
>> >> I think that can only happen if we modify the "globa=
l logger", which<br>
>> >> we may want to but we needn't necessarily. I'll h=
ave a look at the<br>
>> >> mailing list to see if I can find the issue.<br>
>> ><br>
>> > So here's the issue: the logging module has a concept of =
a "log<br>
>> > level", so messages are classified in a severity orderin=
g. INFO is<br>
>> > higher priority than DEBUG, WARNING is higher than INFO, ERRO=
R is<br>
>> > higher than WARNING, and CRITICAL is higher than all. Then yo=
u tell<br>
>> > the logger the minimum level you want to see.<br>
>> ><br>
>> > What had happened in Theano was that someone had modified the=
global<br>
>> > logger state from minimum level NOTSET to minimum level DEBUG=
, so a<br>
>> > bunch of third party modules also started spewing debug messa=
ges. This<br>
>> > is pretty easily avoided.<br>
><br>
><br>
</div></div></blockquote></div><br>
--14dae93405173946aa04ccaad13d--
Message from discussion Printed output on clusters
| Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy |
| ©2013 Google |