CAE implementation - pylearn-dev

The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Message from discussion CAE implementation
Received-SPF: pass (google.com: domain of ca9...@gmail.com designates 209.85.215.42 as permitted sender) client-ip=209.85.215.42;
MIME-Version: 1.0
In-Reply-To: <CAPXn81q716ULia9Vs7K8Ejox46LCvfseC-hKVK3k0BsgiM_...@mail.gmail.com>
References: <CAPXn81pga3DA+f4dEHm5pbFgNF-BoEMSi-P5a=26+cu=0YQ...@mail.gmail.com>
	<20120803030658.GA9...@bob.blip.be>
	<CALiffSNBE-c=Z244UQb6YEeHTyh+8k_WE1opnqn-S8Ukj7O...@mail.gmail.com>
	<CAPXn81pARx3xwaQmrbnm=dP_4KvRHC=+6kdwiMkwD5Aqoxs...@mail.gmail.com>
	<CADKKbtgsfukQ7ddvNB0ST4w-tDyKLyKQmT3kNN+xSp-5ODa...@mail.gmail.com>
	<CALc6Xo4hmTomRZHpMs74PriRfve6=UPB1a7SfuwD8QTDHzK...@mail.gmail.com>
	<CADKKbthX6++0NK0RNJAJxa7n2k0kjWMUJW9PfxbJE=AEDQ0...@mail.gmail.com>
	<CALc6Xo6AmYDKzZ-OJB1QJ5nBOisW5e0Wb8SkJwsoodJybxJ...@mail.gmail.com>
	<CAPXn81p45tAv=Um4bhoaD849kvTq9idrRzPFVKuLVge1-Bd...@mail.gmail.com>
	<CAARJ_g5Uj6tOzc9tEhBb+uiDWWqPGhn7pg27OxJUf=XsBfF...@mail.gmail.com>
	<CA+DEogMxamo6psrw_DGuuBKz5dFYaScSH49_sX89N3oeq03...@mail.gmail.com>
	<CAPXn81qmFYzSUHdB-M8eh8LVp8yyBtkPoUXi56SyireDvp5...@mail.gmail.com>
	<CA+DEogONff87YF21d=okBfk3p7_XSorO4hMv8xCkneDreWe...@mail.gmail.com>
	<CAPXn81q716ULia9Vs7K8Ejox46LCvfseC-hKVK3k0BsgiM_...@mail.gmail.com>
Date: Mon, 15 Oct 2012 12:45:22 -0400
Message-ID: <CA+DEogO99C=_ZH3V9T6DZbSP=LD+SQiDPeqTuWOPqVLWoUq...@mail.gmail.com>
Subject: Re: [pylearn-dev] CAE implementation
From: =?UTF-8?B?w4dhxJ9sYXIgR8O8bMOnZWhyZQ==?= <ca9...@gmail.com>
To: pylearn-dev@googlegroups.com
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

OK I've made the  pull-request here (Also I had to fix bunch of bugs
in order to test it as well):

https://github.com/lisa-lab/pylearn/pull/127

2012/10/12 Ian Goodfellow <goodfellow....@gmail.com>:
> Thanks.
>
>
> On Fri, Oct 12, 2012 at 9:49 AM, =C3=87a=C4=9Flar G=C3=BCl=C3=A7ehre <ca9=
l...@gmail.com> wrote:
>>
>> OK I'll do the pull request with the improved version.
>>
>> 2012/10/11 Ian Goodfellow <goodfellow....@gmail.com>:
>> > Would either of you like to submit a pull request with the improved
>> > version?
>> >
>> >
>> > On Thu, Oct 11, 2012 at 6:01 PM, =C3=87a=C4=9Flar G=C3=BCl=C3=A7ehre <=
ca9...@gmail.com>
>> > wrote:
>> >>
>> >> I agree with Hani, with the Ian's tip, I got a very significant
>> >> performance increase in my CA implementation as well.
>> >>
>> >> 2012/10/10 Eng.Hani Almousli <hani.mou...@gmail.com>:
>> >> > Hi,
>> >> >  I would like  to  confirm what Ian said. I have applied both metho=
ds
>> >> > in
>> >> > theano and it seems that getting rid off dimshuffle   really makes
>> >> > things
>> >> > faster (at least 2 to 3 times)  because the hidden unit in my case =
is
>> >> > large,
>> >> > so it is not only better for reducing memory consumption but to get
>> >> > better
>> >> > performance as well.
>> >> > Thanks...
>> >> >
>> >> >
>> >> > On Thu, Aug 16, 2012 at 9:41 PM, Ian Goodfellow
>> >> > <goodfellow....@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> pylearn2 computes the Jacobian as:
>> >> >> Z =3D V W + b
>> >> >> H =3D g(Z)
>> >> >> G =3D T.grad( H.sum(), Z)  #this assumes elemwise nonlinearity
>> >> >> J =3D W.dimshuffle('x',0,1) * G.dimshuffle(0,'x',1)
>> >> >>
>> >> >> the penalty is then sq(J).sum(axis=3D1,2).mean()
>> >> >>
>> >> >> let's forget about the mean issue for the moment and just
>> >> >> worry about computing sq(J).sum().
>> >> >>
>> >> >> sum_i,j,k W_jk^2 G_ik^2
>> >> >> =3D sum_i,k G_ik^2 sum_j W_jk^2
>> >> >> =3D sum_i, k G_ik^2 sq(W).sum(axis=3D0)_k
>> >> >> =3D sum_i dot( sq(G), sq(W).sum(axis=3D0) )_i
>> >> >>
>> >> >> OK, so it looks like what we want is an optimization that replaces
>> >> >> T.sqr( W.dimshuffle('x',0,1) * G.dimshuffle(0,'x',1)
>> >> >> ).sum(axis=3D(1,2))
>> >> >> with
>> >> >> T.dot( T.sqr(G), T.sqr(W).sum(axis=3D0) )
>> >> >>
>> >> >> I haven't actually tried running the pylearn2 code, but the
>> >> >> optimization
>> >> >> should be faster as well as use less memory.
>> >> >>
>> >> >> If we say W is of shape (v,h) and Z is of shape (b,h) then the
>> >> >> original
>> >> >> has
>> >> >> space and runtime requirements of O(bhv) while the final has space
>> >> >> and runtime requirements of O( bh + vh ).
>> >> >>
>> >> >> On Thu, Aug 16, 2012 at 8:07 PM, David Warde-Farley
>> >> >> <d.warde.far...@gmail.com> wrote:
>> >> >> > On Thu, Aug 16, 2012 at 3:44 PM, Fr=C3=A9d=C3=A9ric Bastien
>> >> >> > <no...@nouiz.org>
>> >> >> > wrote:
>> >> >> >
>> >> >> >> If you end up having a profile that show this is a bottleneck, =
it
>> >> >> >> would be useful to know what % is spend in the elemwise and in
>> >> >> >> the
>> >> >> >> sum.
>> >> >> >
>> >> >> > I don't think the problem was ever computation time. The problem
>> >> >> > is
>> >> >> > memory.
>> >> >> >
>> >> >> > David
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Caglar GULCEHRE
>> >
>> >
>>
>>
>>
>> --
>> Caglar GULCEHRE
>
>



--=20
Caglar GULCEHRE
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
Account Options
