How to build a simple neural network in 9 lines of Python code

[Sam Chats]

https://medium.com/technology-invention-and-more/how-to-build-a-simple-neural-network-in-9-lines-of-python-code-cc8f23647ca1

[Marko Rauhamaa]

Sam Chats <sam_chat...@gioia.aioe.org>:

> https://medium.com/technology-invention-and-more/how-to-build-a-simpl
> e-neural-network-in-9-lines-of-python-code-cc8f23647ca1

Impressive, but APL's got Python beat. This one-liner implements the
Game of Life:

life←{↑1 ⍵∨.∧3 4=+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵}

<URL: https://en.wikipedia.org/wiki/APL_(programming_language)#Gam
e_of_Life>

"Donc Dieu existe, répondez !"


Marko

[John Ladasky]

On Tuesday, June 27, 2017 at 9:24:07 AM UTC-7, Sam Chats wrote:
> https://medium.com/technology-invention-and-more/how-to-build-a-simple-neural-network-in-9-lines-of-python-code-cc8f23647ca1

OK, that's cheating a bit, using Numpy. It's a nice little program, but it leverages a huge, powerful library.

[Marko Rauhamaa]

John Ladasky <john_l...@sbcglobal.net>:

> OK, that's cheating a bit, using Numpy. It's a nice little program,
> but it leverages a huge, powerful library.

What would *not* be cheating? A language without a library would be
dead.


Marko

[alister]

true but for a realistic comparison you should probably count all the
lines of code in the libruary



--
I cannot believe that God plays dice with the cosmos.
-- Albert Einstein, on the randomness of quantum mechanics

[Chris Angelico]

Sure, but there are different levels of cheating. Using a
general-purpose programming language and its standard library isn't
usually considered cheating, but using a language or library that's
specifically designed for this purpose is less about "hey look how
simple this is" and more about "hey look how awesome this lang/lib
is". Which is a perfectly reasonable thing to brag; Python is
beautifully expressive in the general case, but there are some amazing
tools for special purposes.

So I wouldn't call it cheating; it's a demonstration of the
expressiveness of numpy.

ChrisA

[Marko Rauhamaa]

alister <aliste...@ntlworld.com>:

> On Tue, 27 Jun 2017 22:34:18 +0300, Marko Rauhamaa wrote:
>
>> John Ladasky <john_l...@sbcglobal.net>:
>>> OK, that's cheating a bit, using Numpy. It's a nice little program,
>>> but it leverages a huge, powerful library.
>>
>> What would *not* be cheating? A language without a library would be
>> dead.
>

> true but for a realistic comparison you should probably count all the
> lines of code in the libruary

How's that defined? Lines of Python? Lines of C? Lines of assembly?
Lines of microcode?

What is a line?


Marko

[Marko Rauhamaa]

Chris Angelico <ros...@gmail.com>:

You saw the APL example, right? APL's standard runtime/library contains
most of Numpy functionality because that's what APL has been designed
for.

Is that cheating?


Marko

[Chris Angelico]

No, for the same reason. What it demonstrates is the terseness of APL
when used for its exact use case.

Ultimately, everything is implemented using lower level code. Isn't it
impressive how easily a computer can add two integers? Trying to do
the same thing in dominoes takes a lot of code:

http://www.numberphile.com/videos/domino_circuit.html

Should we count program code in domino equivalence?

ChrisA

[Steve D'Aprano]

On Wed, 28 Jun 2017 02:23 am, Sam Chats wrote:

>
https://medium.com/technology-invention-and-more/how-to-build-a-simple-neural-network-in-9-lines-of-python-code-cc8f23647ca1

The derivative of the sigmoid curve given is completely wrong.

def __sigmoid(self, x):
return 1 / (1 + exp(-x))

def __sigmoid_derivative(self, x):
return x * (1 - x)


Should be:

def __sigmoid_derivative(self, x):
return exp(x) / (1 + exp(x))**2



http://mathworld.wolfram.com/SigmoidFunction.html


I wish these neural networks would give a tutorial on how to do something
non-trivial where:

- your training data is not just a couple of bits;

- the function you want the neural network to perform is non-trivial.


E.g. I have a bunch of data:

['cheese', 'pirate', 'aardvark', 'vitamin', 'egg', 'moon']

and a set of flags:

[True, False, True, True, False, True]


and I want the network to predict that:

'calcium' -> False
'aluminium' -> True
'wood' -> True
'plastic' -> False


(The function is, "is there a duplicated vowel?")


--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[Steve D'Aprano]

Its not really nine lines of Python. It's nine lines of Python, to call a
library of many thousands of lines of C, Fortran and Python.

And its not really a neural *network* as such, more like just a single neuron.

Starting from scratch with just the Python language and standard library, no
external third party code, how quickly can you write a neural network? I think
the answer is probably a wee bit more than "nine lines".

[Steve D'Aprano]

Microcode isn't text so it doesn't come in lines.

Obviously we're talking about source code, not object code, byte code or machine
code, since source code is the only one that is measured in lines rather than
bytes.

[Steve D'Aprano]

On Wed, 28 Jun 2017 06:22 am, Marko Rauhamaa wrote:

> You saw the APL example, right? APL's standard runtime/library contains
> most of Numpy functionality because that's what APL has been designed
> for.
>
> Is that cheating?

Of course not. That demonstrates beautifully (or perhaps "unreadably tersely")
that the APL language primitives are extremely powerful (too powerful?).

Apart from just your usual contrariness *wink* I don't know why you are
objecting to this. It should be obvious that if you allow the use of external
libraries that can contain arbitrary amounts of code, *without* counting that
external code towards your measure of code complexity, you get a bogus
measurement of code complexity.

(That's like manufacturers who can make things artificially cheaply because they
don't have to pay the full costs of their input and processes: they get their
raw materials subsidised, putting part of the cost on tax payers, and don't
have to pay for their pollution, making others pay the cost.)

Suppose I said that I can write a full featured, advanced GUI web browser,
complete with Javascript and a plug-in system, in just *two* lines of Python
code:

import webbrowser
webbrowser.open('http://www.example.com')


What an astonishingly powerful language Python must be! Other languages require
dozens of lines of code just to parse Javascript, let alone run it.

Except... I'm not really measuring the capability of *Python*, as such. Not if I
word it as I have. I haven't really implemented a web browser, I'm just using
an external web browser. If I wanted to change the parameters, lets say by
changing the Javascript keyword "true" to "truish", then my code would expand
from two lines to millions of lines.

My demo of launching an external process to do all the work is of no help at all
to assist a developer in estimating the expressiveness and power of the
language.

If I were more honest, I would say:

"Here is how I can launch a web browser using the Python standard library in
just two lines of code."

which is an honest description of what I did, and can allow people to make fair
comparisons. E.g. comparing Python to AcmeScript, where you write:


program myprogram
begin program
load library webbrowser
new string url copied from 'http://www.example.com'
with webbrowser
begin
method = open
method.call url
end
end program

(I made that language up, for the record, don't bother googling for it.)

That gives you a fair comparison of the language expressiveness and power.

Chris is right: saying that we can create a simple neural network in 9 lines of
Python plus numpy allows fair comparison to other combinations of language and
library. Implying that it's just Python alone does not.

[Ian Kelly]

On Tuesday, June 27, 2017, Steve D'Aprano <steve+...@pearwood.info>
wrote:

> On Wed, 28 Jun 2017 02:23 am, Sam Chats wrote:
>
> >
> https://medium.com/technology-invention-and-more/how-to-
> build-a-simple-neural-network-in-9-lines-of-python-code-cc8f23647ca1
>
>
> The derivative of the sigmoid curve given is completely wrong.
>
> def __sigmoid(self, x):
> return 1 / (1 + exp(-x))
>
> def __sigmoid_derivative(self, x):
> return x * (1 - x)
>
>
> Should be:
>
> def __sigmoid_derivative(self, x):
> return exp(x) / (1 + exp(x))**2
>
>
>
> http://mathworld.wolfram.com/SigmoidFunction.html
>
>

Actually, it's not stated clearly but x is not the same in each function.
The return value of sigmoid is the input of sigmoid_derivative. The article
that you linked confirms that this identity holds.

[Chris Angelico]

On Wed, Jun 28, 2017 at 12:28 PM, Steve D'Aprano
<steve+...@pearwood.info> wrote:
> E.g. comparing Python to AcmeScript, where you write:
>
>
> program myprogram
> begin program
> load library webbrowser
> new string url copied from 'http://www.example.com'
> with webbrowser
> begin
> method = open
> method.call url
> end
> end program
>
> (I made that language up, for the record, don't bother googling for it.)

Just to be contrary, I did. And Google suggested that maybe I meant
"acnescript".

Hmmmmmmmmmmmm.

ChrisA

[Rustom Mody]

On Wednesday, June 28, 2017 at 1:04:46 AM UTC+5:30, Marko Rauhamaa wrote:
> John Ladasky

> > OK, that's cheating a bit, using Numpy. It's a nice little program,
> > but it leverages a huge, powerful library.
>
> What would *not* be cheating? A language without a library would be
> dead.

One man's powerful is another's simple
A beginning C programmer treats lists as a "data structure" for an advanced
course on data structures
A beginning python programmer starts using them in his first hour or at worst day

Likewise python does this with the help of numpy
From which one could conclude one of
- This (numpy-using) 9 lines is advanced
- Numpy needs to be in python's core

My guess of Marko's intent of bringing up APL is just to show a language in
which numpy is effectively in the language core

My other guess is that these same 9 lines could be translated to R with no import

PS I am not advocating numpybuiltintopython

[Gregory Ewing]

Steve D'Aprano wrote:
> It should be obvious that if you allow the use of external
> libraries that can contain arbitrary amounts of code, *without* counting that
> external code towards your measure of code complexity, you get a bogus
> measurement of code complexity.

Numpy isn't really doing a lot here, just saving you from
having to write some explicit loops. The code would be
maybe 2 or 3 times bigger at most without it.

What this example says to me is not so much how powerful
Python is, but how simple in essence neural networks are.

--
Greg

[sohca...@gmail.com]

Agreed.

It's equivalent to saying "Hey you can create an HTTP server in only 3 lines!" when you're just making a call to SimpleHTTPServer.

If I create a AAA-quality game and pack it up so all I have to do is "import mygame; mygame.start()", then have I created a AAA-quality game in only two lines?

[John Ladasky]

On Tuesday, June 27, 2017 at 12:34:46 PM UTC-7, Marko Rauhamaa wrote:
> John Ladasky <j...y@s...l.net>:

> > OK, that's cheating a bit, using Numpy. It's a nice little program,
> > but it leverages a huge, powerful library.
>
> What would *not* be cheating? A language without a library would be
> dead.

Python's standard library is huge, and useful. Numpy is not part of Python's STANDARD library.

I love Numpy, and use it in almost everything I write. But I see that many people won't need it, would find it overwhelming, and would find its way of doing things rather different than standard Python (vectorized mathematical operations, fancy slicing, broadcasting, etc.).

So to use Numpy in a post that says, "implement a neural network in Python with just nine lines of code!" seems a little misleading to me. Python is IMHO the best programming language in use today -- and it's the best one for a beginner to start learning. But I wouldn't want to oversell the language to a novice programmer. It's going to take you a long time to understand exactly what those nine lines of code are doing, if you're new to this.

[John Ladasky]

On Tuesday, June 27, 2017 at 7:28:58 PM UTC-7, Steve D'Aprano wrote:
> On Wed, 28 Jun 2017 06:22 am, Marko Rauhamaa wrote:
>
> > You saw the APL example, right? APL's standard runtime/library contains
> > most of Numpy functionality because that's what APL has been designed
> > for.
> >
> > Is that cheating?
>
>
> Of course not. That demonstrates beautifully (or perhaps "unreadably tersely")
> that the APL language primitives are extremely powerful (too powerful?).

Interesting how this discussion has detoured into APL. I think that these comments underscore my point that Numpy's way of doing things isn't quite Python's way of doing things, that it's a little esoteric.

I don't know APL, but... some years ago, I remember reading that the concept of a scalar did not exist in APL. The equivalent of a scalar in APL is (if I understand correctly) a one-dimensional array with one element in it.

When I first read that, I thought it was a needless complication.

After ten years of Numpy and three months of TensorFlow, it's starting to dawn on me why that might actually make programming sense... if you're thinking in matrix algebra all the time, which increasingly, I find myself doing.