Merge of new-backend into develop

[Álvaro Justen [Turicas]]

Hi folks,
I've just merged feature/new-backend into develop.
Except for the pipeliner (that actually resides on pypln/client.py),
it is fully tested (it did a lot of kind of tests and corrected some
bugs). The pipeliner will be tested on issue
28[https://github.com/NAMD/pypln/issues/28].

Note that:
- I've changed some directories and *a lot* of files (most were
deleted, some added), please stash your changes (or discard them/move
to another place) before pulling from GitHub
- The current documentation is deprecated. I've added a "warning" on
all *.rst and created an issue to recreate based on new
backend[https://github.com/NAMD/pypln/issues/34]

If you have some problem with the new backend, please don't push/force
to GitHub. Describe the problem here and I'll help.

I also deleted some branches that were merged into master but were not
deleted. Please, delete local *and remote* branches when finish
working on it. The receipt is:
git branch -f branch-name #delete local one
git push origin :branch-name #delete remote
The branches deleted were:
-feature/cluster_manager
- feature/mimetypes
- hot_fixes/cluster_bugs

[]s
--
 Álvaro Justen "Turicas"
   http://blog.justen.eng.br http://twitter.com/turicas
   http://CursoDeArduino.com.br http://github.com/turicas
   +55 21 9898-0141

[Flavio Coelho]

Alvaro,

after I installed the package, most tests started to pass (I also had to delete my existing PYPLN databaseon Mongo). Even the import error in the broker is gone.

Now Ihave been able to catch some bugs in the testing:

hard coded number of workers:

Traceback (most recent call last):
  File "/home/fccoelho/Documents/Projects_software/pypln/tests/test_broker.py", line 374, in test_broker_should_insert_monitoring_information_about_workers
    self.assertEquals(broker_process['active workers'], 4)
AssertionError: 8 != 4

should have a try/except around the file deletion:

line 76, in tearDownClass
    unlink(worker + 'c') # .pyc
OSError: [Errno 2] No such file or directory: './pypln/workers/dummy.pyc'

that's it for now.

Flávio

--
You received this message because you are subscribed to the Google Groups "PyPLN" group.
To post to this group, send email to py...@googlegroups.com.
To unsubscribe from this group, send email to pypln+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pypln?hl=en.

--
Flávio Codeço Coelho
================
+55(21) 3799-5567
Professor
Escola de Matemática Aplicada 
Fundação Getúlio Vargas
Rio de Janeiro - RJ
Brasil

[Álvaro Justen [Turicas]]

On Sat, Jun 9, 2012 at 6:04 PM, Flavio Coelho <fcco...@gmail.com> wrote:
> Alvaro,
>
> after I installed the package, most tests started to pass (I also had to
> delete my existing PYPLN databaseon Mongo). Even the import error in the
> broker is gone.

This is weird since In tests I use database 'pypln_test'. Probably the
problem was with the installed package.

> Now Ihave been able to catch some bugs in the testing:
>
> hard coded number of workers:
>
> Traceback (most recent call last):
>   File
> "/home/fccoelho/Documents/Projects_software/pypln/tests/test_broker.py",
> line 374, in test_broker_should_insert_monitoring_information_about_workers
>     self.assertEquals(broker_process['active workers'], 4)
> AssertionError: 8 != 4

Fixed, thanks. It was working on my machine because it has 4 cores. :)

> should have a try/except around the file deletion:
>
> line 76, in tearDownClass
>     unlink(worker + 'c') # .pyc
> OSError: [Errno 2] No such file or directory: './pypln/workers/dummy.pyc'
>
> that's it for now.

This will only happen if, for some reason, workers/__init__.py does
not import all the workers. I added try/except but trying to figure
out why it's happening if all other tests are passing. If the worker
initializer didn't create .pyc files for workers, probably there is an
error an some tests should fail.

[Flavio Coelho]

On Sat, Jun 9, 2012 at 11:41 PM, Álvaro Justen [Turicas] <alvaro...@gmail.com> wrote:

On Sat, Jun 9, 2012 at 6:04 PM, Flavio Coelho <fcco...@gmail.com> wrote:
> Alvaro,
>
> after I installed the package, most tests started to pass (I also had to
> delete my existing PYPLN databaseon Mongo). Even the import error in the
> broker is gone.

This is weird since In tests I use database 'pypln_test'. Probably the
problem was with the installed package.

The Funny thing was that it tried to create a "pypln" databased but stopped saying that another database called "PYPLN" already existed.  

> Now Ihave been able to catch some bugs in the testing:
>
> hard coded number of workers:
>
> Traceback (most recent call last):
>   File
> "/home/fccoelho/Documents/Projects_software/pypln/tests/test_broker.py",
> line 374, in test_broker_should_insert_monitoring_information_about_workers
>     self.assertEquals(broker_process['active workers'], 4)
> AssertionError: 8 != 4

Fixed, thanks. It was working on my machine because it has 4 cores. :)

ok

> should have a try/except around the file deletion:
>
> line 76, in tearDownClass
>     unlink(worker + 'c') # .pyc
> OSError: [Errno 2] No such file or directory: './pypln/workers/dummy.pyc'
>
> that's it for now.

This will only happen if, for some reason, workers/__init__.py does
not import all the workers. I added try/except but trying to figure
out why it's happening if all other tests are passing. If the worker
initializer didn't create .pyc files for workers, probably there is an
error an some tests should fail.

Maybe this worker loading mechanism is "too clever". You know the saying right?

"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." --Brian Kernighan

;-)

Flávio
 

--
 Álvaro Justen "Turicas"
   http://blog.justen.eng.br http://twitter.com/turicas
   http://CursoDeArduino.com.br http://github.com/turicas
   +55 21 9898-0141

--
You received this message because you are subscribed to the Google Groups "PyPLN" group.
To post to this group, send email to py...@googlegroups.com.
To unsubscribe from this group, send email to pypln+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pypln?hl=en.

[Álvaro Justen [Turicas]]

On Sun, Jun 10, 2012 at 8:00 AM, Flavio Coelho <fcco...@gmail.com> wrote:
> On Sat, Jun 9, 2012 at 11:41 PM, Álvaro Justen [Turicas]
> <alvaro...@gmail.com> wrote:
>>
>> On Sat, Jun 9, 2012 at 6:04 PM, Flavio Coelho <fcco...@gmail.com> wrote:
>> > Alvaro,
>> >
>> > after I installed the package, most tests started to pass (I also had to
>> > delete my existing PYPLN databaseon Mongo). Even the import error in the
>> > broker is gone.
>>
>> This is weird since In tests I use database 'pypln_test'. Probably the
>> problem was with the installed package.
>
>
> The Funny thing was that it tried to create a "pypln" databased but stopped
> saying that another database called "PYPLN" already existed.

When you run ./pypln/broker.py, it'll use database 'pypln' (and
MongoDB will say that 'PYPLN' already exists). When you run the tests,
it'll only use 'pypln_test' database. So, the problem was created when
running broker.py manually, not the tests.

>> This will only happen if, for some reason, workers/__init__.py does
>> not import all the workers. I added try/except but trying to figure
>> out why it's happening if all other tests are passing. If the worker
>> initializer didn't create .pyc files for workers, probably there is an
>> error an some tests should fail.
>
> Maybe this worker loading mechanism is "too clever". You know the saying
> right?
>
> "Debugging is twice as hard as writing the code in the first place.
> Therefore, if you write the code as cleverly as possible, you are, by
> definition, not smart enough to debug it." --Brian Kernighan
>
> ;-)

Well, I don't think this code is complicated. One reason for that is I
can write automated tests.
If we can't create automated tests, so probably the code is
complicated or too complex. If we create it using TDD, probably the
architecture will be more flexible (hackable) and simple.
BTW, I think it's better to the project having to worker discovery
complex than having the worker development complex, so anyone can
contribute with new workers.

[Flavio Coelho]

On Sun, Jun 10, 2012 at 8:51 PM, Álvaro Justen [Turicas] <alvaro...@gmail.com> wrote:

On Sun, Jun 10, 2012 at 8:00 AM, Flavio Coelho <fcco...@gmail.com> wrote:
> On Sat, Jun 9, 2012 at 11:41 PM, Álvaro Justen [Turicas]
> <alvaro...@gmail.com> wrote:
>>
>> On Sat, Jun 9, 2012 at 6:04 PM, Flavio Coelho <fcco...@gmail.com> wrote:
>> > Alvaro,
>> >
>> > after I installed the package, most tests started to pass (I also had to
>> > delete my existing PYPLN databaseon Mongo). Even the import error in the
>> > broker is gone.
>>
>> This is weird since In tests I use database 'pypln_test'. Probably the
>> problem was with the installed package.
>
>
> The Funny thing was that it tried to create a "pypln" databased but stopped
> saying that another database called "PYPLN" already existed.

When you run ./pypln/broker.py, it'll use database 'pypln' (and
MongoDB will say that 'PYPLN' already exists). When you run the tests,
it'll only use 'pypln_test' database. So, the problem was created when
running broker.py manually, not the tests.

ok. 

>> This will only happen if, for some reason, workers/__init__.py does
>> not import all the workers. I added try/except but trying to figure
>> out why it's happening if all other tests are passing. If the worker
>> initializer didn't create .pyc files for workers, probably there is an
>> error an some tests should fail.
>
> Maybe this worker loading mechanism is "too clever". You know the saying
> right?
>
> "Debugging is twice as hard as writing the code in the first place.
> Therefore, if you write the code as cleverly as possible, you are, by
> definition, not smart enough to debug it." --Brian Kernighan
>
> ;-)

Well, I don't think this code is complicated. One reason for that is I
can write automated tests.
If we can't create automated tests, so probably the code is
complicated or too complex. If we create it using TDD, probably the
architecture will be more flexible (hackable) and simple.
BTW, I think it's better to the project having to worker discovery
complex than having the worker development complex, so anyone can
contribute with new workers.

Well the best is to have everything as simple as possible. But since I am not very familiar with this  new codebase, I really cannot say if it is overly complex or not, it is just an impression. 

--
 Álvaro Justen "Turicas"
   http://blog.justen.eng.br http://twitter.com/turicas
   http://CursoDeArduino.com.br http://github.com/turicas
   +55 21 9898-0141

--
You received this message because you are subscribed to the Google Groups "PyPLN" group.
To post to this group, send email to py...@googlegroups.com.
To unsubscribe from this group, send email to pypln+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pypln?hl=en.