speed of creating a list?

[Terrence Brannon]

I have a 30-record database table which takes less than a second to read. However rendering it to the screen in Reahl takes about 45 seconds.

1. Is Reahl streaming out my data as I add children to the DOM? Or does it render all the DOM once it is all done?
2. I suppose creating all the bookmarks is what also makes it slow?

The 30-record database table is models as a SQLAlchemy table. Here is the relevant code for pulling from the SA table and writing out as HTML:

class MainPage(HTML5Page):

    def __init__(self, view, main_bookmarks):

        super(MainPage, self).__init__(view, style='basic')

        self.use_layout(PageColumnLayout('main'))

        self.layout.header.add_child(Menu.from_bookmarks(

            view, main_bookmarks).use_layout(HorizontalLayout()))

class RecordParserPanel(Panel):

    def __init__(self, view, obfl_ui):

        super(RecordParserPanel, self).__init__(view)

        for row in _Session.query(t_QMx_OBFL_Parser):

            self.add_child(RecordParserEntry(view, row, obfl_ui))

class RecordParserEntry(Widget):

    def __init__(self, view, row, obfl_ui):

        super(RecordParserEntry, self).__init__(view)

        bookmark = obfl_ui.get_edit_bookmark(row=row, description='edit')

        par = self.add_child(P(view, text='%s: %s ' % (row.Parser_Name, row.isEnabled)))

        par.add_child(A.from_bookmark(view, bookmark))

class OBFLParserUI(UserInterface):

    def assemble(self):

        add = self.define_view('/add', title='Add a record parser')

        add.set_slot('main', RecordParserAddForm.factory())

        edit = self.define_view('/edit', view_class=RecordParserEditView, parser_id=IntegerField())

        parsers = self.define_view('/', title='Record Parsers')

        parsers.set_slot('main', RecordParserPanel.factory(self))

        bookmarks = [f.as_bookmark(self) for f in [parsers, add]]

        self.define_page(MainPage, bookmarks)

        self.define_transition(RecordParser.events.save, add, parsers)

        self.define_transition(RecordParser.events.update, edit, parsers)

        self.edit = edit

    def get_edit_bookmark(self, row, description=None):

        return self.edit.as_bookmark(self, parser_id=row.ID, description=description)

[Iwan Vosloo]

Hi Terrence,

I'll have to study this one when I have more time to think, and get back to you on the weekend. 30 records is nothing... it should be quite snappy, but I will have to really understand your program and preferably also run it here.

Regards
- Iwan

--
You received this message because you are subscribed to the Google Groups "Reahl discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reahl-discus...@googlegroups.com.
To post to this group, send email to reahl-...@googlegroups.com.
Visit this group at http://groups.google.com/group/reahl-discuss.
For more options, visit https://groups.google.com/d/optout.

-- 
Reahl, the Python only web framework: http://www.reahl.org

[Iwan Vosloo]

Hi Terrence,

I have tried to reproduce your speed issue on this end, using the code you sent, with some boilerplate added to make it run and be representative of what I think you have on that end. I cannot reproduce any performance issue, on my ageing laptop that page loads in less than a second on a sqlite database with 60 rows.

Let me answer your questions first, though:

On 23/04/2015 01:56, Terrence Brannon wrote:

1. Is Reahl streaming out my data as I add children to the DOM? Or does it render all the DOM once it is all done?
   

That depends on the kind of Resource you are requesting. In this case, all the DOM is first rendered to a string and then that string is streamed back in chunks. (With other resources that may be bigger, such as files served from disk, it works differently.)  This is done so that we can encounter any errors or exceptions in your code before we start streaming back.  (Once you have started streaming something back, you cannot change your mind about what the HTML response code is.)

1. I suppose creating all the bookmarks is what also makes it slow?

Creating many Bookmarks *can* be a problem in some situations. In your case you *do* incur some unnecessary database overhead which can be avoided. (I'll explain that plan at the end of this email). But still, on a database with only 30 rows you should not even be aware of what is happening.  When we had to fine-tune things related to Bookmarks in the past, we were talking about shaving a second off of a response time that was probably just over 1 second to start with and hence considered slow (on a much larger database).

I'd start searching for the problem by printing out (or otherwise collect) what queries are all run, and how long they take in total.

So, extract the query in:

         for row in _Session.query(t_QMx_OBFL_Parser):

Like this:

         start = time.time()
         rows = _Session.query(t_QMx_OBFL_Parser).all()
         print( 'selected list: %s' % (time.time()-start) )
         for row in rows:

... and do something similar with the query I suspect you also have in the .assemble() of RecordParserEditView.

That would yield interesting info to investigate further.



Now about the bookmarks:
------------------------------------

In our tutorial.parameterised1 example, we have the following for the parameterised (edit) view:

class EditView(UrlBoundView):
    def assemble(self, address_id=None):
        try:
            address = Session.query(Address).filter_by(id=address_id).one()
        except NoResultFound:
            raise CannotCreate()

        self.title = 'Edit %s' % address.name
        self.set_slot('main', EditAddressForm.factory(address))

I suspect you did something similar for your RecordParserEditView.

We also have:

    def get_edit_bookmark(self, address, description=None):
        return self.edit.as_bookmark(self, address_id=address.id, description=description)

And I suspect you did the same.

While this is the simplest way to create a bookmark, it is not the most efficient: When you do as_bookmark on a ViewFactory as is happening here, it actually creates the View and then delegates the .as_bookmark call to the actual View instance. When creating the View instance, its .assemble() is called in turn which results in an unnecessary database access. (You already had the row selected from the DB, now you're doing it again.)

You could circumvent this extra DB access (and unnecessary creation of the View entirely) by creating the Bookmark yourself in get_edit_bookmark:

    def get_edit_bookmark(self, row):
           return Bookmark(self.base_path, '/edit?parser_id=%s' % row.ID, description='edit')

This requires a bit deeper understanding of Bookmarks, but not that much. The Views you define on a UserInterface are defined using URLs relative to the UserInterface where you define them. The second arg (ignoring self) to Bookmark.__init__ is called relative_path. You need to compute what that will be in this case. The first arg, base_path, is the path of the entire UserInterface in your current app. These two are concatenated by the Bookmark to yield an href.
By default, a Bookmark also uses the title of the target View as its description - in this case that is not needed, since you always just use the word 'edit'.

I prefer to keep get_bookmark() methods on the UserInterface. The UserInterface already has knowledge of what Views it contains, and which URLs they are on. That is the same knowledge used to create the Bookmark. In fact, the FiewFactory you hold onto in self.edit also has code on it that can recreate its relative_url, given the objects you expect to get out of the URL as view parameters. You can do a bit better and use this fact if you use a method which I see is not documented (but should be):

    def get_edit_bookmark(self, row, description=None):
           return Bookmark(self.base_path, self.edit.get_relative_path(parser_id=row.ID), description=row.Parser_Name)

Aside:
   Actually, this make me think... we ought to add an easier way to do this,
   like a method you can call that will create the bookmark without creating

   the view. Then you could write:

    def get_edit_bookmark(self, row, description=None):

           return self.edit.get_quick_bookmark(row.Parser_Name, parser_id=row.ID)
   
   Note, supplying a description would be mandatory.



All that said - the initial inefficient implementation of get_edit_bookmark we started out with should have overheads so small you should not notice it in your app. Perhaps your queries are slower than you think when executed in this environment for some reason yet to be found?

Regards
- Iwan