get all the expressions driven by a parameter

[Jeremie Passerin]

Hi guys, 

I was wondering what is your technique to get all the expressions driven by a parameter. 

I'm trying to find the fastest way to do it. 

here is my code : 

def getExpressionsDrivenByParameter( param ):

allExpressions = xsi.FindObjects(None, "{12723EB2-7DDB-11D0-A57F-00A0C91412DE}")

result = []

for exp in allExpressions:

for port in exp.InputPorts:

if port.Target2.IsEqualTo(param) and exp.OutputPorts.Count:

result.append(exp)

break

return result

With around 5000 expressions in my scene it takes about 2sec to get the result. Anyone got a faster solution ? 

cheers, 

Jeremie

[Matt Lind]

Although minor, you could move exp.OutputPorts.Count a little earlier in your code as it’s testing the same thing for each input port on the operator.    You should only have to test that once per expression, like this:

 

def getExpressionsDrivenByParameter( param ):

 

     allExpressions = xsi.FindObjects(None, "{12723EB2-7DDB-11D0-A57F-00A0C91412DE}")

    

     result = []

     for exp in allExpressions:

if exp.OutputPorts.Count:

               for port in exp.InputPorts:

                   if port.Target2.IsEqualTo(param):

                        result.append(exp)

                        break

                  

     return result

 

 

 

Other than that, you’re dealing with an O(n^2) runtime.  The only way to make it significantly faster is to unroll the loops, but may not be possible for this task.  You could try reading the “target” and “definition” parameters of the expression, but those may be ambiguous for some cases and also result in string comparisons which may or may not be faster.

 

 

Matt

[Alok]

Maybe globally store the expressions with their params in a hash (dictionary in Python) and then do the reverse look up. Will not run faster but will save you time for each call to getExpressionsDrivenByParameter()

In this case:
PARAM_EXPR_DICT = {}
for expr in Application.FindObjects(None, "{12723EB2-7DDB-11D0-A57F-00A0C91412DE}"):
    ports = expr.InputPorts
    if not ports.Count:
        continue
    for port in in ports :
        paramName = port.Target2.FullName
        if not PARAM_EXPR_DICT.has_key(paramName): PARAM_EXPR_DICT[paramName] = []
        PARAM_EXPR_DICT[paramName].append(expr)
        
        
Then you can test it for parameter simply by:
def getExpressionsDrivenByParameter( param ):
    return (PARAM_EXPR_DICT[param.FullName] if PARAM_EXPR_DICT.has_key(param.Name) else None)

Alok.

No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.2197 / Virus Database: 2437/5158 - Release Date: 07/27/12

[Jeremie Passerin]

Thanks Matt !

Interesting solution Alok, my issue is not the 2sec that it takes to perform the seach but that I am looping on this method about 20 times... so it makes sense to try something like that ! I'll give it a try right now !

[Jeremie Passerin]

Okay you are my hero Alok !

with your approach looping 10 times on this function went from 1min5sec to 6sec !

thanks a lot !

[jo benayoun]

Hey Jeremie,
Here is what I am coming with on a character with thousands of expressions:
# method 1 took 5.42011094882 and find 3 expressions.
# method 2 took 2.31037755754 and find 3 expressions.
# method 3 took 5.41616293425 and find 3 expressions.
I would recommend you to take a look at a paradigm that is called Data-Oriented Design and which is specially considered to avoid CPU usage and fast computation.
Let me know if this fit your needs :)
-- Jo




import time


SI_EXPRESSION_ID = 49


def Jerems( param ):
     allExpressions = Application.FindObjects(None, "{12723EB2-7DDB-11D0-A57F-00A0C91412DE}")

     result = []
     for exp in allExpressions:

        if exp.OutputPorts.Count:
           for port in exp.InputPorts:
               if port.Target2.IsEqualTo(param):
                    result.append(exp)
                    break
     return result


def exprs_from_parameters(parameters):
    if type(parameters) not in (list, tuple):
        parameters = (parameters, )

    res = list()
    parameters = [p for p in parameters if p is not None]
    nparameters = len(parameters)
    if nparameters == 0:
        return res

    expressions = Application.FindObjects2(SI_EXPRESSION_ID)

    res = [list() for i in xrange(nparameters)]
    for exp in expressions:
        sources = [iport.Target2 for iport in exp.InputPorts]
        for i in xrange(nparameters):
            b = [src.IsEqualTo(parameters[i]) for src in sources]
            if any(b):
                res[i].append(exp)
    return tuple(res)


def Aloks(param):

    PARAM_EXPR_DICT = {}
    for expr in Application.FindObjects(None, "{12723EB2-7DDB-11D0-A57F-00A0C91412DE}"):
        ports = expr.InputPorts
        if not ports.Count:
            continue

        for port in ports:

            paramName = port.Target2.FullName
            if not PARAM_EXPR_DICT.has_key(paramName):
                PARAM_EXPR_DICT[paramName] = []
            PARAM_EXPR_DICT[paramName].append(expr)

    return PARAM_EXPR_DICT[param.FullName] if PARAM_EXPR_DICT.has_key(param.Name) else None



msg = "method {0} took {1} and find {2} expressions."     


def timethat1():
    ppg = Application.Selection(0)
    res = []
    t = time.clock()
    for param in ppg.Parameters:
        res.append(Jerems(param))
    t = time.clock() - t
    print msg.format(1, t, sum([len(r) for r in res]))


def timethat2():
    ppg = Application.Selection(0)
    t = time.clock()
    res = exprs_from_parameters(tuple(ppg.Parameters))
    t = time.clock() - t
    print msg.format(2, t, sum([len(r) for r in res]))


def timethat3():
    ppg = Application.Selection(0)
    res = []
    t = time.clock()
    for param in ppg.Parameters:
        res.append(Aloks(param))
    t = time.clock() - t
    print msg.format(3, t, sum([len(r) for r in res]))



timethat1()
timethat2()
timethat3()

2012/7/27 Jeremie Passerin <gere...@gmail.com>

[Alok]

Hey Jo,

Nice job ! but my approach did not involve one function approach to speed up as you used in your profiling, rather it was just to globally store the data and look up as needed. The initial data pull was still the same as Jeremy with a little correction from Matt, but after that the subsequent function calls do not need to query data again as it is already available through the hash. So , my approach was really to store the data once, which I think is working good for Jeremy for now.

Cheers!!

Alok.

[Alok]

Glad to help :)

[Eric Thivierge]

Maybe something like this depending on if you use "this_model" (I dont' recommend it as we've seen tons of expression corruptions involving this).

collExpressions = xsi.FindObjects2(c.siExpressionID)

lTgtExprs = [x for x in collExpressions if "null.kine.local.posx" in x.Parameters("Definition").Value]

--------------------------------------------
Eric Thivierge
http://www.ethivierge.com

[jo benayoun]

Hey Alok,
thanks for you feedback, I think you're mistaken in the interpretation of the code layout I chose tho.
This is the purpose of the DOD paradigm to think about how the data should be input and how it should be treated.  This would be too pretentious from me to say it has been used here, but trying to treat the data the more linearly possible is a good start, it saves memory and speed.  I can bet easily, speed increases massively on yours or Jeremie's if lot more than 2 or 3 parameters are treated compared to mine.  One of the effects of using DOD is you're ending by processing the data linearly which avoid any query next and make your code really fast and reactive.

About the function calls thing, I would say, that making one function call instead of many would not influence so much speed results.  A function is a simple object as many other stored on the heap once created and called with arguments (other objects).  There is no special initialisation nor building done each time.  But even by considering this, and making my function called with xsi parameters passed one by one, this is still faster (20perc on my machine) and this for many different other reasons.

If I didnt chose any dictionaries based implementations as you did, this is because the python hash table implementation is generic and use lot of memory with lost of blocks where py list are implemented as linked list and permit really fast appending, O(1).

:)
-- Jo

2012/7/27 Eric Thivierge <ethiv...@gmail.com>

[Alok Gandhi]

Hi Jo,

"This is the purpose of the DOD paradigm to think about how the data should be input and how it should be treated.  This would be too pretentious from me to say it has been used here, but trying to treat the data the more linearly possible is a good start, it saves memory and speed.  I can bet easily, speed increases massively on yours or Jeremie's if lot more than 2 or 3 parameters are treated compared to mine.  One of the effects of using DOD is you're ending by processing the data linearly which avoid any query next and make your code really fast and reactive. "

The DOD is a nice pattern , I agree.

"About the function calls thing, I would say, that making one function call instead of many would not influence so much speed results.  A function is a simple object as many other stored on the heap once created and called with arguments (other objects).  There is no special initialisation nor building done each time.  But even by considering this, and making my function called with xsi parameters passed one by one, this is still faster (20perc on my machine) and this for many different other reasons."

I would like to differ here, please look at the simple logic or semantics. If want to grab data from XSI, it takes some time to query the data. If i query once and store it in a hash, do not have to query it again, I simply look at my stored hash which is way faster. If you wrap a function which is trying to query data inside of itself, every time you make a call to the function you adding the overhead by the data query. As simple as that. You can see the this has already worked for Jeremy from his reply above. 

"If I didn't chose any dictionaries based implementations as you did, this is because the python hash table implementation is generic and use lot of memory with lost of blocks where py list are implemented as linked list and permit really fast appending, O(1)."

In this case here, the reason to use dictionary was for faster look ups. For look ups specially dictionary are a way to go for me always. I would rather use a dictionary then to store in a list/tuple and then have to do dirty labmdas or other stuff to retreive by list.index() and what not.You really can't beat the python hash table. They may have more memory requirements, but these days, it really is not a problem. For tasks such as these, I suspect it really would not make impact even on a machine with 8 gigs. The hash table are good and really good for one thing, fast look ups. No wonder Google uses python as their main scripting languages and the dictionary is one good reason for it. You can easily verify this on Official Google Developers Videos on You Tube http://www.youtube.com/watch?v=haycL41dAhg.

Cheers !

[Jeremie Passerin]

I haven't tested Jo solution, but just wanted to mentioned that with Alok's one looping a hundred time on this method is about the same speed of treating it once (around 6secs on my machine.)

[Alok Gandhi]

Hi Jeremy,

I missed this first time but to make it more elegant you (not a speed up) you can even do:

def getExpressionsDrivenByParameter( param ):
    return PARAM_EXPR_DICT.get(param.FullName)

which is exactly the same as before but cleaner code,

[Alan Fregtman]

Hey guys,

You're all forgetting the ConnectionStack. It tells you what's connected under the hood. No need to scan through all expressions in the scene!

import xml.etree.ElementTree as ET

def getExpressionsDrivenByParameter( param ):
    stack = XSIUtils.DataRepository.GetConnectionStackInfo(param)

    expressions = XSIFactory.CreateObject('XSI.Collection')
    xml = ET.fromstring(stack)
    for conn in xml.findall('connection'):
        if conn.find('type').text == 'out':
            item = conn.find('object').text
            if item.endswith('.Expression'):
                expressions.AddItems(item)

    return expressions

It took 4.5s on my laptop to find 10,752 expressions that were pointing to one single parameter. Fast enough for ya? :) -- In a less ludicrous use case, it's pretty much instant.

   -- Alan

[Alok]

That is awesome !!!

No virus found in this message.
Checked by AVG - www.avg.com

Version: 2012.0.2197 / Virus Database: 2437/5162 - Release Date: 07/29/12

[Eric Thivierge]

Black magic! I've never even heard of that object. :(

--------------------------------------------
Eric Thivierge
http://www.ethivierge.com

[Alan Fregtman]

If you ever looked at how to parse the operator stack you must've seen the doc page for ConstructionHistory:

http://download.autodesk.com/global/docs/softimage2012/en_us/sdkguide/index.html?url=si_om/ConstructionHistory.html,topicNumber=si_om_ConstructionHistory_html

Third paragraph under Description:

"The construction history is one example of the more general concept of "Connection Stack", see DataRepository.GetConnectionStackInfo for details."

[Eric Thivierge]

Good stuff Alan. Thanks for the tip. Must not be reading the docs as thoroughly as I should. :)

--------------------------------------------
Eric Thivierge
http://www.ethivierge.com

[jo benayoun]

Alan, you're opening a new interesting way. :)

Alok, DOD is not a pattern but a paradigm as OOP.  If you have the time to take a look deeper at it, you will understand what I am saying when I reject any ideas concerning further querying on the data.

I don't remember where I've read this (and I regret), someone was publishing simple profiling results of regular python code and was highlighting that 70 perc of the ressources were exclusively consumed by dictionaries and this tends to grow because people are more and more seduced by the convenience of those objects.

By replacing those by simpler data structures or other mechanisms, speed performances have been improved considerably as well as CPU and memory usage (something like 2 times faster if I remember correctly).

I am really worried when reading "They may have more memory requirements, but these days, it really is not a problem." because this is clearly for me an excuse for developers to do code and not good code.

Personally, I don't consider educational material even from Google engineers as a reference as it's educational.

And yes, I am capable of beating appending on hash table in worst case with a single linked list.  Look at Python C implementation to see how they resolve collisions. 

But even considering the query as important, nothing avoid you to pass parameter by parameter to the function to control the flow of your code.  Like that no need of any further querying.

I've done some other tests today and friday and I am surprised that your function produce wrong results on some models (I didn't find the why yet).

But still, considering everything which has been said, here is what I got as last results on my machine at home for a model with still thousand of expressions and launching a search for 11 parameters of the same ppg. (some of those parameters are driving custom operators, that's why there is only 4 expressions).

# INFO : jerems took 16.2187956328 and find 4 expressions.
# INFO : mine took 1.33966050716 and find 4 expressions.
# INFO : aloks took 12.9062194445 and find 0 expressions.
# INFO : alans took 0.297864204575 and find 4 expressions.

Anyways, I think we have all our own way of coding and I respect that.  It's always a pleasure to share with you!

=)

-- Jo

2012/7/29 Eric Thivierge <ethiv...@gmail.com>

[Jeremie Passerin]

Really nice guys thanks a lot for the inputs ! 

Just one note on Alan's code. His current method returns all the expression driven by an object not by a specific parameter.

I had to slightly adapt the code to get the specific local parameter I'm looking for !

really happy, this is so fast now !

[Alan Fregtman]

Care to share your modifications? :)

[Jeremie Passerin]

Sure ! :D

It only works with local parameters of an object (I renamed the method accordingly) that's enough for me right now. I would need to check what it looks like when the driver is not a local parameter.

Also I made the collection unique because one parameter might be involved several time on the same expression.

Thanks again Alan, it really nice to get this performance !

def getExpressionsDrivenByLocalParameter( obj, param="posx" ):

stack = xsiUtils.DataRepository.GetConnectionStackInfo( obj.Parameters(param) )

expressions = xsiFactory.CreateObject("XSI.Collection")

expressions.Unique = True

xmlRoot = etree.fromstring(stack)

for xmlCnx in xmlRoot.findall('connection'):

if xmlCnx.find('type').text == 'out' and xmlCnx.find('localparameter') is not None and xmlCnx.find('localparameter').text == param:

item = xmlCnx.find('object').text

if item.endswith('.Expression'):

expressions.AddItems(item)

return expressions