PythonCommand altering Wedge downstream

[Unai Martínez Barredo]

Hi, in this graph:

When executing the Wedge node, I was expecting to get 2 images, with the numbers 0 and 1. Instead, I get 11 images, with decimal numbers from 0 to 1.

Am I overlooking something? I’m very new to Gaffer, so I might very well be doing something terribly wrong here. Thanks in advance!

[Unai Martínez Barredo]

I can confirm that the behaviour is exactly the same if I connect the nodes like this instead:

[Daniel Dresser]

Hmm.  Altering the graph from a PythonCommand is a little bit uncharted territory.

I do wonder whether there is a structure where you could do this with an expression.  The problem you're trying to solve is that the setup expression is slow, and you only want to run it once, right?  As long as the inputs to expression are constant, it should get cached, and never re-evaluated.

There is a pattern where we've seen this sort of problem with, that looks like, in pseudocode:

Expression1 :

* compute complex data structure

* set wedge based on number of entries

* check a context variable for the current entry

* set some other plugs specific to the current entry

This is bad because one expression both does the really heavy compute, and depends on the context for the current entry, which means we have to keep recomputing it.  This can be refactored like:

Expression1

* compute complex data structure

* set wedge based on number of entries

* store complex data structure on intermediate plug ( Often an AtomicCompoundDataPlug or StringVectorDataPlug )

Expression2

* check a context variable for the current entry

* use intermediate plug to access complex data structure

* read appropriate entry from complex data structure for current entry

* set some other plugs specific to the current entry

This way, expression 2 is quick, since it doesn't need to compute the complex structure, it just reads it.  Expression 1 is slow, but only runs once.

If you're able to structure things that way, it's most likely to be reliable.

If you really need to use the PythonCommand though, you can probably do it.  The trick is that you'll need to move things into the PythonCommand a bit more.  Currently, you're using Gaffer's dispatcher to run everything at once - that will start by looking at the graph, figuring out what needs to be run, and then starting to run it - it doesn't account for the possibility that some task are going to modify the dispatch graph itself.  What you could do instead is only dispatch the PythonCommand, and manually execute the other graph from the PythonCommand - this can be done either with Task.out.execute() ( which only runs one node without following the dispatch graph ) or by creating a temp GafferDispatch.Dispatcher and calling dispatch on it ( which can follow dispatch graph control flow, like wedges ).  If you do go with this approach, you might even want to move more of the logic into the PythonCommand - instead of using a wedge, you could just change input plug values in a loop and then redo the dispatch for each iteration.  Then you're really not benefitting from GafferDispatch stuff, but that's probably where you're at if you need to go this way.

I'm still hopeful that there's probably a way to structure what you need using expressions though.

-Daniel

[Unai Martínez Barredo]

Thanks Daniel. Unfortunately, using two Expressions as you suggest, still makes the expensive one run per frame, per wedge:

import Gaffer
import GafferDispatch
import GafferImage
import IECore
import imath

Gaffer.Metadata.registerValue( parent, "serialiser:milestoneVersion", 0, persistent=False )
Gaffer.Metadata.registerValue( parent, "serialiser:majorVersion", 61, persistent=False )
Gaffer.Metadata.registerValue( parent, "serialiser:minorVersion", 5, persistent=False )
Gaffer.Metadata.registerValue( parent, "serialiser:patchVersion", 0, persistent=False )

__children = {}

parent["variables"].addChild( Gaffer.NameValuePlug( "image:catalogue:port", Gaffer.IntPlug( "value", defaultValue = 0, flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ), "imageCataloguePort", Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic ) )
parent["variables"].addChild( Gaffer.NameValuePlug( "project:name", Gaffer.StringPlug( "value", defaultValue = 'default', flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ), "projectName", Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic ) )
parent["variables"].addChild( Gaffer.NameValuePlug( "project:rootDirectory", Gaffer.StringPlug( "value", defaultValue = '$HOME/gaffer/projects/${project:name}', flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ), "projectRootDirectory", Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic ) )
__children["defaultFormat"] = GafferImage.FormatPlug( "defaultFormat", defaultValue = GafferImage.Format( 1920, 1080, 1.000 ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, )
parent.addChild( __children["defaultFormat"] )
__children["Text"] = GafferImage.Text( "Text" )
parent.addChild( __children["Text"] )
__children["Text"].addChild( Gaffer.V2fPlug( "__uiPosition", defaultValue = imath.V2f( 0, 0 ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["ImageWriter"] = GafferImage.ImageWriter( "ImageWriter" )
parent.addChild( __children["ImageWriter"] )
__children["ImageWriter"].addChild( Gaffer.V2fPlug( "__uiPosition", defaultValue = imath.V2f( 0, 0 ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["Wedge"] = GafferDispatch.Wedge( "Wedge" )
parent.addChild( __children["Wedge"] )
__children["Wedge"]["preTasks"].addChild( GafferDispatch.TaskNode.TaskPlug( "preTask1", flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["Wedge"]["ramp"].clearPoints()
__children["Wedge"]["ramp"].addChild( Gaffer.ValuePlug( "p0", flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["Wedge"]["ramp"]["p0"].addChild( Gaffer.FloatPlug( "x", defaultValue = 0.0, flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["Wedge"]["ramp"]["p0"].addChild( Gaffer.Color3fPlug( "y", defaultValue = imath.Color3f( 0, 0, 0 ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["Wedge"]["ramp"].addChild( Gaffer.ValuePlug( "p1", flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["Wedge"]["ramp"]["p1"].addChild( Gaffer.FloatPlug( "x", defaultValue = 1.0, flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["Wedge"]["ramp"]["p1"].addChild( Gaffer.Color3fPlug( "y", defaultValue = imath.Color3f( 1, 1, 1 ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["Wedge"].addChild( Gaffer.V2fPlug( "__uiPosition", defaultValue = imath.V2f( 0, 0 ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["StoreData"] = Gaffer.Node( "StoreData" )
parent.addChild( __children["StoreData"] )
__children["StoreData"]["user"].addChild( Gaffer.StringVectorDataPlug( "data", defaultValue = IECore.StringVectorData( [  ] ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["StoreData"].addChild( Gaffer.V2fPlug( "__uiPosition", defaultValue = imath.V2f( 0, 0 ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["CalculateData"] = Gaffer.Expression( "CalculateData" )
parent.addChild( __children["CalculateData"] )
__children["CalculateData"]["__in"].addChild( Gaffer.StringPlug( "p0", defaultValue = '$HOME/gaffer/projects/${project:name}', flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["CalculateData"]["__in"].addChild( Gaffer.StringPlug( "p1", defaultValue = 'default', flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["CalculateData"]["__out"].addChild( Gaffer.IntPlug( "p0", direction = Gaffer.Plug.Direction.Out, defaultValue = 11, minValue = 2, flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["CalculateData"]["__out"].addChild( Gaffer.StringVectorDataPlug( "p1", direction = Gaffer.Plug.Direction.Out, defaultValue = IECore.StringVectorData( [  ] ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["CalculateData"].addChild( Gaffer.V2fPlug( "__uiPosition", defaultValue = imath.V2f( 0, 0 ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["ContextVariables"] = Gaffer.ContextVariables( "ContextVariables" )
parent.addChild( __children["ContextVariables"] )
__children["ContextVariables"].setup( GafferImage.ImagePlug( "in", ) )
__children["ContextVariables"]["variables"].addChild( Gaffer.NameValuePlug( "", Gaffer.StringPlug( "value", defaultValue = '', flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ), True, "member1", Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic ) )
__children["ContextVariables"].addChild( Gaffer.V2fPlug( "__uiPosition", defaultValue = imath.V2f( 0, 0 ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["Expression"] = Gaffer.Expression( "Expression" )
parent.addChild( __children["Expression"] )
__children["Expression"]["__in"].addChild( Gaffer.StringVectorDataPlug( "p0", defaultValue = IECore.StringVectorData( [  ] ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["Expression"]["__out"].addChild( Gaffer.StringPlug( "p0", direction = Gaffer.Plug.Direction.Out, defaultValue = '', flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
__children["Expression"].addChild( Gaffer.V2fPlug( "__uiPosition", defaultValue = imath.V2f( 0, 0 ), flags = Gaffer.Plug.Flags.Default | Gaffer.Plug.Flags.Dynamic, ) )
parent["frameRange"]["start"].setValue( 1001 )
parent["frameRange"]["end"].setValue( 1024 )
parent["frame"].setValue( 1001.0 )
parent["variables"]["imageCataloguePort"]["value"].setValue( 38396 )
parent["variables"]["projectName"]["value"].setValue( 'expensiveExpression' )
Gaffer.Metadata.registerValue( parent["variables"]["imageCataloguePort"], 'readOnly', True )
Gaffer.Metadata.registerValue( parent["variables"]["projectName"]["name"], 'readOnly', True )
Gaffer.Metadata.registerValue( parent["variables"]["projectRootDirectory"]["name"], 'readOnly', True )
__children["Text"]["out"].setInput( __children["Text"]["__merge"]["out"] )
__children["Text"]["text"].setValue( '${mydata}\n${frame}' )
__children["Text"]["area"].setValue( imath.Box2i( imath.V2i( 0, 0 ), imath.V2i( 1920, 1080 ) ) )
__children["Text"]["__uiPosition"].setValue( imath.V2f( 4.94926643, 15.5640621 ) )
__children["ImageWriter"]["in"].setInput( __children["ContextVariables"]["out"] )
__children["ImageWriter"]["fileName"].setValue( '${project:rootDirectory}/${project:name}_${wedge:index}/${project:name}_${wedge:index}.####.jpg' )
__children["ImageWriter"]["__uiPosition"].setValue( imath.V2f( 3.44926524, -0.764061749 ) )
__children["Wedge"]["preTasks"][0].setInput( __children["ImageWriter"]["task"] )
__children["Wedge"]["floatSteps"].setInput( __children["CalculateData"]["__out"]["p0"] )
__children["Wedge"]["__uiPosition"].setValue( imath.V2f( 4.14926529, -8.92812347 ) )
__children["StoreData"]["user"]["data"].setInput( __children["CalculateData"]["__out"]["p1"] )
Gaffer.Metadata.registerValue( __children["StoreData"]["user"]["data"], 'nodule:type', '' )
__children["StoreData"]["__uiPosition"].setValue( imath.V2f( -17.2999916, 7.05000067 ) )
Gaffer.Metadata.registerValue( __children["CalculateData"], 'nodeGadget:type', 'GafferUI::StandardNodeGadget' )
__children["CalculateData"]["__in"]["p0"].setInput( parent["variables"]["projectRootDirectory"]["value"] )
__children["CalculateData"]["__in"]["p1"].setInput( parent["variables"]["projectName"]["value"] )
__children["CalculateData"]["__uiPosition"].setValue( imath.V2f( -17.1999969, -8.75000095 ) )
__children["ContextVariables"]["variables"]["member1"]["name"].setValue( 'mydata' )
__children["ContextVariables"]["variables"]["member1"]["value"].setInput( __children["Expression"]["__out"]["p0"] )
__children["ContextVariables"]["in"].setInput( __children["Text"]["out"] )
Gaffer.Metadata.registerValue( __children["ContextVariables"]["in"], 'noduleLayout:section', 'top' )
Gaffer.Metadata.registerValue( __children["ContextVariables"]["out"], 'noduleLayout:section', 'bottom' )
__children["ContextVariables"]["__uiPosition"].setValue( imath.V2f( 4.94926643, 7.40000057 ) )
__children["Expression"]["__in"]["p0"].setInput( __children["StoreData"]["user"]["data"] )
__children["Expression"]["__uiPosition"].setValue( imath.V2f( -6.32661057, 7.39918041 ) )
__children["CalculateData"]["__engine"].setValue( 'python' )
__children["CalculateData"]["__expression"].setValue( 'import datetime\n\nEXPENSIVE_DATA = ["alice", "bob"]\n\nwith open("%s/%s.log" % (parent["__in"]["p0"], parent["__in"]["p1"]), "a") as f:\n    f.write(datetime.datetime.now().isoformat(" ") + "\\n")\n\nparent["__out"]["p0"] = len(EXPENSIVE_DATA)\nparent["__out"]["p1"] = IECore.StringVectorData(EXPENSIVE_DATA)' )
__children["Expression"]["__engine"].setValue( 'python' )
__children["Expression"]["__expression"].setValue( 'data = parent["__in"]["p0"]\nparent["__out"]["p0"] = data[context["wedge:index"]]' )


del __children

[John Haddon]

 Altering the graph from a PythonCommand is a little bit uncharted territory.

Yep, making modifications to the graph during dispatch is certainly not supported. The Dispatcher can't reliably reason about the system if the system changes underneath it during dispatch. In this case, the Dispatcher has already built the entire job graph before the PythonCommand is executed, so it's too late for the PythonCommand to affect anything.

Unfortunately, using two Expressions as you suggest, still makes the expensive one run per frame, per wedge

Typically each task is executed in a separate `gaffer` process, so the expression will inevitably end up being run once per task (if the task depends on the expression, as is the case here).

I presume your example is vastly simplified compared to the reality, but it can actually be refactored such that the expression only runs once during dispatch. Since you eventually make a string context variable, you can use the Wedge in StringList mode and just provide the strings to it directly (example attached). Perhaps in the more complex reality, you can encode the data somehow such that it can be represented as a string?

Alternatively, there is a "blessed" mechanism for modifying a node graph immediately prior to dispatch : `Dispatcher::dispatchSignal()`. This signal is emitted immediately prior to dispatch, and it is legitimate to modify the node graph from within a slot connected to that signal. This is most useful when you want to do something for every dispatch (it's typically used by custom asset management systems), so again, might or might not map well to the specifics of what you're trying to achieve.

Cheers...

John

MILK VISUAL EFFECTS

Tel: +44 (0)20 3697 8448

 

This message is intended solely for the addressee and may contain confidential and/or legally privileged information. Any use, disclosure or reproduction without the sender’s explicit consent is unauthorized and may be unlawful. If you have received this message in error, please notify Milk VFX immediately and permanently delete it. Any views or opinions expressed in this message are solely those of the author and do not necessarily represent those of Milk VFX. 

By engaging in professional correspondence with Milk VFX, we may store your contact information for future reference. This is in line with Milk’s Privacy policy which can be found here. Milk Visual Effects is a registered limited company: 1324 8586. The registered company address is Unit 5, 7 Tyers Gate, London SE1 3HX.

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/a31babd0-57d8-402d-bfd8-0c6fd5a09db5n%40googlegroups.com.

[Unai Martínez Barredo]

Thanks John! Yeah, the example is massively simplified indeed— for the real thing, I will probably use either a Spreadsheet node, or an AtomicCompoundDataPlug as Dave suggested (cheers!).

dispatchSignal does sound like what I want; checking it out now. Thanks again!

PS: Is there any way we can attach Gaffer files instead of copy/pasting them?

[Unai Martínez Barredo]

Hi John,

I’m a little confused about dispatching through the API in general. I have a graph consisting of a Text node and an ImageWriter, and I’m trying to run the following code in the Python Editor:

import pprint

def slot(*args, **kwargs):
    with open('/path/to/dispatchSignal.log', 'w') as f:
        pprint.pprint(args, f)
        pprint.pprint(kwargs, f)

d = GafferDispatch.LocalDispatcher()
d.dispatchSignal().connect(slot)
with root.context():
    d.dispatch([root['ImageWriter']])

However, /path/to/dispatchSignal.log never gets created. What am I missing here?

Thanks,

Unai

[Daniel Dresser]

I think maybe you're just getting caught by a nasty API we haven't fixed quite yet.   `connect(slot)` takes an additional argument "scoped", which defaulted to True, which meant that a handle would be returned, and you must keep that handle alive in order to maintain the connection.  This was a confusing default, which we're working on fixing - in most recent Gaffer, the default has been removed, forcing you to deal with it, rather than silently failing, but that change may not have been released yet.

I think it may work if you do either:

`myConnection = d.dispatchSignal().connect(slot)`

or

`d.dispatchSignal().connect(slot, scoped = False)`

Currently, I think what's happening is that you're using the form that returns a handle, but not storing the handle, so it immediately gets garbage collected, and the connection gets removed ( this is an extremely easy mistake to make, which is why we are going to improve this in the future ).

If that's still not working, then maybe there's a more complex issue.

-Daniel