Jenkins as a render farm
34 views
Skip to first unread message
Mark Raynsford
Dec 18, 2018, 3:56:45 PM12/18/18
to jenkins...@googlegroups.com
Hello!
I've been using Jenkins to build code from hundreds of git repositories
using declarative pipelines. In the setup I'm using, I have a small
number of nodes, and all nodes must successfully build and run all
tests in order for the job as a whole to have been considered
successful.
I'd now like to try using Jenkins to render animations using Blender[0].
The problem that I'm actually trying to solve is that Blender (like all
3D renderers) generally takes a long time to render images. When
rendering an animation, each frame is rendered individually and saved
to a PNG file. When all frames have been rendered, the PNG files can be
stitched together with something like ffmpeg to produce a video file.
Given that there's no data dependency between frames, it's possible to
distribute the workload over many machines by giving each machine a set
of frame numbers to render and then collecting the images from all
machines when they've all finished rendering.
Blender has a full command-line interface, so the 'sh' declaration
in the pipelines will be enough to launch instances of Blender for
rendering. I can place the data to be rendered in a Git repository so
that all nodes can pull from that repository to start building. I would
include a pipeline script in the repository that each node can use to
build. This is how I've been handling building code, and it's been
working very well.
The problem: It's not clear to me how I could assign a set of frame
numbers to each node in the pipeline. I could hard code the numbers
into something that the pipeline code reads in the first stage of
building, but it seems like to do that effectively, I'd need to know
all of the nodes ahead of time (meaning I'd have to rewrite the data if
I add or remove a node later).
What I would really like is for the pipeline in each node to be told
"You are node 1 of 4" "You are node 3 of 4", and so on. Node 3 could
then, for example, use this information to tell Blender "Start
rendering at frame 3, and step forward by 4 frames each time" (4
because there are 4 nodes in total). If each node did this, the entire
range of frames would be covered and the work would be split equally
between the four nodes. The nodes I have are homogeneous, so there's no
need to try to give powerful nodes more work, and weaker nodes less.
Any advice would be appreciated!
[0] https://www.blender.org/
--
Mark Raynsford | http://www.io7m.com
I've been using Jenkins to build code from hundreds of git repositories
using declarative pipelines. In the setup I'm using, I have a small
number of nodes, and all nodes must successfully build and run all
tests in order for the job as a whole to have been considered
successful.
I'd now like to try using Jenkins to render animations using Blender[0].
The problem that I'm actually trying to solve is that Blender (like all
3D renderers) generally takes a long time to render images. When
rendering an animation, each frame is rendered individually and saved
to a PNG file. When all frames have been rendered, the PNG files can be
stitched together with something like ffmpeg to produce a video file.
Given that there's no data dependency between frames, it's possible to
distribute the workload over many machines by giving each machine a set
of frame numbers to render and then collecting the images from all
machines when they've all finished rendering.
Blender has a full command-line interface, so the 'sh' declaration
in the pipelines will be enough to launch instances of Blender for
rendering. I can place the data to be rendered in a Git repository so
that all nodes can pull from that repository to start building. I would
include a pipeline script in the repository that each node can use to
build. This is how I've been handling building code, and it's been
working very well.
The problem: It's not clear to me how I could assign a set of frame
numbers to each node in the pipeline. I could hard code the numbers
into something that the pipeline code reads in the first stage of
building, but it seems like to do that effectively, I'd need to know
all of the nodes ahead of time (meaning I'd have to rewrite the data if
I add or remove a node later).
What I would really like is for the pipeline in each node to be told
"You are node 1 of 4" "You are node 3 of 4", and so on. Node 3 could
then, for example, use this information to tell Blender "Start
rendering at frame 3, and step forward by 4 frames each time" (4
because there are 4 nodes in total). If each node did this, the entire
range of frames would be covered and the work would be split equally
between the four nodes. The nodes I have are homogeneous, so there's no
need to try to give powerful nodes more work, and weaker nodes less.
Any advice would be appreciated!
[0] https://www.blender.org/
--
Mark Raynsford | http://www.io7m.com
Slide
Dec 18, 2018, 4:01:28 PM12/18/18
to Jenkins User Mailing List
I think would be "easy" in a scripted pipeline compared to a declarative pipeline. The reason I say that is because you can use loops directly in your pipeline, so if you knew the number of frames, you could iterate over the number of frames with 4 as an increment and create parallel steps for each iteration (https://jenkins.io/doc/pipeline/examples/#parallel-from-list). You'd use the iterator variable as the start index and the increment as the number of frames. You could then even make the number of frames a parameter to your job. I am not sure how this could be done with declarative, I am still new to using declarative.
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/20181218205530.767b8097%40almond.int.arc7.info.
For more options, visit https://groups.google.com/d/optout.
Website: http://earl-of-code.com
Mark Raynsford
Dec 18, 2018, 4:29:24 PM12/18/18
to Slide, jenkins...@googlegroups.com
On 2018-12-18T14:01:09 -0700
I would probably want it the other way around though: The increment of
4 is derived from the number of nodes, and therefore I'd want this to
be determined at build-time so that I can easily add more nodes
later. The total number of frames that need to be rendered is
effectively constant; it's part of the data in the repository.
Slide <slide...@gmail.com> wrote:
> I think would be "easy" in a scripted pipeline compared to a declarative
> pipeline. The reason I say that is because you can use loops directly in
> your pipeline, so if you knew the number of frames, you could iterate over
> the number of frames with 4 as an increment and create parallel steps for
> each iteration (https://jenkins.io/doc/pipeline/examples/#parallel-from-list).
> You'd use the iterator variable as the start index and the increment as the
> number of frames. You could then even make the number of frames a parameter
> to your job. I am not sure how this could be done with declarative, I am
> still new to using declarative.
I see, thanks! I'll keep it in mind.
> I think would be "easy" in a scripted pipeline compared to a declarative
> pipeline. The reason I say that is because you can use loops directly in
> your pipeline, so if you knew the number of frames, you could iterate over
> the number of frames with 4 as an increment and create parallel steps for
> each iteration (https://jenkins.io/doc/pipeline/examples/#parallel-from-list).
> You'd use the iterator variable as the start index and the increment as the
> number of frames. You could then even make the number of frames a parameter
> to your job. I am not sure how this could be done with declarative, I am
> still new to using declarative.
I would probably want it the other way around though: The increment of
4 is derived from the number of nodes, and therefore I'd want this to
be determined at build-time so that I can easily add more nodes
later. The total number of frames that need to be rendered is
effectively constant; it's part of the data in the repository.
Slide
Dec 18, 2018, 4:31:28 PM12/18/18
to Mark Raynsford, jenkins...@googlegroups.com
I'm not sure you can get the number of nodes that match a label, that might be an interesting step to add though.
Mark Raynsford
Dec 19, 2018, 4:27:57 AM12/19/18
to Slide, jenkins...@googlegroups.com
On 2018-12-18T14:31:07 -0700
variables.
I think it probably knows the current number of nodes for a job, and it
could probably assign each node a number within that set... Anyone know
how easy/difficult it would be to implement this?
Slide <slide...@gmail.com> wrote:
> I'm not sure you can get the number of nodes that match a label, that might
> be an interesting step to add though.
It'd be nice if Jenkins passed this information to me as environment
> I'm not sure you can get the number of nodes that match a label, that might
> be an interesting step to add though.
variables.
I think it probably knows the current number of nodes for a job, and it
could probably assign each node a number within that set... Anyone know
how easy/difficult it would be to implement this?
Mark Raynsford
Dec 19, 2018, 6:58:50 AM12/19/18
to jenkins...@googlegroups.com
On 2018-12-19T09:27:31 +0000
are participating in rendering, then I think I could implement this
using the pipeline-utility-steps plugin [0]. It has a step that can
list all nodes with a given set of labels.
[0] https://plugins.jenkins.io/pipeline-utility-steps
> I think it probably knows the current number of nodes for a job, and it
> could probably assign each node a number within that set... Anyone know
> how easy/difficult it would be to implement this?
If I make the assumption that all offline nodes with the correct label
> could probably assign each node a number within that set... Anyone know
> how easy/difficult it would be to implement this?
are participating in rendering, then I think I could implement this
using the pipeline-utility-steps plugin [0]. It has a step that can
list all nodes with a given set of labels.
[0] https://plugins.jenkins.io/pipeline-utility-steps
Mark Raynsford
Dec 19, 2018, 7:01:55 AM12/19/18
to jenkins...@googlegroups.com
On 2018-12-19T11:58:28 +0000
> If I make the assumption that all offline nodes...
s/offline/online
> If I make the assumption that all offline nodes...
s/offline/online
Slide
Dec 19, 2018, 7:07:17 AM12/19/18
to Jenkins User Mailing List
I thought there was a step for that already, hopefully that will work for you.
--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/20181219120142.1169318d%40almond.int.arc7.info.
For more options, visit https://groups.google.com/d/optout.
Website: http://earl-of-code.com
Mark Raynsford
Dec 20, 2018, 5:34:04 AM12/20/18
to jenkins...@googlegroups.com
On 2018-12-19T11:58:28 +0000
probably nothing surprising to anyone experienced with Jenkins. I was
pretty pleased to be able to cut down my rendering times, though!
https://blog.io7m.com/2018/12/19/low-budget-render-farm.xhtml
> If I make the assumption that all offline nodes with the correct label
> are participating in rendering, then I think I could implement this
> using the pipeline-utility-steps plugin [0]. It has a step that can
> list all nodes with a given set of labels.
This ended up working out quite well. Nothing earth shattering, and
> are participating in rendering, then I think I could implement this
> using the pipeline-utility-steps plugin [0]. It has a step that can
> list all nodes with a given set of labels.
probably nothing surprising to anyone experienced with Jenkins. I was
pretty pleased to be able to cut down my rendering times, though!
https://blog.io7m.com/2018/12/19/low-budget-render-farm.xhtml
Reply all
Reply to author
Forward
0 new messages