www.BinaryAlchemy.de :: View topic - Map Reduce "Race for the frame" functionality.
 SearchSearch   RegisterRegister  ProfileProfile   UsergroupsUsergroups   Log inLog in 
If you create a new post, please use a topic that describes your problem
Documento sin título
 
Map Reduce "Race for the frame" functionality.

 
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.    www.BinaryAlchemy.de Forum Index -> old - RR Requests - v6.x
View previous topic :: View next topic  
Author Message

Cryptite



Joined: 09 Oct 2012
Posts: 10
Location/Company/Country: Green Grass Studios, Dallas, TX

PostPosted: Tue Oct 09, 2012 8:12 pm    Post subject: Map Reduce "Race for the frame" functionality. Reply with quote

My apologies if, and almost certainly, this question has been brought up before, but has there been any consideration towards optimizing the way the final frames of jobs are handled when the farm is "free"?

I speak of the following scenario:

-40 machines on the farm
-500 frame job has been sent.
-The job is at the almost-complete stage, whereby one or two (slow) machines are attempting to finish a significant number of frames (say 20-30 each).

I'm certain this has been brought up before. Many of us have had to wrangle with the min/max machines dialog in order to juke RoyalRender into having the machines handle one frame a piece in order to wrap up rendering the last bits of a job. I'm sure we've all been there.

There is an aspect of the Map Reduce algorithm in which, when the last bits of a divided job are left, (usually being worked on by the slowest machines), the remaining bits of the job are distributed to faster machines. At this point the situation becomes a race to finish. If the slower machines finish, then the faster machines can have their processing canceled. But in situations where there is a significant difference between the fastest and the slowest machines, often the faster machines will receive and finish the job before the slower ones would.

I write just to ask if this functionality has been thought about for RR. Having some faster machines race to finish the last few frames would save quite a lot of waiting time. I would assume that basically whichever machine tells rrServer it finished first, rrServer could abort the job on any other remaining machines trying to render that frame.

The other option I suppose would be if the min/max of a job could dynamically change towards the end of a job in which there are free farm machines sitting around waiting. This would prevent one slow machine from trying by itself to render the last few dozen frames of a job, thereby speeding up the time to complete the job.

Anyway, just another thought. Sorry if this is preaching to the choir, but it seems like a pretty paramount feature that many "dumb render farm managers" have, only because they don't try to divide up frames into chunks, they just sort of power through frames no matter what. (Backburner is an example in mind).

Thanks!
Back to top
View user's profile Send private message Visit poster's website

schoenberger
Site Admin


Joined: 02 Mar 2005
Posts: 3786

PostPosted: Tue Oct 09, 2012 8:27 pm    Post subject: Reply with quote

Hi

An Abort of frames is currently not considered as aborts mean losing frames.

But the frames send to a client are reduced the less frames are missing.
The min/max settings are not touched, but the server sends less frames (of course not less than min).


What have you set as min/max for your jobs?
_________________
Holger Schönberger
Binary Alchemy - digital materialization
Back to top
View user's profile Send private message Send e-mail

Cryptite



Joined: 09 Oct 2012
Posts: 10
Location/Company/Country: Green Grass Studios, Dallas, TX

PostPosted: Wed Oct 10, 2012 12:08 am    Post subject: Reply with quote

I tend not to really change them unless I need to, so the jobs are usually always sent at 3 min 25 max.

I think a specific situation would be a job in which a slow machine would literally take exponentially greater amounts of time to render than a very fast machine. If it were a job that would only take 5 minutes if only fast machines were rendering it, but 30 minutes if a couple of slow machines bogged it down, then the case could be made for trying to send duplicate frames to faster machines as a race.

It's good to know that less frames are sent when a job is almost done.

I see that when a client starts a frame, a 0kb file is written to the drive while the rendering goes; is that the only thing preventing you from trying to send competing frames to multiple machines?
Back to top
View user's profile Send private message Visit poster's website

schoenberger
Site Admin


Joined: 02 Mar 2005
Posts: 3786

PostPosted: Wed Oct 10, 2012 12:49 am    Post subject: Reply with quote

It completely depends on the renderer.
If you tell two machines to render into the same file, then one will crash because the file is in use.
Probably the second one.

And RR cannot control the output for all scenes, especially if you render multiple outputs.
_________________
Holger Schönberger
Binary Alchemy - digital materialization
Back to top
View user's profile Send private message Send e-mail

schoenberger
Site Admin


Joined: 02 Mar 2005
Posts: 3786

PostPosted: Wed Oct 10, 2012 12:51 am    Post subject: Reply with quote

PS: About the splitting of frames.
You can always use the tile frame feature for a whole sequence.
e.g. 3 tiles.
So the small machines finish even faster.
_________________
Holger Schönberger
Binary Alchemy - digital materialization
Back to top
View user's profile Send private message Send e-mail

Cryptite



Joined: 09 Oct 2012
Posts: 10
Location/Company/Country: Green Grass Studios, Dallas, TX

PostPosted: Wed Oct 10, 2012 1:19 am    Post subject: Reply with quote

What if you could intercept the outputs? You already know where the output for frames will end up because that's simply part of job submission. In my mind you could have the clients render their frames to their local drives, then copy over their frames to the network upon completion of their assigned frames. This way, you could send the same frames to two machines, and whichever finishes first, the other could be aborted without worrying about them trying to fight for that filename on the network.

I haven't tried the tile frames yet. Many of the times what's causing scene slowdown on our end for the slower machines are simply memory based, so they'll get stuck even trying to load a scene over the course of hours, whereas the fast machines load the scene and kick out renders quickly.

And for the record, for the time being I'm talking about this largely in the case of 3ds max, and Maya soon. These problems don't so much plague the compositing renders, just the 3d renders.

Also, shouldn't you be asleep?
Back to top
View user's profile Send private message Visit poster's website

schoenberger
Site Admin


Joined: 02 Mar 2005
Posts: 3786

PostPosted: Wed Oct 10, 2012 11:31 am    Post subject: Reply with quote

Hi

With 3dsmax it is not possible to intercept any output if you use render elements.
RR can only set the main output file at render stage.

Maya should be possible as all outputs are driven by one single output line with variables.

>slower machines are simply memory based
If the renderer would take all the memory of the machine, then RR should recognize it and abort the render.
But of course the renderer could have a setting which disallows it to use all memory of a machine.
And use only almost all memory.


>Also, shouldn't you be asleep?
There was an important update which kept me awake until 5am two days ago.
So everything is shifted. And my default working hours are shifted by 3 hours anyway all the time.
_________________
Holger Schönberger
Binary Alchemy - digital materialization
Back to top
View user's profile Send private message Send e-mail

Cryptite



Joined: 09 Oct 2012
Posts: 10
Location/Company/Country: Green Grass Studios, Dallas, TX

PostPosted: Wed Oct 10, 2012 3:05 pm    Post subject: Reply with quote

I think most people that render with 3ds max these days opt to package the render elements in with a single EXR file, rather than truly splitting them apart into separate files. Could it possibly be an option, then, for the job not to try to redistribute when separated render elements are in fact being used? This may be specific to the actual renderer (vray, MR, etc).

Also, at least in vray's case, I'm able to intercept the render elements, but of course that requires opening the max file in order to run the script and then closing it before running the render process. I suppose it would be further plausible to either redirect paths with a pre-render script, or, in the case that one already exists, editing a local copy of the pre-render script and appending the necessary code to the end of it (pending apocalyptic events if that should break something) before letting it run.

I'd be willing to give a hand if this is something that you're interested in trying.
Back to top
View user's profile Send private message Visit poster's website

schoenberger
Site Admin


Joined: 02 Mar 2005
Posts: 3786

PostPosted: Wed Oct 10, 2012 3:19 pm    Post subject: Reply with quote

I will log the request, but it will not be implemented in the next time.
As this would also mean a large change in the structure and there are more features on the list.

I have also logged something that is easier to implement:
Do not use slow clients for the last frames.
_________________
Holger Schönberger
Binary Alchemy - digital materialization
Back to top
View user's profile Send private message Send e-mail

Cryptite



Joined: 09 Oct 2012
Posts: 10
Location/Company/Country: Green Grass Studios, Dallas, TX

PostPosted: Wed Oct 10, 2012 4:06 pm    Post subject: Reply with quote

Right, yeah I expected it would be a little ways off. Thanks Holger!
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.    www.BinaryAlchemy.de Forum Index -> old - RR Requests - v6.x All times are GMT + 1 Hour
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
 
Documento sin título
 



Powered by phpBB © 2001, 2002 phpBB Group



Number of shameful bots caught by Anti-Spam ACP: 1667