Archive for the 'brender' Category

Brender: a new render farm solution for blender needs beta tester.

Hello everyone.

Good news we have decided to release brender as open-source for everyone to enjoy!

I have already mentioned a few times in this blog the use of an in-house software called brender. It was first written in 2005 when I needed a simple render queuing system built around blender. The system is an apache/php/mysql server running a script  and dispatching the different render orders to client machines. There is a web interface for managing the projects, job, clients and orders. It has a slightly more complicated setup time (due to some paths possible confusion) because it allows scene files (and all the dependencies) to be on any machine in the local network, not necessarly on the server. They can be hosted on the editing machine for instance.

So after 5 years of lonely use in Swizerland and 2-3 months of rewrite with Francesco Siddi, we can proudly release the source code to the wild.

The project is hosted on a github repository here. For easier access you can find a download at this page.
Installation procedure and help is on the brender wiki.

We welcome any feedback from beta testers.

for more help you can either :
– email me (Olivier oenvoyage AT gmail) or Francesco (fsiddi AT gmail)
– post a comment/question on this blog
– check the blender artists thread
– join us on IRC freenode #brender

and to get started you can find some help/installation/introduction videos here :

simple project rendering :

installation help :

Please keep in mind it evolved from an in-house (and very hardcoded) solution to something more open, We tried as much as possible to ease the installation process.
Some complicated aspects include the paths needed for projects setup. This is mostly due to the concept of brender to not move the .blend files from server to client. All rendernodes renders through network in a “local” way. It gives less hassle with linked files, textures, simcache etc.

the codesource can be downloaded on github : https://www.github.com/oenvoyage/brender

Advertisements

Brender Render farm queue management system developement

I have written a few times on this blog about Brender. It is a little in-house system that I developed and used for the last 3 or 4 years to help me manage some rendering and scene management on different machines in our studio. To release the code to the public was always a wish to me, but not possible without some cleaning. It was first developed specifically for our network and machines, including a lot of hardcoded variables, machine names, IP’s and blender path.

Luckily recently I got the help of Francesco Siddi,  and we are working hard to have something downloadable (maybe before christmas?) .

A little work in progress screenshot

But be warned, this might not be the easiest render farm solution to set up and install 🙂 It is a system based on an apache web interface and mysql database. If you are interested to test it contact me (oenvoyage AT gmail) or Francesco (francesco.siddi AT gmail)

blender gameengine mysql interface for render farm visualization

Hello and happy new year 2010!
With a new year comes new ideas, and one of those (actually since a long time in my mind) was to develop a fun visualization of the render farm datas like rendernode status etc.
Unfortunately i never had time to dig into python and the gameengine. But christmas holiday brings its amount of free days to study new things.
first of all here is a quick video showing it at work:

you can see a bit how it works. In fact it is quiet simple the script is linked to an “always” actuator with a pulse mode of 20 so it updates about every second.
using the MySQLdb python module it creates a connection to the renderfarm database where it gets a list of clients and there actual status (which can be idle, disabled or rendering) then it tries to see if an object with the name of the renderclient exists in the gamengine scene. If an object exists then the script assigns the status of the client to the corresponding object property called “status” (line ob[‘status’]=row[2]) .

the second part of the script (brenderdb.execute (“select round(sum(end-current)/sum(end-start)*100) as frames from jobs where status=’rendering’ or status=’waiting’ or status=’pause’;”) tries to find out how many renders there are still left to render and assigns that value to an object’s property.

Then the rest is just Game Engine logic bricks, although more powerfull (and only limited by imagination) could be done in scripting. Each fireball has 3 actions that are played when the value of “status” changes to the corresponding action (idle, rendering and disabled)

Now I will try to see what if i could adapt this to 2.5 and the NetRender…maybe during the next holidays 🙂

Feel free to ask for questions here or by email (oenvoyageATgmaaaiil.com)

Brender, home made render farm system

Today I decided to write a bit about brender, a little phph/mysql render farm system I developed a few years ago. At that time I had tried dr queue and royal render for xsi. Was kind of interested in different renderfarming solutions, but also had some time to spend time develop my own as a personal interest and challenge.

So first of all I tried to make different computers communicate through mysql/php. I must precise that I have absolutely not programmer background and my only experience with databasing was webpage page. So that might explain why I did not use sockets or advanced technices but rather a simple database orders system.

Basically the system uses a mysql server database and clients connecting to it. A server (running script brender_server.php) will get the blender jobs (for instance Render file sample.blend frames 1 to 150) and dispatch it as frames sequences to the different availables clients (running brender_clients.php). A render job is then written to database (ex. Client1 render frames 1 to 10 and client2 render 11 to 20). The clients will then see the render job and start to calculate his part. When finished he sends an order to the mysql server saying he finished the render job and is available(or idle) for something else.

 

brender renderfarming system

brender renderfarming system

The main interaction is done through a web server interface. Files are located on a shared network drive.

We have used the system successfully in our last few productions at RGBprod studio. It has been stable although I had to update it when more complicated library linked or composting files are done.

here are some photos of it when i was developping it in my basement, it looks scary but that was the whole point of the installation. It even had mechanical sounds whenever a client finished a render chunk, or server gives new orders.
http://www.flickr.com/photos/oenvoyage/tags/brender/

I will take the system with me to Russia for the blender summer camp in Nizhni Novgorod and maybe will get new idea there 🙂

I can make those files available if anyone if interested. Although I do not know what kind of open source license I should put. Maybe it needs some installation instruction as well 🙂

Rendering on an 8-core with 2 blenders is faster

Today i am going to share some “trick” that i discovered on recent productions. First let me explain that I set up a small render farm environment (based on personal php/mysql system) in our studio.

brender renderfarming system

brender renderfarming system

the basic
Our fastest machine is an 8-core mac. By checking the CPU usage during renders it clearly came out that there were times when the system was not using all cpus at max (either when transfering files, talking with server, getting the blend files, building shadow maps or other maybe non multi-threaded tasks). So the solution was to actually launch 2 instances of the render client script on the same machine and thus having 2 blenders rendering simulteanously . And now i can hear you …you are going to tell me “but hey it drops down your per-frame render time!!”:-) And yes sir you are perfectly correct! it slows down the per-frame rendering…but not enough down.
The numbers talk for themself :
One instance of blender rendering i get an average render time per frame : 38sec
With 2 instances the average per frame is not 1min16 as one would expect but 43sec.

 

A rapid useless but fun calculation tell us that for a 100 frames anim, i get 64mins of rendering (38sec*100frames/60) and with a “dual-blender” i get 36mins (80sec*50frames/60). Voila!
This scene (scene03 from current project) indeed includes quiet a good amount of compositing, which i do believe is not multithreaded yet.

Other tests : scene 04_office
2 blend 37 sec
1 blend 23 sec

And with the blender benchmark file from eofw.org _
2 blend average 31sec.
1 blend 17 secs. ( yes its slower from other mac pro results , but i renderer slightly bigger and OSA 8)

The .exr temp files problem
So we already used this “trick” on the H7 robot movie. But at that time we did not use full sample FSA. This is were we got the temp files problem. I do not really understand how blender works for saving the sample passes in an .exr temporary files, but it seemed that by launching the render client in a same user session, it got access both time on same destination. This made blender not able to either overwrite or simply write, and thus crashing or exiting without rendering.
I tried different solutions like having a separate blender installation for the second render client instance, as well as running the client script in different user in the terminal… strangely it did not affect anything.
The workaround solution we found (thanks to a patagonian yogi guru) was to start an other session on the same computer to run the render client.
I am suspecting my lame render farm settings to be the only mistake in that 🙂 but at least in the end it worked.

So a tip to all people with a 8-core machine, or even 4-core i am sure u might get results, when rendering a long heavy animation: launch 2 instances of blender and use Touch Overwrite 🙂
for more infos on TouchOverwrite : http://www.blender.org/development/release-logs/blender-246/distributed-rendering-new-render-options/

please let me know if you do get some rendering speed improvement.


Categories

Flickr Photos

@oenvoyage latest Twitter


%d bloggers like this: