YOUR ACCOUNT

Home > Forums > Scripting > 

Initialization problem

Active Topics Search
Login or Register to post new topics or replies
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
I'm working on a filter with a rather heavy initialization section. In theory the initialization script part should execute one time (I'm testing with only CPU core), however I counted more than 60 executions :-S

Here is a dedicated test script that shows the problem (also attached, note it is unsafe due to the os.time() call):
Code

function prepare()
   do_initialize = true
   stamp = os.time()
   print("prepare ("..stamp..") ---------------------");
end;

function get_sample(x, y)
   if do_initialize then
      do_initialize = false
      print("initialize ("..stamp.."); x = " .. x ..", y = "..y);
   end
   return 0,0,0,0
end;


Here is the output (Filter Forge x86.log):

Code

prepare (1313930290) ---------------------
initialize (1313930290); x = 0,00083333333333333, y = 0,00083333333333333
initialize (1313930290); x = 0,12583333333333, y = 0,00083333333333333
initialize (1313930290); x = 0,25083333333333, y = 0,00083333333333333
initialize (1313930290); x = 0,37583333333333, y = 0,00083333333333333
initialize (1313930290); x = 0,50083333333333, y = 0,00083333333333333
initialize (1313930290); x = 0,62583333333333, y = 0,00083333333333333
initialize (1313930290); x = 0,75083333333333, y = 0,00083333333333333
initialize (1313930290); x = 0,87583333333333, y = 0,00083333333333333
initialize (1313930290); x = 0,00083333333333333, y = 0,12583333333333
initialize (1313930290); x = 0,12583333333333, y = 0,12583333333333
initialize (1313930290); x = 0,25083333333333, y = 0,12583333333333
initialize (1313930290); x = 0,37583333333333, y = 0,12583333333333
initialize (1313930290); x = 0,50083333333333, y = 0,12583333333333
initialize (1313930290); x = 0,62583333333333, y = 0,12583333333333
initialize (1313930290); x = 0,75083333333333, y = 0,12583333333333
initialize (1313930290); x = 0,87583333333333, y = 0,12583333333333
initialize (1313930290); x = 0,00083333333333333, y = 0,25083333333333
initialize (1313930290); x = 0,12583333333333, y = 0,25083333333333
initialize (1313930290); x = 0,25083333333333, y = 0,25083333333333
initialize (1313930290); x = 0,37583333333333, y = 0,25083333333333
initialize (1313930290); x = 0,50083333333333, y = 0,25083333333333
initialize (1313930290); x = 0,62583333333333, y = 0,25083333333333
initialize (1313930290); x = 0,75083333333333, y = 0,25083333333333
initialize (1313930290); x = 0,87583333333333, y = 0,25083333333333
initialize (1313930290); x = 0,00083333333333333, y = 0,37583333333333
initialize (1313930290); x = 0,12583333333333, y = 0,37583333333333
initialize (1313930290); x = 0,25083333333333, y = 0,37583333333333
initialize (1313930290); x = 0,37583333333333, y = 0,37583333333333
initialize (1313930290); x = 0,50083333333333, y = 0,37583333333333
initialize (1313930290); x = 0,62583333333333, y = 0,37583333333333
initialize (1313930290); x = 0,75083333333333, y = 0,37583333333333
initialize (1313930290); x = 0,87583333333333, y = 0,37583333333333
initialize (1313930290); x = 0,00083333333333333, y = 0,50083333333333
initialize (1313930290); x = 0,12583333333333, y = 0,50083333333333
initialize (1313930290); x = 0,25083333333333, y = 0,50083333333333
initialize (1313930290); x = 0,37583333333333, y = 0,50083333333333
initialize (1313930290); x = 0,50083333333333, y = 0,50083333333333
initialize (1313930290); x = 0,62583333333333, y = 0,50083333333333
initialize (1313930290); x = 0,75083333333333, y = 0,50083333333333
initialize (1313930290); x = 0,87583333333333, y = 0,50083333333333
initialize (1313930290); x = 0,00083333333333333, y = 0,62583333333333
initialize (1313930290); x = 0,12583333333333, y = 0,62583333333333
initialize (1313930290); x = 0,25083333333333, y = 0,62583333333333
initialize (1313930290); x = 0,37583333333333, y = 0,62583333333333
initialize (1313930290); x = 0,50083333333333, y = 0,62583333333333
initialize (1313930290); x = 0,62583333333333, y = 0,62583333333333
initialize (1313930290); x = 0,75083333333333, y = 0,62583333333333
initialize (1313930290); x = 0,87583333333333, y = 0,62583333333333
initialize (1313930290); x = 0,00083333333333333, y = 0,75083333333333
initialize (1313930290); x = 0,12583333333333, y = 0,75083333333333
initialize (1313930290); x = 0,25083333333333, y = 0,75083333333333
initialize (1313930290); x = 0,37583333333333, y = 0,75083333333333
initialize (1313930290); x = 0,50083333333333, y = 0,75083333333333
initialize (1313930290); x = 0,62583333333333, y = 0,75083333333333
initialize (1313930290); x = 0,75083333333333, y = 0,75083333333333
initialize (1313930290); x = 0,87583333333333, y = 0,75083333333333
initialize (1313930290); x = 0,00083333333333333, y = 0,87583333333333
initialize (1313930290); x = 0,12583333333333, y = 0,87583333333333
initialize (1313930290); x = 0,25083333333333, y = 0,87583333333333
initialize (1313930290); x = 0,37583333333333, y = 0,87583333333333
initialize (1313930290); x = 0,50083333333333, y = 0,87583333333333
initialize (1313930290); x = 0,62583333333333, y = 0,87583333333333
initialize (1313930290); x = 0,75083333333333, y = 0,87583333333333
initialize (1313930290); x = 0,87583333333333, y = 0,87583333333333


Variable "do_initialize" should remain false, but for some reason it doesn't. "Prepare" is only executed once, so what the h*** is going on here? Is this related to the rendering "tiles"?

Ini Test.ffxml
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: August 21, 2011 7:41 am   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
This is very annoying :-S I can't prepare the script in function Prepare because get_sample calls are not allowed (I need to analyze multiple image inputs to get average color of the inputs).

So I'm left with the option to analyze the images in the main get_sample function (with a similar initialization setup as described above). But the extreme overhead caused by the multiple executions (as described above) makes it practically unusable.. smile:evil:

We really need a middle way here: some sort of initialization function that allows get_sample calls and can produce data that is shared in all threads/tiles..

FF programmers.. any comments?
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: August 21, 2011 8:11 am   Details E-Mail
Igor Lorents
Filter Forge, Inc.
Posts: 39
Sphinx, thank you for reporting the issue.

The problem seems to be in our architecture. Changes you've made in the Global table which contains definition for all global variables after the first initialization (which is made by calling the prepare() routine) do not propagate properly.
That's why your initialization code in get_sample() function executes each time when the new rendering block is started.
The problem should still persist even if you use single-threaded rendering, and your test sample illustrates it very well.

So, this issue seems to be a pretty large piece of work, and unfortunately I won't have time for it until the release of version 3.0.

Sorry for the inconvenience.
Posted: August 22, 2011 5:27 am   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
I see - after Prepare has been called you assume the global table is not changed and simply copy the table...

Could you possibly add some sort of "super global" table that works across tiles, or allow get_sample calls in prepare?
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: August 22, 2011 7:16 am   Details E-Mail
Igor Lorents
Filter Forge, Inc.
Posts: 39
Quote
Could you possibly add some sort of "super global" table that works across tiles, or allow get_sample calls in prepare?


Unfortunately no, because the behavior of the suggested "super global" table is absolutely equal to the intended behavior of the global table. We'll just have to implement it properly smile:)
Posted: August 22, 2011 7:42 am   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
Sigh... well, that "bug" along with the darn unpersist error blocks several of my projects. Patience..

Thanks for the explanation anyways.
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: August 22, 2011 9:06 am   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
Alright, FF3 released - time to bring this up again smile:-)

The problem still persists:

Global variables are not truly global, but only "global" inside a given rendering block. This will ruin smart logic such as initialization regions.

A typical scenario: collect a set of samples and use their properties to determine global filtering (e.g. auto levels). This initial sample collection is only necessary to do once.
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: November 17, 2011 5:43 am   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
So, any news on this issue? Script filters with heavy prepare/initialization sections are unreasonably slow...
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: November 21, 2014 2:14 am   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
Here is ThreeDee's cool circle packer filter. It uses an initialization section to prepare some stuff based on sampler inputs. Its essentially the same as using the prepare method only you blocked the possibility to use sampler sources for preparation, so he has to use an init variable and do it in get_sample.. nothing weird about that design.

I stripped the actual rendering script so all this filter is doing in theory is that one initialization, however in reality this seems to be called for each of your internal temporary rendering tiles smile:eek:


Code

function get_sample(x, y)

   -- only on first run:
   if init == 0 then
          get_circles() --get_circles set init = 1
   end -- if init == 0
   local r,g,b = 1,1,1
   return r,g,b,1,aa_zone
end;



Circle packing 6B - SLOW INIT.ffxml
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: November 21, 2014 3:20 am   Details E-Mail
Ramlyn
Ramlyn

Posts: 2930
Filters: 691
Humm... so it repeats the same process also when it is not needed.
Posted: November 21, 2014 9:26 am   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
Yes, for each of those rendering blocks you see being completed during rendering.. and if you have multiple pass rendering enabled (that is default) it is number of blocks x number of passes..
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: November 21, 2014 9:28 am   Details E-Mail
ThreeDee
Lost in Space

Posts: 1672
Filters: 112
Hey, I had a thought:

How about using an unsafe script to store the precalculation to a (comma-delimited?) text file when a checkbox is activated. You can set it up so that only the first thread does this calculation and saving of the file by making sure the coordinates are less than 1/16th of the shorter dimension of the image (and init ==0). Other threads do nothing while the checkbox in on.

Then when the checkbox is clicked off the filter reads the precalculation file instead of doing the calculation. Hopefully this can be done during the Prepare() function so it doesn't have to be done for each thread.

This is similar solution to what Ken (with some help from the undersigned) did with the embedded external image:
https://www.filterforge.com/forum/read...sage120745

Won't be able to submit to library as it is an unsafe script. Otherwise I don't see why it wouldn't work.
Posted: November 23, 2014 3:34 pm   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
Yes, that will probably do as a temporary workaround for private filters (since we can't submit unsecure filters).

However the main point of this 3 year, 3 months and 3 days old thread is still very valid: script filters with heavy global initialization are quite inefficiently executed.

I wish they had a "fastmode" render mode that skipped all the render block stuff all together. One render block to rule them all (and that one blocks resolution = the output image resolution). No magic, no preview, no multipass.. just raw to the bone processing.
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: November 24, 2014 2:26 am   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
Yum Yum.. http://lua-users.org/wiki/SaveTableToFile
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: November 25, 2014 5:52 am   Details E-Mail
ThreeDee
Lost in Space

Posts: 1672
Filters: 112
Quote
Sphinx wrote: I wish they had a "fastmode" render mode that skipped all the render block stuff all together.


Couldn't agree with you more. Lack of such an option makes script optimization a bitch.
Posted: November 26, 2014 10:06 am   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
Vlad, GMM anyone... are you going to address this multithreading performance issue in V5?
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: July 12, 2015 1:44 pm   Details E-Mail
GMM
Moderator
Filter Forge, Inc
Posts: 3491
I hope this is going to be fixed at the time we implement LuaJIT which is supposed to be faster and better overall. I'm not sure LuaJIT will make its way into FF5 but it's in the works.
Posted: July 13, 2015 5:40 am   Details E-Mail
trung
Y8 Games
Posts: 1
longish iterative process, and not really necessary if you can use other commands
Posted: September 8, 2015 10:13 pm   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
Filter Forge x86 5.006.38069.21538.

Problem still present - an init call that theoretically should called *once* is called for each rendering tile etc. smile:cry:

Code
function prepare()

   do_initialize = true
   stamp = os.time()
   print("prepare ("..stamp..") ---------------------");
end;

function get_sample(x, y)
   if do_initialize then
      do_initialize = false
      print("initialize ("..stamp.."); x = " .. x ..", y = "..y);
   end

   return 0,0,0,0
end;


Above script produces the following log output. Ideally we should see one line only.

Quote
Filter Forge x86.log Wrote:

Filter Forge x86 5.006.38069.21538 Release, November 19, 2015 09:43 [21-11-2015 08:35:38 Rom, normaltid]
prepare (1448091338) ---------------------
initialize (1448091338); x = 0,00083333333333333, y = 0,00083333333333333
initialize (1448091338); x = 0,12583333333333, y = 0,00083333333333333
initialize (1448091338); x = 0,25083333333333, y = 0,00083333333333333
initialize (1448091338); x = 0,37583333333333, y = 0,00083333333333333
initialize (1448091338); x = 0,50083333333333, y = 0,00083333333333333
initialize (1448091338); x = 0,62583333333333, y = 0,00083333333333333
initialize (1448091338); x = 0,75083333333333, y = 0,00083333333333333
initialize (1448091338); x = 0,87583333333333, y = 0,00083333333333333
initialize (1448091338); x = 0,00083333333333333, y = 0,12583333333333
initialize (1448091338); x = 0,12583333333333, y = 0,12583333333333
initialize (1448091338); x = 0,25083333333333, y = 0,12583333333333
initialize (1448091338); x = 0,37583333333333, y = 0,12583333333333
initialize (1448091338); x = 0,50083333333333, y = 0,12583333333333
initialize (1448091338); x = 0,62583333333333, y = 0,12583333333333
initialize (1448091338); x = 0,75083333333333, y = 0,12583333333333
initialize (1448091338); x = 0,87583333333333, y = 0,12583333333333
initialize (1448091338); x = 0,00083333333333333, y = 0,25083333333333
initialize (1448091338); x = 0,12583333333333, y = 0,25083333333333
initialize (1448091338); x = 0,25083333333333, y = 0,25083333333333
initialize (1448091338); x = 0,37583333333333, y = 0,25083333333333
initialize (1448091338); x = 0,50083333333333, y = 0,25083333333333
initialize (1448091338); x = 0,62583333333333, y = 0,25083333333333
initialize (1448091338); x = 0,75083333333333, y = 0,25083333333333
initialize (1448091338); x = 0,87583333333333, y = 0,25083333333333
initialize (1448091338); x = 0,00083333333333333, y = 0,37583333333333
initialize (1448091338); x = 0,12583333333333, y = 0,37583333333333
initialize (1448091338); x = 0,25083333333333, y = 0,37583333333333
initialize (1448091338); x = 0,37583333333333, y = 0,37583333333333
initialize (1448091338); x = 0,50083333333333, y = 0,37583333333333
initialize (1448091338); x = 0,62583333333333, y = 0,37583333333333
initialize (1448091338); x = 0,75083333333333, y = 0,37583333333333
initialize (1448091338); x = 0,87583333333333, y = 0,37583333333333
initialize (1448091338); x = 0,00083333333333333, y = 0,50083333333333
initialize (1448091338); x = 0,12583333333333, y = 0,50083333333333
initialize (1448091338); x = 0,25083333333333, y = 0,50083333333333
initialize (1448091338); x = 0,37583333333333, y = 0,50083333333333
initialize (1448091338); x = 0,50083333333333, y = 0,50083333333333
initialize (1448091338); x = 0,62583333333333, y = 0,50083333333333
initialize (1448091338); x = 0,75083333333333, y = 0,50083333333333
initialize (1448091338); x = 0,87583333333333, y = 0,50083333333333
initialize (1448091338); x = 0,00083333333333333, y = 0,62583333333333
initialize (1448091338); x = 0,12583333333333, y = 0,62583333333333
initialize (1448091338); x = 0,25083333333333, y = 0,62583333333333
initialize (1448091338); x = 0,37583333333333, y = 0,62583333333333
initialize (1448091338); x = 0,50083333333333, y = 0,62583333333333
initialize (1448091338); x = 0,62583333333333, y = 0,62583333333333
initialize (1448091338); x = 0,75083333333333, y = 0,62583333333333
initialize (1448091338); x = 0,87583333333333, y = 0,62583333333333
initialize (1448091338); x = 0,00083333333333333, y = 0,75083333333333
initialize (1448091338); x = 0,12583333333333, y = 0,75083333333333
initialize (1448091338); x = 0,25083333333333, y = 0,75083333333333
initialize (1448091338); x = 0,37583333333333, y = 0,75083333333333
initialize (1448091338); x = 0,50083333333333, y = 0,75083333333333
initialize (1448091338); x = 0,62583333333333, y = 0,75083333333333
initialize (1448091338); x = 0,75083333333333, y = 0,75083333333333
initialize (1448091338); x = 0,87583333333333, y = 0,75083333333333
initialize (1448091338); x = 0,00083333333333333, y = 0,87583333333333
initialize (1448091338); x = 0,12583333333333, y = 0,87583333333333
initialize (1448091338); x = 0,25083333333333, y = 0,87583333333333
initialize (1448091338); x = 0,37583333333333, y = 0,87583333333333
initialize (1448091338); x = 0,50083333333333, y = 0,87583333333333
initialize (1448091338); x = 0,62583333333333, y = 0,87583333333333
initialize (1448091338); x = 0,75083333333333, y = 0,87583333333333
initialize (1448091338); x = 0,87583333333333, y = 0,87583333333333
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: November 21, 2015 2:43 am   Details E-Mail
Indigo Ray
Adam

Posts: 1442
Filters: 82
FF 7.001: Still a problem. Really hampers the scripting feature in FF.

I adapted Sphinx's code to output an image instead of text ( smile:!: unsafe script):

script_init.ffxml
Posted: May 24, 2017 10:36 pm   Details E-Mail
Indigo Ray
Adam

Posts: 1442
Filters: 82
If function prepare() was called once, this image would be completely black.
Instead, there are 64 boxes, meaning it was called 64 times! For a little 600x600 image!
Posted: May 24, 2017 10:39 pm   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
Any news on this issue?

This is a major performance issue in advanced scripts...
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: October 12, 2017 2:42 am   Details E-Mail
GMM
Moderator
Filter Forge, Inc
Posts: 3491
I'm afraid no news for FF7 in this matter.
Posted: October 12, 2017 7:02 am   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
That is bad news smile:-(

Here is a simple graphic summary of the issue - and I'm not sure the estimated 64 tiles is really correct as previous tests showed even more init calls

Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: October 12, 2017 7:47 am   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
GMM/Vlad - is this issue going to be addressed in V9 at some point?
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: August 2, 2019 6:46 am   Details E-Mail
GMM
Moderator
Filter Forge, Inc
Posts: 3491
I'm afraid not in V9 smile:(
Posted: August 2, 2019 6:52 am   Details E-Mail
Sphinx.
Filter Optimizer

Posts: 1750
Filters: 39
Sigh! I guess its more profitable adding new bells and whistles than polishing/fixing the existing ones (Psst... loop sample cache!). Oh well, true global initialization would also be needed for green to gray conversion I guess. Fingers crossed. Thanks for the answer smile:-)
Njyldgarkn init/prepare process x no. of render blocks x no. of passes
Posted: August 2, 2019 7:56 am   Details E-Mail
Rico
Rico

Posts: 21
Filters: 2
Wow, this thread is nearly 11 years old!

I am trying to work on circle-packing, like many others before me. I have made some good headway building on ThreeDee's excellent work, but the primary problem is the inability to cache properly, which is tied to this initialisation issue. I want to build my tables once and once only.

Dear moderators can you tell us if this will ever be fixed please? Thank you.
Posted: March 28, 2022 3:54 am   Details E-Mail
GMM
Moderator
Filter Forge, Inc
Posts: 3491
Quote
Rico wrote:
Dear moderators can you tell us if this will ever be fixed please?


Ever? Probably yes.
Soon? I'm afraid not.
Posted: March 28, 2022 8:52 am   Details E-Mail

Join Our Community!

Filter Forge has a thriving, vibrant, knowledgeable user community. Feel free to join us and have fun!

33,711 Registered Users
+18 new in 30 days!

153,531 Posts
+39 new in 30 days!

15,347 Topics
+72 new in year!

Create an Account

Online Users Last minute:

23 unregistered users.

Recent Forum Posts:

Join our Forum Join our Forum
Over 30 000 members!
Join us on Facebook Join us on Facebook
Over 42 000 likes!
Follow us on Twitter Follow us on Twitter
Get updates quickly!
Follow us on Instagram Follow us on Instagram
Tag us #filterforge