Filter Forge - Table memory limit?

Sphinx.

Filter Optimizer

Posts: 1344
Filters: 38

I'm getting some strange results with large tables. Do you have some limit to how large tables can be or memory usage in general for a script?

Njyldgarkn sample cache!

Posted: June 8, 2010 7:52 am

Details E-Mail

Vladimir Golovin
Administrator

Filter Forge, Inc.

Posts: 3106
Filters: 54

Uh, no idea. We'll look into this.

Edit: Could you post the sample script here?

Posted: June 8, 2010 7:54 am

Details E-Mail

Sphinx.

Filter Optimizer

Posts: 1344
Filters: 38

Yep - it may very well be errors in the script though smile;)

script:

Quote

function setPixRGBA(buf,cx,cy,r,g,b,a)
cx = math.floor(cx)
cy = math.floor(cy)
--zero indexed!
if (cx >= 0) and (cx < OUTPUT_WIDTH) and
(cy >= 0) and (cy < OUTPUT_HEIGHT)then
i = (cx + cy * OUTPUT_WIDTH) * 4 + 1
buf[i] = r
buf[i + 1] = g
buf[i + 2] = b
buf[i + 3] = a
end
end;

function getPixRGBA(buf,cx,cy)
cx = math.floor(cx)
cy = math.floor(cy)
--zero indexed!
if (cx >= 0) and (cx < OUTPUT_WIDTH) and
(cy >= 0) and (cy < OUTPUT_HEIGHT)then
local i = (cx + cy * OUTPUT_WIDTH) * 4 + 1
local r = buf[i]
local g = buf[i + 1]
local b = buf[i + 2]
local a = buf[i + 3]
return r,g,b,a
end
return {0,0,0,0}
end;

function createBitmap(buf, src)
local sx = 1 / OUTPUT_WIDTH
local sy = 1 / OUTPUT_HEIGHT

for j=0,OUTPUT_HEIGHT-1 do
for i=0,OUTPUT_WIDTH-1 do
local r,g,b,a = get_sample_map(i*sx, j*sy, src)
setPixRGBA(buf, i, j, r,g,b,a)
end
end
end

function prepare()
bitmap = {}
initialize = true
end;

function get_sample(x, y)
if (initialize == true) then
createBitmap(bitmap, SOURCE)
initialize = false
end

-- do not work:
--local r,g,b,a = getPixRGBA(bitmap, x * OUTPUT_WIDTH, y * OUTPUT_HEIGHT)

-- works, but with error:
local r,g,b,a = getPixRGBA(bitmap, 300, 300)

return r,g,b,a
end;

Bitmap Buffer.ffxml

Njyldgarkn sample cache!

Posted: June 8, 2010 7:59 am

Details E-Mail

Sphinx.

Filter Optimizer

Posts: 1344
Filters: 38

Hmm, I think its something else that fails, even a very small bitmap buffer causes the same problem.. still learning this lua stuff

Njyldgarkn sample cache!

Posted: June 8, 2010 8:17 am

Details E-Mail

Dmitry Sapelnikov

Filter Forge, Inc. AKA Egret

Posts: 76
Filters: 5

Heh, I've fixed this =)
Script code after bug fixing:

Quote

function setPixRGBA(buf, cx, cy, r, g, b, a)
local cx = math.floor(cx)
local cy = math.floor(cy)
--zero indexed!
if (cx >= 0) and (cx < OUTPUT_WIDTH) and
(cy >= 0) and (cy < OUTPUT_HEIGHT)then
local i = (cx + cy * OUTPUT_WIDTH) * 4 + 1
buf[i] = r
buf[i + 1] = g
buf[i + 2] = b
buf[i + 3] = a
end
end;

function getPixRGBA(buf,cx,cy)
local cx = math.floor(cx)
local cy = math.floor(cy)
--zero indexed!
if (cx >= 0) and (cx < OUTPUT_WIDTH) and
(cy >= 0) and (cy < OUTPUT_HEIGHT)then
local i = (cx + cy * OUTPUT_WIDTH) * 4 + 1
local r = buf[i]
local g = buf[i + 1]
local b = buf[i + 2]
local a = buf[i + 3]
return r, g, b, a
end
-- Fixed(Egret): return {0, 0, 0, 0} is incorrect here
return 0, 0, 0, 0
end;

function createBitmap(buf, src)
-- Fixed(Egret): it's a proper conversion from the filter coords to
-- the image coords
local scale = 1 / SIZE

for j=0, OUTPUT_HEIGHT-1 do
for i=0, OUTPUT_WIDTH-1 do
local r,g,b,a = get_sample_map(i*scale, j*scale, src)
setPixRGBA(buf, i, j, r, g, b, a)
end
end
end

function prepare()
bitmap = {}
initialize = true
end;

function get_sample(x, y)
if (initialize == true) then
createBitmap(bitmap, SOURCE)
initialize = false
end

-- Fixed(Egret): it's a proper conversion from the filter coords to
-- the image coords
local r,g,b,a = getPixRGBA(bitmap, x * SIZE, y * SIZE)
return r,g,b,a
end;

Posted: June 9, 2010 1:32 am

Details E-Mail

Dmitry Sapelnikov

Filter Forge, Inc. AKA Egret

Posts: 76
Filters: 5

The problem which caused nils was tiny.

But an approach used in the script makes it good "don't do it!" example (Sphinx, I'm sorry smile:(

Of course it's not your fault, just lack of info about scripting)

For all FF script programmers:
Never create tables or another kind of caches without really serious purpose. Caches in scripts consumes memory really hard because caches are independently created for each rendering thread. (For exapmle, if you've got double-core processor and "use all CPU(s)" option is enabled in FF options, script component creates at least 3 copies of one cache: 2 for filter preview rendering threads, and 1 for component thumbnail rendering.
BTW, I've got independent infrastructure for components which need caching. And we haven't implemented it in script components yet.

Posted: June 9, 2010 2:11 am

Details E-Mail

Dmitry Sapelnikov

Filter Forge, Inc. AKA Egret

Posts: 76
Filters: 5

Another point of discussion is coordinate systems used in FF.
We are going to post an wiki article about it this week.

Posted: June 9, 2010 2:15 am

Details E-Mail

Vladimir Golovin
Administrator

Filter Forge, Inc.

Posts: 3106
Filters: 54

Egret, good point -- we should mention per-thread Lua state duplication in the help articles.

Posted: June 9, 2010 2:37 am

Details E-Mail

Sphinx.

Filter Optimizer

Posts: 1344
Filters: 38

Ah, of course! Remains of my first implementation where I tried to store channels in a small table for each rgba pixel/sample, but that didn't work out (maan I wish I could drag off the script tab and maximize it smile;)

hint hint)

About the memory consumption - yes I did notice things got slow with the "large" table. I am really just testing/learning things by trying out familiar concepts from earlier projects.

Are you saying that the complete script is copied for each thread or what is going on? It would be quite useful if you somehow could declare a shared variable cross threads and script components, perhaps in some sort of read only state to avoid changing data problems. Or the other way around: declare that a script is not suitable for multithreaded processing.

I think that you will see several attempts to implement bitmap filters, perhaps something can be done in this regard (some sort of bitmap filter script component which is wrapped by your general bitmap filter framework, with cache tiles etc).

Njyldgarkn sample cache!

Posted: June 9, 2010 3:31 am

Details E-Mail

Igor Lorents

Filter Forge, Inc.

Posts: 39

Also, I'd like to mention an important thing about component sampling.

Filter Forge core can take samples not only in [0..Size-1] bounds. We have to take some samples outside this area to make the proper supersampling of border pixels.
That's why, your execution path went through the line I marked with bold below.

Quote
function getPixRGBA(buf,cx,cy) local cx = math.floor(cx) local cy = math.floor(cy) --zero indexed! if (cx >= 0) and (cx < OUTPUT_WIDTH) and (cy >= 0) and (cy < OUTPUT_HEIGHT)then -- skipped.... return r, g, b, a end return 0, 0, 0, 0 end;

You can easily check that by adding the following line before the second return statement:

Quote
error (string.format("%d, %d", cx, cy))

After that, you'll see something like this in the log window:

Quote
Script error (line 28): -1, -1.

Generally, I agree with Vladimir and Egret. Thanks for your post, Sphinx. Your code sample uncovers a lot of things which must be thoroughly explained in our help articles.

Posted: June 9, 2010 3:45 am

Details E-Mail

Vladimir Golovin
Administrator

Filter Forge, Inc.

Posts: 3106
Filters: 54