In the "Lua Programming Manual" it states
I had some troubles with one of my program loops where I wanted to subdivide the picture into discrete cells. Occasionally I would have overlapping cells. As the map input range is scaled in the range [0..1] for the Y axis I needed to change this to an integer value. It took me some time to find that the trouble in my algorithm lay with the maths.floor() function giving incorrect results. Contrary to what the manual says, it may be necessary at times to make allowances for small rounding errors. For example the following code shows how maths.floor() does not give the expected answer.
The output from this is:
To ensure you get the correct results you should add a small offset
Such a small offset should not affect your application but will ensure that the results are as expected. Using this offset the results are:
which is correct.
After I modified my math.floor() functions in this way my algorithm worked correctly.
I hope this helps someone who has troubles with such logic.
|Posted: February 24, 2011 6:15 am|
Ah - Good to know!
Floating point numbers can be rather strange in representation - my guess is that 0.8 * 10 actually is represented like 0.7999999.. which is why the floor is 7.
Njyldgarkn sample cache!
|Posted: February 24, 2011 6:41 am|
|Great info, thanks for that. I'm using lots of floors in one of my filters and I found it looked better when I added a little, but I couldn't work out why. I thought maybe it was a pixel sampling issue (like, sampling from the center rather than from the corner but this might have been part of the problem as well.|
|Posted: February 27, 2011 10:58 pm|
Filter Forge has a thriving, vibrant, knowledgeable user community. Feel free to join us and have fun!
16,218 Registered Users
+10 new last day!
+36 new last day!
+19 new in 7 days!