Fringe at alpha edge

Forgot to attach the filter.

AlphaTest.ffxml

I guess this is what happens:

You set up FF with a new image that is 100% transparent white.
When the anti-alias process kicks in, the white color from the outer transparent region contributes to the anti-alias sum of the semi transparent edge region.

If this is really what is going on, it could be a sign on flawed anti-alias calculation in FF (a sub sample should be scaled by its alpha when added to the sample sum, and the sample sum should be divided by the sum of alpha values from the sub samples)

To fix it for this case, ensure that the background is 100% transparent black when you set up New Image background color..

Great I first thought it was the blending of the filter output onto the input image that was wrong, but it works as it should.

I'd say this is a bug, but hopefully a programmer will comment on that..

- Btw.: The Blur component can do similar things.. I think that could be improved with the same solution too..

No, the span is never more than one pixel width: think of the pixel area being divided into several subpixels which then is averaged.. If the span was larger than one pixel, it would give you a blurred result.

With your ellipse example, the actual edge of the ellipse goes through the pixel area, so inside one "pixel" you can have both outer transparent region and ellipse. Thus when the aa sum is calculated it uses both ellipse and out region samples.

EDIT: In the attached image, imagine that the blue grid represent pixel cells in the final rendering - each cell will end up being one solid color (the pixel value). You can easily see whats going on in the cells both covering ellipse edge and background - they will be reduced to just one pixel value and that value will be mixed ellipse and background.

You can only solve this when you have a simple solid color outer region (like black for the ellipse), so you can set up the new image with this color (100% transparent).

So actually there is a problem when you have variation in the edge colors..
I can't see how it is possible to solve that without fabricating a special transparent version of the rendering (which FF is not capable of) with another program.

You can try this:
1. Render a temporary version of your filter that is completely opaque.
2. Open image in PS and give it a 100% transparent mask (you need to convert BG to layer)
3. Save in format that supports alpha (for example PNG 24)
4. Open image in FF and render your original masked version of the filter onto that

This workaround should do the trick (but I haven't tested it) - and don't worry about the lower bitdepth of the temporary PNG, its just to get the edges right, it will not degrade things visually

I can confirm that both workarounds work perfectly. I like the high coolness factor of the first method but the second method is so much easier.
I can't thank you enough for working this out and the clear explanation of what is happening.
I will attach the filters I was using for testing in case anyone else needs to do this.

Method 1 test:

AlphaTest2.ffxml

Thanks for the info, I posted about this a few years back but from memory there wasn't an easy way to resolve it code-wise. This is way better