out from selection

Yes, I worked a bit on that some time ago. Its a tough problem and I didn't manage to make it perfect. Check out the discussion here.

Yeah, I am thinking of a photoshop selection....but I am confused as to how to get the filter to recognize the selection and project the pattern out accordingly with regard to the edges of the selection.

Rawn

Think about how the coordinate x and y dimensions look respectively for that mapping, i.e. what would you need to plug into a Lookup component to get the pattern.

One dimension would look like gradient going from black to white outwards from the shape (or inverted). This can be faked with a blur (ideal is a distance transform).
If the selection is a disc/circle, this would be a radial gradient.

The other dimension would look like a repeating gradient around the shape path. This is the "hard" problem I tried to solve (see the link).
If the selection is a disc/circle this would look like a repeating angular gradient with center in the center of the disc.

Follow?

Quote

One dimension would look like gradient going from black to white outwards from the shape (or inverted). This can be faked with a blur (ideal is a distance transform). If the selection is a disc/circle, this would be a radial gradient. The other dimension would look like a repeating gradient around the shape path. This is the "hard" problem I tried to solve (see the link). If the selection is a disc/circle this would look like a repeating angular gradient with center in the center of the disc.

Gah, again, another effect that a Distance Transform is perfect for. It's really frustrating, it's impossible to implement without the bitmap-based Script component.
After passing the area through the distance transform, one direction is the normal of the resulting heightmap, and the other is simply that normal rotated 90 degrees (or the "orthonormal"), at least, I think, i'd have to actually see if that worked or not .

OK: here you go. This is based off of the filter thread that Sphinx pointed to above. Lacking a proper distance transform, I just made one in Pshop, and loaded it via a selection channel.

Here is the source image, which gets wrapped around the shape:

Here is the result:

And here is the filter. (again, based off of Sphinx's original.)

You can control where the shape starts and stops by messing w/ the cuves. You can change the 'circularity' of it by changing the control after the derivatives. You can rotate the whole thing by changing the curve driving the Repeat A shape. If you shift the phase on this, you roate the pattern around the shape.

Basic Angular Expansion PATTERN.ffxml