Thanks for recommending that I upload an example, because in the course of building the example filter I planned to post, I actually realized what was going on, and what I'll need to do. Also, I didn't know about the Ctrl + L shortcut, that's pretty handy.
Short answer:
The FF Median Component never returns an average, as near as I can tell. The averaged-pixel-looking artifacts I was seeing resulted from the Median Component handling the RGB channels independently. (I can't believe that didn't occur to me sooner.) The solution, to rounding the contours of the colored regions via median, is to use unique shades of gray (rather than colors) to represent the region types.
Thanks again Morgantao and SpaceRay for the feedback. It looks like I've got things working correctly now.
The Details:
So, what was confusing me was that the Median Component looked like it would occasionally output an averaged pixel along boundaries, but not always. A median function only returns an average if it is given an even number of values.
e.g. median( 1 , 2 , 3 , 600 ) == 2.5
What struck me as odd was that if the FF Median Component samples are taken from a square region around a center pixel, then the number of samples should always be odd. (e.g. 3x3, 5x5, 7x7, and so on, and of course odd times odd is always odd.)
I couldn't figure out how it could sometimes return an average.
Well, it wasn't returning an average.
When I started building the example I was going to post here, I used only three colors. When I took my example Red, Green, and Blue color map, and ran it through the median, I started seeing these bid black areas show up between colors.
What I was seeing was the fact that the Median Component handles the Red, Green, and Blue channels separately. As far as the Median Component was concerned, each channel was just a gray scale image. Now it just so happens that the particular colors I was using in my actual full scale project were really good at hiding that fact, so I didn't realize it until I started building the RGB example I gave in the first post. (Yeah, I felt pretty stupid after realizing that of course the median handles the channels independently.)
Another way to look at it is that, for each region type, I had a specific color, essentially a 24-bit ID number (8-bit RGB) representing that region type (e.g. FF0000 = metal, 00FF00 = marble, etc). By running it through the Median Component, I was breaking the 24-bit number into 3 8-bit chunks, and applying a median to the low, middle, and high chunks separately.
This whole time I've been using the color map as what amounts to a texture index channel. While you can pass a grayscale index channel through a median with no problem and get the region smoothing effect I was looking for, it doesn't work so well if you try it with three separate RGB channels and expect the results to be cohesive.
The solution. I can extract a Hue channel from my color map, and use that as the aforementioned grayscale index channel. As long as the Saturation and Lightness channels are single solid shades of gray throughout, I'll get only the colors I started with when I recombine everything.
An alternative solution, is that I could actually combine the 8-bit R, G, and B channels of my original color map into a single 24-bit grey channel by stacking the bits in each channel end to end. Then each color would be represented as a unique shade of gray. I'm not sure if the Median Component can handle a single channel at 24-bit resolution, but it does have a "High Precision" option, so that might work, (as long as High Precision doesn't mean 16-bits.)
Anyway, thanks again Morgantao and SpaceRay for the feedback.