This effect was generated in Filter Forge, a Photoshop plugin that allows you to create your own filters.
A snippet for a new BlendMode that only affects the Luminance, Brightness or Y component of two images. You can either add, subtract or muliply them together,
blending blend mode add subtract multiply luminance brightness Y snippet