Hi Marko,

It sounds like you want to do something a little like the spatially 
adaptive filtering in Davatzikos' paper:
   http://dx.doi.org/10.1006/nimg.2000.0655
so it might be worth asking if he has code available.

> I tried NaN's (also with a dedicated 
> conv2nan function) but the effects are far too widespread.

I'm not sure I follow you here... did you write a new conv2 function 
that did something special when the kernel included NaNs? If so, what 
did you do, and in what way were the results too widespread?

It sounds to me like you'd probably need to do this in 3D, as the 
kernel will no longer be separable if you have "holes" in it where the 
NaNs are. Likewise, the non-separable kernel means you can't use 
clever tricks like Fast (Fourier) Convolution, so I think you'd need a 
brute-force approach, something like:
- loop over every voxel in the image
- if kernel (on current voxel) includes any NaNs, set kernel weight on 
these voxels to zero, AND renormalise the kernel to have unit volume 
(I think this renormalisation avoids the problem you mentioned of 
zeroed NaNs affecting neighbours)
- in output image, voxel gets sum of result of multiplying modified 
kernel with original image (NaNs set to zero here).
- for voxels which are NaN, probably want to set value in output image 
to the original value before NaN-masking, i.e. I think you'll need to 
pass an original image AND a NaN-mask image marking voxels to exclude.

Does that sound reasonable? I don't know if this kind of thing has 
been investigated much in the signal processing literature... I think 
usually, when people move away from standard smoothing, they start 
looking at more complicated anisotropic diffusion models, which I'm 
afraid I don't know too much about... (some of the refs in the 
Davatzikos paper above should help here)

Best of luck, and happy Christmas/days!

Ged.

P.S. Just had an idea... might be completely wrong, but I think there 
might be a simple shortcut to the above approach...
- create an image with zeros (not NaNs), and smooth this as usual.
- now binarize this image, and smooth this version as usual (the 
result of this will be 1 everywhere where the kernel didn't include 
any zeros, and a reduced value where it did.
- divide (voxelwise) the first smoothed image by the second, this 
should effectively be equivalent to the renormalisation of the kernel 
mentioned above (I think/hope!). You will get NaNs where the kernel 
fell entirely on zeros, but these will be overwritten when you...
- replace output voxels with the original values in your masked regions.

This sounds like it should work... but I might be missing something... 
Let me know how you get on (and say if you need help coding this up)