I noticed a peculiar behaviour when running on more than one GPU node.

When I run on just one GPU node (with 4 GPUs) and set MPI processes to 5, I see one GPU process per GPU as expected.

However, when I do the same but ask 2 GPU nodes when I submit the job, the first node shows not one but two GPU processes on GPU ID 0. Other GPUs on the first node (IDs 1,2,3) and all GPUs on the second node (IDs 0,1,2,3) show just one GPU process. 

I was using "--gpu -j 1". Is there a way to map the GPU processes properly across several GPU nodes?