CUDA: Thread and Array Allocation

I have read many times about CUDA Thread/Blocks and Array, but still don't understand point: how and when CUDA starts to run multithread for kernel function. when host calling kernel function, or inside kernel function.

For example I have this example, It just simple transpose an array. (so, it just copy value from this array to another array).

__global__
void transpose(float* in, float* out, uint width) {
    uint tx = blockIdx.x * blockDim.x + threadIdx.x;
    uint ty = blockIdx.y * blockDim.y + threadIdx.y;
    out[tx * width + ty] = in[ty * width + tx];
}

int main(int args, char** vargs) {
    /*const int HEIGHT = 1024;
    const int WIDTH = 1024;
    const int SIZE = WIDTH * HEIGHT * sizeof(float);
    dim3 bDim(16, 16);
    dim3 gDim(WIDTH / bDim.x, HEIGHT / bDim.y);
    float* M = (float*)malloc(SIZE);
    for (int i = 0; i < HEIGHT * WIDTH; i++) { M[i] = i; }
    float* Md = NULL;
    cudaMalloc((void**)&Md, SIZE);
    cudaMemcpy(Md,M, SIZE, cudaMemcpyHostToDevice);
    float* Bd = NULL;
    cudaMalloc((void**)&Bd, SIZE); */
    transpose<<<gDim, bDim>>>(Md, Bd, WIDTH);   // CALLING FUNCTION TRANSPOSE
    cudaMemcpy(M,Bd, SIZE, cudaMemcpyDeviceToHost);
    return 0;
}

(I have commented all lines that not important, just have the line calling function transpose)

I have understand all lines in function main except the line calling function tranpose. Does it true when I say: when we call function transpose<<<gDim, bDim>>>(Md, Bd, WIDTH), CUDA will automatically assign each elements of array into one thread (and block), and when we calling "one time" tranpose, CUDA will running gDim * bDim times tranpose on gDim * bDim threads.

This point makes me feel frustrated so much, because it doesn't like multithread in java, when I use :( Please tell me.

Thanks :)