Asus Zenfone AR correlate depth and colour image

Question

Hello, I am using the depth and colour images from Google Tango, so that I can load the image into Meshlab. There is a related question, where the goal is to find the colour of each point in the Tango Point Cloud. However, I would like to go the other way. For each pixel of the colour image, how do I find the corresponding depth?

I have upsampled the depth image and saved the result in the TangoDepthBuffer. I have used the OpenGL readPixels() method, to get the colour image and store the RGB values in an array called pixels[]. I then correlate the x, y, z values with the RGB values using the following code:

index_rgb = 0;
index_pixels = 0;
for(int i = 0; i < color_camera_width; i++)
{
    for(int j = 0; j < color_camera_height; j++)
    {
        red     [index_rgb] = pixels[color_camera_width * color_camera_height * 3 - 3 - index_pixels];
        green   [index_rgb] = pixels[color_camera_width * color_camera_height * 3 - 2 - index_pixels];
        blue    [index_rgb] = pixels[color_camera_width * color_camera_height * 3 - 1 - index_pixels];

        z[index_rgb] = render_point_cloud_buffer->depths[j * color_camera_width + i];
        x[index_rgb] = (double) (i - color_camera_width/2);
        y[index_rgb] = (double) (j - color_camera_height/2);

        x[index_rgb] = (x[index_rgb] / color_camera_width) *  depth_camera_horizontal_fov;
        y[index_rgb] = (y[index_rgb] / color_camera_height) * depth_camera_vertical_fov;

        x[index_rgb] = z[index_rgb] * tan(x[index_rgb]);
        y[index_rgb] = z[index_rgb] * tan(y[index_rgb]);

        index_rgb++;
        index_pixels += 3;
    }
}

I would expect the result to align the depth and colour images. However, when I load the result into Meshlab, the depth pixels are shifted down and to the left of the corresponding colour pixels. The manner in which this shift occurs varies based on the depth. However, I cannot find a depth where there is no shift.

How do you find the transformation required to fix this? Will it work for any depth? Alternatively, how do you find the depth at each specific colour pixel?