Acessing large3D matrix incredibly slow

I have a code in which i wish to remove certain points from a large matrix, calculation of which points to remove which requires large number of calculations only takes a fraction of the time it takes to change a point in the matrix. i timed it like so:

d=top_cell_array{i};
        a=d(1);
        b=d(2);
        c=d(3);
        tic;
        centroids_3D(a,b,c)=0;
        time_of_exec=toc

what i get as a result is sometimes a few ms, but sometimes exceeds half a second, just to excess and change the point in the matrix, the matrix is of size (512,512,520); my calculations which require hundreds if not thousands of operations(which include excessing the large matrix) only take a few ms everytime, im incredibly confused as to why matlab takes it time in changing it, and what can i do about it.

[Daniel ziv]

Ok The issue was that i was recreating the large matrix over and over again, what i did was take the code that removes points out of the function so that the new matrix is created only once, not sure if there is a way to pass a pointer in matlab so that i can change a matrix inside a function instead of creating a new matrix and returning it.

[Stephen23]

Instead you could easily leave the matrix where it is, and return the indices.