Plotting a giant surface exceeding array limitations

Hi all

I have a need to plot a large surface.

Im trying to create a multi dimensional array.

Picture the following.

A typical X and Y plot that is a vector that has 40,000 data points. If you plot that you get a simple traditional XY graph.

Now, i have say 100 of the X and Y vectors and want to plot them all as a surface.

I tried something like this to build the array first. According to the matlab documentation the 3rd variable is like a page variable. Which is exactly what i need.

So to start i just tried

test = zeros(40,000,40,000,100)

And i got an error saying that my array will take 775 gig of space lol

So how does one go about doing this?

Thanks

[Robert Scott]

I looked into that but there are very poor examples for tall arrays mostly using data that already exists in some file.

Answer 1

[Matt J]

First of all, you should consider downsampling from 40000 points to perhaps 400. Having very dense data beyond a certain point will not help you better visualize the surface.

Aside from that, though, your surface will have 40000*100 points, so a trisurf plot would only require you to form 40000x100 arrays.

[Xsurf,Ysurf,Zsurf]=deal(zeros(4e4,100)); %pre-allocate

[Robert Scott]

400000 points is not because of resolution.

My resolution is already course. I just have a large range

[Matt J]

Okay, but the display area on your monitor is fixed, and the renderer will have to cram 40000 points into it. Unless you have an extra large monitor, I don't see how you will see a difference between 400 and 40000.

[Robert Scott]

To clarify,

what i have is 65 single X and Y plots

I want to combine my 65 X and Y plots into a single surface.

So think of it like this.

Think of each of the 65 X and Y plots pertain to a single temperature.

So i have 40,000 points of X and Y data at Temp = 1

Then again at Temp = 2

Then again at Temp = 3 etc etc all the way to temp = 65

I want to plot my X and Y data and add the third access of temperature to the z axis to form a surface or 3dimensional plot.

The Z data for each page will all be the same because it represents the same temperature.

so it would be like have 65 individual arrays that are 40,000 X 40,000

[Matt J]

so it would be like have 65 individual arrays that are 40,000 X 40,000

Everything is and has been clear except for why you think the data size will be 40000x40000? You said you have 100 (now 65) X,Y plots. Each X,Y plot has 80000 numbers associated with it, not 40000^2.

[Robert Scott]

i can make the third value whatever i want. Sorry i reduced it to 65. When i asked originally, i just threw out the number 100 for general question.

But yes, what you said above is correct.

How can i plot this? Can you please recommend an approach?

[Matt J]

Attach your 65 X,Y data sets in a .mat file (preferably as Nx65 matrices), so that solutions can be demonstrated. Shorten N from 40000 to something that will fit within the 5MB file attachment limit.

[Robert Scott]

I was just trying to reference this page here

https://www.mathworks.com/help/matlab/math/multidimensional-arrays.html

So i was treating each X and Y plot at a given temperature as a single page per this document

So i was trying zeros (40,000,40,000,65)

So 40k rows and 40k cols per 65 different temperatures

[Robert Scott]

I dont have any specifc data yet because i cant get it to run.

Could you possibly please demonstrate just creating maybe a few random arrays.

And we could reduce 100 down to say 5 just for the example.

[Matt J]

[M,N]=deal(4000,100);

X=linspace(0,1,M)'+rand(1,N)/10;  %fake X,Y,T data
Y=X.^2+linspace(0,1,N);
T=repmat(log(1:N),M,1);

whos X Y T

  Name         Size               Bytes  Class     Attributes

  T         4000x100            3200000  double              
  X         4000x100            3200000  double              
  Y         4000x100            3200000  double              

dt=delaunay(X,Y);

trisurf(dt,X,Y,T,'EdgeColor','none'); xlabel X; ylabel Y; zlabel T

[Robert Scott]

any chance you can offer some explination or code commenting?

I have literally no idea what you did.

Can you do this will a cell array?

[Robert Scott]

Im trying to do something like this but cant get the plot to work

a = 1
b = 2

surface_matrix = {100,2};

for k =1:100

test_x_data = (b-a).*rand(40000,1) + a;
test_y_data = (b-a).*rand(40000,1) + a;

single_matrix = [test_x_data,test_y_data];

surface_matrix{k,1} = [single_matrix];
surface_matrix{k,2} = [k];

clear single_matrix


for k = 1;100

trisurf(surface_matrix{1:end,1},surface_matrix{1:end,2},surface_matrix{k,2})

hold on
end


end

[Robert Scott]

i sincerly wished you explained anything.

You might be a level 10 MVP but because you didnt explain a thing you did not help.

[Matt J]

any chance you can offer some explination or code commenting?

Well, most of the lines of code in my last comment was just generating hypothetical X,Y,T data, which I did note in a comment. The only real work once the data is given was these 2 lines, which is really just calling trisurf:

dt=delaunay(X,Y);
trisurf(dt,X,Y,T,'EdgeColor','none'); xlabel X; ylabel Y; zlabel T

I don't know what more I can say about that. If you've never used trisurf and are having trouble understanding the documentation for it, then you can say what it is you are stuck on.

Can you do this will a cell array?

Undoubtedly, but you have wasted a lot of time not providing your own example data, and asking me to propose example data instead. Obviously, we cannot be expected to know in advance the data organization you have in mind.

Im trying to do something like this but cant get the plot to work

Is this supposed to be the temperature data?

surface_matrix{k,2} = [k];

[Matt J]

a = 1;
b = 2;

surface_matrix = cell(100,2);

for k =1:100

    test_x_data = (b-a).*rand(40000,1) + a;
    test_y_data = (b-a).*rand(40000,1) + a;
    
    single_matrix = [sort(test_x_data),test_y_data];
    
    surface_matrix{k,1} = single_matrix;
    surface_matrix{k,2} = k;

end

%%%%%

temp=cat(3, surface_matrix{:,1});
X=squeeze(temp(:,1,:));
Y=squeeze(temp(:,2,:));
T=[surface_matrix{:,2}];  T=repmat(T,height(X),1); %temperature?

dt=delaunay(X,Y);
trisurf(dt,X,Y,T,'EdgeColor','none'); xlabel X; ylabel Y; zlabel T

[Robert Scott]

Hello

Thank you for your continued help

So i have adopted your first version of code

I almost have it working.

However, when i run the delaunay command

Im getting a warning and i dont know why

Warning: Duplicate data points have been detected and removed.

Some point indices will not be referenced by the triangulation

What can i do about this? I obviously dont want it removing data points

[Matt J]

Why not? the warning says they are duplicates. The real question is why do you have duplicate X,Y points?

[Robert Scott]

ok taking a step back

it turns out that i screwed up.

My data set is actually much smaller then i realized. I was thinking about this incorrect.

My data set in general is less then 100

So here is what i have tried to do.

I want to take your approach of having all the X in one matrix

and all the Ys in another. and all the Ts in another. I have my code setup like that per your recomendation.

How can i get this to work with just a simple meshgrid.

So i have put all the X and Ys into a mesh grid but dont know how to get the Zs in the proper dimension

Can you explain? Im going to post the code here in just one second.

[Robert Scott]

Now trying something like this since my data set is much much smaller

But not sure how to get z dimensions to work

[M,N]=deal(40000,100);

X=linspace(0,1,M)'+rand(1,N)/10;  %fake X,Y,T data
Y=X.^2+linspace(0,1,N);
T=repmat(log(1:N),M,1);
whos X Y T
              

[X,Y] = meshgrid(X,Y);
Z= T
surf(X,Y,Z);

[Matt J]

How can i get this to work with just a simple meshgrid.

I don't think you can, because in your original problem description, your X,Y data is scattered, not gridded. Ifthat's no longer the case, it would seem that you have a whole new problem and you should accordingly post a whole new question.

[Robert Scott]

Thanks, Matt

Well, i have no idea how it works but it works.

I basically just took 3 giant vectors X,Y and T and put them in the surf command and it worked.

This turned out to be a total waste of a day sadly. My misinterprestation of the data set really screwed me up and caused me to write many nonsensical scripts.

In the end, your original solution or my cell array solution would work for something like this.

I thank you for your persistance in helping me esepcailly on a sunday.