Matlab is significantly slower than Julia on simple evaluation

More and more frequently some people open the question about speed comparison between Matlab and Julia. The main Julia developers goal was to create high-level language for scientific computing with computing speed comparable to C/C++.

I just read the interesting blog: Testing Julia: Fast as Fortran, Beautiful as Python which shows simple exp() evaluation comparison between Python, Julia and Fortran (details are here).

The corresponding Matlab functions to compare Julia with Matlab:

function [M] = test_eval_exp(N)

disp('Loop-for version:')
for m = 1:10
    n = m*N;
    tic;
    M = eval_exp_loop(n);
    tcpu = toc;
    disp(['N = ' num2str(n) ' time = ' num2str(tcpu)]);

end    

disp('Vectorized version:')
for m = 1:10
    n = m*N;
    tic;
    M = eval_exp_vec(n);
    tcpu = toc;
    disp(['N = ' num2str(n) ' time = ' num2str(tcpu)]);    
end

end

function M = eval_exp_loop(n)
a = linspace(1,2*pi,n);
M = complex(zeros(n,n));
for j = 1:n
  for i = 1:n
    M(i,j) = exp(1j * (100+1j) * sqrt( a(i)^2 + a(j)^2 ));
  end
end
end   

function M = eval_exp_vec(n)
a = linspace(1,2*pi,n);
a2 = a.^2;
M = exp(1j*(100+1j)*sqrt(a2 + a2'));
end

And results:

>> test_eval_exp(1e3);
Loop-for version:         %... sigle Thread 
N = 1000 time = 0.067138
N = 2000 time = 0.2546
N = 3000 time = 0.54324
N = 4000 time = 0.96516
N = 5000 time = 1.5097
N = 6000 time = 2.161
N = 7000 time = 2.9356
N = 8000 time = 3.8305
N = 9000 time = 4.8475
N = 10000 time = 5.9773    
Vectorized version:      % ... multi Thread (8 threads)
N = 1000 time = 0.036739
N = 2000 time = 0.03424
N = 3000 time = 0.073091
N = 4000 time = 0.12321
N = 5000 time = 0.18877
N = 6000 time = 0.26854
N = 7000 time = 0.36426
N = 8000 time = 0.47832
N = 9000 time = 0.61006
N = 10000 time = 0.75371

% Julia loop-for (1 thread) for N= 1e4 ... 3sec 
% Julia vectorized (8 thread) for N= 1e4... 0.6 sec
% Julia vectorized + optimized (8 thread) for N= 1e4... 0.3 sec

These results really supports the general answer, that Julia (at least on this specific task) is significantly faster in a case when Julia and Matlab code are realised simply by for-loop (1-thread ~ 2x). In a case when Matlab code is vectorized and run on 8 core CPU the Julia vectorized for-loop is still faster (~1.25x) . Finaly, the highly optimized Julia code is 2x faster than Matlab vectorized code.

My questions are:

1. Is there any possibility how to speed up (optimize) loop-for and vectorized Matlab code to get similar speed as Julia?
2. MATLAB JIT engine produce typically significantly slower simple for-loop code than Julia LLVM. Why? Is possible in future MATLAB releases expect some significant additonal speed up?

My questions are motivated to get the right answers for those people who argue by simply comparing the speed of Matlab vs Julia and completely avoid the other benefits of Matlab.

[Jon]

Sorry if I overlooked it, but I'm not finding your code defining the functions eval_exp_loop(n) and eval_exp_vec(n) I think I would need to see that to judge whether you had implemented those as efficiently as possible in order to give a good comparison.

[Jon]

Oh, I guess you are showing those functions with your commented out code below the function call. Is that correct?

[Michal]

Just use the commented parts of code to create separate functions.

Answer 1

[Bobby Cheng]

First of all, thank you for creating the benchmark and raising this performance issue. Indeed, we are seeing more and more performance comparison like this, what I would call a focused micro-benchmark. At MathWorks, we found this very informative. More often than not, it highlights the less optimized part of the MATLAB language, and it is exactly what this benchmark is picking up. The bad news is that, like Jan, I can confirm that Julia is doing better than MATLAB in the current release for this benchmark, and there is not much you can do within MATLAB to optimize it further. The good news is that the issues discovered here are on our plan to be addressed in a future release.

When I took a look at your benchmark, the most time consuming items are:

1. Complex scalar execution overhead in MATLAB . Complex is an attribute of certain classes, not a datatype of its own. MATLAB math tends to return the most mathematically correct answer, like sqrt(-1) returns 1i instead of NaN as required by the IEEE standard, and in MATLAB, 1i – 1i returns real 0, instead of 0+0i. This makes MATLAB easy to use, but it does add overhead in execution. We need to minimize this overhead.
2. There are faster algorithms to compute the exponential function in the literature. But using those algorithms is not a matter of simply ripping out one piece of code, inserting new code, compiling it, and saying "Ship it to users!" We need to ensure that the performance, accuracy, stability, and portability (among other considerations) meet our standards before we publish them as part of a new release. Sometimes this decision is not easy: for example, when the faster and more accurate SUM for single precision data was released, as my colleague Christine noted on Loren Shure's blog (link to https://blogs.mathworks.com/loren/2021/09/07/a-faster-and-more-accurate-sum/) it created noise in our internal test suite and led us to watch for user feedback before we made the same modification to double precision SUM. Not everyone was happy with the change but we could justify why we made the change.

I can't tell you the exact timeline for each of these issues to be addressed because each issue has its own unique challenge, but rest assured that we plan to improve in these areas. Keep checking on MATLAB and keep us on our toes.

Thank you again for the benchmark.

[Michal]

Thanks for honest and professional answer!

Answer 2

[Jan]

I can confirm your observation that Julia runs faster than Matlab for some problems. Answering your three questions is not easy due to the restriction "completely avoid the other benefits of Matlab". It is questionable if a comparison of two tools is useful, if only one specific feature is considered. This is like comparing a racing care with a limousine: Of course they have a different speed. As soon as I want to transport a cardboard box, this difference is not relevant.

• Is there any possibility how to speed up (optimize) loop-for and vectorized Matlab code to get similar speed as Julia?

I tried it without success. I frequently post C-mex functions in this forum, because this is my way to squeeze bottlenecks in Matlab. But some C code cannot be a fair answer to a question about Matlab and Julia.

• MATLAB JIT engine produce typically significantly slower simple for-loop code than Julia LLVM. Why?

Because I do not have the possibility to control the way Matlab's JIT is working, I cannot guess, why it is slower than Julia.

• Is possible in future MATLAB releases expect some significant additonal speed up?

Yes. An acceleration is possible.

I've answered some questions concerning the acceleration of code in this forum. The speed of the used computer is less relevant. Even for a lot of money increasing the speed by a factor of 4 is really hard (if neither the number of cores or the RAM is the limitting factor). In opposite to this, accelerating some code by a factor of 100 to 1000 was possible in many cases just by avoiding repeated calculations and using a smarter representation of the data (like columnwise operations and avoiding the dynamic creation of variables by eval()).

Of course the speed of the programming language has an effect also. You have demonstrated an advantage of Julia a tiny test problem. This is interesting and it is a good strategy to use Julia, if it solves a problem efficiently. For my work this is not an option, because I use exhaustively tested tools, e.g. one for clinical decision making with more than 300'000 lines of Matlab code (plus comments), which was tested over 2 decaded with thousands of patients. Converting this to Julia would be a waste of time.

You see, "completely avoid the other benefits of Matlab" is a really hard limitation for writing a meaningful answer to your question. The time to solve a problem consists of:

t_solution = t_design + t_programming + t_debugging + t_documenting + ...
             t_runtime

Therefor re-using existing toolboxes has a much more effect than the actaul run time.

But your point is important: If I create a new program e.g. to calculate collisions of galaxies using some simple loops to obtain the accelerations (and not a leap-frog-toolbox, which performs the actual calculations in an assembler routine, such that Matlab is responsible for defining the initial values only and to visualize the results), I'd try this in Julia. Then it would not help to discuss, if a future version of Matlab might be faster, because waiting for years cannot compete with minutes of runtime.

For my daily work, the power of Matlab is more important than its speed. Therefore I'm not the right one to answer your question, but I wanted to make a start at least.

[Michal]

Thanks for detailed answer. I completely understand that Matab has a lot of other advantages and comparison based only on one feature (speed) may be confusing. But if you need brute-force computing power, Julia is really good choice. On the other hand the Matlab has extraordinary well mainteined documentation and perfect compatibility with all toolboxes. Julia packages very often suffer by mutual incompatibility due to the uncontroled development without well mainteined documetation, for example.

[Michal]

But once again: If computation speed is crucial, and in many cases definetly is! Then the question: "Why is Matlab in many cases so slower than Julia?" is fully legitimite for many Matlab users. The majority of Matlab users does not want to write separate programs in Julia and call them from Matlab or switch entirely to the Julia, because of many important reasons you mentioned above.

So crucial question is: Is the computation speed of Julia real challange for TMW, or not?

P.S. Maybe I should ask TMW directly ...

[Jan]

I've observed a remarkable drop of speed in Matlab R2018b, if (U)INT64 data are used compared to (U)INT32 and DOUBLE. In a very tiny function FEX: NoverK the processing time grows for 64 bit integers, even if only the validity of the input checks are performed:

if ~(nargin == 2 && isscalar(n) && isscalar(k) && ...
      strcmp(class(n), class(k)) && isnumeric(n))
...
if ~((isinteger(n) && n >= 0) || (floor(n) == abs(n) && floor(k) == k))

The only meaningful explanation I've found is, that UINT64 are implemented in a different way than the other types. On one hand this shows a potential for speed improvements. On the other hand this demonstrates, that the same code can be applied to different underlying classes. Even if I define my own numerical class and run this function, it will work. Of course this flexibility costs run time. Power is usually a drawback for efficiency.

I have much less experiences with Julia than with Matlab. Therefore I cannot assess its flexibility and power sufficiently. I assume, that Matlab has limitations in the design, which impede the application of the same JIT acceleration methods applied in Julia. E.g. the automatic parallelization of loops is impossible as long as an eval can redefine variables and functions during the run time.

In the last years MathWorks has removed several details like the automatic sharing of variables defined inside nested function. I think this helps to avoid clutter in the lookup tables of variables to accelerate the processing.

It is a good idea to ask TMW directly. My current opinion (based on rough estimations!) is, that Julia is and will be faster for basic algorithms and that it is less useful for my scientific work. For long term studies I let some Matlab code run, which I have written 1999. It is interpreted much faster in modern Matlab version. Simple graphics (2 buttons, 2 diagrams with 4 line objects) need 2 to 10 times longer to be displayed, but complex graphics with many 2D or 3D objects profit from the upgrades. The stable backward compatibility might slow down the processing severly, but it is essential for my work.

[Bobby Cheng]

Thank you again for posting here. I love reading about what customers do and what issues they encounter. 64 bit integer is an area that we need to do better. R2022a prerelease is out. We make some incremental improvement to the area. Hopefully more to come soon. Let us know what you think.

[Michal]

Bobby, speaking about future speed improvments in Matlab. I am using very often Symbolic toolbox, especially vpa functionality. If is some part of Matlab really slow, it is definitely vpa evaluation of elementary and special functions (at least at quadraple precision). Is that any chance to expect in near future any significant improvement? First thing is abvious, vpa is so far not multi-threaded. Am I right?

Try to compare vpa evaluation performance with Advanpix Multi-precision toolbox. From this point of view is vpa really terribly slow.

[Bobby Cheng]

Michal,

I don't know enough to comment, but I will forward your comment to the development team. I know your issue applies to the overall vpa performance. However, would be mind giving me a specific code example to demonstrate the issue? It would make it easier for me to create an enhancement request. Thanks!

[Michal]

Bobby,

testing trial version of recent release Advanpix multi-precision toolbox (Windows platform) is available here.

Testing codes for Advanpix toolbox (mp) and Matlab (vpa) are attached.

Elementary and some special functions:

>> timing_elementary_advanpix
>> timing_elementary_vpa

Linear algebra:

>> mp_test
>> vpa_test

You should be prepared, that VPA evaluations takes really (really!!!) long time :)

Anybody who test the Advanpix MCT and compare it with VPA, will be really surprised, how slow VPA actually is. I think there is a real reason to significantly speed up VPA, because in the current state of development the VPA is really not suitable for any serious "big-data" applications which requires multi-precision computing.

On the other hand VPA is very well connected with rest of Matlab functionality, but ... it is slow!

[Bobby Cheng]

Thanks.

[Oliver Josephs]

Hi Bobby,

You mentioned on the 10th that the R2022a prerelease was out. Normally I can download this but it seems not at the moment.

Should it be available?

Oliver

[Bobby Cheng]

Sorry Oliver for the delay response. I was on vacation. You should try download again. There were some issues over the holiday. It should be available now. If you have any issue, you should contact support directly for faster response.