The "Auth fail" message is shown when all authentication mechanisms supported by the server failed; the password dialog will be shown if public key authentication failed but additional authentication methods are available on the server. One possible reason for this public-key authentication failure is MATLAB finding ~/.ssh/id_rsa or ~/.ssh/id_dsa or ~/.ssh/identity file(s) and attempting to authenticate over SSH with them, but none of these files contained a key which MATLAB was able to work with.
MATLAB's Git client requires your private key to be in the (older) RSA format and it cannot work with the (newer) OPENSSH format. Please open your private key file in a text editor; to be compatible with MATLAB it should start with:
-----BEGIN RSA PRIVATE KEY-----
And not with:
-----BEGIN OPENSSH PRIVATE KEY-----
Now why might you have different keys in different formats?
This is a combination of factors:
- Older ssh-keygen versions always created keys in the RSA format, so then it would always have worked fine with MATLAB.
- Then at some point in time an "-o" option was added to ssh-keygen, when indeed called with this "-o" option, it would now produce keys in the OPENSSH format. At the time, online instructions (on websites like git-scm.com, github.com, gitlab.com, etc.) about using ssh-keygen varied by website, some did instruct you to explicitly add "-o", some did not. If you did call ssh-keygen with "-o" you would have gotten a key incompatible with MATLAB, if you did not, the key was compatible with MATLAB.
- In current ssh-keygen versions the OPENSSH format is in fact the default, you no longer have to explicitly add "-o" to get a key in that format. Nowadays you explicitly have to add "-m PEM" to get a key in the RSA format. Most online instructions now instruct you to use ssh-keygen without the "-o" option (since it has no effect anymore anyway) but also without the "-m PEM" option. So nowadays when following most of the online instructions you would get a key which is incompatible with MATLAB.
What is our advice right now?
When creating new keys, you can in the basis follow online instructions for your preferred platform on how to use ssh-keygen, for example with regards to which key size to use; however:
- If these instructions include adding "-o" to the command-line then actually omit this option.
- If these instructions do not include "-m PEM" then please do add this option.
So you would for example get:
ssh-keygen -t rsa -b 4096 -m PEM
Further, it should be possible to convert from OPENSSH to RSA format if you had already created your key in the OPENSSH format:
ssh-keygen -p -m PEM -f ~/.ssh/id_rsa
Where you may need to replace ~/.ssh/id_rsa with your specific private key file.