The "Auth fail" message is shown when all authentication mechanisms supported by the server failed; the password dialog will be shown if public key authentication failed but additional authentication methods are available on the server. One possible reason for this public-key authentication failure is MATLAB finding ~/.ssh/id_rsa or ~/.ssh/id_dsa or ~/.ssh/identity file(s) and attempting to authenticate over SSH with them, but none of these files contained a key which MATLAB was able to work with. This may for example occur if your ~.ssh/id_rsa key is in OpenSSH format instead of the RSA format.
For Pre-OpenSSH 7.8 it is recommended to use the -o option to generate the keys, while for newer versions using ssh-keygen will produce the valid keys.
If you are using Pre-OpenSSH 7.8 with ssh-keygen -o:
1. To check the format, open the file in a text editor, the file should start with:
-----BEGIN RSA PRIVATE KEY-----
-----BEGIN OPENSSH PRIVATE KEY-----
2. If your RSA private key is in the OpenSSH format, you may either generate a new key using the RSAformat, or convert your existing key to the RSA format. In either case, you may first want to create a backup copy of your "id_rsa" and "id_rsa.pub" files. Then, open a command prompt window and run one of the following commands:
To generate a new key in the valid format, use "ssh-keygen" with the "-m PEM" format setting:
ssh-keygen -m PEM -t rsa -b 4096
To convert your existing key from OpenSSH to RSA, use the command below. If your private key is not stored at "~/.ssh/id_rsa", change the last parameter accordingly. This will update the format of the file without changing the key. Note that it will also prompt you to set a new password -- if you set a password for your SSH key, you will need to enter that password whenever using the key.
ssh-keygen -p -m PEM -f ~/.ssh/id_rsa
You can verify the format of the new key by checking the first line of the file as described above. After making changes to your SSH keys, please restart MATLAB in order to reload the keys being used. If you generated a new key, you will also need to update the remote repository with the new public key. Please refer to the documentation of that repository for their specific workflow for adding public SSH keys.
If you generate your keys with ssh-keygen (no -o option) you should be able to interact with Git remote correctly.