How can i access .mat files, stored in an encrypted amazon S3 bucket via Matlab (API Key available)

Using Matlab R2018b on Windows 7

Already tried the following:

url = "https://se.eu-central-1.amazonaws.com/my_bucket/my_file.mat";
opt = weboptions('keyName','my_key_id','keyValue','my_api_key');
file = webread(url,opt);

[David Reiter]

The exception i got:

Error using readContentFromWebService (line 62)

The server returned the status 403 with message "Forbidden" in response to the request to URL

https://s3.eu-central-1.amazonaws.com/bogie-diagnostic-978814483201/Parametrization/RRX/configWheelWear.mat.

Answer 1

[Aylin]

Hi David, it might be easier to use FileDatastore with the load function to do this:

% Set up S3 credentials.
setenv('AWS_ACCESS_KEY_ID', MY_ACCESS_KEY_ID); 
setenv('AWS_SECRET_ACCESS_KEY', MY_SECRET_ACCESS_KEY); 
setenv('AWS_REGION', MY_REGION);

% Construct a FileDatastore with a S3 URI.
fds = fileDatastore("s3://rdmello/*.mat", "ReadFcn", @load);

% Read from a single MAT file.
data = read(fds);

% Read all the data from all the MAT files on S3.
all_data = readall(fds);

Note here that MY_ACCESS_KEY_ID, MY_SECRET_ACCESS_KEY, and MY_REGION are strings that you need to provide in your code. See the Working with Remote Data doc page for more information about this.

Using a datastore here provides some benefits...each file on Amazon S3 is only downloaded only when needed, and you can build a tall array over it for easier scalability on parallel clusters.

[David Reiter]

Hi Rylan,

thanks for your answer! I appretiate it.

Apperently I have trouble with my API-Key. When loading my .mat file via AWS-CLI I get an "access denied" error, which tells me that I still do not have necessary permissions. I tried loading the file with your code anyway. Matlab throws the following error:

Error using fileDatastore (line 105)
Cannot find files or folders matching: 's3-URI'.

I suppose I get this error due to my lack of permission? As soon as I get my new access key, I am going to comment on this page whether I had success or not.

Regards,

David

[David Reiter]

Hi again.

I got my new API-Key and guess what?! It worked just fine. Thanks again for your input. Here is the code that worked for me:

% Set up S3 credentials.
MY_ACCESS_KEY_ID = 'my_access_id';
MY_SECRET_ACCESS_KEY = 'my_access_key';
MY_REGION = 'my_region';
setenv('AWS_ACCESS_KEY_ID', MY_ACCESS_KEY_ID); 
setenv('AWS_SECRET_ACCESS_KEY', MY_SECRET_ACCESS_KEY); 
setenv('AWS_REGION', MY_REGION);
% Construct a FileDatastore with a S3 URI.
% file_key can be obtained directly on S3 when navigating to the deisred file or folder
fds = fileDatastore('s3://<my_bucket>/<file_key>.<file_extendsion>', "ReadFcn", @load,'FileExtensions','.mat');
% Read from a single MAT file.
data = read(fds);

[Maitreyee Dey]

Hi Rylan,

I am trying read parquet file format from S3 using Mac os. Receiving error every time. When I am trying to read csv using windows os is able to do it. But with mac its not working. can you please help? Thanks

%% See the code below

setenv('AWS_ACCESS_KEY_ID', 'my_credentials');

setenv('AWS_SECRET_ACCESS_KEY', 'my_credentials');

setenv('AWS_DEFAULT_REGION', 'EU (Ireland) eu-west-1');

pds = parquetDatastore('s3://{mybucket_path}/');

data = readall(pds);

% Below the error msg i am getting everytime

Error using matlab.io.datastore.ParquetDatastore (line 316)

Cannot find files or folders matching: