How can I fix error in working directory location change when using a parallel cluster?

This issue is a little complicated, I think. So here's the general rundown:

• Local machine, R2014a, Windows 7
• Off-site computing cluster that only accepts batch() commands, some linux distro, requires ssh connection
• Matlab-authored config scripts were provided
• Files for the job are stored on my local machine, not on the server
• When running a script, the files get copied over into a temp directory on the server, but matlab attempts to change the working directory from the server to my personal matlab's current working directory

Here's the output from c=parcluster() for that cluster

>> c=parcluster()
c = 
Generic Cluster

    Properties: 

                          Profile: ghpcc_remote_r2014a
                         Modified: false
                             Host: MYPERSONALPC.uml.edu
                       NumWorkers: 128

               JobStorageLocation: C:\Users\Ronen\AppData\Local\Temp\MdcsDataLocation\umass\ghpcc\R2014a\remote
                ClusterMatlabRoot: /share/pkg/matlab/R2014a
                  OperatingSystem: unix

             IndependentSubmitFcn: [1x3 cell]
           CommunicatingSubmitFcn: [1x3 cell]
                   GetJobStateFcn: @profiles.umass.ghpcc.getJobStateFcn
                     CancelJobFcn: []
                    CancelTaskFcn: []
                     DeleteJobFcn: @profiles.umass.ghpcc.deleteJobFcn
                    DeleteTaskFcn: []
 RequiresMathWorksHostedLicensing: false

    Associated Jobs: 

                   Number Pending: 0
                    Number Queued: 0
                   Number Running: 0
                  Number Finished: 1

That completed job that is listed errored out with the following:

>> j
j = 
 Job

      Properties: 

                   ID: 8
                 Type: pool
             Username: Ronen
                State: finished
           SubmitTime: Tue May 12 12:57:16 EDT 2015
            StartTime: Tue May 12 12:59:07 EDT 2015
     Running Duration: 0 days 0h 0m 10s
      NumWorkersRange: [11 11]

      AutoAttachFiles: true
  Auto Attached Files: List files
        AttachedFiles: {}
      AdditionalPaths: {}

    Associated Tasks: 

       Number Pending: 0
       Number Running: 0
      Number Finished: 11
    Task ID of Errors: [1]
   Task with properties: 

                   ID: 1
                State: finished
             Function: @parallel.internal.cluster.executeScript
               Parent: Job 8
            StartTime: Tue May 12 12:59:08 EDT 2015
     Running Duration: 0 days 0h 0m 7s

      ErrorIdentifier: MATLAB:undefinedVarOrClass
         ErrorMessage: Undefined variable "tester" or class "tester.m".
>> diary(j)
--- Start Diary ---
Warning: Unable to change to requested folder: 'C:\Users\Ronen\Google Drive\School\Masters Thesis 22.743\Spring 2015\Optimization'. Current folder is: '/home/myserverfolder'.
Reason: Cannot CD to C:\Users\Ronen\Google Drive\School\Masters Thesis 22.743\Spring 2015\Optimization (Name is nonexistent or not a directory).

--- End Diary ---

[Ronen Yudzinsky]

I just want to clarify that my only goal at this point is to force the cluster to use the .m files that it downloaded from my computer, instead of downloading them and then immediately trying to CD to my remote computer to get my work directory. My work directory doesn't need to have anything to do with this.

Answer 1

[Edric Ellis]

The message in the diary output there is simply a warning that the worker couldn't cd to that location. You can use the CurrentFolder argument to batch to avoid that warning, as described in the batch reference page.

The reason for the error is that tester.m isn't getting attached to your job. Does it help to list that in the AttachedFiles argument to batch? Also, what happens when you select the "list attached files" link in the job display? Does that show tester.m?

[Ronen Yudzinsky]

1: Yes, it is a warning, but I'm also running a GA that is contained in tester.m, so a 20 second solution doesn't seem feasible for the solution set I'm seeking. Plus, the journal should have outputs from the tester (solution time-stamps). These would appear in the diary, wouldn't they?

2: I attempted to use the following code to set up the job. The warning did not appear in the diary, but the error persisted, and the job was not done.

j = c.batch('tester.m','Pool',10,'CurrentFolder','/home/ry54l/matlab')

3: This is the output from the "List files" link under the "Auto attached files" field. I assume that since it is here (and the log file for Task 1 shows the copying of the files to a "/tmp/..../tester.m" location) that I don't need to manually attach the files? also, the files are stored in that /home/ry54l/matlab directory.

>> j = c.batch('tester.m','Pool',10)
additionalSubmitArgs =
-n 11  -W 01:00  -R "rusage[mem=1024]" -R "rusage[mem=1024]" -q long -R "rusage[mdcs=11:duration=15]"

j = 
Job
  Properties: 

                   ID: 1
                 Type: pool
             Username: Ronen
                State: queued
           SubmitTime: Fri May 15 11:34:11 EDT 2015
            StartTime: 
     Running Duration: 0 days 0h 0m 0s
      NumWorkersRange: [11 11]

      AutoAttachFiles: true
  Auto Attached Files: List files
        AttachedFiles: {}
      AdditionalPaths: {}

    Associated Tasks: 

       Number Pending: 11
       Number Running: 0
      Number Finished: 0
    Task ID of Errors: []

Files that are automatically added for the job with ID 1:
csecarea.m
csecplot.m
ixxc.m
massstress.m
math2code.m
nonlinbound.m
pcrit.m
saveDat.m
tester.m
C:\Users\Ronen\Google Drive\Work\Matlab Codes\circle.m
C:\Users\Ronen\Google Drive\Work\Matlab Codes\circlepolar.m

[Edric Ellis]

Your call to the batch command is incorrect. The first argument should be the name of the script without .m, like this:

j = c.batch('tester','Pool',10,'CurrentFolder','/home/ry54l/matlab')

[Ronen Yudzinsky]

...well, I guess that solves my issue. Thanks for catching that Edric!