The ability to programmatically check whether MATLAB has a display is not available in MATLAB 7.6 (R2008a).
As a workaround, you can do one of the following:
1. Manually set or unset an environment variable in your shell script that launches MATLAB, so you can query it from inside MATLAB. For instance, setting the ISDISPLAY environment variable in your shell script before launching MATLAB:
setenv ISDISPLAY no
matlab -r "foo; quit"
(here shown with C Shell syntax), means that the MATLAB command
will return the string 'no' in any MATLAB processes that are spawned from this shell.
2. Query the "ScreenSize" property of the root object inside MATLAB:
When there is no display, this returns [1 1 1 1] instead of an actual screen size.
Note that this will only work in releases prior to R2014b.