How can I record mouse clicks (left and right) with coordinates and time?

Hello,

I am trying to capture mouse movements outside of the GUI, on second display while users are using another program. So far, I manage to record the coordinates, every 0.1 seconds period, by using timer function. I need to add mouse clicks (left or right button down and ups). Does anyone know how can I do this? I try to Buttondownfnc but it only works with a figures not outside of the Matlab..or is there any workaround?

fileID = fopen('motion.txt','wt');

t = timer('ExecutionMode', 'fixedRate','Period', 0.1, 'TasksToExecute', 200, ... 
          'TimerFcn', @(~,~) fprintf(fileID,'(X, Y) = (%g, %g)\n', get(0, 'PointerLocation')));
start(t);

[Adam Danz]

I'm not aware of a way to capture mouse clicks in Matlab outside of clicking on graphics objects produced by Matlab.

[Doli Swey]

Maybe buttondownfnc can be assign in other application rather than figure? I am trying to find a way..

[Adam Danz]

The buttondownfcn is a propery of the object it's assigned to and will only respond to interactions with that object.

Example (click inside and outside the figure)

fig = figure();
fig.ButtonDownFcn = @(~,~)disp('hit')

[Doli Swey]

can we assign a program as object? or else maybe create transparent object (figure as screen szie) which goes onto the program that I am using? what do you think Adam?

[Adam Danz]

No to both questions 😕.

> can we assign a program as object?

That's what OOP is for (see Create a Simple Class). But ButtonDownFcn's are only properties of graphics objects.

> or else maybe create transparent object...

Axes can be transparent but figures cannot be transparent. You can't have an axis without a figure. Since you must click on the object for a buttondownfcn to respond, the object cannot be under something else.

If you add some concise details to explain the goal I might be able to think of some stuff to try but it doesn't sound like Matlab has anything that would work with what I imagine you're trying to do. There are definitely 3rd party mouse tracking software that records and stores mouse clicks.

[Doli Swey]

Okay. I am trying to record the movement of subjects when they are drawing on a specific drawing tablet (Wacom One) with pen. We are using one of the Adobe softwares for drawing. I would like to know duration and coordinates of the movements, therefore I want to record the coordinates of the mouse and clicks (right/left button up and down). Although, some of the 3rd party macro recorders are incredible, somehow they are failing to record mouse clicks inside of the Adobe program. I have tried 18 different programs so far. Thats why I am looking a solution on Matlab, maybe I can figure this problem by some workaround.

I just found this: https://ch.mathworks.com/matlabcentral/fileexchange/31437-windowapi Trying to figure out if I can make a transparent figure with this. Do you have any knowledge on this? Do you think this would be a good option to give it a try?

[Mario Malic]

Some of the Adobe programs support the Active X controls IIRC, maybe your software supports that, if it does, maybe there are methods that track mouse movement.

[Doli Swey]

Hi Mario, thank you for your suggestion. What is Active X controls IIRC? I dont know anything about it and i couldnt find so much information online. Can you maybe give me a link to understand?

Answer 1

[Jan]

You can emulate this with WindowsAPI. Setting the window completely invisble does not work, because with Alpha=0.0 the mouse events are not caught anymore. But with Alpha= 0.01 you cannot see the window anymore, but clicks are still caught:

FigH = figure;
FigH.ButtonDownFcn = @(FigH, Event) disp(Event);
drawnow;
% Set inner window covers complete screen on monitor m'th monitor:
m = 1;
WindowAPI(FigH, 'Position', 'full', m);

% Allmost transparent (I do not see anything, do you):
WindowAPI(FigH, 'Alpha', 0.01);

% Crop window border, such this does not cover the margins
% of other monitors:
WindowAPI(FigH, 'Clip', true);

% Set window to top or catch at least the focus:
WindowAPI(FigH, 'Top');
WindowAPI(FigH, 'SetFocus');

for k = 1:100
    pause(0.1)
    disp(get(groot, 'PointerLocation'))
end
delete(FigH);  % Important: Otherwise the window steals all events!

[Doli Swey]

Dear Jan, thank you so much for your answer. This code is working quite well for creating transparent figure and collecting the coordinates. However, since it is running on top of the application, it is not allow me to use the application. And when I dont make it on top, but still catch the focus, it is not recording anymore. Is there any way to run this on the background, so I can still use the application?

[Jan]

You can catch the mouse position and the clicks in the almost transparent Matlab window at first. Then you give the focus to the wanted Window and use jawa.awt.Robot to repeat the recorded clicks. Or maybe the WindowAPI can forward the clicks also.

I've tried to implement this based on : https://www.mathworks.com/matlabcentral/answers/408669-how-do-i-close-a-desktop-window-using-matlab-commands but was not successful yet to insert:

SendMessage(childWindow, WM_LBUTTONUP, new IntPtr(MK_LBUTTON), NULL);

[Doli Swey]

Hı Jan, thank you for your reply. But I dont think this will work in my case. Briefly, I would like to record the movement while a person drawing in tablet, one of those Adobe softwares. So I have to find a way to run the code behind while they are drawing, and analyze the movement. If I put transparet Matlab window, Adobe software doesnot work.. Do you have any other suggestions that I can find a workaround?

[Jan]

My suggestion would be to record the mouse clicks in the transparent window, move the Acrobat window to top an to emulate the formerly recorded mouse clicks.

For fast double clicks this might fail. With SendMessage there is no need for a time consuming movement of the Acrobat window to the top. But as said already, I did not get the Mex-function to work.

There are some mouse recording softwares, which track the mouse position and the clicks. Do you have a good reason to do this in Matlab?

[Doli Swey]

Yes because I am using tablet with pen, the third party mouse recorders somehow do not track the touch (left clck) and pen button (right click).

My suggestion would be to record the mouse clicks in the transparent window, move the Acrobat window to top an to emulate the formerly recorded mouse clicks.

I have tried but it is not possible to move the Adobe software on top of the transparent window once I start run the script. I will try to find a workaround with the code you sent. Thank you so much. I will keep you updated.

Appriciate the help!