Results for
Wouldn’t it be great if your laptop AI app could convert itself into an agent for everything? I mean for your laptop and for the entire web? Done.
Setup
My setup is a MacBook with MATLAB and various MCP servers. I have several equivalent Desktop AI apps configured. I will focus on the use of Claude App but fully expect Perplexity App to behave similarly. See How to set up and use AI Desktop Apps with MATLAB and MCP servers.
Warning: My setup grants Claude access to various macOS system services and may have unforeseen consequences. Try this entirely at your own risk, and carefully until comfortable.
My MacOS permissions include
Settings=>Privacy & Security=> Accessibility
where matlab-mcp-core-server and MATLAB enabled, and in
Settings=>Privacy & Security=> Automation,
matlab-mcp-core-server has been enabled to access on a case-by-case basis the applications Comet, Messages, Safari, Terminal, Keynote, Mail, System Events, Google Chrome, Microsoft Excel, Microsoft PowerPoint, and Microsoft Word. These include just a few of my 85 MacOS applications available, those presently demonstrated to be operable by Claude. Contacts remain disabled for privacy, so I am AI texting carefully.
MCP services are the following:
Server
Command
Associated Tools/Commands
ollama
npx ollama-mcp
ollama_chat, ollama_generate, ollama_list, ollama_show, ollama_pull, ollama_push, ollama_create, ollama_copy, ollama_delete, ollama_embed, ollama_ps, ollama_web_search, ollama_web_fetch
filesystem
npx @modelcontextprotocol/server-filesystem
read_file, read_text_file, read_media_file, read_multiple_files, write_file, edit_file, create_directory, list_directory, list_directory_with_sizes, directory_tree, move_file, search_files, get_file_info, list_allowed_directories
matlab
matlab-mcp-core-server
evaluate_matlab_code, run_matlab_file, run_matlab_test_file, check_matlab_code, detect_matlab_toolboxes
fetch
npx mcp-fetch-server
fetch_html, fetch_markdown, fetch_txt, fetch_json
puppeteer
npx @modelcontextprotocol/server-puppeteer
puppeteer_navigate, puppeteer_screenshot, puppeteer_click, puppeteer_fill, puppeteer_select, puppeteer_hover, puppeteer_evaluate
shell
uvx mcp-shell-server
Allowed commands: osascript, open, sleep, ls, cat, pwd, echo, screencapture, cp, mv, mkdir, touch, file, find, grep, head, tail, wc, date, which, convert, sips, zip, unzip, pbcopy, pbpaste, ps, curl, mdfind, say
Here, mcp-shell-server has been authorized for a fairly safe set of commands, while Claude inherits from MATLAB additional powers, including MacOS shortcuts. Ollama-mcp is an interface to local Ollama LLMs, filesystem reads and writes files in a limited folder Documents/MATLAB, MATLAB executes MATLAB helper codes and runs scripts, fetch fetches web pages as markdown text, puppeteer enables browser automation in a headless Chrome, and shell runs allowed shell commands, especially osascript (AppleScript to control apps).
Operation of local apps
Once rolling, the first thing you want to do is to text someone what you’ve done! A prompt to your Claude app and some fiddling demonstrates this.
Now you can use Claude to create a Keynote, Excel presentation, or Word document locally. You no longer need to access Office 365 online using a late-2025 AI-assistant-enabled slower browser like Claude Chrome Extension or Perplexity Comet, and I like Keynote better.
Let’s edit a Keynote:
Next ,you might want to operate a cloud AI model using any local browser - Safari, Chrome, or Firefox or some other favorite.
Next, let’s have a conversation with another AI using its desktop application. Hey, sometimes an AI gets a little closed-minded and need a push.
How about we ask some other agent to book us a yoga class? Delegate to Comet or Claude Chrome Extenson to do this.
How does this work?
The key to agentic AI is feedback to enable autonomous operation. This feedback can be text, information about an application state or holistic - images of the application or webpage that a human would see. Your desktop app can screensnap the application and transmit that image to the host AI (hosted by Anthropic for Claude), but faces an API bottleneck. Matlab or an OS-dependent data compression application is an important element. Your AI can help you design pathways and even write code to assist. With MATLAB in the loop, for example, image compression is a function call or two, so with common access to a part of your filesystem, your AI can create and remember a process to get ‘er done efficiently. There are multiple solutions. For my operation, Claude performed timing tests on three such paths and selected the optimal one - MATLAB.
Can one literally talk and listen, not type and read?
Yes. Various ways. On MacOS, one can simply enable dictation and use hot-keys for voice-to-text input. One can also enable audio and have the response read back to you.
How can I do this?
My advice is to build it your way bottom-up with Claude help, and the try-it. There are many details and optimizations in my setup defining how Claude responds to various instructions and confronts circumstances like the user switching or changing the size of windows while operations are on-going, some of which I expect to document along with example helper codes on File Exchange shortly but these are OS-dependent and which will evolve.
Inspired in part by Christmas Trees, I'm curious about people's experience using AI to generate Matlab code.
1. Do you use AI to generate production code or just for experimentation/fun code?
2. Do you use the AI for a complete solution? Or is it more that the AI gets you most of the way there and you have to apply the finishing touches manually?
3. What level of quality would you consider the generated code? Does it follow "standard" Matlab coding practices? Is it well commented? Factored into modular functions? Argument checking? Memory efficient? Fast execution? Etc.?
4. Does the AI ever come up with a good or clever solution of which you wouldn't have thought or maybe of which you weren't even aware?
5. Is it easy/hard to express your requirements in a manner that the AI tool effectively translates into something useful?
6. Any other thoughts you'd care to share?
Give your LLM an easier time looking for information on mathworks.com: point it to the recently released llms.txt files. The top-level one is www.mathworks.com/llms.txt, release changes use www.mathworks.com/help/relnotes. How does it work for you??
Thank you to everyone who attended the workshop A Hands-On Introduction to Reinforcement Learning! Now that you all have had some time to digest the content, I wanted to create a thread where you could ask any further questions, share insights, or discuss how you're applying the concepts to your work. Please feel free to share your thoughts in the thread below! And for your reference, I have attached a PDF version of the workshop presentation slides to this post.
If you were interested in joining the RL workshop but weren't able to attend live (maybe because you were in one of our other fantastic workshops instead!), you can find the workshop hands-on material in this shared MATLAB Drive folder. To access the exercises, simply download the MATLAB Project Archive (.mlproj) file or copy it to your MATLAB Drive, extract the files, and open the project (.prj). Each exercise has its own live script (.mlx file) which contains all the instructions and individual steps for each exercise. Happy (reinforcement) learning!
Please share with us how you are using AI in your control design workflows and what you want to hear most in our upcoming talk, 4 Ways to Improve Control Design Workflows with AI.
Arkadiy
“Hello, I am Subha & I’m part of the organizing/mentoring team for NASA Space Apps Challenge Virudhunagar 2025 🚀. We’re looking for collaborators/mentors with ML and MATLAB expertise to help our student teams bring their space solutions to life. Would you be open to guiding us, even briefly? Your support could impact students tackling real NASA challenges. 🌍✨”
In the past two years, large language models have brought us significant changes, leading to the emergence of programming tools such as GitHub Copilot, Tabnine, Kite, CodeGPT, Replit, Cursor, and many others. Most of these tools support code writing by providing auto-completion, prompts, and suggestions, and they can be easily integrated with various IDEs.
As far as I know, aside from the MATLAB-VSCode/MatGPT plugin, MATLAB lacks such AI assistant plugins for its native MATLAB-Desktop, although it can leverage other third-party plugins for intelligent programming assistance. There is hope for a native tool of this kind to be built-in.
If you are interested in AI, Autonomous Systems and Robotics, and the future of engineering, don't miss out on MATLAB EXPO 2024 and register now.
You will have the opportunity to connect with engineers, scientists, educators, and researchers, and new ideas.
Featured Sessions:
- From Embedded to Empowered: The Rise of Software-Defined Products - María Elena Gavilán Alfonso, MathWorks
- The Empathetic Engineers of Tomorrow - Dr. Darryll Pines, University of Maryland
- A Model-Based Design Journey from Aerospace to an Artificial Pancreas System - Louis Lintereur, Medtronic Diabetes
Featured Topics:
- AI
- Autonomous Systems and Robotics
- Electrification
- Algorithm Development and Data Analysis
- Modeling, Simulation, Verification, Validation, and Implementation
- Wireless Communications
- Cloud, Software Factories, and DevOps
- Preparing Future Engineers and Scientists
The AI Chat Playground at MATLAB Central has two new upgrades: OpenAI GPT-4o mini and MATLAB R2024b!
GPT-4o mini is a new language model from OpenAI and brings general knowledge up to October 2023. GPT-4o mini surpasses GPT-3.5 Turbo and other small models on academic benchmarks across both textual intelligence and reasoning. Our goal is to keep improving the output of the AI Chat Playground. This upgrade is available now: https://www.mathworks.com/matlabcentral/playground/
One more thing... we also updated the system to the latest release of MATLAB. This is R2024b and comes with hundreds of updates and new plot types to explore.Check out Mike Croucher's blog post about the latest version of MATLAB: https://blogs.mathworks.com/matlab/2024/09/13/the-latest-version-of-matlab-r2024b-has-just-been-released/
We are looking forward to your feedback on the updates to the AI Chat Playground. Let us know what you think and how you use this community app.
I recently discovered a 2-minute video that introduces MatGPT, and I believe it's a resource worth sharing. The creator highlights MatGPT's impressive capabilities by demonstrating how it tackles the classic Travelling Salesman Problem.
With more than 13,000 downloads on File Exchange, MatGPT is gaining traction among users. I strongly recommend taking it for a spin to experience its potential firsthand.
The MATLAB AI Chat Playground is now open to the whole community! Answer questions, write first draft MATLAB code, and generate examples of common functions with natural language.
The playground features a chat panel next to a lightweight MATLAB code editor. Use the chat panel to enter natural language prompts to return explanations and code. You can keep chatting with the AI to refine the results or make changes to the output.
Give it a try, provide feedback on the output, and check back often as we make improvements to the model and overall experience.
In the past year, we've witnessed an exponential growth of ChatGPT and other Generative AI tools. AI has quickly become a transformative force across industries, from tech giants to small startups, and even community sites like ours. For instance, Stack Overflow announced its plan to leverage AI tools to draft a question or tag content; Quora built a ChatGPT bot to answer questions; and GitHub is piloting the AI tool for personalized content.
This trend in the community landscape makes me wonder what MATLAB Central community, especially in MATLAB Answers, can do to integrate AI and enhance the community.
Share with us your ideas in the comment session. Ideally one comment per idea, so that others can vote on a secific idea or have deeper discussions about it.
A research team found a way to trick a number of AI systems by injecting carefully placed nonsense -- for example being able able to beat DeepMind's Go game.
MathWorks is hosting the QIT Boston event to discuss Ethical AI for the LGBTQ+ community on its Lakeside campus, hosted by our own Heather Gorr, PhD.
AI is known to learn biases from the various sources that affects our daily lives, and in particular minority groups like LGBTQ+.
Join the discussion to what the impacts are, where the biases come from, and how to mitigate as you develop AI.
Sign up here
https://www.eventbrite.com/e/oit-boston-panel-discussion-ethical-ai-for-the-lgbtq-community-registration-496419814027
AI and ACoP, hype or real?
