Dependency management & package manager like pip in ML

Hi all,

Matlab is officially still lacking a proper package manager like npm or pip. I do not understand how people structure projects with dependencies. I understand, there are

• ML projects
• ML toolboxes
• several mpm versions

Toolboxes have a version number attached, projects have not. Both can be shared as a package. Projects can reference other projects, but that 1. requires the same folder structure on every developer PC and 2. requires that the correct version is checked out from the VCS (which is GIT in our case). There's no way to define "project abc@v1.2.0" as a dependency. Just folders/prj files.

How do you do complex app developments? Like a frontent GUI app that needs several self-created toolboxes and projects? The only way is to use GIT (submodules in worst case) to do dependency managment, and that's definitely a red flag.

Then, there's Matlab's mpm matlab-dockerfile/MPM.md at main · mathworks-ref-arch/matlab-dockerfile (github.com) but that's meant to be used to install different product versions and addons, and there's a third party mpm mobeets/mpm: Simple Matlab package management inspired by pip (github.com) which seems to be interesting, but there's no direct support from Mathworks.

Then, it seems like there's another mpm built-in in ML 2022b that mentions a Matlab -packages flag which is very interesting:

mpm

Error using mpm
This command is not supported when the packages feature is not enabled. Start MATLAB with the -packages command line switch.

There's no information about that. Starting ML with -packages reveals more:

Which seems to support different versions. But there's no documentation anywhere, and it's a built-in function © 2022.

What does that mean? Will there be a Mathworks' supported package manager? What exactly is a package by the way in ML? So far, packages were a synonym to namespaces, using the +package folders. How does that work together with ML projects?

I'm in the process of re-structuring code due to a migration from Bitbucket to Gitlab and I'm at a point where a proper dependency management must take place, and that must work in CI/CD, too.

Would be very happy to hear your thoughts.

Thanks,

Jan

[Jan Kappen]

Any updates from Mathworks?

I'd really like to know if theres something on its way or if I have to create my own dependency manager, probably in Python.

Thanks

Answer 1

[Michelle Hirsch]

We are working on a package management system for MATLAB. The mpm comand you found in MATLAB was from an early beta test we ran in R2022b to flesh out some of the basic principles of the system we are designing.

I think the best bet would be for us to connect you with an appropriate expert or two at MathWorks to talk through ideas of how to best approach meeting your needs now while setting yourself up to adopt our system when it is ready. I'll email you through your author profile to make the connection.

[Ernst van der Pols]

Thanks Jan for raising this question and Michelle for giving an update.

We at KROHNE have a Nexus server for storage and distribution of software artifacts, and are still in the process of introducing this technology in the various software-creating communities within the company. Mainly .NET (nuget), and a little PowerShell (nuget) and Python (pip) are already there.

Leveraging this for our MATLAB software would mean a great boost for improving the quality and ease of sharing the abundance of MATLAB scripts and models.

I looked into some options to distribute MATLAB Toolbox packages via Nexus. Since a MATLAB Toolbox (.mltbx) is an OPC container, I tested the combination of cloaking an mltbx as NuGet package, which uses also OPC technology. With cloaking I mean building the package such that it can be handled by both MATLAB and NuGet, with at most only a manual fix of the package file name extension (from .nupkg to .mltbx vv.). Alas, this had no satisfactory results.

NuGet is already not only used for .NET packaging, but also for PowerShell packaging. Currently using the Semantic Versioning standard and a rather simpel Software-Bill-Of-Material (nuspec), it is easy to use, and to integrate in a build process.

Next best option would be to wrap the mltbx in a NuGet package, but before starting that up we first focus on the creation of a Toolbox in the first place. Hopefully we can expect some managed versioning solution that we can employ in-house soon.

[Michelle Hirsch]

@David Sampson do you have any advice you could offer Ernst? I'm guessing you've supported others using NuGet.

[Jan Kappen]

@Michelle Hirsch I'd be very interested in that, too.

[Michelle Hirsch]

@Ernst van der Pols in cases it's helpful, here's some input I received from one of MathWorks consultants who has helped a lot of users set up package management tooling. He hasn't worked directly with NuGet with MLTBX, but has heard from some users who have. Here are his observations:

• It’s pretty simple to put an mltbx in a NuGet package, I don’t know of anyone making an mltbx be a NuGet package.
• NuGet packages can have dependencies and pull them in automatically.
• There is no requirement in .NET for one version of any given library, so it makes no effort to resolve dependency ranges.
• NuGet will allow you to search packages and pull the mltbx files, plus dependencies, into a folder, but can’t install them.
• You need to write a bunch of MATLAB code to take the folder of mltbxs and install them (and with dependencies, work out which ones to install)
• And more code to avoid downloading everything every time.

Let me know if you'd like to get on a call to discuss how our consultants might be able to help you in the short term, and to learn more about your usage to inform our future development plans. We do expect to support integration between the forthcoming MATLAB package management system and artifact repositories, but don't have a timeline yet.

[Ashley Trowell]

I'm glad to see activity on this.

[Jan Kappen]

Just FYI, I'm very happy that the package manager has arrived (I'm on R2024b, but might be available in R2024a already): Package Management - MATLAB & Simulink

Only missing item is the Git integration. And a self-hostable package/registry server in GitLab (similar to the mentioned NuGet integration). @Ernst van der Pols have you progressed with the NuGet server? Any experience with the package manager in MATLAB?

Thanks :)

[Michelle Hirsch]

Thanks, @Jan Kappen. What do you have in mind with Git integration, beyond a self-hostable package registry server in GitLab or the like? MATLAB already has decent git integration for the development of the package source code. I do think that the biggest missing piece is flexible support for package repositories beyond just a shared file system, e.g. Artifactory, or a GitLab package registry.

[Jan Kappen]

Dear @Michelle Hirsch,

> I do think that the biggest missing piece is flexible support for package repositories beyond just a shared file system, e.g. Artifactory, or a GitLab package registry.

I agree :)

Until this exists, a local package meta data repository could make sense, something like a big json file telling MATLAB "this package is available in this and that version and can be found under this git link + revision". If the package does not exist in a local cache, it gets downloaded.

That way, we would avoid that all packages must be accessible from all MATLAB PCs, and instead just the meta data file can be distributed.

Or even easier: have a dependency list in each project with the link to git in the current version (url + rev), similar to buildinspace/peru: a generic package manager, for including other people's code in your projects

[Jan Kappen]

Dear @Michelle Hirsch are there any shareable news regarding the self-hosted pacakge registry on Artifactory or GitLab? I'd like to avoid developing something on my own and get it from MathWorks in the next release ;)

Thanks!

[Michelle Hirsch]

Sorry, but we don't yet have a firm date for either of these.

[Ernst van der Pols]

Dear @Jan Kappen and @Michelle Hirsch,

Sorry for my delayed response, finally found time to look into the new Package Manager. And I must say that it fulfills my expectations to a great extent, as I hoped for in my request last year. So, thank you MathWorks!

Two features are, however, missed:

1 - As you already mentioned, the repository side is currently rather bare.

• Having the packages zipped and a kind of index file would already improve the file system based repository.
• Being able to locate and access the repository on an web-based server is the main missing feature for us to use the MPM on our company network.

2 - Integration with the matlab GUI

• Having the installed packages show up in the Add-on of similar list
• Having apps in the package being installed, like when they are packaged in a Toolbox

To summarize my own progress: I did not pursue on the NuGet solution, because our project development turned a bit in another direction. Instead of building an embedded software package (like a NuGet package) with the matlab front-end software enclosed, we are now in a multidisciplinary team (physicists, hw/sw engineers) all primarily writing matlab code, including device drivers and the like, aiming at code generation for the final embedded solution. So, for now we have chosen to use Toolboxes as package container with manual dependency management at build time. Users have a simple installation experience, this way.

As long as we have just a few toolboxes, this can work. But for the long term the MPM will be essential!

--

When testing, I detected a small issue with the addFolder() function in the current MPM (R2024b Update 5):

>> cd \data\matlab

>> mkdir test

>> cd test

>> mkdir doc

>> mkdir test

>> pkg = mpmcreate("TestPackage",".",IncludeSubfolders=true,Install=false)

pkg =

Package with properties:

Package Definition

Name: "TestPackage"

DisplayName: "TestPackage"

Version: 1.0.0 (1×1 Version)

Summary: ""

Description: ""

Provider: "" (1×1 Provider)

Folders: [../doc ../test] (1×2 PackageFolder)

Dependencies: ""

ReleaseCompatibility: ""

FormerNames: [1×0 string]

ID: "13f6dc57-8665-4e12-8b0a-729d44380a97"

Package Installation

Installed: 0

Editable: 1

InstalledAsDependency: 0

PackageRoot: "D:\data\matlab\test"

InstalledDependencies: ""

MissingDependencies: ""

Repository

Repository: [0×0 Repository]

help TestPackage

>> mkdir source

>> pkg.addFolder("source")

Error using matlab.mpm.Package/addFolder (line 14)

Unable to modify Package 'TestPackage' located at 'D:\data\matlab\test'.

Caused by:

Error using matlab.mpm.Package/addFolder (line 12)

The folder '../doc' does not exist relative to package root folder 'D:\data\matlab\test'.

The "../doc" is relative to the "./resources" folder, as specified in the json file there. "source" is relative to the package root, the current directory, so there must be something wrong there.

[Michelle Hirsch]

Thanks for the response and all of these details, @Ernst van der Pols.

Regarding the missing features:

Package repository:

• Could you elaborate on why packages being zipped would be an improvement? And what benefits you'd expect from an index file?
• Can you say any more about what your web-based repository might look like? e.g. are you using something like sharepoint or s3 to host web-accessible files, or something like Artifactory for managing artifacts accessible via a URL endpoint?

Integration with MATLAB UI:

• Installed packages will be integrated with the list of add-ons starting in R2025a
• Installing apps, like we do with toolbox: Good suggestion. I'll pass this along to the team.

The addFolder error: I see the same behavior on my machine in a test version of R2025a. I'll bring this to the team and will let you know if I learn anything.

[Ernst van der Pols]

Thanks @Michelle Hirsch for the reply and the update on the integration of the packages in the UI in the upcoming release. That will make things more clear and easy for colleagues (and myself as well).

Zipped packages:

• Semantically: a package ("an object or group of objects wrapped in paper or plastic, or packed in a box") is typically something wrapped in an enclosure, to protect it during transport.
• Practically: the package content in the repository is both transmitted over time (the storage function of the repository) and transmitted over space (to client machines); in both cases a zipped container has the obvious benefits of size reduction and integrity control.

I know digest is added for checking package integrity, but zip container typically allows for secure transport as well with signed certificates.

In MATLAB the OPC zipfiles are widely used. The Wikipedia only mentions .slx files, but toolboxes, live scripts, etc. could be mentioned as well. So, perhaps I could state that a package is currently the odd one out collection of related files in the MATLAB environment.

Index file or server benefits:

• Scalability: imagine a repository with 1000 packages with each e.g. 100 versions. Without an index of some kind, worst case mpmsearch or mpminstall have to read 100000 files to find the single package you are looking for. And then again for the dependencies.

web-based repository server:

• As mentioned earlier, we have a Sonatype Nexus repository for storing our binary packages and as gateway for onlie repositories. It is similar to Artifactory. It supports a large number of package formats and companion transfer protocols, among which NuGet (for .NET and Powershell) , PyPi (for Python), and so on.
• It also supports "raw" files, meaning that you download the files simply via their "folder" URL on the server. That method can be used for MATLAB Toolboxes: dependent toolboxes can be automatically installed via a web URL.

For local teams, having a fileserver-based repository can work out, but for company-wide sharing we prefer the Nexus server.

[Michelle Hirsch]

Thanks, @Ernst van der Pols. This is all quite helpful. I'll pass it along to the team.

I agree that lack of support for proper file system repositories is still the biggest gap with the package manager. We knew that before we shipped - our interviews suggested maybe half of interested users could at least make do with a file system repository. I wish support for Artifactory and others were coming faster, but it's taking a while. The teams are putting a lot of effort into unifying our underlying installation technology to ensure we have a very solid foundation that can support installing MathWorks products and user-authored packages long into the future.

[Ernst van der Pols]

I fully understand, @Michelle Hirsch, the problems that have to be solved for such a feature and perhaps some frustration on your side just offering "half" a solution. Although, I can only image the effort it takes with each new release of MATLAB to have all the diverse topics and features again in a trustworthy product that so many peoply in the world rely on. And that already over such a long period of time. So, don't worry on taking some time for it. We are still in the process of making colleagues aware of the benefits of having tested matlab code available from a shared location, and the Toolbox is already helping in that way.