Why doesn't concatLayer in Deep Learning Toolbox concatenate the 'T' dimension?

Hello,

While implementing a ViT transformer in Matlab, I found at that the concatLayer does not concatenate over the T dimension. This is needed to concatenate the class token with patch tokens, since the natural representation is CBT with C corresponding to features, B to batch and T to token within a batch (this is also the canonical representation in the attention function).

It's possible to work around this by hacking to e.g. SCB, but then other problems pop up which also need to be hacked around.

Thx

Answer 1

[Ben]

You can create a layer that concatenates on the T dimension with functionLayer

sequenceCatLayer = functionLayer(@(x,y) cat(3,x,y));

This will work in dlnetwork to concatenate two CBT dlarray-s.

Since you're concatenating the class token, it might also be worth considering creating a custom layer that has the class token embedding as a Learnable property, and performs the concatenation in the predict method.

[John Smith]

Thx.

That's what I ended up doing. It'd be helpful if Matlab supported this natively.

[Catalytic]

@John Smith - Since Ben's answer yielded a solution for you, you should hit the Accept this Answer button, and likewise with other answers you might not have accepted.

[Artem Lensky]

Are there any plans to make concatenationLayer support concatetnation along the T dimension?