I wonder if you could not do some operator overloading on the TS side to do some rewriting to get things like tensor addition on tensor types.
Heck, if you are doing that, maybe convert to webgpu automatically as well.
Someone very enterprising might do this in bun using zig.