zlacker

>>daniel+(OP)
It’s hard to elaborate just how wild this model might be if it performs as claimed. The claims are this can perform close to Sonnet 4.5 for assisted coding (SWE bench) while using only 3B active parameters. This is obscenely small for the claimed performance.

>>skhame+l9
I experimented with the Q2 and Q4 quants. First impression is that it's amazing we can run this locally, but it's definitely not at Sonnet 4.5 level at all.

Even for my usual toy coding problems it would get simple things wrong and require some poking to get to it.

A few times it got stuck in thinking loops and I had to cancel prompts.

This was using the recommended settings from the unsloth repository. It's always possible that there are some bugs in early implementations that need to be fixed later, but so far I don't see any reason to believe this is actually a Sonnet 4.5 level model.

>>Aurorn+A21
> I experimented with the Q2 and Q4 quants.

Of course you get degraded performance with this.

>>cubefo+Fa1
Obviously. That's why I led with that statement.

Those are the quant thresholds where people with mid-high end hardware can run this locally at reasonable speed, though.

In my experience Q2 is flakey, but Q4 isn't dramatically worse.

>>Aurorn+Ot1
> Obviously. That's why I led with that statement.

Then why did you write this?

> It's always possible that there are some bugs in early implementations that need to be fixed later, but so far I don't see any reason to believe this is actually a Sonnet 4.5 level model.