> Generally, it's best to have this optimization be done automatically by an optimizing compiler.

What compiler should be optimizing this operation?

There are comments on the commit reporting errors under certain conditions.

A "safe" way to do this is still straightforward, I think.

    from copy import copy
    def _forward(self, x, context=None):
        x = x.contiguous() if x.device.type == 'mps' else x
        x = copy(x)
        x += self.attn1(self.norm1(x))
        x += self.attn2(self.norm2(x), context=context)
        x += self.ff(self.norm3(x))
        return x

The problem here comes down to not knowing whether you’re allowed to modify a value in-place or not, because it’s not clear who owns it: it wasn’t written down anywhere, and in stable-diffusion alone it was fine to mutate it, but textual-inversion did something so it wasn’t (perhaps passing it something it expected to not be mutated). This is a moderately common type of bug that can be extraordinarily difficult to diagnose—it’s unusually easy to pinpoint here because it promptly raises a RuntimeError—and which is statically impossible in Rust, because the whole “am I allowed to mutate it” thing is resolved in the type system.

If the engineers that originally implemented the function intentionally chose the slower version, a quick comment as to why would have prevented this from happening in the first place.

    z = y + x
    x = z

`x += y` just adds y to x directly without any intermediary.

You could write this in any language pretty easily. For example, in Rust:

    let x = "abc".to_string();
    let y = "123".to_string();
    let x = x + &y;

    let mut x = "abc".to_string();
    let y = "123".to_string();
    x.push_str(&y);

The problem is thinking `+' and `+=' are the same, they are not and `+' should not be used when `+=' can be used.

Btw there's ongoing work to automatically optimize expressions like this. See the XLA compiler for example. Right now deep learning has a ton of seemingly obvious compute/memory optimisations that are not done automatically.

The Python part only runs a handful of times so JIT vs. non-JIT doesn't really make a difference.

https://old.reddit.com/r/MachineLearning/comments/xa75km/p_p...

The difference is that `x += y` modifies `x` inplace, where `x = x + y` creates a new object. In other words, if anybody had a reference to `x` before the update, the "optimized" code would break things.

Even so, there are absolutely silly things which can hint JS JITs to optimize (or to not deoptimize). Like defining and instantiating a class rather than just creating POJOs with the same values, or assigning NaN instead of null to uninitialized numeric variables/properties. Conditional control flow can deopt, but generally performs better around different function calls than within a single function. Even creating and throwing errors for control flow (which is generally expensive, and terrible for maintenance) can be optimal if your try/catch is the whole body of the function it resides in. And all of those might vary between JITs.

It would be interesting to get a benchmark using CPython vs Nuitka related to this change.

[1] https://github.com/Nuitka/Nuitka

  x = x + y

  x += y

https://github.com/CompVis/stable-diffusion/blob/69ae4b35e0a...

This is rarely an option that has presented itself to me. If there's a clear performance issue in my code then I probably picked the wrong algorithm or my code has a bug, unless you decided for some reason to do heavy calculations in raw python. If you're doing operations on big chunks of data you should always use something like numpy or jax.

Even OPs issue the clear reason is that it's doing an operation in place instead of creating a copy, for ML models this can only be done at inference time and not training time since you need to keep track of the whole network, hence why the code was in it's unoptimized state.

Jokes aside this is pytorch so this is compiled to C++ or cuda, the problem likely comes from the different functions that are called for += vs +

    >>> def f(x): x += 1
    ... 
    >>> def g(x): x = x + 1
    ... 
    >>> dis.dis(f)
      1           0 LOAD_FAST                0 (x)
                  3 LOAD_CONST               1 (1)
                  6 INPLACE_ADD         
                  7 STORE_FAST               0 (x)
                 10 LOAD_CONST               0 (None)
                 13 RETURN_VALUE        
    >>> dis.dis(g)
      1           0 LOAD_FAST                0 (x)
                  3 LOAD_CONST               1 (1)
                  6 BINARY_ADD          
                  7 STORE_FAST               0 (x)
                 10 LOAD_CONST               0 (None)
                 13 RETURN_VALUE