Scale affine transformation [WIP]

[pabloppp]

I have been experimenting with how to add a couple of extra values for scaling the Fourier features. I still think my implementation is wrong because what should look like a smooth zoom-in looks more like if we were changing the FoV of the camera, so things in the center of the image scale at a slightly different rate. Some tweaks need to be done for this to work seamlessly, but the results are still pretty cool. With some refinement, would you consider adding this as a configurable parameter? (once i manage to make it work as I want, I could open a PR)

pe.mp4

pe.mp4

pe.mp4

[rosinality]

Interesting! But I wonder it will compatible with antialiasing.

[pabloppp]

Yeah, that could be a problem because by scaling the Fourier features you are altering the frequencies.
My thought was that, if we keep the frequencies below the cutoff value it should be fine, so I only allow scaling to values > 1 (so you can zoom in but not zoom out) by using a threshold.
The face does get distorted when zooming, but I don't think that is caused by aliasing, since I don't see wave/sticky effects. To me, this deformation seems to be caused by not applying properly the scaling, like if some frequencies were being scaled more than others, or idk...

pe.mp4

[rosinality]

I think achieving scale equivariance will be hard. (If it is possible) But definitely it will be very interesting & important development.

I think it will be nice if model could have options for frequency modulation.

[pabloppp]

Even if we do not achieve equivariance, and we do have those weird deformations, allowing the model to modify the scale can be beneficial, in the same way as your current configuration gives the model freedom to modify the rotation, but it's not rotation equivariant, but having this freedom allows the model to be much more flexible, right?

For example, take a look at this interpolation, the model is clearly taking advantage of the scale to deform the face into the shape it wants, and the result is still pretty smooth!

pe.mp4

pe.mp4

[rosinality]

Yes, of course! As this is further/additional improvements to the model, it would be nice to have it as a optional/additional model choices.

[MHRosenberg]

This last movie looks closer to what I'm aiming for. Is it possible to use that method to transition between entirely distinct faces? If so, I'd love to see what modifications to the code are necessary for that.

[pabloppp]

@MHRosenberg It should be possible, and the changes are quite simple:

First, make it so the self.affine_fourier linear layer returns 6 values instead of 4, two for rotation, two for shift, and two for scale (initialize the ones for the scale to 1).
Next, in the FourierFeature class you need to divide coord_h and coord_w by these new scale values, but first, make sure you don't normalize them, and second, you need to threshold them so they can't take a value lower than 1 (if they do, then you'll end up with frequencies above the cutoff and you'll have aliasing)

I did my own custom implementation of the FourierFeatures class based on the one from this repo (for personal purposes) here https://github.com/pabloppp/pytorch-tools/blob/master/torchtools/nn/fourier_features.py#L42 and I included the scale as an optional affine transformation, maybe you can use some of this code as a base.

The main issue that I found is that, for some reason, after training the scale was not homogeneous, meaning that if I scaled x1.5 on both axes, somehow the face ended up being longer, or wider, so I needed to maybe scale x1.5 on the height and x1.2 on the width or something like that. I also think that since when scaling the Fourier features you're changing the frequencies, in order for it to work seamlessly you'd need to also change the parameters of the AliasFree activation, to reflect those frequency changes, but I didn't try any of that yet. A 'hacky' solution is to train your GAN with images with a lot of RandomResidedCropping augmentation, so the model is forced to learn to properly make faces at different scales.

I would love to hear if you make any progress with this! 🙇