This is with the inverted matrix. It doesn't look like this is right at all.

There are some individual sky maps, with the option set to zero.

There are some individual sky maps, with the option set to -1.67.

These are the averages over 200 skymaps