Three by three matrix problem

Question

Hints

Answer