Hidden quadratic problem

Question

Hints

Answer