Considering the Complex Least Squares Problem, one can write the cost function for estimating H (where Y = XH + Z) as:
Reference: Mimo-OFDM Wireless Communications with MATLAB book (page 190)
$J(\boldsymbol H) = ||\boldsymbol Y - \boldsymbol X \boldsymbol H ||^2 = (\boldsymbol Y - \boldsymbol X \boldsymbol H)^H(\boldsymbol Y - \boldsymbol X \boldsymbol H) = \boldsymbol Y^H \boldsymbol Y - \boldsymbol Y^H \boldsymbol X \boldsymbol H - \boldsymbol H^H \boldsymbol X^H \boldsymbol Y + \boldsymbol H^H \boldsymbol X^H \boldsymbol X \boldsymbol H$
Calculating the derivative of the above function with respect to H gives:
$\frac{\partial J( \boldsymbol H)}{\partial \boldsymbol H} = -2(\boldsymbol X^H \boldsymbol Y)^* + 2(\boldsymbol X^H \boldsymbol X \boldsymbol H)^*$
How do I get to the expression above? Do I need to use some properties? I am trying to do that deriving each term separately.