2

I'm trying to implement the gradient descent method for the problem of minimising the following function:

$$f(x) = \frac{1}{2}(x-m)^{T}A(x-m)-\sum\limits_{i=1}^n\log\left(x_i^{2}\right),$$

where $x \in R^n$ is a vector; $m \in R^n$ is a fixed vector; and $A$ is a fixed positive definite matrix.

The only applications of gradient descent I have come across is for linear regression! So, as a starting point for helping me to solve this, I'd like to know in what situations this cost function would be applied. Does anyone out there recognise it?

Davide Giraudo
  • 172,925
Barton
  • 67
  • Your objective function is a special case of a quadratic cost function with a regularization term that uses the $\log$ of the coefficients. $L_1$ or $L_2$ regularization cost functions are more typical, but you can impose a gentler penalty with a $\log$ cost function. – Godfather Nov 12 '18 at 22:25
  • 1
    Thanks @AdityaDua. This response has been useful to me – Barton Nov 14 '18 at 21:52
  • @AdityaDua do you know what the role of the matrix A would be in this cost function? – Barton Nov 17 '18 at 17:48
  • I'd suggest reading up on "generalised least squares". The matrix A allows you to deal with correlated errors and unequal variances (Heteroscedasticity). – Godfather Nov 18 '18 at 03:09

1 Answers1

1

From the question, we have that f is a convex quadratic function

$$f(x) = \frac{1}{2}(x-m)^{\mathrm T} \mathrm A (x-m)-\sum\limits_{i=1}^n\log\left(x_i^{2}\right),$$

where $x \in R^n$ is a vector; $m \in R^n$ is a fixed vector, and $A$ is a fixed positive definite matrix (symmetric and positive semidefinite).

$$\nabla f ( x) = \frac{\mathrm T+1}{2} \mathrm A (x-m)^{\mathrm T}-\sum\limits_{i=1}^n\left(\frac{2x}{x_i^{2}ln(10)}\right)$$

Using gradient descent with step $\mu$,

$$ x_{k+1} = x_k - \mu \nabla f ( x_k)$$

Choose $\mu$ such that $f(x_k) < f(x_{k+1}) $,then do a loop until we find $x^{*}: f(x^{*}_k) - f(x^{*}_{k+1}) \sim 0$

This is a general way to take gradient descent for a convex quadratic function in n-dimensional space. Hope it is helpful.

AnNg
  • 45