ML lec 03 - Linear Regression의 cost 최소화 알고리즘의 원리 설명

Simplified hypothesis

H(x) = Wx

$cost(W)=\frac{1}{m}\sum _{i=1}^{m}W{x}^{(i)}-y^{(i)}$

 

 

W = 1, cost(W) = 0

$\frac{1}{3}((1\ast 1-1{)}^2+(1\ast 2-1{)}^2+(1\ast 3-1{)}^2)$

W = 0, cost(W) = 4.67

$\frac{1}{3}((0\ast 1-1{)}^2+(0\ast 2-1{)}^2+(0\ast 3-1{)}^2)$

W = 2, cost(W) = 4.67

$\frac{1}{3}((2\ast 1-1{)}^2+(2\ast 2-1{)}^2+(2\ast 3-1{)}^2)$

What cost(W) looks like?

$cost(W)=\frac{1}{m}\sum _{i=1}^m(W{x}^{(i)}-y^{(i)}{)}^2$

Gradient descent algorithm

• Minimize cost function
• Gradient descent is used many minimization problems
• For a given cost function, cost(W, b),it will be find W, b to minimize cost
• It can be applied to more general function: cost(w1, w2, ...)
• Gradinet descent algorithm(경사하강 알고리즘) cost를 최소로 만드는 W,b를 찾는다. 일반적인 함수에 적용된다.

How it works?

• Start with initial guesses
  • Start aat 0,0 (or any other value)  //아무 곳에서나 시작할 수 있다
  • Keeping changing W and b a little bit to try and reduce cost(W,b) // W,b를 조금씩 변경해서 cost를 줄인다
• Each time you change the parameters, you select the gradient which reduces cost(W,b) the most possible // 
• Repeat //반복
• Do so until you converge to local minimum 
• Has an interesting propery
  • Where you start can determine which minumum you end up

미분

$$cost(W)=\frac{1}{m}\sum _{i=1}^m(W{x}^{(i)}-y^{(i)}{)}^2$$

$$V$$

$$cost(W)=\frac{1}{2m}\sum _{i=1}^m(W{x}^{(i)}-y^{(i)}{)}^2$$

 

Formal definition

$$cost(W)=\frac{1}{2m}\sum _{i=1}^m(W{x}^{(i)}-y^{(i)}{)}^2$$

$$W:=W-\alpha \frac{\partial }{\partial W}cost(W)$$

 

cost(W) : 기울기 '-' 작은 쪽으로 움직이겠다 W가 큰값으로 움직이겠다.

$$W:=W-\alpha \frac{\partial }{\partial W}\frac{1}{2m}\sum _{i=1}^m(W{x}^{(i)}-y^{(i)}{)}^2$$

$$W:=W-\alpha \frac{1}{2m}\sum _{i=1}^{m}2(W{x}^{(i)}-y^{(i)})x^{(i)}$$

$$W:=W-\alpha \frac{1}{m}\sum _{i=1}^m(W{x}^{(i)}-y^{(i)})x^{(i)}$$

 

cost함수의 모양이 Convex function 이어야한다.  그래야 어디서든 정답에 도달한다.