Calculus

Here are a bunch of random things related to calculus.

Lagrange Multipliers

Consider a function \( f: \mathbb{R}^n \to \mathbb{R} \) and a constraint function \( g: \mathbb{R}^n \to \mathbb{R} \). Then, consider an optimisation problem of finding the maxima/minima of \( f \) subject to the constraint \( g(x) = c \) for some constant \( c \). This is equivalent to solving the following system of equations: $$ \begin{cases} \nabla f(x) &= \lambda \nabla g(x) \\ g(x) &= c \end{cases} $$ where \( \lambda \) is TBD and is known as the Lagrange multiplier.

Now, why did we add the first constraint? Suppose that \( f \) intersects \( g(x) = c \) at some point \( p \). Then, because \( g(x) \) is a level set at \( c \), the gradient \( \nabla g(p) \) is orthogonal to the level set at the point \( p \). This is a simple result by differentiation along the level set.

Assume instead that \( \nabla f(p) \) is not orthogonal to the level set of \( g(x)=c \) at \( p \), then we can take a unit tangent vector along the level set of \( g(x) = c \) at \( p \) which we denote by \( u \). Then, we have the directional derivative of \( f \) at point \( p \) along direction \( u \) given by $$ \nabla f(p) \cdot u = \| \nabla f(p) \| \cos \theta $$ where \( \theta \) is the angle between \( \nabla f(p) \) and \( u \). Here, \( \theta \neq \frac{\pi}{2} \) due to our assumption and therefore there exists a value along the level set of \( g \) at \( p \) where the directional derivative is positive and non-zero for \( f \). This is a contradiction and therefore \( \nabla f(p) \) is orthogonal to the level set of \( g \) at \( p \) and thus it is parallel to \( \nabla g(p) \).

Jacobians

Consider a coordinate system \( (x_1, x_2, \ldots, x_n) \) in \( \mathbb{R}^n \) and a function \( f: \mathbb{R}^n \to \mathbb{R} \). The (differential form) Jacobian matrix of \( f \) at \( p \) is defined as the matrix of all first-order partial derivatives of \( f \): $$ J_{f(p)} = \begin{bmatrix} \frac{\partial f}{\partial x_1} & \frac{\partial f}{\partial x_2} & \cdots & \frac{\partial f}{\partial x_n}. \end{bmatrix} $$ We can integrate \( f \) over an open region \( D \subseteq \mathbb{R}^n \) to obtain the volume under the surface defined by \( f \) over the region \( D \): $$ V = \int_D f(x) \, dx.$$ Suppose we have a homeomorphism between \( D \) and another open region \( D' \subseteq \mathbb{R}^n \) defined by a smooth function \( \phi: D \to D' \). We can write $$ \phi(x) = ( \phi_1(x), \phi_2(x), \ldots, \phi_n(x) ) $$ where each \( \phi_i: D \to \mathbb{R} \) is a smooth function and represents in a change in coordinate from \( x \mapsto \phi(x) \). By Taylor expansion, we have $$ \phi^i(x+dx) = \phi^i(x) + \sum_{j=1}^n \frac{\partial \phi^i}{\partial x_j}(x) \, dx_j + O(\|dx\|^2) $$ and thus we can approximate the change in \( \phi \) by $$ d\phi = J_\phi(x) \, dx$$ where \( d\phi = (d\phi^1, d\phi^2, \ldots, d\phi^n) \) and not the differential form. Note that the Jacobian is a linear transformation \( J_{\phi(p)}: \mathbb{R}^n \to \mathbb{R}^n \) at each point \( p \in D \) and thus by considering an infinitesimal hypercube at point \( p \) with side lengths \( dx_1, dx_2, \ldots, dx_n \), we can see that the transformation of the hypercube under \( J_{\phi(p)} \) will be some parallelopiped. By noting that the determinant gives the volume of the parallelipiped formed by the column vectors of the matrix, we have \begin{align*} J_{\phi(p)}\begin{bmatrix} dx_1 & 0 & \cdots & 0 \\ 0 & dx_2 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & dx_n \end{bmatrix} &= dV\\ \implies d\phi_1 d\phi_2 \cdots d\phi_n &= |\det(J_{\phi(p)})| \cdot dx_1 dx_2 \cdots dx_n. \end{align*} Where \( d\phi_1 d\phi_2 \cdots d\phi_n \) represents the volume element \( dV \). Rearranging gives us $$ dx_1 dx_2 \cdots dx_n = \frac{1}{|\det(J_{\phi(p)})|} \, d\phi_1 d\phi_2 \cdots d\phi_n. $$ Therefore, when we change variables in the integral, we have \begin{align*} V &= \int_D f(x) \, dx \\ &= \int_{D'} f(\phi^{-1}(u)) \cdot \frac{1}{|\det(J_{\phi(\phi^{-1}(u))})|} \, du . \end{align*} Note that \( d\phi_1 d\phi_2 \cdots d\phi_n \) should not be treated as a product of differentials but rather as the volume element in \( \mathbb{R}^n \) after the change of variables.

Now, \( J_{\phi(p)} \) is invertible as \( \phi \) is a homeomorphism, so we have \( \det(J_{\phi(p)}) \neq 0 \). Moreover, as it is a linear transformation, the inverse transformation is given by the inverse matrix \( J_{\phi(p)}^{-1} \) which is precisely the Jacobian of the inverse function \( \phi^{-1} \) at the point \( \phi(p) \) and thus we have $$ V = \int_{D'} f(\phi^{-1}(u)) \cdot |\det(J_{\phi^{-1}(u)})| \, du.$$