Practice problems for Midterm 1 ECE 490/590 Spring 2025

Date: Oct 9, 2025¶

Instructor: Vikas Dhiman (vikas.dhiman@maine.edu)¶

Total marks are 75.
Total time allowed is 75 min.
One page 8"x11" cheatsheet is allowed.
Calculators are allowed.
Computers are not allowed. You must know approximately know what the Python code will output. Minor formatting errors will not be penalized.

1. Write your name here:¶

2. Write your email here:¶

Python Basics¶

The midterm will be on paper, no computers will be allowed. Make sure you know what the python code output should be.

Python questions will be restriced to content covered in Python_1.ipynb, Python_2.ipynb, Python_3.ipynb

Python Basics¶

The midterm will be on paper, no computers will be allowed. Make sure you know what the python code output should be.

Python questions will be restriced to content covered in Python_1.ipynb, Python_2.ipynb, NumpyTutorial.ipynb

Q1. What will the following code print?¶

hello = "'Hello'"
name = '"ECE"'
pi = 3.1419
print(f'{hello:s} {name}. pi is {pi:.03f}')  # string formatting

'Hello' "ECE". pi is 3.142

Q2. What will the following code print?¶

xs = [1, 2, 3, 'hello', [4, 5, 6]]    # Create a list
print(xs[-1])

[4, 5, 6]

Q3. What will the following code print?¶

nums = list(range(5))    # range is a built-in function that creates a list of integers
print(nums[-2:])

[3, 4]

Q4. Which code is faster for very large lists and dictionaries ? Option 1 or Option 2? Why?¶

# Code Option 1:
d = {'cat': 'cute', 'dog': 'furry'}  # Create a new dictionary with some data
print(d['dog'])
# Code option 2:
keys = ['cat', 'dog'] # Create the dictionary with keys as lists
values = ['cute', 'furry'] # # Create the dictionary with values as lists
print(values[keys.index('dog')])

furry
furry

Q5. Which code is faster for very large lists and dictionaries ? Option 1 or Option 2? Why?¶

# Code Option 1:
d = {0: 'cute', 1: 'furry'}  # Create a new dictionary with some data
print(d[1])
# Code option 2:
values = ['cute', 'furry'] # # Create the dictionary with values as lists
print(values[1])

furry
furry

Q6. What is the output of the following code?¶

class Value:
    def __init__(self, v):
        self.v = v

    def __add__(self, other):
        return self.v * other

print(Value(3) + 2)

Numpy basics¶

Python questions will be restriced to content covered in NumpyTutorial.ipynb

Q7: What is the output of the following code?¶

import numpy as np
x = np.array([[1, 2], [3, 4]])
y = np.array([[5, 6]])
np.concatenate((x.T, y.T), axis=-1)

array([[1, 3, 5],
       [2, 4, 6]])

Q8. What is the output of the following code?¶

x = np.array([[1, 2], [3, 4]])
y = np.array([[5, 6]])
x @ y.T

array([[17],
       [39]])

Q9. What is the output of the following code?¶

x = np.array([[1, 2], [3, 4]])
y = np.array([[5, 6]])
(x * y).sum(axis=-1)

array([17, 39])

Linear algebra and it’s geometry¶

Q10.¶

Show that for any vector $\bfa = [a_1, a_2, \dots, a_n]$ , it’s magnitude squared is same as dot product with itself i.e. $\|\bfa\|^2 = \bfa^\top \bfa$

A10. The mangitude of n-D vector is given by $\|\bfa\| = \sqrt{a_1^2 + a_2^2 + \dots + a_n^2}$ and dot product the vector with itself is given by $\bfa^\top \bfa = a_1 a_1 + a_2 a_2 + \dots + a_n a_n = a_1^2 + a_2^2 + \dots + a_n^2$ . Squaring the magnitude gives us $\|\bfa\|^2 = a_1^2 + a_2^2 + \dots + a_n^2$ , which is same as $\bfa^\top \bfa$ .

Q11. For given vectors $\bfv$ and $\bfu$ find the projection of $\bfv$ on $\bfu$ $\proj_\bfu \bfv$ . Also find the equation of dotted line which is perpendicular to $\bfu$ and passes through $\bfv$ . Convert the equation of line to the form $y = m x + c$ .

$\bfv = \begin{bmatrix} 2 \\ 3 \end{bmatrix}$ and $\bfu = \begin{bmatrix} 3 \\ 2 \end{bmatrix}$ .

A11.

$\proj_\bfu \bfv = \bfv^\top \frac{\bfu}{\|\bfu\|} = \frac{12}{\sqrt{13}}$
The dotted line is the set of all points $\bfx \in \bbR^2$ that satisfy $\bfu^\top \bfx = \bfu^\top \bfv$
Let $\bfx = [x, y]$ . Then the above equation of line can be written as $[3, 2] \begin{bmatrix} x \\ y \end{bmatrix} = 12$ or $3x + 2y = 12$

Q12.¶

Convert the following scalar equation into vector form. Your end result should contain $\bfm = [m; c]$ , $\bfy = [y_1; y_2; \dots; y_n]$ and $\bfx = [x_1; x_2, \dots, x_n]$ . You can define other vectors and matrices as needed, included a vector of ones like $\mathbb{1}_n$ .

e(m, c, (x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)) = (y_1 - (x_1 m + c))^2 + (y_2 - (x_2 m + c))^2 + \dots + (y_n - (x_n m + c))^2

(1)

A12. Recall that the magnitude of a vector $\|\bfv\| = \sqrt{v_1^2 + v_2^2 + \dots + v_n^n}$ has a similar form to the error function. This suggests that we can define an error vector with the signed error for each data point as it’s elements

\bfe = \begin{bmatrix}y_1 - (mx_1 + c)\\ y_2 - (mx_2 + c)\\ \vdots \\ y_n - (mx_n + c)\end{bmatrix}

(2)

The total error is same as minimizing the square of error vector magnitude which is further same as vector product with itself.

e(m, c, (x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)) = \|\bfe\|^2 = \bfe^\top \bfe

(3)

Let us define $\bfx = [x_1; \dots; x_n]$ to denote the vector of all x coordinates of the dataset and $\bfy = [y_1; \dots; y_n]$ to denote y coordinates. Then the error vector is:

\bfe = \bfy - (\bfx m + \mathbf{1}_n c)

(4)

where $\mathbf{1}_n$ is a n-D vector of all ones. Finally, we vectorize parameters of the line $\bfm = [m; c]$ . We will also need to horizontally concatenate $\bfx$ and $\mathbf{1}_n$ . Let’s call the result $\bfX = [\bfx, \mathbf{1}_n] \in \bbR^{n \times 2}$ . Now, the error vector looks like this:

\bfe = \bfy - \bfX \bfm

(5)

Expanding the error magnitude:

\|\bfe\|^2 = (\bfy - \bfX \bfm)^\top (\bfy - \bfX \bfm) \\ = \bfy^\top\bfy + \bfm^\top \bfX^\top \bfX \bfm - 2\bfy^\top \bfX \bfm

(6)

Q13:¶

Convert the following scalar equation into vector form. Your end result should contain $\bfm = [a; b; c]$ , $\bfz = [z_1; z_2; \dots; z_n]$ , $\bfy = [y_1; y_2; \dots; y_n]$ and $\bfx = [x_1; x_2, \dots, x_n]$ . You can define other vectors and matrices as needed, included a vector of all ones like $\mathbb{1}_n$ .

e(a, b, c, (x_1, y_1, z_1), (x_2, y_2, z_2), \dots, (x_n, y_n, z_n)) = (z_1 - (x_1 a + y_1 b + c))^2 + (z_2 - (x_2 a + y_2 b + c))^2 + \dots + (z_n - (x_n a + y_n b + c))^2

(7)

A13: A variation of A12

Q14¶

Convert the following vector equation into even more vectorized form.

e(m_0, \bfm, (\bfx_1, y_1), (\bfx_2, y_2), \dots, (\bfx_n, y_n)) = (y_1 - (\bfx_1^\top \bfm + m_0))^2 + (y_2 - (\bfx_2^\top \bfm + m_0))^2 + \dots + (y_n - (\bfx_n^\top \bfm + m_0))^2

(8)

where $\bfm = [m_1; m_2; \dots; m_p] \in \bbR^p$ is a p-dimensional vector and $\bfx_i = [x_{i1}; x_{i2}; \dots; x_{ip}] \in \bbR^p$ are p-dimensional vectors for all $i = \{1, 2, \dots n\}$

Your end result should contain $\bfq = [m_0, m_1, m_2, \dots, m_p] \in \bbR^{p+1}$ , $\bfy = [y_1; y_2; \dots; y_n]\in \bbR^n$ and

\begin{align} \bfX = \begin{bmatrix}x_{11} & x_{12} & \dots & x_{1p}\\ x_{21} & x_{22} & \dots & x_{2p} \\ \vdots & \vdots & \ddots & \vdots\\ x_{n1} & x_{n2} & \dots & x_{np} \end{bmatrix} = \begin{bmatrix} \bfx_1^\top \\ \bfx_2^\top \\ \vdots\\ \bfx_n^\top \end{bmatrix} \in \bbR^{n \times p} \end{align}

(9)

You can define other vectors and matrices as needed, included a vector of all ones like $\mathbb{1}_n$ .

A15.

Recall that the magnitude of a vector $\|\bfv\| = \sqrt{v_1^2 + v_2^2 + \dots + v_n^n}$ has a similar form to the error function. This suggests that we can define an error vector with the signed error for each data point as it’s elements

\bfe = \begin{bmatrix}y_1 - (\bfx_1^\top\bfm + m_0)\\ y_2 - (\bfx_2^\top\bfm + m_0)\\ \vdots \\ y_n - (\bfx_2^\top\bfm_2 + m_0)\end{bmatrix}

(10)

The total error is same as minimizing the square of error vector magnitude which is further same as vector product with itself.

e(m_0, \bfm, (\bfx_1, y_1), (\bfx_2, y_2), \dots, (\bfx_n, y_n)) = \|\bfe\|^2 = \bfe^\top \bfe

(11)

Let us define $\bfX = [\bfx_1^\top; \dots; \bfx_n^\top]$ to denote the vector of all x coordinates of the dataset and $\bfy = [y_1; \dots; y_n]$ to denote y coordinates. Then the error vector is:

\bfe = \bfy - (\mathbf{1}_n m_0 + \bfX \bfm)

(12)

where $\mathbf{1}_n$ is a n-D vector of all ones. Finally, we call parameters of the line $\bfq = [m_0; \bfm]$ . We will also need to horizontally concatenate $\bfX$ and $\mathbf{1}_n$ . Let’s call the result $\bar{\bfX} = [\mathbf{1}_n, \bfX] \in \bbR^{n \times (p+1)}$ . Now, the error vector looks like this:

\bfe = \bfy - \bar{\bfX} \bfq

(13)

Expanding the error magnitude:

\|\bfe\|^2 = (\bfy - \bbfX \bfq)^\top (\bfy - \bbfX \bfq) \\ = \bfy^\top\bfy + \bfq^\top \bbfX^\top \bbfX \bfq - 2\bfy^\top \bbfX \bfq

(14)

Q16:¶

Convert the following scalar equation into vector form. Your end result should contain $\bfm = [m; c]$ , the matrix $\bfW = \Diag([w_1; w_2; \dots; w_n])$ , $\bfy = [y_1; y_2; \dots; y_n]$ and $\bfx = [x_1; x_2, \dots, x_n]$ . You can define other vectors and matrices as needed, included a vector of all ones like $\mathbb{1}_n$ .

e(m, c, (x_1, y_1, w_1), (x_2, y_2, w_2), \dots, (x_n, y_n, w_n)) = w_1^2(y_1 - (x_1 m + c))^2 + w_2^2(y_2 - (x_2 m + c))^2 + \dots + w_n^2(y_n - (x_n m + c))^2

(15)

The matrix $\bfW$ is defined as $\Diag([w_1; w_2; \dots; w_n])$ which indicates that $\bfW$ is diagonal matrix of $[w_1; w_2; \dots; w_n]$ .

\bfW = \Diag([w_1; w_2; \dots; w_n]) = \begin{bmatrix}w_1 & 0 & \dots & 0\\ 0 & w_2 & \dots & 0\\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & w_n\end{bmatrix}

(16)

A16:

Recall that the magnitude of a vector $\|\bfv\| = \sqrt{v_1^2 + v_2^2 + \dots + v_n^n}$ has a similar form to the error function. This suggests that we can define an error vector with the signed error for each data point as it’s elements

\bfe = \begin{bmatrix}y_1 - (mx_1 + c)\\ y_2 - (mx_2 + c)\\ \vdots \\ y_n - (mx_n + c)\end{bmatrix}

(17)

and let $\bfW = \Diag([w_1; w_2; \dots; w_n])$ .

Note that

\bfW \bfe = \begin{bmatrix}w_1(y_1 - (mx_1 + c))\\ w_2(y_2 - (mx_2 + c))\\ \vdots \\ w_3(y_n - (mx_n + c))\end{bmatrix}

(18)

The total error is same as the square of error vector magnitude

e(m, c, (x_1, y_1, w_1), (x_2, y_2, w_2), \dots, (x_n, y_n, w_n)) = w_1^2(y_1 - (x_1 m + c))^2 + w_2^2(y_2 - (x_2 m + c))^2 + \dots + w_n^2(y_n - (x_n m + c))^2 = \|\bfW \bfe\|^2

(19)

The square of error vector magnitude is same as dot product with itself,

\|\bfW \bfe\|^2 = (\bfW\bfe)^\top(\bfW\bfe) = \bfe^\top \bfW^\top \bfW \bfe

(20)

Let us define $\bfx = [x_1; \dots; x_n]$ to denote the vector of all x coordinates of the dataset and $\bfy = [y_1; \dots; y_n]$ to denote y coordinates. Then the error vector is:

\bfe = \bfy - (\bfx m + \mathbf{1}_n c)

(21)

where $\mathbf{1}_n$ is a n-D vector of all ones. Finally, we vectorize parameters of the line $\bfm = [m; c]$ . We will also need to horizontally concatenate $\bfx$ and $\mathbf{1}_n$ . Let’s call the result $\bfX = [\bfx, \mathbf{1}_n] \in \bbR^{n \times 2}$ . Now, the error vector looks like this:

\bfe = \bfy - \bfX \bfm

(22)

Expanding the error magnitude:

\|\bfW\bfe\|^2 = (\bfy - \bfX \bfm)^\top \bfW^\top\bfW (\bfy - \bfX \bfm) \\ = \bfy^\top\bfW^\top\bfW\bfy + \bfm^\top \bfX^\top \bfW^\top\bfW\bfX \bfm - 2\bfy^\top \bfW^\top\bfW\bfX \bfm

(23)

Q17:¶

Using vector derivatives find the minimum of the following vector quadratic function in $\bfm$ :

\arg~\min_{\bfm} e(\bfm) = \bfy^\top\bfW^\top\bfW\bfy + \bfm^\top \bfX^\top \bfW^\top\bfW\bfX \bfm - 2\bfy^\top \bfW^\top\bfW\bfX \bfm

(24)

The dimensions of the each of the variables are given $\bfm \in \bbR^p$ , $\bfy \in \bbR^n$ , $\bfW \in \bbR^{n \times n}$ , $\bfX \in \bbR^{n \times p}$ .

A17:

\begin{align} \mathbf{0}^\top &= \frac{\p }{\p \bfm} (\bfy^\top\bfW^\top\bfW\bfy + \bfm^\top \bfX^\top \bfW^\top\bfW\bfX \bfm - 2\bfy^\top \bfW^\top\bfW\bfX \bfm)\\ &= 2 {\bfm^*}^\top \bfX^\top \bfW^\top \bfW \bfX - 2\bfy^\top \bfW^\top \bfW \bfX \end{align}

(25)

This gives us the solution

\bfm^* = (\bfX^\top \bfW^\top \bfW \bfX)^{-1} \bfX^\top \bfW^\top \bfW \bfy

(26)

Vector derivatives¶

Q18:¶

Find the derivative of $f(\bfx) = (\bfx - \bfa_1)^\top A (\bfx - \bfa_2)$ with respecto to $\bfx$ .

You can assume $A \in \bbR^{n\times n}$ to be symmetric. The size of vectors are $\bfx, \bfa_1, \bfa_2, \bfa_3, \bfb \in \bbR^n$

A18

f(\bfx) = (\bfx - \bfa_1)^\top A (\bfx - \bfa_2)\\ = \bfx^\top A \bfx - \bfa_1^\top A \bfx - \bfx^\top A \bfa_2 + \bfa_1^\top A\bfa_2\\

(27)

Note that $\bfx^\top A \bfa_2$ is a scalar. That’s why we can replace it with its transpose $\bfx^\top A \bfa_2 = \bfa_2^\top A \bfx$

= \bfx^\top A \bfx - (\bfa_1 + \bfa_2)^\top A \bfx + \bfa_1^\top A\bfa_2

(28)

\frac{\partial f}{\partial \bfx} = 2\bfx^\top A - (\bfa_1 + \bfa_2)^\top A\\ = (2\bfx - (\bfa_1 + \bfa_2))^\top A

(29)

Q19:¶

Find the quadratic approximation of the following function near the point $\bfx_0$ :

f(\bfx) = \left((\bfx - \bfa_1)^\top A (\bfx - \bfa_2)\right) \left((\bfx - \bfa_3)^\top \bfb\right)

(30)

You can assume $A \in \bbR^{n\times n}$ to be symmetric. The size of vectors are $\bfx, \bfa_1, \bfa_2, \bfa_3, \bfb \in \bbR^n$

A19:

[\nabla_\bfx f(\bfx)]^\top = \left((\bfx - \bfa_1)^\top A (\bfx - \bfa_2)\right) \bfb^\top+ ((\bfx - \bfa_3)^\top \bfb)\left((2\bfx - (\bfa_1 + \bfa_2))^\top A\right)

(31)

\nabla_\bfx f(\bfx) = \left((\bfx - \bfa_1)^\top A (\bfx - \bfa_2)\right) \bfb + ((\bfx - \bfa_3)^\top \bfb)\left(A (2\bfx - (\bfa_1 + \bfa_2))\right)

(32)

\nabla_\bfx f(\bfx_0) = \left((\bfx_0 - \bfa_1)^\top A (\bfx_0 - \bfa_2)\right) \bfb + ((\bfx_0 - \bfa_3)^\top \bfb)\left(A (2\bfx_0 - (\bfa_1 + \bfa_2))\right)

(33)

H f(\bfx) = \nabla^2_\bfx f(\bfx) = \bfb (2\bfx - (\bfa_1 + \bfa_2))^\top A + \left(A (2\bfx - (\bfa_1 + \bfa_2))\right)\bfb^\top + \left((\bfx - \bfa_3)^\top \bfb\right)(2A)

(34)

H f(\bfx_0) = \nabla^2_\bfx f(\bfx_0) = \bfb (2\bfx_0 - (\bfa_1 + \bfa_2))^\top A + \left(A (2\bfx_0 - (\bfa_1 + \bfa_2))\right)\bfb^\top + \left((\bfx_0 - \bfa_3)^\top \bfb\right)(2A)

(35)

The quadratic approximation by Taylor series is:

f(\bfx) = f(\bfx_0) + [\nabla_\bfx f(\bfx_0)]^\top (\bfx - \bfx_0) + (\bfx - \bfx_0)^\top H f(\bfx_0) (\bfx - \bfx_0)

(36)

Q20¶

Show that for $\bfc, \bfx \in \bbR^n$

\begin{align} \frac{\p }{ \p \bfx} \bfc^\top \bfx = \bfc^\top\\ \end{align}

(37)

A20: Let $\bfc = [c_1, c_2, \dots, c_n]$ and $\bfx = [x_1, x_2, \dots x_n]$

Let $f(\bfx) = \bfc^\top \bfx = c_1 x_1 + c_2 x_2 + \dots c_n x_n$

\frac{\p f}{\p x_1} = c_1\\ \frac{\p f}{\p x_2} = c_2\\ \vdots\\ \frac{\p f}{\p x_n} = c_n\\

(38)

By Jacobian convention, we arrange the partial derivatives in a row vector:

\begin{align} \frac{\p }{ \p \bfx} \bfc^\top \bfx = \begin{bmatrix} \frac{\p f}{\p x_1} & \frac{\p f}{\p x_2} & \dots & \frac{\p f}{\p x_n}\end{bmatrix} \\ = \begin{bmatrix} c_1 & c_2 & \dots & c_n\end{bmatrix} = \bfc^\top \\ \end{align}

(39)

Q21:¶

Show that for $\bfA \in \bbR^{n \times n}$ , $\bfx \in \bbR^n$

\begin{align} \frac{\p }{ \p \bfx} \bfA \bfx = \bfA\\ \end{align}

(40)

A21: Let $\bfx = [x_1; x_2; \dots x_n]$

Let $\bfA = \begin{bmatrix} a_{11} & a_{12} & \dots & a_{1n} \\ a_{21} & a_{22} & \dots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & \dots & a_{nn} \end{bmatrix} = \begin{bmatrix} \bfa_1^\top \\ \bfa_2^\top \\ \vdots \\ \bfa_n^\top \end{bmatrix}$ , where $\bfa_i^\top \in \bbR^{1 \times n}$ are the row vectors of matrix $\bfA$ .

Then

\bfA \bfx = \begin{bmatrix} \bfa_1^\top \\ \bfa_2^\top \\ \vdots \\ \bfa_n^\top \end{bmatrix}\bfx = \begin{bmatrix} \bfa_1^\top \bfx \\ \bfa_2^\top \bfx \\ \vdots \\ \bfa_n^\top\bfx \end{bmatrix}

(41)

Let $$\bff(\bfx) =

\begin{bmatrix} f_1(\bfx) \\ f_2(\bfx) \\ \vdots\\ f_n(\bfx) \end{bmatrix}

(42)

= \bfA \bfx = \begin{bmatrix} \bfa_1^\top \bfx \ \bfa_2^\top \bfx \ \vdots \ \bfa_n^\top\bfx \end{bmatrix}$$

By Jacobian convention we arrange the partial derivatives of each function component column-wise

\frac{\p \bff(\bfx)}{\p \bfx} = \begin{bmatrix} \frac{\p f_1(\bfx)}{\p \bfx} \\ \frac{\p f_2(\bfx)}{\p \bfx} \\ \vdots \\ \frac{\p f_n(\bfx)}{\p \bfx} \end{bmatrix} = \begin{bmatrix} \frac{\p \bfa_1^\top \bfx}{\p \bfx} \\ \frac{\p \bfa_2^\top \bfx}{\p \bfx} \\ \vdots \\ \frac{\p \bfa_2^\top \bfx}{\p \bfx} \end{bmatrix} = \begin{bmatrix} \bfa_1^\top \\ \bfa_2^\top \\ \vdots \\ \bfa_n^\top \end{bmatrix} = \bfA

(43)

Q22:¶

Use vector-derivative chain rule:

\frac{\p \bff(\bfg(\bfx)) }{\p \bfx} = \frac{\p \bff}{\p \bfg}\frac{\p \bfg}{\p \bfx}

(44)

,

for any function $\bfg : \bbR^n \mapsto \bbR^m$ and $\bff : \bbR^m \mapsto \bbR^o$ .

Show that for $\bfx \in \bbR^n$ amd $\bfA \in \bbR^{n \times n}$

\begin{align} \frac{\p }{ \p \bfx} \bfx^\top \bfA \bfx = \bfx^\top (\bfA^\top + \bfA)\\ \end{align}

(45)

A22:

For product of any two vectors

\begin{align} \frac{\p }{\p \bfx} \bfx^\top \bfy = \bfy^\top \end{align}

(46)

If $\bfy$ is a function of $\bfx$ , then

\begin{align} \frac{\p }{\p \bfx} \bfx^\top \bfy &= \bfy^\top + \left(\frac{\p }{\p \bfy} \bfx^\top \bfy \right) \left(\frac{\p \bfy }{\p \bfx} \right)\\ &= \bfy^\top + \bfx^\top \left(\frac{\p \bfy }{\p \bfx}\right) \end{align}

(47)

If $\bfy = \bfA \bfx$ , then

\frac{\p \bfy}{\p \bfx} = \frac{\p }{\p \bfx} \bfA \bfx = \bfA

(48)

and

\frac{\p }{\p \bfx} \bfx^\top \bfA \bfx = \bfy^\top + \bfx^\top \left(\frac{\p \bfy }{\p \bfx}\right) = \bfx^\top \bfA^\top + \bfx^\top \bfA = \bfx^\top (\bfA^\top + \bfA)

(49)

Perceptron¶

Q23:¶

You are given 2D points and corresponding labels as a training dataset $\{ (x_1, y_1, l_1), (x_2, y_2, l_2), \dots, (x_n, y_n, l_n) \}$ , where $x_i \in \bbR$ , $y_i \in \bbR$ and the labels $l_i \in \{-1, 1\}$ . Use the model $\hat{l}_i = \sign(y_i - (m x_i + c))$ to construct a loss (or error) function. Find the gradient of the loss function with respect to the vector $\bfm = [m; c]$ .

A23

e(y_i, x_i; m,c) = \begin{cases} 0 &\text{ if }\sign(y_i - (m x_i + c)) = l_i\\ |y_i - (m x_i + c)| &\text{ if } \sign(y_i - (m x_i + c)) \ne l_i \end{cases}

(50)

\bfm = \begin{bmatrix}m \\ c\end{bmatrix}

(51)

e(y_i, x_i;\bfm) = \begin{cases} 0 &\text{ if } \sign( y_i - \begin{bmatrix} x_i& 1\end{bmatrix}\bfm) = l_i\\ |y_i - \begin{bmatrix} x_i& 1\end{bmatrix}\bfm| &\text{ if } \sign( y_i - \begin{bmatrix} x_i& 1\end{bmatrix}\bfm) \ne l_i \end{cases}

(52)

If $l_i \in \{-1, 1\}$ , then $\sign( y_i - \begin{bmatrix} x_i& 1\end{bmatrix}\bfm) = l_i$ is same as saying $l_i( y_i - \begin{bmatrix} x_i& 1\end{bmatrix}\bfm) > 0$ .

e(y_i, x_i;\bfm) = \begin{cases} 0 &\text{ if } l_i( y_i - \begin{bmatrix} x_i& 1\end{bmatrix}\bfm) > 0\\ |l_i( y_i - \begin{bmatrix} x_i& 1\end{bmatrix}\bfm)| &\text{ if } l_i( y_i - \begin{bmatrix} x_i& 1\end{bmatrix}\bfm) < 0 \end{cases}

(53)

Also when $z < 0$ , then $|z| = -z$ .

e(y_i, x_i;\bfm) = \begin{cases} 0 &\text{ if } l_i( y_i - \begin{bmatrix} x_i& 1\end{bmatrix}\bfm) > 0\\ -l_i( y_i - \begin{bmatrix} x_i& 1\end{bmatrix}\bfm) &\text{ if } l_i( y_i - \begin{bmatrix} x_i& 1\end{bmatrix}\bfm) < 0 \end{cases}

(54)

e(y_i, x_i;\bfm) = \max\{0, - l_i (y_i - \begin{bmatrix} x_i& 1\end{bmatrix}\bfm)\}

(55)

\nabla_\bfm e(y_i, x_i;\bfm) = \max\{0, l_i(\begin{bmatrix} x_i& 1\end{bmatrix})\}

(56)

For the entire dataset, we have $\bfy = [y_1; \dots; y_n]$ and $\bfx = [x_1; \dots; x_n]$ , $\ell = [l_1; \dots; l_n]$ the average error is:

e(\bfx, \bfy; \bfm) = \frac{1}{n}{\bfone_n^\top}\max\{0, - \ell \odot (\bfy - \begin{bmatrix}\bfx & \bfone_n\end{bmatrix}\bfm )\},

(57)

where $\odot$ is the element-wise product. and $\bfone_n$ is a vector of ones.

and the average gradient is:

\nabla_\bfm^\top e(\bfx, \bfy; \bfm) = \frac{1}{n}{\bfone_n^\top}\max\{0, \ell \odot ( \begin{bmatrix}\bfx & \bfone_n\end{bmatrix} )\}

(58)

Q24¶

You are given p-D points $\bfx_i \in \bbR^p$ and corresponding labels as a training dataset $\{ (\bfx_1, l_1), (\bfx_2, l_2), \dots, (\bfx_n, l_n) \}$ , where $\bfx_i \in \bbR^p$ , and the labels $l_i \in \{-1, 1\}$ . Use the model $\hat{l}_i = \sign(\bfx_i^\top \bfm + m_0))$ to construct a loss (or error) function. Find the gradient of the loss function with respect to the vector $\bfq = [m_0; \bfm]$ .