Instructor's Solution Manual Optimization Models 1st edition Giuseppe C. Calafiore, Laurent El Ghaou

SOLUTION MANUAL

G I U S E P P E C A L A F I O R E AND L A U R E N T E L G H A O U I

O P T I M I Z AT I O N M O D E L S SOLUTIONS MANUAL

CAMBRIDGE

Ver. 0.1 - Oct. 2014

DISCLAIMER This is the first draft of the Solution Manual for exercises in the book "Optimization Models"

by Calafiore & El Ghaoui.

This draft is under construction. It is still incomplete and it is very likely to contain errors.

This material is offered "as is," non commercially, for personal use of instructors.

Comments and corrections are very welcome.

Contents 2.

Vectors

3.

Matrices

4.

Symmetric matrices

5.

Singular Value Decomposition

6.

Linear Equations

7.

Matrix Algorithms

8.

Convexity

9.

Linear, Quadratic and Geometric Models

81

10. Second-Order Cone and Robust Models

94

4 10 21 35

46 58

68

11. Semidefinite Models

107

12. Introduction to Algorithms 13. Learning from Data

140

14. Computational Finance 15. Control Problems 16. Engineering Design

124

152

174 183

4

2.

Vectors

Exercise 2.1 (Subpaces and dimensions) Consider the set S of points such that x1 + 2x2 + 3x3 = 0, 3x1 + 2x2 + x3 = 0.

Show that S is a subspace. Determine its dimension, and find a basis for it.

Solution 2.1 The set S is a subspace, as can be checked directly: if x, y ∈ S , then for every λ, µ ∈ R, we have λx + µy ∈ S . To find the dimension, we solve the equation and find that any solution to the equations is of the form x1 = −1/2x2 , x3 = −1/3x2 , where x2 is free. Hence the dimension of S is 1, and a basis for S is the vector (−1/2, 1, −1/3).

Exercise 2.2 (Affine sets and projections) Consider the set in R3 , defined by the equation n o P = x ∈ R3 : x1 + 2x2 + 3x3 = 1 .

1. Show that the set P is an affine set of dimension 2. To this end, express it as x (0) + span( x (1) , x (2) ), where x (0) ∈ P , and x (1) , x (2) are linearly independent vectors.

2. Find the minimum Euclidean distance from 0 to the set P , and a point that achieves the minimum distance.

Solution 2.2 1. We can express any vector x ∈ P as x = ( x1 , x2 , 1/3 − x1 /3 − 2x2 /3), where x1 , x2 are arbitrary. Thus x = x (0) + x 1 x (1) + x 2 x (2) , where 

     0 1 0       x (0) =  0  , x (1) =  0  , x (2) =  1  .

1 − 13 − 23 3

Since x (1) and x (2) are linearly independent, P is of dimension 2.

2. The set P is defined by a single linear equation a> x = b, with a> = [1 2 3] and b = 1, i.e., P is a hyperplane. The minimum Euclidean distance from 0 to P is the `2 norm of the projection of 0 onto P , which can be determined as discussed in Section 2.3.2.2.

That is, the projection x ∗ of 0 onto P is such that x ∗ ∈ P and x ∗

5 is orthogonal to the subspace generating P (which coincides with the span of a), that is x ∗ = αa. Hence, it must be that a> x ∗ = 1, thus αk ak22 = 1, and α = 1/k ak22 . We thus have that x∗ =

a , k ak22

√ and the distance we are seeking is k x ∗ k2 = 1/k ak2 = 1/ 14.

Exercise 2.3 (Angles, lines and projections) 1. Find the projection z of the vector x = (2, 1) on the line that passes through x0 = (1, 2) and with direction given by vector u = (1, 1).

2. Determine the angle between the following two vectors:     1 3     x =  2 , y =  2 .

3 1 Are these vectors linearly independent?

Solution 2.3 1. We can observe directly that u> ( x − x0 ) = 0, hence the projection of x is the same as that of x0 , which is z = x0 itself.

Alternatively, as seen in Section (2.3.2.1), the projection is z = x0 +

u > ( x − x0 ) u u> u

which gives z = x0 .

Another method consists in solving min k x0 + tu − x k22 t

= min t2 u T u − 2tu> ( x − x0 ) + k x − x0 k22 t

= min (u> u)(t − t0 )2 + constant, t

where t0 = ( x − x0 )> u/(u T u). This leads to the optimal t∗ = t0 , and provides the same result as before.

2. The angle cosine is given by cos θ = which gives θ ≈ 41◦ .

x> y 10 = , k x k2 k y k2 14

The vectors are linearly independent, since λx + µy = 0 for λ, µ ∈ R implies that λ = µ = 0. Another way to prove this is to observe that the angle is not 0◦ nor 180◦ .

6 Exercise 2.4 (Inner product) Let x, y ∈ Rn . Under which condition on α ∈ Rn does the function n

f ( x, y) = ∑ αk xk yk k =1

define an inner product on Rn ?

Solution 2.4 The axioms of 2.2 are all satisfied for any α ∈ Rn , except the conditions f ( x, x ) ≥ 0; f ( x, x ) = 0 if and only if x = 0.

These properties hold if and only if αk > 0, k = 1, . . . , n. Indeed, if the latter is true, then the above two conditions hold. Conversely, if if there exist k such that αk ≤ 0, setting x = ek (the k-th unit vector in Rn ) produces f (ek , ek ) ≤ 0; this contradicts one of the two above conditions.

Exercise 2.5 (Orthogonality) Let x, y ∈ Rn be two unit-norm vectors, that is, such that k x k2 = kyk2 = 1. Show that the vectors x − y and x + y are orthogonal. Use this to find an orthogonal basis for the subspace spanned by x and y.

Solution 2.5 When x, y are both unit-norm, we have

( x − y)> ( x + y) = x > x − y> y − y> x + x > y = x > x − y> y = 0, as claimed.

We can express any vector z ∈ span( x, y) as z = λx + µy, for some λ, µ ∈ R. We have z = αu + βv, where α=

λ+µ λ−µ , β= .

2 2

Hence z ∈ span(u, v). The converse is also true for similar reasons.

Thus, (u, v) is an orthogonal basis for span( x, y). We finish by normalizing u, v, replacing them with (u/kuk2 , v/kvk2 ). The desired orthogonal basis is thus given by (( x − y)/k x − yk2 , ( x + y)/k x + yk2 ).

Exercise 2.6 (Norm inequalities) 1. Show that the following inequalities hold for any vector x:

√ 1 √ k x k2 ≤ k x k ∞ ≤ k x k2 ≤ k x k1 ≤ n k x k2 ≤ n k x k ∞ . n Hint: use the Cauchy-Schwartz inequality.

7 2. Show that for any non-zero vector x,

k x k21 , k x k22

card( x ) ≥

where card( x ) is the cardinality of the vector x, defined as the number of non-zero elements in x. Find vectors x for which the lower bound is attained.

Solution 2.6 1. We have

n

k x k22 = ∑ xi2 ≤ n · max xi2 = n · k x k2∞ . i

i =1

Also, k x k∞ ≤

q

x12 + . . . + xn2 = k x k2 .

The inequality k x k2 ≤ k x k1 is obtained after squaring both sides, and checking that n

n

i =1

i =1

n

∑ xi2 ≤ ∑ xi2 + ∑ |xi x j | = ∑ |xi | i6= j

Finally, the condition k x k1 ≤ inequality

i =1

√

!2

= k x k21 .

nk x k2 is due to the Cauchy-Schwartz

| z > y | ≤ k y k2 · k z k2 , applied to the two vectors y = (1, . . . , 1) and z = | x | = (| x1 |, . . . , | xn |).

2. Let us apply the Cauchy-Schwartz inequality with z = | x | again, and with y a vector with yi = 1 if xi 6= 0, and yi = 0 otherwise. √ We have kyk2 = k, with k = card( x ). Hence

| z > y | = k x k1 ≤ k y k2 · k z k2 =

√

k · k x k2 ,

which proves the result. The bound is attained for vectors with k non-zero elements, all with the same magnitude.

Exercise 2.7 (Hölder inequality) Prove Hölder’s inequality (2.4). Hint: consider the normalized vectors u = x/k x k p , v = y/kykq , and observe that

| x > y | = k x k p k y k q · | u > v | ≤ k x k p k y k q ∑ | u k v k |. k

Then, apply Young’s inequality (see Example 8.10) to the products |uk vk | = |uk ||vk |.

8 Solution 2.7 The inequality is trivial if one of the vectors x, y is zero.

We henceforth assume that none is, which allows us to define the normalized vectors u, v. We need to show that

∑ |uk vk | ≤ 1. k

Using the hint given, we apply Young’s inequality, which states that for any given numbers a, b ≥ 0 and p, q > 0 such that 1 1 + = 1, p q it holds that ab ≤

1 p 1 q a + b . p q

We thus have, with a = |uk | and b = |vk |, and summing over k:

∑ |uk vk | k

≤ = =

1 1 |uk | p + ∑ |vk |q p∑ q k k 1 1 p q kuk p + kvkq p q 1 1 + = 1, p q

where we have used the fact that kuk p = kvkq = 1.

Exercise 2.8 (Linear functions) 1. For a n-vector x, with n = 2m − 1 odd, we define the median of x as the scalar value x a such that exactly n of the values in x are ≤ x a and n are ≥ x a (i.e., x a leaves half of the values in x to its left, and half to its right). Now consider the function f : Rn → R, with values f ( x ) = x a − n1 ∑in=1 xi . Express f as a scalar product, that is, find a ∈ Rn such that f ( x ) = a> x for every x. Find a basis for the set of points x such that f ( x ) = 0.

2. For α ∈ R2 , we consider the “power law” function f : R 2++ → R, α with values f ( x ) = x1 1 x2α2 . Justify the statement: “the coefficients αi provide the ratio between the relative error in f to a relative error in xi ”.

Solution 2.8 (Linear functions) TBD Exercise 2.9 (Bound on a polynomial’s derivative) In this exercise, you derive a bound on the largest absolute value of the derivative of a polynomial of a given order, in terms of the size of the coefficients1 . For w ∈ Rk+1 , we define the polynomial pw , with values

See the discussion on regularization in Section 13.2.3 for an application of this result.

1

9 . p w ( x ) = w1 + w2 x + . . . + w k +1 x k .

Show that, for any p ≥ 1

∀ x ∈ [−1, 1] :

dpw ( x ) ≤ C (k, p)kvk p , dx

where v = (w2 , . . . , wk+1 ) ∈ Rk , and    k k3/2 C (k, p) =   k ( k +1) 2

p = 1, p = 2, p = ∞.

Hint: you may use Hölder’s inequality (2.4) or the results from Exercise 2.6.

Solution 2.9 (Bound on a polynomial’s derivative) We have, with z = (1, 2, . . . , k), and using Hölder’s inequality: dpw ( x ) dx

w2 + 2w3 x + . . . + kwk+1 x k−1

=

≤ | w2 | + 2 | w3 | + . . . + k | w k +1 | = |v> z|

≤ kvk p · kzkq .

When p = 1, we have

kzkq = kzk∞ = k.

When p = 2, we have

k z k q = k z k2 = When p = ∞, we have

p

1 + 4 + . . . + k2 ≤

√

k z k q = k z k1 = 1 + 2 + . . . + k =

k · k2 = k3/2 .

k ( k + 1) .

2

10

3.

Matrices

Exercise 3.1 (Derivatives of composite functions) 1. Let f : Rm → Rk and g : Rn → Rm be two maps. Let h : Rn → Rk be the composite map h = f ◦ g, with values h( x ) = f ( g( x )) for x ∈ Rn . Show that the derivatives of h can be expressed via a matrix-matrix product, as Jh ( x ) = J f ( g( x )) · Jg ( x ), where Jh ( x ) is the Jacobian matrix of h at x, i.e., the matrix whose (i, j) element ∂hi ( x ) is ∂x . j

2. Let g be an affine map of the form g( x ) = Ax + b, for A ∈ Rm,n , b ∈ Rm . Show that the Jacobian of h( x ) = f ( g( x )) is Jh ( x ) = J f ( g( x )) · A. 3. Let g be an affine map as in the previous point, let f : Rn → R (a scalar-valued function), and let h( x ) = f ( g( x )). Show that

∇ x h( x ) =

∇2x h( x ) =

A> ∇ g f ( g( x ))

A> ∇2g f ( g( x )) A.

Solution 3.1 1. We have, by the composition rule for derivatives:

[ Jh ( x )]i,j

=

m ∂hi ( x ) ∂f ∂g = ∑ i (x) l (x) ∂x j ∂gl ∂x j l =1

=

∑ [ J f ( g(x))]i,l [ Jg (x)]l,j ,

m

l =1

which proves the result.

2. Since gi ( x ) = ∑nk=1 aik xk + bi , i = 1, . . . , m, we have that the (i, j)-th element of the Jacobian of g is

[ Jg ( x )]ij =

∂gi ( x ) = aij , ∂x j

hence Jg ( x ) = A, and the desired result follows from applying point 1. of this exercise.

3. For a scalar-valued function, the gradient coincides with the transopose of the Jacobian, hence the expression for the gradient of h w.r.t. x follows by applying the previous point. For the Hessian,

11 we have instead

[∇2x h( x )]ij

= = =

∂2 h ( x ) ∂ ∂h( x ) ∂ > = = a ∇ g f ( g( x )) ∂xi ∂x j ∂x j ∂xi ∂x j i m ∂ m ∂ f ( g( x )) ∂ ∂ f ( g( x )) aik = ∑ aik ∑ ∂x j k=1 ∂gk ∂x j ∂gk k =1

m m ∂ ∂ f ( g( x )) ∂g p ( x ) ∑ aik ∑ ∂g p ∂gk ∂x j p =1 k =1 m

= =

m

k =1

∂2 f ( g( x )) ∂g p ( x ) ∂g p ∂gk ∂x j p =1

m

m

∑ aik ∑

∂2 f ( g( x )) a ∂g p ∂gk pj p =1

∑ aik ∑

k =1

= ai> ∇2g f ( g( x )) a j , which proves the statement.

Exercise 3.2 (Permutation matrices) A matrix P ∈ Rn,n is a permutation matrix if its columns are a permutation of the columns of the n × n identity matrix.

1. For a n × n matrix A, we consider the products PA and AP. Describe in simple terms what these matrices look like with respect to the original matrix A. 2. Show that P is orthogonal.

Solution 3.2 1. Given the matrix A, the product PA is the matrix obtained by permuting the rows of A; AP corresponds to permuting the columns.

2. Every pair of columns ( pk , pl ) of P is of the form (ek , el ), where ek , el are the k-th and the l-th standard basis vectors in Rn . Thus, k pk k2 = 1, and p> k pl = 0 if k 6 = l, as claimed.

Exercise 3.3 (Linear maps) Let f : Rn → Rm be a linear map. Show how to compute the (unique) matrix A such that f ( x ) = Ax for every x ∈ Rn , in terms of the values of f at appropriate vectors, which you will determine.

Solution 3.3 For i = 1, . . . , n, let ei be the i-th unit vector in Rn . We have f (ei ) = Aei = ai , where ai is the i-th column of A. Hence we can compute the matrix A column-wise, by evaluating f at the points e1 , . . . , en .

12 Exercise 3.4 (Linear dynamical systems) Linear dynamical systems are a common way to (approximately) model the behavior of physical phenomena, via recurrence equations of the form2

Such models are the focus of Chapter 15.

2

x (t + 1) = Ax (t) + Bu(t), y(t) = Cx (t), t = 0, 1, 2, . . . , where t is the (discrete) time, x (t) ∈ Rn describes the state of the system at time t, u(t) ∈ R p is the input vector, and y(t) ∈ Rm is the output vector. Here, matrices A, B, C, are given.

1. Assuming that the system has initial condition x (0) = 0, express the output vector at time T as a linear function of u(0), . . . , u( T − 1); that is, determine a matrix H such that y( T ) = HU ( T ), where   u (0)  .  ..  U (T ) =  .   u ( T − 1) contains all the inputs up to and including at time T − 1.

2. What is the interpretation of the range of H? Solution 3.4 1. We have x (1)

=

Bu(0)

x (2)

=

Ax (1) + Bu(1) = ABu(0) + Bu(1)

x (3)

=

Ax (2) + Bu(2) = A2 Bu(0) + ABu(1) + Bu(2).

We now prove by induction that, for T ≥ 1: T −1

x ( T ) = ∑ Ak Bu( T − 1 − k) = k =0

h

A T −1 B

...

AB

B

i

U ( T ).

The formula is correct for T = 1. Let T ≥ 2. Assume the formula is correct for T − 1; we have ! x ( T ) = Ax ( T − 1) + Bu( T − 1)

=

A

T −2

∑ Ak Bu(T − 2 − k)

k =0

=

+ Bu( T − 1)

T −2

∑ Ak+1 Bu(T − 2 − k) + Bu(T − 1)

k =0

= =

T −1

∑ Ak Bu(T − 1 − k) + Bu(T − 1)

k =1 T −1

∑ Ak Bu(T − 1 − k),

k =0

13 as claimed. Finally, we have y( T ) = HU ( T ), with h i H = C · A T −1 B . . . AB B .

2. The range of H is the set of output vectors that are attainable at time T by the system by proper choice of the sequence of inputs, starting from the initial state x (0) = 0.

Exercise 3.5 (Nullspace inclusions and range) Let A, B ∈ Rm,n be two matrices. Show that the fact that the nullspace of B is contained in that of A implies that the range of B> contains that of A> .

Solution 3.5 Assume that the nullspace of B is contained in that of A. This means that Bx = 0 =⇒ Ax = 0.

Let z ∈ R( A> ): there exist y ∈ Rm such that z = A> y. We have thus, for any element x ∈ N ( A), z> x = y> Ax = 0. Hence, z is orthogonal to the nullspace of A, so it is orthogonal to the nullspace of B. We have obtained R( A> ) ⊆ N ( B)⊥ = R( B> ), as claimed. Here, we have used the fundamental theorem of linear algebra (3.1).

Exercise 3.6 (Rank and nullspace) Consider the image in Figure 3.6, a gray-scale rendering of a painting by Mondrian (1872-1944). We build a 256 × 256 matrix A of pixels based on this image by ignoring grey zones, assigning +1 to horizontal or vertical black lines, +2 at the intersections, and zero elsewhere. The horizontal lines occur at row indices 100, 200 and 230, and the vertical ones, at columns indices 50, 230.

1. What is nullspace of the matrix?

2. What is its rank?

Solution 3.6 1. Denote by ei the i-th unit vector in R256 , by z1 ∈ R256 the vector with all first 50 components equal to one, by z2 ∈ R256 the vector with all last 26 components equal to one, and by z3 ∈ R256 the vector with all last 56 components equal to one. Finally, 1 denotes the vector of all ones in R256 . We can express the matrix as > > M = e100 z1> + e200 1> + e230 z3> + 1e50 + z3 e230 .

The condition Mx = 0, for some vector x ∈ Rn , translates as > > x )z3 = 0. (z1> x )e100 + (1> x )e200 + (z3> x )e230 + (e50 x )1 + (e230

Figure 3.1: A gray-scale rendering of a painting by Mondrian.

14 Since the vectors (e100 , e200 , e230 , 1, z3 ) are linearly independent, we obtain that the five coefficients in the above must be zero: > > 0 = z1> x = 1> x = z3> x = e50 x = e230 x.

It is easy to check that the corresponding subspace of R256 is of dimension 256 − 5 = 251. Indeed, two elements of x are zero (x50 = x230 = 0), the remaining ones satisfy three independent equality constraints. From these we can express (say) x1 , x201 , x51 from the remaining variables, which then are free of any constraints. We can eliminate a total of five variables from the above five conditions, so the nullspace is of dimension 251.

2. The rank is 5.

Exercise 3.7 (Range and nullspace of A> A) Prove that, for any matrix A ∈ Rm,n , it holds that

N ( A> A) = N ( A)

R ( A > A ) = R ( A > ).

(3.1)

Hint: use the fundamental theorem of linear algebra.

Solution 3.7 First, suppose x ∈ N ( A), then Ax = 0 and obviously A> Ax = 0. Conversely, suppose x ∈ N ( A> A), we show by contradiction that it must be x ∈ N ( A), hence proving the first claim. Indeed, suppose x ∈ N ( A> A) but x 6∈ N ( A). Define then v = Ax 6= 0.

Such a v is by definition in the range of A, and A> v = A> Ax = 0, so v is also in the nullspace of A> , which is impossible since, by the fundamental theorem of linear algebra, R( A)⊥ N ( A> ). Next,

R ( A > ) = N ( A ) ⊥ = N ( A > A ) ⊥ = R ( A > A ), which proves (3.1).

Exercise 3.8 (Cayley-Hamilton theorem) Let A ∈ Rn,n and let . p(λ) = det(λIn − A) = λn + cn−1 λn−1 + · · · + c1 λ + c0 be the characteristic polynomial of A. 1. Assume A is diagonalizable. Prove that A annihilates its own characteristic polynomial, that is p( A) = An + cn−1 An−1 + · · · + c1 A + c0 In = 0.

Hint: use Lemma 3.3.

15 2. Prove that p( A) = 0 holds in general, i.e., also for non-diagonalizable square matrices. Hint: use the facts that polynomials are continuous functions, and that diagonalizable matrices are dense in Rn,n , i.e., for any e > 0 there exist ∆ ∈ Rn,n with k∆k F ≤ e such that A + ∆ is diagonalizable.

Solution 3.8 1. The result is immediate from Lemma 3.3: if A = UΛU −1 is a diagonal factorization of A, then p(Λ) = 0, since by definition eigenvalues are roots of the characteristic polynomial, hence p( A) = U p(Λ)U −1 = 0.

2. The map p : Rn,n → Rn,n with values p( A) = An + cn−1 An−1 + · · · + c1 A + c0 In is continuous on Rn,n . This map is identically zero on the dense subset of Rn,n formed by diagonalizable matrices (proved in the previous point of the exercise), hence by continuity it must be zero everywhere in Rn,n .

Exercise 3.9 (Frobenius norm and random inputs) Let A ∈ Rm,n be a matrix. Assume that u ∈ Rn is a vector-valued random variable, with zero mean and covariance matrix In . That is, E{u} = 0, and E{uu> } = In .

1. What is the covariance matrix of the output, y = Au? 2. Define the total output variance as E{ky − ŷk22 }, where ŷ = E{y} is the output’s expected value. Compute the total output variance and comment.

Solution 3.9 1. The mean of the output is zero: ŷ = Ey = AEu = 0. Hence the covariance matrix is given by E(yy> )

= E( Auu> A> ) =

AE(uu> ) A>

=

AA> .

2. The total variance is E( y > y )

= trace E(yy> ) = trace( AA> ) = k Ak2F .

16 The total output variance is the square of the Frobenius norm of the matrix. Hence the Frobenius norm captures the response of the matrix to a class of random inputs (zero mean, and unit covariance matrix).

Exercise 3.10 (Adjacency matrices and graphs) For a given undirected graph G with no self-loops and at most one edge between any pair of nodes (i.e., a simple graph), as in Figure 3.2, we associate a n × n matrix A, such that ( 1 if there is an edge between node i and node j, Aij = 0 otherwise.

This matrix is called the adjacency matrix of the graph3 .

1. Prove the following result: for positive integer k, the matrix Ak has an interesting interpretation: the entry in row i and column j gives the number of walks of length k (i.e., a collection of k edges) leading from vertex i to vertex j. Hint: prove this by induction on k, and look at the matrix-matrix product Ak−1 A. 2. A triangle in a graph is defined as a subgraph composed of three vertices, where each vertex is reachable from each other vertex (i.e., a triangle forms a complete subgraph of order 3). In the graph of Figure 3.2, for example, nodes {1, 2, 4} form a triangle.

Show that the number of triangles in G is equal to the trace of A3 divided by 6. Hint: For each node in a triangle in an undirected graph, there are two walks of length 3 leading from the node to itself, one corresponding to a clockwise walk, and the other to a counter-clockwise walk.

Solution 3.10 1. We can prove the result by induction on k. For k = 1, the result follows from the very definition of A. Let Lk (i, j) denote the number of paths of length k between nodes i and j, and assume that the result we wish to prove is true for some given h ≥ 1, so that Lh (i, j) = [ Ah ]i,j . We next prove that it must also hold that Lh+1 (i, j) = [ Ak+1 ]i,j , thus proving by inductive argument that Lk (i, j) = [ Ak ]i,j for all k ≥ 1.

Indeed, to go from a node i to a node j with a walk of length h + 1, one needs first reach, with a walk of length h, a node l linked to j by an edge. Thus: Lh+1 (i, j) =

∑ Lh (i, l ),

l ∈V ( j )

1

2 3

5

4

Figure 3.2: An undirected graph with n = 5 vertices.

The graph in Figure 3.2 has adjacency matrix   0 1 0 1 1  1 0 0 1 1     A=  0 0 0 0 1 .  1 1 0 0 0  1 1 1 0 0

3

17 where V ( j) is the neighbor set of j, which is the set of nodes connected to the j-th node, that is, nodes l such that Al,j 6= 0. Thus: n

Lh+1 (i, j) = ∑ Lh (i, l ) Al,j . l =1

But we assumed that Lh (i, j) = [ Ah ]i,j , hence the previous equation can be written as n

Lh+1 (i, j) = ∑ [ Ah ]i,l Al,j . l =1

In the above we recognize the (i, j)-th element of the product Ah A = Ah+1 , which proves that Lh+1 (i, j) = [ Ah+1 ]i,j , and hence concludes the inductive proof.

2. Following the hint, we observe that for each node in a triangle in an undirected graph there are two walks of length 3 leading from the node to itself, one corresponding to a clockwise walk, and the other to a counter-clockwise walk. Therefore, each triangle in the graph produces 6 walks of length 3 (two walks for each vertex composing the triangle). From the previous result, the number of walks of length 3 from node i to itself is given by [ A3 ]i,i , hence the total number of of walks of length 3 from each node to itself is ∑in=1 [ A3 ]i,i = trace( A3 ), and therefore the number of triangles is trace( A3 )/6.

Exercise 3.11 (Nonnegative and positive matrices) A matrix A ∈ Rn,n is said to be nonnegative (resp. positive) if aij ≥ 0 (resp. aij > 0) for all i, j = 1, . . . , n. The notation A ≥ 0 (resp. A > 0) is used to denote nonnegative (resp. positive) matrices.

A nonnegative matrix is said to be column (resp. row) stochastic, if the sum of the elements along each column (resp. row) is equal to one, that is if 1> A = 1> (resp. A1 = 1). Similarly, a vector x ∈ Rn is said to be nonnegative if x ≥ 0 (element-wise), and it is said to be a probability vector, if it is nonnegative and 1> x = 1. The set of probability vectors in Rn is thus the set S = { x ∈ Rn : x ≥ 0, 1> x = 1}, which is called the probability simplex. The following points you are requested to prove are part of a body of results known as the Perron-Frobenius theory of nonnegative matrices.

1. Prove that a nonnegative matrix A maps nonnegative vectors into nonnegative vectors (i.e., that Ax ≥ 0 whenever x ≥ 0), and that a column stochastic matrix A ≥ 0 maps probability vectors into probability vectors.

18 2. Prove that if A > 0, then its spectral radius ρ( A) is positive. Hint: use the Cayley-Hamilton theorem.

3. Show that it holds for any matrix A and vector x that

| Ax | ≤ | A|| x |, where | A| (resp. | x |) denotes the matrix (resp. vector) of moduli of the entries of A (resp. x). Then, show that if A > 0 and λi , vi is an eigenvalue/eigenvector pair for A, then

|λi ||vi | ≤ A|vi |.

4. Prove that if A > 0 then ρ( A) is actually an eigenvalue of A (i.e., A has a positive real eigenvalue λ = ρ( A), and all other eigenvalues of A have modulus no larger than this “dominant” eigenvalue), and that there exist a corresponding eigenvector v > 0. Further, the dominant eigenvalue is simple (i.e., it has unit algebraic multiplicity), but you are not requested to prove this latter fact.

Hint: For proving this claim you may use the following fixed-point theorem due to Brouwer: if S is a compact and convex set4 in Rn , and f : S → S is a continuous map, then there exist an x ∈ S such that . f ( x ) = x. Apply this result to the continuous map f ( x ) = 1>AxAx , with S being the probability simplex (which is indeed convex and compact).

5. Prove that if A > 0 and it is column or row stochastic, then its dominant eigenvalue is λ = 1.

Solution 3.11 (Nonnegative and positive matrices) 1. Let A ≥ 0, x ≥ 0, y = Ax, and denote with ai> the i-th row of A. Then obviously n

yi = ai> x = ∑ aij x j ≥ 0,

i = 1, . . . , n,

j =1

which shows that a nonnegative matrix maps nonnegative vectors into nonnegative vectors. Further, if x is a probability vector and A is stochastic, then 1> y = 1> Ax = 1> x = 1, which shows that y is also a probability vector.

2. Suppose by contradiction that A > 0 and ρ( A) = 0. This would imply that A has an eigenvalue of maximum modulus in λ = 0,

See Section 8.1 for definitions of compact and convex sets.

4

19 thus, all eigenvalues of A are actually zero. This means that the characteristic polynomial of A is p A (s) = sn and, by the CayleyHamilton theorem, it must hold that An = 0, which is impossible since An is the n-fold product of positive matrices, hence it must be positive.

3. By the triangle inequality, we have that, for i = 1, . . . , n,

| ai> x | ≤ =

n

n

j =1

j =1

∑ |aij x j | = ∑ |aij ||x j |

| ai> || x |,

whih proves the first part. If A > 0 the above relation reads | Ax | ≤ A| x | which, for x = vi , becomes A|vi | ≥ | Avi | = |λi vi | = |λi ||vi |. .

4. Let S be the probability simplex, and f ( x ) = 1>AxAx . From Brouwer’s fixed-point theorem there exist v ∈ S such that f (v) = v, that is such that .

Av = (1> Av)v = λv, λ = 1> Av. Moreover, since A > 0, it holds that λ > 0 and v > 0; thus A has a positive eigenvalue and a corresponding positive eigenvector. We next apply the same result to A> , obtaining that there exist w ∈ S such that A> w = (1> A> w)w = µw,

. µ = 1> A> w,

where again µ > 0 and w > 0. Now, v> w > 0, and λv> w = v> A> w = µv> w, which implies that µ = λ, whence A> w = λw.

Next, consider any eigenvalue/eigenvector pair λi , vi for A, and apply the result of point 3. in this exercise, to obtain that

|λi ||vi | ≤ A|vi |,

i = 1, . . . , n.

Multiply both sides on the left by w> to get w> |λi ||vi | ≤ w> A|vi | = λw> |vi |, from which we obtain that

|λi | ≤ λ,

i = 1, . . . , n,

which proves that λ (which is real and positive, as shown above) is indeed a maximum modulus eigenvalue of A (thus, λ = ρ( A)), and the corresponding eigenvector v is positive.

20 5. By definition, if A is column stochastic then 1> A = 1> , which means that λ = 1 is an eigenvalue of A. Next, recall from Section 3.6.3.1 that the spectral radius of A is no larger than its induced `1 norm: m

ρ( A) ≤ k Ak1 = max ∑ | aij | = 1, j=1,...,n i =1

hence λ = 1 is indeed the dominant eigenvalue. An analogous argument applies to row stochastic matrices.

21

4.

Symmetric matrices

Exercise 4.1 (Eigenvectors of a symmetric 2 × 2 matrix) Let p, q ∈ Rn be two linearly independent vectors, with unit norm (k pk2 = kqk2 = .

1). Define the symmetric matrix A = pq> + qp> . In your derivations, . it may be useful to use the notation c = p> q.

1. Show that p + q and p − q are eigenvectors of A, and determine the corresponding eigenvalues.

2. Determine the nullspace and rank of A. 3. Find an eigenvalue decomposition of A, in terms of p, q. Hint: use the previous two parts.

4. What is the answer to the previous part if p, q are not normalized?

Solution 4.1 1. We have Ap = (cp + q), Aq = p + cq, from which we obtain A( p − q) = (c − 1)( p − q), A( p + q) = (c + 1)( p + q).

Thus u± := p ± q is an (un-normalized) eigenvector of A, with eigenvalue c ± 1.

2. The condition on x ∈ Rn : Ax = 0, holds if and only if 0 = (q> x ) p + ( p> x )q = 0.

Since p, q are linearly independent, the above is equivalent to p> x = q> x = 0. The nullspace is the set of vectors orthogonal to p and q.

The range is the span of p, q. The rank is thus 2.

3. Since the rank is 2, there is a total of two non-zero eigenvalues.

Note that, since p, q are normalized, c is the cosine angle between p, q; |c| < 1 since p, q are independent. We have found two linearly independent eigenvectors u± = p ± q that do not belong to the nullspace (since |c| < 1). We can complete this set with eigenvectors corresponding to the eigenvalue zero; simply choose an orthonormal basis for the nullspace.

Then, the eigenvalue decomposition is > A = ( c − 1) v − v > − + ( c + 1) v + v + ,

22 where v± are the normalized vectors v± = u± /ku± k2 . We have v± = p

1 2(1 ± c )

( p ± q ),

so that the eigenvalue decomposition amounts to the trivial identity

1 A= ( p + q)( p + q)> − ( p − q)( p − q)> .

2

4. We can always scale the matrix: with p̄ = p/k pk2 , q̄ = q/kqk2 , we have

A = k pk2 kqk2 p̄q̄> + q̄ p̄> .

The eigenvalues are scaled accordingly: with c = ( p/k pk2 )> (q/kqk2 ), λ ± = k p k2 k q k2 ( c ± 1) = p > q ± k p k2 k q k2 .

Note that, since p, q are independent, none of these eigenvalues is zero; one is positive, the other negative. This is due to the Cauchy-Schwartz inequality, which says that | p> q| ≤ k pk2 kqk2 , with equality if and only if p, q are linearly dependent.

The corresponding (un-normalized) eigenvectors are p̄ ± q̄. The eigenvalue decomposition obtained before leads to A=

k p k2 k q k2 ( p̄ + q̄)( p̄ + q̄)> − ( p̄ − q̄)( p̄ − q̄)> .

2

Exercise 4.2 (Quadratic constraints) For each of the following cases, determine the shape of the region generated by the quadratic constraint x > Ax ≤ 1. # " 2 1 .

1. A = 1 2 # " 1 −1 2. A = . −1 1 " # −1 0 3. A = .

0 −1 Hint: use the eigenvalue decomposition of A, and discuss depending on the sign of the eigenvalues.

Solution 4.2 The region determined by x > Ax ≤ 1 is best understood in terms of an eigenvalue decomposition of A: A = UΛU > , with Λ = diag (λ1 , λ2 ) are the two eigenvalues of A, with by convention λ1 ≥ λ2 . We have x > Ax ≤ 1 ⇐⇒ λ1 x̄12 + λ2 x̄22 ≤ 1,

23 where x̄ = U > x (so that x = U x̄). Several cases can occur.

If λi > 0, i = 1, 2, in the rotated space where x̄ lives, the above set takes the shape of an ellipsoid centered at zero, with semi-axes √ lengths given by 1/ λi , and principal directions x̄ (i) = ei (the i-th unit vector), i = 1, 2. The principal directions in the original space are given by x (i) = Uei = ui , where ui is the i-th column of U, i = 1, 2.

Those vectors are nothing else than the eigenvectors of A. If λi < 0, i = 1, 2, the set is the whole space R2 .

If λ1 > 0, λ2 < 0, the set is an hyperboloid centered at zero, with principal directions given by the eigenvectors.

If one of the eigenvalues is zero, the set is a cylinder, again centered at zero. If λ1 > λ2 = 0, then the set is characterized by √ | x̄1 | ≤ 1/ λ1 . If 0 = λ1 > λ2 , the set is the outside of the set √ characterized by | x̄2 | ≤ 1/ −λ2 .

1. The eigenvalues are characterized by 0 = det(λI − A) = (λ − 2)2 − 1, which gives λ = 3, 1. We then solve for u ∈ R2 in Au = λu, which gives two equations that are redundant by construction. We obtain from the first one u1 = u2 (if λ = 3) and u1 = −u2 (if λ = 1). Two normalized eigenvectors for λ = 3, 1 respectively, are " # " # 1 1 1 1 , u2 = √ . u1 = √ 2 1 2 −1 We obtain 3 A= 2

"

1 1

#"

1 −1

#>

1 + 2

"

1 −1

#"

1 1

#>

.

The region determined by x > Ax ≤ 1 is 3x̄12 + x̄22 ≤ 1,

√ √ with x̄1 = ( x1 − x2 )/ 2, x̄2 = ( x1 + x2 ) 2. This corresponds to an ellipsoid centered at zero with semi-axis length of 3, 1, with √ length 1/ 3 associated with the direction (1, 1) and the length 1 associated with the direction (1, −1); the ellipsoid is rotated anticlockwise by a 45◦ degree angle.

2. The set is characterized by 1 ≥ x > Ax = x1 ( x1 − x2 ) + x2 (− x1 + x2 ) = ( x1 − x2 )2 , which translates as the slab −1 ≤ x1 − x2 ≤ 1. The slab is parallel to the direction (1, 1), and is between the two lines x2 = x1 ± 1.

24 3. The set is characterized by − x12 − x22 ≤ 1, which is the whole space R2 .

Exercise 4.3 (Drawing an ellipsoid) 1. How would you efficiently draw an ellipsoid in R2 , if the ellipsoid is described by a quadratic inequality of the form n o E = x > Ax + 2b> x + c ≤ 0 ,

where A is 2 × 2 and symmetric, positive-definite, b ∈ R2 , and c ∈ R? Describe your method as precisely as possible.

2. Draw the ellipsoid n o E = 4x12 + 2x22 + 3x1 x2 + 4x1 + 5x2 + 3 ≤ 1 .

Solution 4.3 1. Since A is positive-definite, it admits a Cholesky decomposition, if the form A = R> R, with R upper-triangular and invertible. In terms of the new variable x̄ = Rx, we have x ∈ E if and only if 1 ≥ x̄ > x̄ − 2x̄0> x̄ + c = k x̄ − x̄0 k22 + c − k x̄0 k22 . where x̄0 = −( R> )−1 b. The set is empty when 1 + k x̄0 k22 < c. Otherwise, the inequality writes k x̄ − x̄0 k2 ≤ ρ, with ρ2 = 1 + k x̄0 k22 − c. In the x̄-space, the set is a circle with center x0 and radius ρ.

The set in the x-space is then obtained by the linear transformation x = R−1 x̄. The resulting set is an ellipsoid is centered at x0 = R−1 x̄0 = − A−1 b, and principal axes and directions given by the singular value decomposition of R (we don’t need to detail those to draw the ellipsoid).

To produce a set of N points on the ellipsoid, encoded in a 2 × N matrix X, the method starts by generating a set of N points on a unit circle, encoded in a 2 × N matrix Z; the set of points in xspace is then obtained by scaling by ρ, adding x̄0 defined above to each column, and multiplying the resulting matrix by the matrix R−1 . That is:

X = R−1 ρ · Z + x̄0 1> , with 1 the vector of ones in R N .

2. We have now A=

"

4 3/2

3/2 2

#

, b=

"

2 5/2

#

, c = 2.

25 A Cholesky decomposition of A is A = R T R, with " # 2 3/4 R= .

0 1.19 We have x̄0 = −( R> )−1 b = (−1, −1.4596), and ρ = 1.4596. The result is shown in Fig 4.3.

Exercise 4.4 (Minimizing a quadratic function) Consider the unconstrained optimization problem p∗ = min x

1 > x Qx − c> x 2

where Q = Q> ∈ Rn,n , Q 0, and c ∈ Rn are given. The goal of this exercise is to determine the optimal value p∗ and the set of optimal solutions, X opt , in terms of c and the eigenvalues and eigenvectors of the (symmetric) matrix Q.

Figure 4.3: An ellipsoid in two dimensions.

1. Assume that Q 0. Show that the optimal set is a singleton, and that p∗ is finite. Determine both in terms of Q, c.

2. Assume from now on that Q is not invertible. Assume further that Q is diagonal: Q = diag (λ1 , . . . , λn ), with λ1 ≥ . . . ≥ λr > λr+1 = . . . = λn = 0, where r is the rank of Q (1 ≤ r < n). Solve the problem in that case (you will have to distinguish between two cases).

3. Now we do not assume that Q is diagonal anymore. Under what conditions (on Q, c) is the optimal value finite? Make sure to express your result in terms of Q and c, as explicitly as possible.

4. Assuming that the optimal value is finite, determine the optimal value and optimal set. Be as specific as you can, and express your results in terms of the pseudo-inverse5 of Q. Solution 4.4 1. When Q 0, it admits a Cholesky decomposition Q = R> R, with R upper-triangular and invertible. We can define the new variable x̄ = Rx, which leads to the problem min x̄

1 > x̄ x̄ − c̄> x̄, 2

where c̄ = ( R−1 )> c. We can express the objective in the above problem as 1 k x̄ − c̄k22 − kc̄k22 , 2

5

See Section 5.2.3.

26 from which it is clear that the unique minimizer is x̄ = c̄. In terms of the x-variable, the unique solution is x = R−1 c̄ = Q−1 b.

The same result can be obtained by invoking the fact that the minimizers of a convex differentiable function f without any constraints, are characterized by the optimality condition 6 ∇ f ( x ) = 0. In our case, we have ∇ f ( x ) = Qx − c.

2. The objective function writes

n r 1 2 λi xi − ci xi + ∑ ci xi . f (x) = ∑ 2 i =r +1 i =1 If any element ci , i = r + 1, . . . , n, is non-zero, the optimal value is −∞. Otherwise, that is, when c is in the range of Q, the optimal value is obtained with xi = ci /λi , i = 1, . . . , r, and the other variables xr+1 , . . . , xn free. That value is p∗ = −

1 r c2i .

2 i∑ λ =1 i

3. We use the eigenvalue decomposition of Q: Q = UΛU > , with Λ = diag (λ1 , . . . , λn ). The problem is more conveniently formulated in terms of the new variable x̄ = U > x: 1 p∗ = min x̄ > Λ x̄ − c̄> x̄, x̄ 2 . with c̄ = Uc. Assuming as before λ1 ≥ . . . ≥ λr > λr+1 = . . . = λn = 0, where r is the rank of Q (1 ≤ r < n), we are lead to the similar conclusions. In particular, the optimal value is finite if and only if the last n − r components of c̄ are zero. This means that c must be in the range of Q, in order for the value to be finite.

4. Let us assume that the optimal value is finite. In that case, the optimal set is the set of points U x̄, where x̄r+1 , . . . , x̄n are free, and xi = c̄i /λi , i = 1, . . . , r. The solution corresponding to x̄r+1 = . . . = x̄n = 0 is nothing else than Q† c, where Q† = UΛ† U > is the pseudo-inverse of Q, defined via Λ† = diag (1/λ1 , . . . , 1/λr , 0, . . . , 0) .

The optimal value is then p∗ = −

1 r c̄2i 1 = − c> Q† c, 2 i∑ λ 2 =1 i

and the optimal set is of the form x = Q† c + ( I − QQ† )v, with v a free variable.

6

See Section 8.4.1.

27 Exercise 4.5 (Interpretation of covariance matrix) As in Example 4.2, we are given m data points x (1) , . . . , x (m) in Rn , and denote by Σ the sample covariance matrix: . 1 m (i ) Σ= ( x − x̂ )( x (i) − x̂ )> , m i∑ =1 where x̂ ∈ Rn is the sample average of the points: . 1 m (i ) x̂ = x . m i∑ =1 We assume that the average and variance of the data projected along a given direction does not change with the direction. In this exercise we will show that the sample covariance matrix is then proportional to the identity.

We formalize this as follows. To a given normalized direction w ∈ Rn , kwk2 = 1, we associate the line with direction w passing through the origin, L(w) = {tw : t ∈ R }. We then consider the projection of the points x (i) , i = 1, . . . , m, on the line L(w), and look at the associated coordinates of the points on the line. These projected values are given by . ti (w) = arg min ktw − x (i) k2 , i = 1, . . . , m. t

We assume that for any w, the sample average t̂(w) of the projected values ti (w), i = 1, . . . , m, and their sample variance σ2 (w), are both constant, independent of the direction w. Denote by t̂ and σ2 the (constant) sample average and variance. Justify your answer to the following questions as carefully as you can.

1. Show that ti (w) = w> x (i) , i = 1, . . . , m.

2. Show that the sample average x̂ of the data points is zero.

3. Show that the sample covariance matrix Σ of the data points is of the form σ2 In . Hint: the largest eigenvalue λmax of the matrix Σ can be written as: λmax = maxw {w> Σw : w> w = 1}, and a similar expression holds for the smallest eigenvalue.

Solution 4.5 1. The result is obtained by noting that, when kwk2 = 1, we have for any x ∈ Rn :

ktw − x k22 = (t − t∗ )2 + constant, where t∗ = x > w. Minimizing the above over t gives t = t∗ .

28 2. We have that

m

∑ ti (w) = x̂> w

i =1

is a constant on the unit sphere, independent of w. This is only possible if x̂ = 0. Indeed, taking w = ei (the i-th unit vector in Rn ), we see that all the elements of x̄ must be equal, so that x̄ = β1, with β ∈ R and 1 the vector of ones in R. If β 6= 0, we obtain that the linear fuction w → 1> w is constant over the unit sphere, a contradiction. Hence β = 0, and thus x̂ = 0.

3. The sample variance of the projected values is given by σ2 ( w ) =

1 m 1 m t i ( w )2 = (w> x (i) )2 = w> Σw, ∑ m i =1 m i∑ =1

where we have used that x̂ = 0, which implies Σ=

1 m (i ) (i ) > x (x ) . m i∑ =1

Since σ2 (w) is a constant on the unit sphere, the function w → w> Σw is also constant on the unit sphere. Thus, the largest and smallest eigenvalues coincide, and Σ is proportional to the identity: Σ = αI, with α ∈ R. The proportion factor is σ2 , since σ2 = w> Σw = αw> w = α.

Exercise 4.6 (Connected graphs and the Laplacian) We are given a graph as a set of vertices in V = {1, . . . , n}, with an edge joining any pair of vertices in a set E ⊆ V × V. We assume that the graph is undirected (without arrows), meaning that (i, j) ∈ E implies ( j, i ) ∈ E. As in Section 4.1, we define the Laplacian matrix by    −1 if (i, j) ∈ E, Lij = d(i ) if i = j,   0 otherwise.

Picture 1.png 250×165 pixels

Figure 4.4: Example of an undirected graph.

Here, d(i ) is the number of edges adjacent to vertex i. For example, d(4) = 3 and d(6) = 1 for the graph in Figure 4.4.

1. Form the Laplacian for the graph shown in Figure 4.4.

2. Turning to a generic graph, show that the Laplacian L is symmetric.

3. Show that L is positive-semidefinite, proving the following identity, valid for any u ∈ Rn : . 1 ( u i − u j )2 . u> Lu = q(u) = 2 (i,j∑ )∈ E

http://cnx.org/content/m29399/1.1/Picture%201.png

29 Hint: find the values q(k), q(ek ± el ), for two unit vectors ek , el such that (k, l ) ∈ E. 4. Show that 0 is always an eigenvalue of L, and exhibit an eigenvector. Hint: consider a matrix square-root7 of L. 5. The graph is said to be connected if there is a path joining any pair of vertices. Show that if the graph is connected, then the zero eigenvalue is simple, that is, the dimension of the nullspace of L is 1. Hint: prove that if u> Lu = 0, then ui = u j for every pair (i, j) ∈ E. Solution 4.6 1. The Laplacian for the graph is  2 −1 0   −1 3 −1   0 −1 2 L=  0 0 −1    −1 −1 0 0 0 0

0 0 −1 3 −1 −1

 −1 0  −1 0   0 0  . −1 −1    3 0  0 1

2. L is symmetric, by definition of the edge set E. 3. Let us prove that the expression is valid by looking at the values of the quadratic form at specific points. Let . 1 q(u) = ( u i − u j )2 .

2 (i,j∑ )∈ E Clearly, q is of the form u> Qu for some symmetric matrix Q. Note that q(u) ≥ 0 for every u, hence Q is positive semi-definite. Let us show that Q = L, which will imply that L is positive semi-definite.

Expanding, we obtain q(u) =

∑ d(i)u2i −

(i,j)∈ E

∑

ui u j .

i 6= j, (i,j)∈ E

We have for every k ∈ {1, . . . , n}: Qkk = q(ek ) =

∑

1 = d ( k ),

j : (k,j)∈ E

We next find Qkl for k 6= l, thanks to the formula Qkl =

1 (q(ek + el ) − q(ek − el )).

2

7

See Section 4.4.4.

30 If (k, l ) ∈ E, k 6= l, we have q(ek + el ) = d(k) + d(l ) − 1, q(ek − el ) = d(k) + d(l ) + 1, which implies that Qkl = −1. Otherwise, both terms are zero, and Qkl = 0. This proves the desired result.

4. Since L is PSD, we can write L = R> R, for some matrix R. Let 1 be the vector of ones in Rn . We have q(1) = k R1k22 = 0, which implies R1 = 0, and in turn, L1 = 0. Hence 0 is always an eigenvalue of L, associated for the eigenvector 1.

5. If u is a vector such that q(u) = 0, then from the expression of q, uk = ul for every pair of vertices (k, l ) ∈ E. Now let (k, l ) ∈ V × V, (k, l ) 6∈ E be a pair of vertices that are not directly connected.

Since the graph is connected, there is a path of vertices connecting vertex k and l, that is, a sequence of indices such that i0 = k ≤ i1 ≤ . . . ≤ im = l, for some path length m > 1, and with (is , is+1 ) ∈ E, s = 0, . . . , m − 1. Since uis = uis+1 , we obtain by induction on s that uk = ui0 = uim = ul , which shows that u is proportional to the vector of ones, as claimed.

Exercise 4.7 (Component-wise product and PSD matrices) Let A, B ∈ Sn be two symmetric matrices. Define the component-wise product of A, B, by a matrix C ∈ Sn with elements Cij = Aij Bij , 1 ≤ i, j ≤ n.

Show that C is positive semidefinite, provided both A, B are. Hint: prove the result when A is rank-one, and extend to the general case via the eigenvalue decomposition of A. Solution 4.7 If A is rank-one, we can express it as A = vv> for some v ∈ Rn . We then have, for an arbitrary vector z ∈ Rn : z> Cz = ∑ zi z j Aij Bij = ∑ zi z j vi v j Bij = y> By, i,j

i,j

where y is the n-vector with i-th component vi zi , i = 1, . . . , n. The proof follows from the positive semi-definiteness of B. When A is of arbitrary rank r ≤ n, we can write r

A = ∑ v(k) ( v(k) )> , k =1

for some vectors v(k) , k = 1, . . . , r. For an arbitrary vector z ∈ Rn : r

(k) (k)

r

z> Cz = ∑ ∑ zi z j vi v j Bij = ∑ (y(k) )> By(k) , k=1 i,j

k =1

31 where, for k = 1, . . . , r, y(k) is the n-vector with i-th component (k)

vi zi , i = 1, . . . , n. The proof again follows from the positive semidefiniteness of B. Exercise 4.8 (A bound on the eigenvalues of a product) Let A, B ∈ Sn be such that A 0, B 0.

1. Show that all eigenvalues of BA are real and positive (despite the fact that BA is not symmetric, in general).

. > 2. Let A 0, and let B−1 = diag k a1> k1 , . . . , k a> n k1 , where ai , i = 1, . . . , n, are the rows of A. Prove that 0 < λi ( BA) ≤ 1,

∀ i = 1, . . . , n.

3. With all terms defined as in the previous point, prove that ρ( I − αBA) < 1,

∀α ∈ (0, 2).

Solution 4.8 (A bound on the eigenvalues of a product) 1. This point is an immediate consequence of Corollary 4.5.

2. From the first point, we have that 0 < λmax ( BA) = max |λi ( BA)| = ρ( BA) ≤ k BAk∞ , i =1,...,n

where the last inequality is proved in Section 3.6.3.1. Further, since B is positive and diagonal,

k BAk∞ = max[ B| A|1]i = max Bii | ai> |1 = 1, i

i

whence 0 < λmax ( BA) ≤ 1, as desired.

3. We have that ρ( I − αBA) < 1 iff |1 − αλi ( BA)| < 1 for all i, iff 0 < αλi ( BA) < 2,

for all i.

Since 0 < λi ( BA) ≤ 1 for all i, the above condition requires α > 0 and α max λi ( BA) < 2, i

which is satisfied if α < 2, since maxi λi ( BA) ≤ 1.

32 Exercise 4.9 (Hadamard’s inequality) Let A ∈ Sn be positive semidefinite. Prove that n

det A ≤ ∏ aii . i =1

Hint: Distinguish the cases det A = 0 and det A 6= 0. In the lat. ter case, consider the normalized matrix à = DAD, where D =

−1/2 1/2 diag a11 , . . . , a− , and use the geometric/arithmetic mean innn equality (see Example 8.9).

Solution 4.9 (Hadamard’s inequality) First observe that A 0 implies aii ≥ 0 for all i, hence ∏i aii ≥ 0. Thus, the inequality is satisfied if det A = 0. Suppose next det A 6= 0, that is, A is invertible (hence positive definite), thus aii > 0 for all i. Let then

−1/2 1/2 D = diag a11 , . . . , a− and observe that nn

.

à = DAD

has all diagonal entries equal to one, and det à ≤ 1 if an only if det A ≤ ∏i aii . Therefore, it sufficies to prove that det à ≤ 1. Indeed, !n n 1 1 det à = ∏ λi ( Ã) ≤ = 1, λi ( Ã) = trace à n∑ n i i where the first inequality follows from the geometric/arithmetic mean inequality.

Exercise 4.10 (A lower bound on the rank) Let A ∈ Sn+ be a symmetric, positive semi-definite matrix.

1. Show that the trace, trace A, and the Frobenius norm, k Ak F , depend only on its eigenvalues, and express both in terms of the vector of eigenvalues.

2. Show that

(trace A)2 ≤ rank( A)k Ak2F .

3. Identify classes of matrices for which the corresponding lower bound on the rank is attained.

Solution 4.10 1. We begin with an eigenvalue decomposition of A = UΛU > , where Λ = diag (λ, . . . , λn ) contains the (non-negative) eigenvalues, and the columns of the matrix U are eigenvectors. Since UU > = I, we have n

trace A = trace(UΛU > ) = trace(ΛU > U ) = trace Λ = ∑ λi . i =1

33 Since λi ≥ 0, i = 1, . . . , n, we obtain that the trace is simply the . `1 -norm of the vector of eigenvalues λ = (λ1 , . . . , λn ): trace A = kλk1 .

Likewise, the Frobenius norm depends only on the eigenvalues:

k Ak2F

= trace( A> A) = trace(UΛU > UΛU > ) = trace(UΛ2 U > ) = trace(Λ2 ) n

=

∑ λ2i .

i =1

We obtain k Ak F = kλk2 .

2. The rank of A is simply the number of eigenvalues that are nonzero. Without loss of generality, we assume that the eigenvalues are ordered in decreasing fashion: λ1 ≥ . . . ≥ λr > λr+1 = . . . = λn = 0.

Define the vector µ = (λ1 , . . . , λr ) ∈ Rr . We have trace A = kµk1 , k Ak F = kµk2 .

The inequality follows from the Cauchy-Schwartz inequality: with 1r the vector of ones in Rr , we have √ k µ k1 = µ > 1r ≤ k 1r k2 · k µ k2 = r k µ k2 , which proves the desired result.

√ 3. The lower bound is attained when kµk1 = r kµk2 . According to the Cauchy-Schwartz inequality result8 , this implies that µ, 1r are collinear, which means that µ = α1r for some α ∈ R+ . The matrices for which the lower bound on the rank is attained are of the form . r A = αB, B = ∑ u(k) (u(k) )> , k =1

where α ≥ 0, u(k) , k = 1, . . . , r are r orthonormal vectors in Rn . Up

to a non-negative scaling factor α, such matrices are orthogonal projections on subspaces (with basis given by the vectors u(k) ).

Exercise 4.11 (A result related to Gaussian distributions) Let Σ ∈ Sn++ be a symmetric, positive definite matrix. Show that Z √ 1 > −1 e− 2 x Σ x dx = (2π )n/2 det Σ.

Rn

8

See Section 2.2.

34 You may assume known that the result holds true when n = 1. The above shows that the function p : Rn → R with (non-negative) values 1 > −1 1 √ p( x ) = e− 2 x Σ x (2π )n/2 · det Σ

integrates to one over the whole space. In fact, it is the density function of a probability distribution called the multivariate Gaussian (or Normal) distribution, with zero mean and covariance matrix Σ. Hint: you may use the fact that for any integrable function f , and invertible n × n matrix P, we have Z

x ∈Rn

f ( x )dx = | det P| ·

Z

f ( Pz)dz.

z ∈Rn

Solution 4.11 Let Σ = R> R be Cholesky decomposition of Σ 0, so that Σ = R> R, with R a n × n upper-triangular, invertible matrix. We have Σ−1 = R−1 ( R−1 )> , and thus q( x )

. = x > Σ −1 x

= x > R −1 ( R −1 ) > x = z> z, where R> z = x. The transformation x → z is a valid change of variable, and the infinitesimal volume dx is expressed as γdz, with √ γ = | det R> | = | det R| = det Σ.

We have, using the hint, Z

1 > −1 x

Rn

e− 2 x Σ

dx

=

Z

1

Rn

= γ·

Z

e− 2 k( R

−1 ) > x k 2 2 dx

1

Rn

2

e− 2 kzk2 dz

= γ · Πin=1 Z = γ·

Z

1 2

e− 2 zi dzi n − 12 η 2 dη . e

z i ∈R

η ∈R

The result follows from the fact that, as implied from the case n = 1, Σ = 1: Z √ 1 2 e− 2 η dη = 2π. η ∈R

35

5.

Singular Value Decomposition

Exercise 5.1 (SVD of an orthogonal matrix) Consider the matrix   −1 2 2 1  A =  2 −1 2  .

3 2 2 −1

1. Show that A is orthogonal.

2. Find a singular value decomposition of A. Solution 5.1 1. The columns are easily shown to be mutually orthogonal, and have unit Euclidean norms.

2. Since A is orthogonal, a valid SVD of A is UΣV > , with U = A, Σ = V = I. As usual, there is no unicity: we can also choose U = I, V = A> .

Exercise 5.2 (SVD of a matrix with orthogonal columns) Assume a matrix A = [ a1 , . . . , am ] has columns ai ∈ Rn , i = 1, . . . , m that are orthogonal to each other: ai> a j = 0 for 1 ≤ i 6= j ≤ n. Find an SVD for A, in terms of the ai ’s. Be as explicit as you can.

Solution 5.2 Assume first that none of the ai ’s is zero.Let σi = k ai k2 , . ui = ai /σi , i = 1, . . . , n. By assumption the matrix U = (u1 , . . . , un ) is orthogonal. In addition, we have A = UΣ, where Σ = diag (σ1 , . . . , σn ). The above is an SVD of A, with right singular vector matrix V = I. Exercise 5.3 (Singular values of augmented matrix) Let A ∈ Rn,m , with n ≥ m, have singular values σ1 , . . . , σm .

1. Show that the singular values of the (n + m) × m matrix " # A .

à = Im are σ̃i =

q

1 + σi2 , i = 1, . . . , m.

2. Find an SVD of the matrix Ã.

Solution 5.3

36 1. We have Ã> à = A> A + I. Since the eigenvalues of A> A are σi2 , i = 1, . . . , m, those of the shifted matrix Ã> à are σi2 + 1, i = 1, . . . , m. Hence, the singular values of à satisfy the desired relation with those of A. 2. Let A = USV > be an SVD of A, with the familiar notation. The SVD of à = Ũ S̃Ṽ > has S̃ = diag (σ̃1 , . . . , σ̃m ); matrix Ũ contain the eigenvectors of à Ã> , while Ṽ contains those of Ã> Ã. Since Ã> à = A> A + I, we can choose Ṽ = V. The condition for (ũ, ṽ) to be a pair of left- and right-singular vectors of à are that both vectors must be unit-norm, and Ãṽ = σ̃ũ, Ã> ũ = σ̃ṽ.

We have seen that we can choose ṽ = v to be an eigenvector of A> A (that is, a right singular vector of A). Further, decomposing ũ = (ũ1 , ũ2 ), with ũ1 ∈ Rn , we obtain Av = σ̃ũ1 , v = σ̃ũ2 , A> ũ1 + ũ2 = σ̃v.

Solving for the second equation: ũ2 = v/σ̃, we obtain from the third A> ũ1 = (σ̃ − 1/σ̃)v. Multiplying by A, and with the first equation, we then obtain AA> ũ1 = σ̃(σ̃ − 1/σ̃)ũ1 = σu1 .

This shows that we can set ũ1 to be proportional to a left singular vector u of A, and ũ2 = v/σ̃ proportional to v. We have " # αu ũ = , 1 σ̃ v where α must be chosen so that the above has unit Euclidean norm, that is: σ α= √ .

2 σ +1 We have obtained that a generic pair of left- and right singular √ vectors (ũ, ṽ) of à corresponding to the singular value σ2 + 1, can be constructed from a generic pair of left- and right singular vectors (u, v) of A corresponding to the singular value σ, with the choice " # √ σ u 2 +1 σ ũ = , ṽ = v. √ 1 v 2 σ +1

Exercise 5.4 (SVD of score matrix) An exam with m questions is given to n students. The instructor collects all the grades in a n × m

37 matrix G, with Gij the grade obtained by student i on question j. We would like to assign a difficulty score to each question, based on the available data.

1. Assume that the grade matrix G is well approximated by a rankone matrix sq> , with s ∈ Rn and q ∈ Rm (you may assume that both s, q have non-negative components). Explain how to use the approximation to assign a difficulty level to each question. What is the interpretation of vector s?

2. How would you compute a rank-one approximation to G? State precisely your answer in terms of the SVD of G. Solution 5.4 1. We have Gij ≈ si q j , i = 1, . . . , n, j = 1, . . . , m. In this rank-one model, each student i is characterized by an ability level si , and each question j has a difficulty level j.

2. We solve for the rank-one approximation problem min

q≥0, s≥0

k G − sq> k F

Ignoring the non-negativity constraints, we obtain s, q by the SVD of G. If σ1 is the largest singular value, and (u1 , v1 ) corresponding √ √ left- and right singular vectors, we set s = σ1 u1 , q = σ1 v1 .

Exercise 5.5 (Latent semantic indexing) Latent semantic indexing is an SVD-based technique that can be used to discover text documents similar to each other. Assume that we are given a set of m documents D1 , . . . , Dm . Using a “bag-of-words” technique described in Example 2.1, we can represent each document D j is described by an n-vector d j , where n is the total number of distinct words appearing in the whole corpus. In this exercise, we assume that the vectors d j are constructed as follows: d j (i ) = 1 if word i appears in document D j , and 0 otherwise. We refer to the n × m matrix M = [d1 , . . . , dm ] as the “raw” term-by-document matrix. We will also use a normalized9 version of that matrix: M̃ = [d˜1 , . . . , d˜m ], where d˜j = d j /kd j k2 , j = 1, . . . , m.

Assume we are given another document, referred to as the “query document,” which is not part of the collection. We describe that query document as a n-dimensional vector q, with zeros everywhere, except a 1 at indices corresponding to the terms that appear in the query. We seek to retrieve documents that are “most similar” to the query, in some sense. We denote by q̃ the normalized vector q̃ = q/kqk2 .

In practice, other numerical representation of text documents can be used. For example we may use the relative frequencies of words in each document, instead of the l2 -norm normalization employed here.

9

38 1. A first approach is to select the documents that contain the largest number of terms in common with the query document. Explain how to implement this approach, based on a certain matrix-vector product, which you will determine.

2. Another approach is to find the closest document by selecting the index j such that kq − d j k2 is the smallest. This approach can introduce some biases, if for example the query document is much shorter than the other documents. Hence a measure of similarity based on the normalized vectors, kq̃ − d˜j k2 , has been proposed, under the name of “cosine similarity”. Justify the use of this name for that method, and provide a formulation based on a certain matrix-vector product, which you will determine.

3. Assume that the normalized matrix M̃ has an SVD M̃ = UΣV > , with Σ a n × m matrix containing the singular values, and the unitary matrices U = [u1 , . . . , un ], V = [v1 , . . . , vm ] of size n × n, m × m respectively. What could be an interpretation of the vectors ul , vl , l = 1, . . . , r? Hint: discuss the case when r is very small, and the vectors ul , vl , l = 1, . . . , r, are sparse.

4. With real-life text collections, it is often observed that M is effectively close to a low-rank matrix. Assume that a optimal rank-k approximation (k min(n, m)) of M̃, M̃k , is known. In the Latent Semantic Indexing approach10 to document similarity, the idea is to first project the documents and the query onto the sub-space generated by the singular vectors u1 , . . . , uk , and then apply cosine similarity approach to the projected vectors. Find an expression for the measure of similarity.

Solution 5.5 1. If d ∈ Rn is a specific document, with di indicating the presence or absence of word i in the document, then q> d is the number of common terms between the document and the query.

For j = 1, . . . , m, the number of common terms between the query q and the j-th document D j is given by q> Me j , with e j the j-th unit vector in Rm . Hence the m-vector M> q gives the number of co-occurring terms between the query and each document. We select the closest documents by selecting indices j that achieve the maximum, that is j ∈ arg max ( M> q) j .

1≤ j ≤ m

In practice, it is often observed that this method produces better results than cosine similarity in the original space, as in part 2.

10

39 2. With d˜ the normalized vector corresponding to a generic document, we have

kq̃ − d˜k22

= kq̃k22 + kd˜k22 − 2q̃> d˜

= kq̃k22 + 1 − 2 cos(θ ),

where θ is the angle between the two vectors11 d, q. This explains the name of the method, which seeks the document D j such that the angle between q and d j is the smallest.

The cosine similarity approach is based on the matrix-vector product M̃> q̃: it involves ordering that vector by decreasing magnitude, and selecting the index j with the largest: j ∈ arg max ( M̃> q̃) j .

1≤ j ≤ m

3. We can write r

M = ∑ σl ul v> l , l =1

which can be interpreted as follows. Each ul corresponds to a nvector assigning a (positive or negative) weight to each term in the dictionary. Hence ul corresponds to a synthetic document.

The vectors vl assign a weight to each document, hence can be understood as a “concept”, if we accept the idea that a concept is a weighted list of words. Indeed, if vl is sparse, the indices where it is non-zero correspond to a short list of terms that can be understood as a concept. The term-by-document matrix M is a linear combination of concept and synthetic document pairs.

4. For a generic document D j in the text collection, we have r

d j = Me j = ∑ σl (v> l e j )ul . l =1

Thus, the coordinates of d j in the orthonormal basis (u1 , . . . , un ) are (σl (v> l e j ))1≤ j≤n , with the convention σr +1 = . . . = σn = 0. The projection onto the subspace spanned by the first k vectors of the basis is the k-vector dˆj with the same k first coordinates as d: k

dˆj = ∑ σl (v> l e j )ul . l =1

For a query document q, we can write n

q = ∑ (q> ul )ul . l =1

11

See Section 2.2.2.2.

40 The projection q̂ onto the subspace spanned by the (u1 , . . . , uk ) is k

q̂ = ∑ (q> ul )ul . l =1

The Latent Semantic Indexing approach is based on evaluating the angles between the projected query q̂ and the projected documents dˆj , j = 1, . . . , m.

Exercise 5.6 (Fitting a hyperplane to data) We are given m data points d1 , . . . , dm ∈ Rn , and we seek an hyperplane .

H(c, b) = { x ∈ Rn : c> x = b} where c ∈ Rn , c 6= 0, and b ∈ R, that best “fits” the given points, in the sense of a minimum sum of squared distances criterion.

Formally, we need to solve the optimization problem m

min c,b

4

3

2

1

0

−1

−2

−3

−4 −3

∑ dist (di , H(c, b)) : kck2 = 1,

−2

−1

0

1

2

3

2

i =1

where dist(d, H) is the Euclidean distance from a point d to H. Here the constraint on c is imposed without loss of generality, in a way that does not favor a particular direction in space.

1. Show that the distance from a given point d ∈ Rn to H is given by dist(d, H(c, b)) = |c> d − b|.

2. Show that the problem can be expressed as min

b,c : kck2 =1

f 0 (b, c)

where f 0 is a certain quadratic function, which you will determine.

3. Show that the problem can be reduced to min

c> ( D̃ D̃ > )c

s.t.:

kck2 = 1,

c

where D̃ is the matrix of centered data points: the i-th column of .

D̃ is di − d,¯ where d¯ = (1/m) ∑im=1 di is the average of the data points. Hint: you can exploit the fact that at optimum, the partial derivative of the objective function with respect to b must be zero, a fact justified in Chapter 8.4.1.

4. Explain how to find the hyperplane via SVD.

Figure 5.5: Fitting an hyperplane to data.

41 Solution 5.6 1. We have from (2.6) that >

dist(d, H) = |c d − b| = [d

>

− 1]

"

c b

#

.

2. Letting D ∈ Rn,m be the matrix having di as columns, D = [d1 · · · dm ], the problem objective becomes #!2 c ∑ [di> − 1] b i =1 " # 2 c [ D> − 1] b 2 " #> " #" # > c DD −md¯ c b −md¯> m b m

f0

= =

=

"

= c> ( DD > )c − 2mbc> d¯ + mb2 , where

m

1 1 d¯ = D1 = di m m i∑ =1 is the barycenter (average) of the given points.

3. Since the problem is unconstrained in variable b, we can partially minimize f 0 with respect to b by simply imposing that ∂ f0 = −2mc> d¯ + 2mb = 0, ∂b resulting in b = c> d,¯ which, substituted back in f 0 gives f˜0 (c) = c> Hc, Notice that matrix

.

H = DD > − md¯d¯> = D

Im −

1 > 11 m

D> .

1 .

E = Im − 11> m

is symmetric and idempotent, that is EE = E, therefore H = D̃ D̃ > , where

1 > ¯ > = D − d1 D̃ = DE = D Im − 11 m represent the matrix of centered data points, i.e., the i-th column of D̃ is di − d,¯ where d¯ is the average of the data points.

42 4. From Theorem 4.3, the optimal objective value of our problem min

c> ( D̃ D̃ > )c

s.t.:

kck2 = 1,

c

is the minimum eigenvalue of H = D̃ D̃ > , which coincides with the smallest singular value of D̃. The optimal value is f 0∗ = λmin ( D̃ D̃ > ) = σn2 , with σn the smallest singular value of D̃. The optimal value is achieved for c = un , being un the left singular vector of D̃ corresponding to the smallest singular value σn .

To summarize, the problem of finding the hyperplane that best fits given points d1 , . . . , dm can be solved as follows: we construct the centered data points matrix D̃ ∈ Rn,m , and find its minimum singular value σn and the corresponding left singular vector un .

Then, the best fitting hyperplane is > ¯ H = { x : u> n x = u n d },

where d¯ is the barycenter of the data points.

An interesting interpretation of this result is that we found a direction in data space, un , along which the centered data have the least variation, in the mean-square sense, meaning that the mean-square deviation of the centered data from the flat H, along the direction un , is minimal and equal to σn2 .

Exercise 5.7 (Image deformation) A rigid transformation is a mapping from Rn to Rn that is the composition of a translation and a rotation. Mathematically, we can express a rigid transformation φ as φ( x ) = Rx + r, where R is an n × n orthogonal transformation and r ∈ Rn a vector.

We are given a set of pairs of points ( xi , yi ) in Rn , i = 1, . . . , m, and wish to find a rigid transformation that best matches them. We can write the problem as m

∑ k Rxi + r − yi k22 : R> R = In , R ∈R , r ∈Rn min n,n

(5.2)

i =1

where In is the n × n identity matrix.

The problem arises in image processing, to provide ways to deform an image (represented as a set of two-dimensional points) based on the manual selection of a few points and their transformed counterparts.

Figure 5.6: Image deformation via rigid transformation. The image on the left is the original image, and that on the right is the deformed image.

Yellow dots indicate points for which the deformation is chosen by the user.

43 1. Assume that R is fixed in problem (5.2). Express an optimal r as a function of R. 2. Show that the corresponding optimal value (now a function of R only) writes as the original objective function, with r = 0 and xi , yi replaced with their centered counterparts, x̄i = xi − x̂, x̂ =

1 m 1 m x j , ȳi = yi − ŷ, ŷ = yj. ∑ m j =1 m j∑ =1

3. Show that the problem can be written as min k RX − Y k F : R> R = In , R

for appropriate matrices X, Y, which you will determine. Hint: explain why you can square the objective; then expand.

4. Show that the problem can be further written as max trace RZ : R> R = In , R

for an appropriate n × n matrix Z, which you will determine.

5. Show that R = VU > is optimal, where Z = USV > is the SVD of Z. Hint: reduce the problem to the case when Z is diagonal, and use without proof the fact that when Z is diagonal, In is optimal for the problem.

6. Show the result you used in the previous question: assume Z is diagonal, and show that R = In is optimal for the problem above.

Hint: show that R> R = In implies | Rii | ≤ 1, i = 1, . . . , n, and using that fact, prove that the optimal value is less than or equal to trace Z. 7. How woud you apply this technique to make Mona Lisa smile more? Hint: in Figure 5.6, the two-dimensional points xi are given (as yellow dots) on the left panel, while the corresponding points yi are shown on the left panel. These points are manually selected.

The problem is to find how to transform all the other points in the original image.

Solution 5.7 1. For fixed R, denoting by zi = yi − Rxi , i = 1, . . . , m, we obtain the problem in variable r m

min ∑ kr − zi k22 r

i =1

44 The solution to this unconstrained least-squares problem can be obtained by setting the derivative to zero. This yields a (unique) optimal point r ∗ r ∗ = ẑ =

1 m zi = ŷ − R x̂ m i∑ =1

2. At optimum, the objective is m

m

m

i =1

i =1

i =1

∑ kzi − ẑk22 = ∑ kyi − ŷ − R(xi − x̂)k22 = ∑ kȳi − Rx̄i k22 ,

as claimed.

3. We now consider the problem m

minn,n ∑ k R x̄i − ȳi k22 : R> R = In ,

R ∈R

i =1

We can express the objective as stated, with X = [ x̄1 , . . . , x̄m ] ∈ Rn,m , Ȳ = [ȳ1 , . . . , ȳm ] ∈ Rn,m .

Indeed, the columns of the matrix RX − Y are R x̄i − ȳi , i = 1, . . . , m.

4. We can safely square the objective, since it is always non-negative.

We have, for any R with R> R = In :

k RX − Y k2F

= trace( RX − Y )> ( RX − Y )

= trace X > R> RX − 2 trace Y > RX + trace Y > Y

= c − 2 trace Y > RX,

where c := trace X > X + trace Y > Y is a constant. We note that trace Y > RX = trace RXY > . Hence, the problem can be expressed as claimed, with Z = XY > ∈ Rn,n .

5. Let Z = USV > be the SVD of Z, with U, V n × n and orthogonal, and S diagonal. Then the objective of the problem writes trace RZ = trace R(USV > ) = trace(V > RU )S = trace MS, where M is the new variable M = V > RU. We note that R> R = In translates as M> M = In . We have reduced the problem to the case when Z is replaced with the diagonal matrix S: max trace MS : M> M = In .

M

Using the fact that M = I is optimal in that case, we obtain V > RU = In at optimum, which leads to the desired result.

45 6. The fact that M = In is optimal for the above problem stems from the fact that since M is orthogonal, | Mii | ≤ 1, i = 1, . . . , n. Indeed, for every i = 1, . . . , m, defining ei to be the i-th unit vector in Rn : n

2 1 = ( M> M )ii = ei> M> Mei = k Mei k22 = ∑ Mki ≥ Mii2 . k =1

Thus, with p ≤ m the rank of Z, and σ1 , . . . , σp its singular values: p

p

i =1

i =1

trace MS = ∑ Mii σi ≤ ∑ σi .

The result follows from the fact that the upper bound is attained when M = In .

7. For the image deformation problem, we find the matrix R and vector r based on the input-output pairs ( xi , yi ), i = 1, . . . , m. We then apply the transformation to every point x ∈ R2 on the original image (left panel in Fig. 5.6), to obtain the transformed points y = Rx + r, and form a new image (right panel in Fig. 5.6).

46

6.

Linear Equations

Exercise 6.1 (Least-squares and total least-squares) Find the least squares line and the total least-squares12 line for the data points ( xi , yi ), i = 1, . . . , 4, with x = (−1, 0, 1, 2), y = (0, 0, 1, 1). Plot both lines on the same set of axes.

Solution 6.1 A perfect line would have yi = w1 xi + w2 , i = 1, 2, 3, 4, for some scalars w1 , w2 .

To find the least-squares line, we solve min k X > w − yk2 , x

where w = (w1 , w2 ), and

X=

"

x1 1

x2 1

x3 1

x4 1

#



 y1  y    , y =  2 .  y3  y4

Since not all coordinates of x are equal, X is full row rank, and hence the LS solution is # " 0.4 . wLS = ( XX > )−1 Xy = 0.3 To find the TLS solution, we have the formula 2 I )−1 Xy, wTLS = (( XX > − σmin

with σmin the smallest singular value of [ X > , y]. We obtain " # 0.4207 wTLS = .

0.3217 As expected, both coefficients are slightly larger in magnitude.

Exercise 6.2 (Geometry of least-squares) Consider a least-squares problem p∗ = min k Ax − yk2 , x

R( A), so that p∗ > 0.

Show that, at optimum, the residual vector r = y − Ax is such that r > y > 0, A> r = 0. Interpret geometrically the result. Hint: use the SVD of A. You can assume that m ≥ n, and that A is full column rank.

where A ∈ Rm,n , y ∈ Rm . We assume that y 6∈

12

See Section 6.7.5.

47 Solution 6.2 Assuming m ≥ n and using the SVD of the full columnrank matrix A: " # Σ > A = U Σ̃V , Σ̃ = , Σ = diag (σ1 , . . . , σn ) , 0 we can easily reduce the problem to the case when A is diagonal.

Precisely, we set ȳ = U > y = (ȳ1 , ȳ2 ), with ȳ1 ∈ Rn , and ȳ2 6= 0. The optimal point is x ∗ = V x̄ ∗ , with x̄ ∗ = Σ−1 ȳ1 , and the corresponding optimal residual r ∗ = y − Ax ∗ satisfies " # 0 U > r ∗ = U > (y − Ax ∗ ) = ȳ − Σ̃ x̄ ∗ = . ȳ2 We do have A> r ∗ = 0, since > ∗

>

> ∗

VA r = Σ̃ U r =

h

Σ

0

i

"

0 y2

#

= 0,

and y> r ∗ = ȳ> (U > r ∗ ) = kȳ2 k22 > 0.

The geometrical interpretation is apparent from Figure 6.7. The residual vector r = y − Ax forms an acute angle with the point y. In addition, it is orthogonal to the range of A. Exercise 6.3 (Lotka’s law and least-squares) Lotka’s law describes the frequency of publication by authors in a given field. It states that X a Y = b, where X is the number of publications, Y the relative frequency of authors with X publications, and a and b are constants (with b > 0) that depend on the specific field. Assume that we have data points ( Xi , Yi ), i = 1, . . . , m, and seek to estimate the constants a and b.

1. Show how to find the values of a, b according to a linear leastsquares criterion. Make sure to define precisely the least-squares problem involved.

2. Is the solution always unique? Formulate a condition on the data points that guarantees unicity.

Solution 6.3 1. If the model was exactly true, we would have b = xia yi , i = 1, . . . , m, or taking logarithms: log b = a log xi + log yi , i = 1, . . . , m.

0

Figure 6.7: Projection onto the range of A.

48 This leads to a least-squares problem with variables θ = ( a, log b): min k Aθ − zk2 , θ

where z = −(log y1 , . . . , log ym ) is the response vector, and   log x1 −1  .. ..   A= . . .  log xm −1 2. The answer is unique if and only if A is full column rank. This is always the case except when all the xi ’s are equal.

Exercise 6.4 (Regularization for noisy data) Consider a least-squares problem min k Ax − yk22 , x

in which the data matrix A ∈ Rm,n is noisy. Our specific noise model assumes that each row ai> ∈ Rn has the form ai = âi + ui , where the noise vector ui ∈ Rn has zero mean and covariance matrix σ2 In , with σ a measure of the size of the noise. Therefore, now the matrix A is a function of the uncertain vector u = (u1 , . . . , un ), which we denote by A(u). We will write  to denote the matrix with rows âi> , i = 1, . . . , m. We replace the original problem with min Eu {k A(u) x − yk22 }, x

where Eu denotes the expected value with respect to the random variable u. Show that this problem can be written as min k Âx − yk22 + λk x k22 , x

where λ ≥ 0 is some regularization parameter, which you will determine. That is, regularized least-squares can be interpreted as a way to take into account uncertainties in the matrix A, in the expected value sense. Hint: compute the expected value of (( âi + ui )> x − yi )2 , for a specific row index i.

Solution 6.4 Consider a fixed row index i, and drop the dependence of the vectors on that index. Define r = y − â> x. We consider the expected value

Eu (( â + u)> x − y)2 = E (u> x )2 − 2r (u> x ) + r2

= E ( u > x )2 + r 2

= σ2 k x k22 + r2 ,

49 where we have used Eu u = 0 in the second line, and in the third, the expression Eu (u> x )2 = E( x > uu> x ) = E trace(uu> )( xx > ) = trace xx > = x > x.

Summing we obtain the desired result, with regularization parameter λ = σ2 .

Exercise 6.5 (Deleting a measurement in least-squares) In this exercise, we revisit Section 6.3.5, and assume now that we would like to delete a measurement, and update the least-squares solution accordingly13 .

We are given a full column rank matrix A ∈ Rm,n , with rows ai> , i = 1, . . . , m, and a vector y ∈ Rm , and a solution to the least-squares problem m

x ∗ = arg min ∑ ( ai> x − yi )2 = arg min k Ax − yk2 . x

x

i =1

Assume now we delete the last measurement, that is, replace ( am , ym ) by (0, 0). We assume that the matrix obtained after deleting any one of the measurements is still full column rank.

1. Express the solution to the problem after deletion, in terms of the original solution, similar to the formula (6.15). Make sure to explain why any quantities you invert are positive.

2. In the so-called leave-one-out analysis, we would like to efficiently compute all the m solutions corresponding to deleting one of the m measurements. Explain how you would compute those solutions computationally efficiently. Detail the number of operations (flops) needed. You may use the fact that to invert a n × n matrix costs O(n3 ).

Solution 6.5 1. We have A=

"

A− a>

#

,

y=

"

y− η

#

where A− , y− corresponds to the least-squares problem with the last measurement deleted, a = am and η = ym corresponds to the . . last measurement. With H = A> A, H− = A> − A− , the current so∗ − 1 > lution expresses x = H A y, while the solution with the last ∗ = H −1 A> y . We would like to exmeasurement deleted is x− − − − ∗ in a computationally efficient way, as a modification of press x− x.

This is useful in the context of crossvalidation methods, as evoked in Section 13.2.2.

13

50 We have > > > > H− = A> − A− = AA − aa , A− y− = A y − aη.

Using the rank-one perturbation formula (3.10), we obtain −1 H− = H −1 +

1 −1 > −1 .

H aa H , γ = 1 − a> H −1 a. γ

Note that γ > 0, since the matrix H− is full rank.

We then have ∗ −1 > x− = H− A− y−

=

H −1 +

= x∗ +

1 −1 > −1 > H aa H A y − aη γ

( a > x ∗ − η ) −1 H a. γ

Compare with the similar formula (6.15), which was obtained in the context of adding a measurement. .

2. In a leave-one-out approach, we can first find K = H −1 , where H = A> A corresponds to all the measurements. Then we set P = KA> , and x ∗ = Py. We then obtain the solution corresponding to the i-th measurement deleted as xi∗ = x ∗ +

( ai> x ∗ − yi ) pi , i = 1, . . . , m, γi

where pi = H −1 ai is the i-th column of P, and γi = 1 − a> pi , i = 1, . . . , m.

Let us compare the complexity of the above method, with a more brute-force approach that involves solving each least-squares problem with one measurement deleted. The complexity of solving a least-squares problem with n variables and m measurements is C (m, n) = O(mn2 + n3 ), so solving m such problems is mC (m, n).

The above method, in contrast, requires: • forming n × n matrix H (O(n2 m)) and its inverse K (O(n3 ));

• forming n × m matrix P = KA> (O(n2 m)); • forming x ∗ and xi∗ , i = 1, . . . , m (O(nm)).

In total, the complexity is C (m, n) = O(mn2 + n3 ), which is the same as one least-squares problem.

Exercise 6.6 The Michaelis-Menten model for enzyme kinetics relates the rate y of an enzymatic reaction, to the concentration x of a substrate, as follows: β1 x , y= β2 + x where β i , i = 1, 2, are positive parameters.

51 1. Show that the model can be expressed as a linear relation between the values 1/y and 1/x.

2. Use this expression to find an estimate β̂ of the parameter vector β using linear least-squares, based on m measurements ( xi , yi ), i = 1, . . . , m.

3. The above approach has been found to be quite sensitive to errors in input data. Can you experimentally confirm this opinion?

Solution 6.6 (Enzyme kinetics model) The Michaelis-Menten model for enzyme kinetics relates the rate y of an enzymatic reaction, to the concentration x of a substrate, as follows: y=

β1 x , β2 + x

where β i , i = 1, 2, are parameters.

1. We have

1 w 1 β2 = w1 + 2 , w1 = , w2 = . y x β1 β1

2. The least-squares problem takes the form min k X > w − zk2 , w

where z = (1/y1 , . . . , 1/ym ), and " 1/x1 . . .

X= 1 ...

1/xm 1

#

.

Once w∗ is found, we can set β∗ = (1/w1∗ , w2∗ /w1∗ ).

3. According to section (6.5.4), the sensitivity of the solution to an LS problem depends on the condition number of the matrix A = X > . We can experiment with random data and observe that the condition number can be very high. Fishy.

Exercise 6.7 (Least norm estimation on traffic flow networks) You want to estimate the traffic (in San Francisco for example, but we’ll start with a smaller example). You know the road network as well as the historical average of flows on each road segment.

1. We call qi the flow of vehicles on each road segment i ∈ I. Write down the linear equation that corresponds to the conservation of vehicles at each intersection j ∈ J. Hint: think about how you might represent the road network in terms of matrices, vectors, etc.

52 2. The goal of the estimation is to estimate the traffic flow on each of the road segment. The flow estimates should satisfy the conservation of vehicles exactly at each intersection. Among the solutions that satisfy this constraint, we are searching for the estimate that is the closest to the historical average, q, in the l2 -norm sense. The vector q has size I and the i-th element represent the average for the road segment i. Pose the optimization problem.

3. Explain how to solve this problem mathematically. Detail your answer (do not only give a formula but explain where it comes from).

1" 5"

a"

6"

10" 14"

e" 19"

b"

7"

f" 20"

c"

8"

12"

11" 15"

4"

3"

2"

16"

g" 21"

d"

9"

Figure 6.8: Example of traffic estimation problem. The intersections are labeled a to h. The road segments are labeled 1 to 22. The arrows indicate the direction of traffic.

13" 17"

h"

18"

22"

4. Formulate the problem for the small example of Figure 6.8 and solve it using the historical average given in Table 6.1. What is the flow that you estimate on road segments 1, 3, 6, 15 and 22?

5. Now, assume that besides the historical averages, you are also given some flow measurements on some of the road segments of the network. You assume that these flow measurements are correct and want your estimate of the flow to match these measurements perfectly (besides matching the conservation of vehicles of course).

The right column of Table 6.1 lists the road segments for which we have such flow measurements. Do you estimate a different flow on some of the links? Give the difference in flow you estimate for road segments 1,3, 6, 15 and 22. Also check that you estimate gives you the measured flow on the road segments for which you have measured the flow.

Solution 6.7 1. At each intersection, the sum of all incoming flows must be equal to the sum of all outgoing flows. We construct the matrix A ∈ R J × I such that the element on the jth line and ith column is • 0 if link i does not arrive or leave intersection j;

segment 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

average 2047.6 2046.0 2002.6 2036.9 2013.5 2021.1 2027.4 2047.1 2020.9 2049.2 2015.1 2035.1 2033.3 2027.0 2034.9 2033.3 2008.9 2006.4 2050.0 2008.6 2001.6 2028.1

measured 2028 2008 2035 2019

2044

2043

2030 2025 2045

Table 6.1: Table of flows: historical averages q (center column), and some measured flows (right column).

53 • 1 if link i arrives at intersection j; • −1 if link i leaves intersection j.

With such a construction, the conservation of flow is written Aq = 0.

2. The problem can be formulated as follows: min

q ∈R I

||q − q||2 subject to Aq = 0.

3. To solve this problem we introduce the variable x = q − q. With this change of variable, we solve the following optimization problem: min || x ||2 subject to Ax = b, x ∈R I

where b = − Aq. Our estimation problem is a least norm problem.

The matrix A is full row rank, that is, the matrix AA T is invertible.

We can solve the minimum norm problem in closed-form x ∗ = A T ( AA T )−1 b. We get the flow estimate as q∗ = x ∗ + q.

4. We first need a function that constructs the matrix A that will represent the incidence matrix of links at intersections. function A = construct_network A = zeros(8,22); for i=1:4 A(i,i) = (-1)^i; A(i,4+i) = 1; A(i,5+i) = -1; A(i,9+i) = -(-1)^i; end for i=5:8 A(i,9+i-4) = (-1)^i; A(i,13+i-4) = 1; A(i,14+i-4) = -1; A(i,18+i-4) = -(-1)^i; end

Then, we solve the estimation problem as a least norm problem.

The function below loads the historical flows, constructs the incidence matrix and estimates the flow that is closest to the historical averages while satisfying the conservation of vehicles. function x_hat = estimate_historic

54

A = construct_network; hist = load(’historical.mat’); hist = hist.historical; b = zeros(8,1); z_hat = A\(b-A*hist); x_hat = z_hat + hist;

5. The additional flow measurements are incorporated into the constraints. Let B be the matrix of size N times I where N represents the number of flow measurements. This matrix has a one on row n and column i if the nth measurement concerns link i. It has zeros everywhere else. We call qm the vector of measurements. The optimization problem reads: min

q ∈R I

||q − q||2 subject to Aq = 0, Bq = qm .

With the same change of variable as before (x = q − q), defining à the matrix obtained by stacking A and B vertically, à = [ A T B T ] T and b̃ the vector obtained by stacking b and qm − Bq, b̃ = [b T (qm − Bq) T ] T , we solve again a least norm problem: min

x ∈R I

|| x ||2 subject to Ãx = b̃.

In the data provided, the measurements are such that à is still full row rank, and we can solve the problem with the same method as before. In the general case, à may not be full row rank anymore.

In this case, two situations may arise: • If b̃ is not in the range of Ã, then the problem is not feasible (the optimal value is +∞).

• Otherwise, any solution of the linear equation Ãx = b̃ can be written x = x ∗ + z. Here, z is any element of the nullspace of Ã.

Also, x ∗ is the element in the orthogonal of the nullspace which is solution to the linear equation i.e. Ãx ∗ = b̃ and for all z in the nullspace of Ã, z T x ∗ = 0.

Exercise 6.8 (A matrix least-squares problem) We are given a set of points p1 , . . . , pm ∈ Rn , which are collected in the n × m matrix P = [ p1 , . . . , pm ]. We consider the problem λ . m min F ( X ) = ∑ k xi − pi k22 + ∑ k xi − x j k22 , 2 X i =1 1≤i,j≤m

55 where λ ≥ 0 is a parameter. In the above, the variable is a n × m matrix X = [ x1 , . . . , xm ], with xi ∈ Rn the i-th column of X, i = 1, . . . , m. The above problem is an attempt at clustering the points pi ; the first term encourages the cluster center xi to be close to the corresponding point pi , while the second term encourages the xi ’s to be close to each other, with a higher grouping effect as λ increases.

1. Show that the problem belongs to the family of ordinary leastsquares problem. You do not need to be explicit about the form of the problem.

2. Show that 1 k xi − x j k22 = trace XHX > , 2 1≤∑ i,j≤m where H = mIm − 11> is a m × m matrix, with Im the m × m identity matrix, and 1 the vector of ones in Rm .

3. Show that H is positive semi-definite.

4. Show that the gradient of the function F at a matrix X is the n × m matrix given by

∇ F ( X ) = 2( X − P + λXH ).

Hint: for the second term, find the first-order expansion of the function ∆ → trace(( X + ∆) H ( X + ∆)> ), where ∆ ∈ Rn,m .

5. As mentioned in Remark 6.1, optimality conditions for a leastsquares problem are obtained by setting the gradient of the objective to zero. Using the formula 3.10, show that optimal points are of the form xi =

1 mλ p + p̂, i = 1, . . . , m, mλ + 1 i mλ + 1

where p̂ = (1/m)( p1 + . . . + pm ) is the center of the given points.

6. Interpret your results. Do you believe the model considered here is a good one to cluster points?

Solution 6.8 1. The objective function is a sum of squares. Inside the squares are linear or affine functions of the variables. There are no constraints.

Hence the problem is an ordinary least-squares problem.

56 2. We have 1 k xi − x j k22 2∑ i,j

=

1 k xi k22 + k x j k22 − 2xi> x j ∑ 2 i,j !> !

= m ∑ k xi k22 − i

∑ xi

∑ xi

i

>

i

>

= m trace( X X ) − ( X1) ( X1)

= m trace( XX > ) − trace( X1)( X1)> = trace( X (mIm − 11> ) X > ),

as claimed.

3. H is positive semi-definite, since for any vector z ∈ Rm , we have 1 1 > z Hz = z T z − (∑ zi )2 = ∑(zi − ẑ)2 ≥ 0, m m i i

where ẑ = (1/m)(z1 + . . . + zm ).

4. The objective function writes F ( X ) = k X − Pk2F + λ trace( XHX > ).

The gradient of the first term is X − P. For the second term, we start by looking at the following first-order expansion: for any ∆ ∈ Rn,m , we have trace(( X + ∆) H ( X + ∆)> ) − trace( XHX > )

= 2 trace(∆HX > ) + h.o.t. = 2 trace( HX > ∆) + h.o.t.

= 2 trace(( XH )> ∆) + h.o.t., which shows that the gradient of the second term is 2λXH. The formula for ∇ F ( X ) follows.

5. Optimality conditions are (1/2)∇ F ( X ) = X − P + λXH = 0, or X ( I + λH ) = P. Since H is positive semi-definite, the matrix I + λH is invertible whenever λ ≥ 0. In fact, we have

−1 ( I + λH )−1 = (1 + mλ) I − λ11> −1

1 .

I − θ11> (with θ = λ/(1 + mλ)) = 1 + mλ

1 θ > = I+ 11 (from formula 3.10) 1 + mλ 1 − θ1> 1 1 = I + θ11> .

1 + mλ The formula for the optimal points then follows from the identity P1 = m p̂.

57 6. The model will simply shrink the points towards their averages.

This is not a desirable effect; a good clustering algorithm would group points in several different clusters, rather than sending them all towards the global mean p̂.

58

7.

Matrix Algorithms

Exercise 7.1 (Sparse matrix-vector product) Recall from Section 3.4.2 that a matrix is said to be sparse if most of its entries are zero. More formally, assume a m × n matrix A has sparsity coefficient γ( A) 1, . where γ( A) = d( A)/s( A), where d( A) is the number of nonzero elements in A, and s( A) is the size of A (in this case, s( A) = mn).

1. Evaluate the number of operations (multiplications and additions) that are required to form the matrix-vector product Ax, for any given vector x ∈ Rn and generic, non-sparse A. Show that this number is reduced by a factor γ( A), if A is sparse.

2. Now assume that A is not sparse, but is a rank-one modification of a sparse matrix. That is, A is of the form à + uv> , where à ∈ Rm,n is sparse, and u ∈ Rm , v ∈ Rm are given. Devise a method to compute the matrix-vector product Ax that exploits sparsity.

Solution 7.1 1. We have to form m scalar products between the m rows of A and the n-vector x. The total number of flops is then s( A) = mn. When A is sparse, the number is d( A).

2. We have Ax = y + z, with y = Ãx and z = (v> x )u. The cost of forming v> x is n flops; forming (v> x )u is m flops. Thus, the total cost required is d( Ã) + m + n, which is much less than s( A).

Exercise 7.2 (A random inner product approximation) Computing the standard inner product between two vectors a, b ∈ Rn requires n multiplications and additions. When the dimension n is huge (say, e.g., of the order of 1012 , or larger), even computing a simple inner product can be computationally prohibitive.

Let us define a random vector r ∈ Rn constructed as follows: choose uniformly at random an index i ∈ {1, . . . , n}, and set ri = 1, and r j = 0 for j 6= i. Consider the two scalar random numbers ã, b̃ that represent the “random projections” of the original vectors a, b along r: . . ã = r > a = ai , b̃ = r > b = bi .

Prove that nE{ ãb̃} = a> b, that is, n ãb̃ is an unbiased estimator of the value of the inner product a> b. Observe that computing n ãb̃ requires very little effort, since it is just equal to nai bi , where i is the randomly chosen index. Notice,

59 however, that the variance of such an estimator can be large, as it is given by

2 n var{n ãb̃} = n ∑ a2i bi2 − a> b k =1

(prove also this latter formula). Hint: Let ei denote the i-th standard basis vector of Rn ; the random vector r has discrete probability distribution Prob{r = ei } = 1/n, i = 1, . . . , n, hence E{r } = n1 1. Further, observe that the products rk r j are equal to zero for k 6= j and that the . vector r2 = [r12 , . . . , rn2 ]> has the same distribution as r.

Generalizations of this idea to random projections onto k-dimensional subspaces are indeed applied for matrix-product approximation, SVD factorization and PCA on huge-scale problems. The key theoretical tool underlying these results is known as the JohnsonLindenstrauss lemma.

Solution 7.2 We have that ãb̃ = r > ar > b = a> rr > b, hence >

>

E{ ãb̃} = a E{rr }b = a

n

1 ∑ n ei ei> i =1

>

!

b=

1 > a b, n

which proves the first statement. Next, observe that, since ri r j = 0 with probability one for i 6= j, then n

ãb̃

= a> rr > b = ∑ ri2 ai bi ,

w.p. 1

i =1

n

=

∑ r i a i bi ,

w.p. 1,

i =1

and

( ãb̃)

2

n

=

∑ r i a i bi

i =1 n

=

∑ ri a2i bi2 ,

!2

n

= ∑ ri2 a2i bi2 ,

w.p. 1

i =1

w.p. 1.

i =1

Thus, var{ ãb̃}

1 > 2 a b) } n 2 1 E{( ãb̃)2 } − ( a> b)E{ ãb̃}} + 2 ( a> b)2 n n n 2 1 1 ∑ E{ri }a2i bi2 − n (a> b) n (a> b) + n2 (a> b)2 i =1

= E{( ãb̃ − E{ ãb̃})2 } = E{( ãb̃ − = = =

1 n 2 2 1 a i bi − 2 ( a > b ) 2 , n i∑ n =1

60 which proves the second statement (recalling that var{αx } = α2 var{ x }).

Exercise 7.3 (Power iteration for SVD with centered, sparse data) In many applications such as Principal Component Analysis (see Section 5.3.2), one needs to find the few largest singular values of a centered data matrix. Specifically, we are given a n × m matrix X = [ x1 , . . . , xm ] of m data points in Rn , i = 1, . . . , m, and define the centered matrix X̃ to be . x̃i = xi − x̄, i = 1, . . . , m,

X̃ = [ x̃1 · · · x̃m ],

with x̄ = m1 ∑im=1 xi the barycenter of the data points. In general, X̃ is dense, even if X itself is sparse. This means that each step of the power iteration method involves two matrix-vector products, with a dense matrix. Explain how to modify the power iteration method in order to exploit sparsity, and avoid dense matrix-vector multiplications.

Solution 7.3 We have X̃ = X − x̄1> .

The power iteration updates take the form u ( k + 1)

=

v ( k + 1)

=

X̃v(k) , k X̃v(k)k2 X̃ > u(k + 1) . k X̃ > u(k + 1)k2

Consider the problem of a matrix-vector multiplication u = X̃v: we can write it as y = 1> v, u = Xv − x̄y, where the opetations only involve a simple scalar product and a matrix-vector product Xv that involves the sparse matrix X. A similar result holds with the transpose operation v = X̃ > u, which can be written as z = x̄ > u, v = Xu − 1z.

The normalizations involved are also simple. We write the power iteration method as y(k) ũ(k + 1) u ( k + 1) z ( k + 1) ṽ(k + 1) v ( k + 1)

= 1 > v ( k ), = Xv(k) − x̄y(k), ũ( k + 1) = , kũ(k + 1))k2 = x̄ > u(k + 1), = Xu(k + 1) − 1> y(k), ṽ(k + 1) = . kṽ(k + 1)k2

61 Exercise 7.4 (Exploiting structure in linear equations) Consider the linear equation in x ∈ Rn Ax = y

where A ∈ Rm,n , y ∈ Rm . Answer the following questions to the best of your knowledge.

1. The time required to solve the general system depends on the sizes m, n and the entries of A. Provide a rough estimate of that time as a function of m, n only. You may assume that m, n are of the same order.

2. Assume now that A = D + uv> , where D is diagonal, invertible, and u ∈ Rm , v ∈ Rn . How would you exploit this structure to solve the above linear system, and what is a rough estimate of the complexity of your algorithm?

3. What if A is upper-triangular?

Solution 7.4 1. The count is O(n3 ). This can be seen from the flop count for the QR algorithm14 .

14

See section 7.3.1.

15

See Section 6.

2. We can write the system as Dx + uz = y, z = v> x.

Assuming D is invertible, we can solve for x: x = D −1 (y − uz).

Then z = v> x = v> D −1 (y − uz) =⇒ z =

v > D −1 y .

1 + v > D −1 u

The flop count in this case is O(n).

3. When A is upper-triangular, we can use the backward substitution algorithm15 . The flop count is then O(n2 ), as explained in Remark 7.1.

Exercise 7.5 (Jacobi method for linear equation) Let A = ( aij ) ∈ Rn,n , b ∈ Rn , with aii 6= 0 for every i = 1, . . . , n. The Jacobi method for solving the square linear system Ax = b

62 consists in decomposing A as a sum: A = D + R, where D = diag ( a11 , . . . , ann ), and R contains the off-diagonal elements of A, and then applying the recursion x (k+1) = D −1 (b − Rx (k) ), k = 0, 1, 2, . . . , with initial point x̂ (0) = D −1 b.

The method is part of a class of methods known as matrix splitting, where A is decomposed as a sum of a “simple,” invertible matrix and another matrix; the Jacobi method uses a particular splitting of A. 1. Find conditions on D, R that guarantee convergence from an arbi. trary initial point. Hint: assume that M = − D −1 R is diagonalizable.

2. The matrix A is said to be strictly row diagonally dominant if

∀ i = 1, . . . , n : | aii | > ∑ | aij |. j 6 =i

Show that when A is strictly row diagonally dominant, the Jacobi method converges.

Solution 7.5 1. Assume that M = − D −1 R is diagonalizable: M = V −1 EV, with E a diagonal matrix containing the eigenvalues λ1 , . . . , λn on the diagonal, and V a matrix with columns equal to corresponding eigenvectors. Defining z0 = D −1 b, the recursion writes xk+1 = Mxk + z0 , k = 0, 1, 2, . . .

This can be written in terms of the vectors z̄ = Vz0 , and x̄k = Vxk , as x̄k = E x̄k + z̄, k = 0, 1, 2, . . .

The above is a set of n independent scalar recursions of the form ξ k+1 = λi ξ k + z̄i , k = 0, 1, 2, . . . , the convergence of which is guaranteed for any initial point when |λi | < 1. Our condition thus writes ρ( D −1 R) < 1, where ρ is the modulus of the largest eigenvalue of its matrix argument.

2. Let λ, v be an eigenvalue-eigenvector pair for M: λDv = Rv, with v 6= 0. Let i be such that |vi | = max j |v j | = kvk∞ 6= 0. We have

|λ|| aii ||vi | = ∑ aij v j ≤ ∑ | aij ||v j | ≤ kvk∞ ∑ | aij |. j 6 =i

j 6 =i

j 6 =i

63 Dividing by | aii ||vi | (a non-zero quantity since | aii | > 0):

|λ| ≤

∑ j6=i | aij | < 1, | aii |

which proves that any eigenvalue of M has modulus strictly less than one.

Exercise 7.6 (Convergence of linear iterations) Consider linear iterations of the form x (k + 1) = Fx (k) + c,

k = 0, 1, . . . ,

(7.3)

where F ∈ Rn,n , c ∈ Rn , and the iterations are initialized with x (0) = x0 . We assume that the iterations admit a stationary point, i.e., that there exist x̄ ∈ Rn such that

( I − F ) x̄ = c.

(7.4)

In this exercise, we derive conditions under which x (k ) tends to a finite limit for k → ∞. We shall use these results in Exercise 7.7, to set up a linear iterative algorithm for solving systems of linear equations.

1. Show that the following expressions hold for all k = 0, 1, . . .: x ( k + 1) − x ( k ) x (k) − x̄

= =

F k ( I − F )( x̄ − x0 ) k

F ( x0 − x̄ ).

(7.5) (7.6)

2. Prove that, for all x0 , limk→∞ x (k) converges to a finite limit if and only if F k is convergent (see Theorem 3.5). When x (k ) converges, its limit point x̄ satisfies (7.4).

Solution 7.6 (Convergence of linear iterations)

1. Applying (7.4) iteratively we obtain that ! x ( k ) = F k x0 +

k −1

∑ Fi

c,

k = 1, 2, . . .

i =0

hence, for x̄ satisfying (7.4), x ( k + 1) − x ( k )

= = =

F k ( F − I ) x0 + F k c = F k ( c − ( I − F ) x0 )

F k (c − ( I − F )( x0 − x̄ ) − ( I − F ) x̄ )

F k ( I − F )( x̄ − x0 ),

64 which proves the first expression. Also, since x̄ is a stationary point, it holds for all k ≥ 1 that ! x̄ = F k x̄ +

k −1

∑ Fi

c,

i =0

thus k

x (k ) − x̄ = F x0 +

k −1

∑F

i =0

i

!

k

c − F x̄ −

k −1

∑F

i =0

i

!

c = F k ( x0 − x̄ ),

which proves the second expression.

2. Suppose first that, for any x0 , x (k ) converges16 for k → ∞. Then, 0 = lim x (k + 1) − x (k) = lim F k ( I − F )( x̄ − x0 ), k→∞

k→∞

thus limk→∞ F k ( I − F )( x̄ − x0 ) = 0 for all x0 , which happens if and only if limk→∞ F k ( I − F ) = 0, which implies (by Theorem 3.5) that F k is convergent.

Conversely, suppose that F k is convergent. Then, limk→∞ F k = F̄, and lim x (k) − x̄ = lim F k ( x0 − x̄ ) = F̄ ( x0 − x̄ ), k→∞

The point of convergence, x̄ ( x0 ) may depend on x0 . In any case, since, in the limit, x (k + 1) = x (k ) = x̄ ( x0 ), we see that the convergence point must satisfy the stationarity equations (7.4).

16

k→∞

thus x (k) converges to x̂ = x̄ + F̄ ( x0 − x̄ ). This point must a fortiori satisfy the stationarity conditions, that is x̂ = F x̂ + c. This property can also be verified directly, since F x̂ + c

=

F x̄ + F F̄ ( x0 − x̄ ) + c = x̄ + F F̄ ( x0 − x̄ )

= x̄ + F̄ ( x0 − x̄ ) = x̂,

where we have used the fact17 that F F̄ = F̄.

Exercise 7.7 (A linear iterative algorithm) In this exercise we introduce some “equivalent” formulations of a system of linear equations Ax = b,

A ∈ Rm,n ,

(7.7)

and then study a linear recursive algorithm for solution of this system.

1. Consider the system of linear equations Ax = AA† b,

Any convergent matrix F can be factored as F = Udiag I, F̃ U −1 , where U = [U1 Ũ ] and U1 contains by columns the eigenvectors of F associated to the eigenvalue λ = 1 (if F has such eigenvalue). Moreover, ρ( F̃ ) < 1. It follows that the limit matrix is F̄ = Udiag ( I, 0) U −1 , and one can verify directly that F F̄ = F̄F = F̄.

17

(7.8)

where A† is any pseudoinverse of A (that is, a matrix such that AA† A = A). Prove that (7.8) always admits a solution. Show that every solution of equations (7.7) is also a solution for (7.8).

Conversely, prove that if b ∈ R( A), then every solution to (7.8) is also a solution for (7.7).

65 2. Let R ∈ Rn,m be any matrix such that N ( RA) = N ( A). Prove that .

A† = ( RA)† R is indeed a pseudoinverse of A. 3. Consider the system of linear equations RAx = Rb,

(7.9)

where R ∈ Rn,m is any matrix such that N ( RA) = N ( A) and Rb ∈ R( RA). Prove that, under these hypotheses, the set of solutions of (7.9) coincides with the set of solutions of (7.8), for A† = ( RA)† R. 4. Under the setup of the previous point, consider the following linear iterations: for k = 0, 1, . . ., x (k + 1) = x (k ) + αR(b − Ax (k)),

(7.10)

where α 6= 0 is a given scalar. Show that if limk→∞ x (k) = x̄, then x̄ is a solution for the system of linear equations (7.9). State appropriate conditions under which x (k) is guaranteed to converge.

5. Suppose A is positive definite (i.e., A ∈ Sn , A 0). Discuss how to find a suitable scalar α and matrix R ∈ Rn,n satisfying the conditions of point 3., and such that the iterations (7.10) converge to a solution of (7.9). Hint: use Exercise 4.8.

6. Explain how to apply the recursive algorithm (7.10) for finding a solution to the linear system Ãx = b̃, where à ∈ Rm,n with m ≥ n and rank à = n. Hint: apply the algorithm to the Normal equations.

Solution 7.7 (A linear iterative algorithm) 1. We recall18 preliminary that AA† is an orthogonal projector onto R( A), therefore, AA† b ∈ R( A), which means that (7.8) always admits a solution.

Suppose next that x satisfies (7.7). Then, multiplying this equation by A† on the left, we have A† Ax = A† b, and multiplying it once again on the left by A we obtain AA† Ax = AA† b, which proves that (7.8) is satisfied by x, since AA† A = A.

18

See Section 5.2.3

66 Finally, let b ∈ R( A) and suppose (7.8) is satisfied for some x.

We write b = AA† z for some z, hence Ax = AA† b = AA† AA† z = AA† z = b, which shows that (7.7) is satisfied at x.

2. Recall that, for any matrix X, R( X ) = R( XX † ), and N ( X )⊥ = R( X † X ). Then, by the fundamental theorem of linear algebra, we have that Rn

= N ( RA) ⊕ N ( RA)⊥ = N ( RA) ⊕ R(( RA)† ( RA)) = N ( A) ⊕ R(( RA)† ( RA)).

Therefore, any x ∈ Rn can be written as x = x1 + x2 , with x1 ∈ R(( RA)† ( RA)) and x2 ∈ N ( A). Hence, we can write x1 = ( RA)† ( RA)z for some z, and A( RA)† ( RA) x

=

A( RA)† ( RA) x1 = A( RA)† ( RA)( RA)† ( RA)z

=

A( RA)† ( RA)z = Ax1 = Ax,

which shows that A(( RA)† R) A = A, thus ( RA)† R is a pseudoinverse of A. 3. Any solution of (7.9) is of the form x = x̄ + z,

x̄ = ( RA)† Rb, z ∈ N ( RA) = N ( A),

where x̄ is a solution, since it is assumed that Rb ∈ R( RA). Then, Ax

=

A x̄ + Az = A x̄ = A( RA)† Rb = AA† b,

which shows that x satisfies (7.8). Conversely, any solution of (7.8) is of the form x = x̄ + z,

x̄ = A† b, z ∈ N ( A),

Then, RAx

=

RAA† b + RAz = RAA† b = ( RA)( RA)† Rb = Rb,

where the last passage follows from the fact that ( RA)( RA)† is a projector onto R( RA), but Rb ∈ R( RA) by assumption, hence the projection leaves this vector unchanged.

4. The solution to this point is easily obtained by applying the statements of Exercise 7.6, considering that the iterations have the form (7.3), for F = I − αRA and c = αRb. The iterations converge if and only if F k is convergent. A sufficient condition for this is given by ρ( I − αRA) < 1.

67 5. Let .

R = diag

1

k a1> k1

,...,

1

k a> n k1

!

,

where ai> are the rows of A. Since A 0 and R 0, this choice of R satisfies the conditions of point 3. Next, applying the results in points 2. and 3. of Exercise 4.8, we have that ρ( I − αRA) < 1 for any α ∈ (0, 2) hence, for such values of α, ( I − αRA)k converges to zero and the iterations (7.10) converge to the (unique) solution of (7.9).

6. All solutions of Ãx = b̃ are also solutions to the Normal equations Ax = b, with A = Ã> Ã, b = Ã> b̃. If à ∈ Rm,n with m ≥ n and rank à = n, then A ∈ Sn is positive definite, and we can apply the results of the previous point to the system of Normal equations.

68

8.

Convexity

Exercise 8.1 (Quadratic inequalities) Consider the set defined by the following inequalities

( x1 ≥ x2 − 1 and x2 ≥ 0) or ( x1 ≤ x2 − 1 and x2 ≤ 0) .

1. Draw the set. Is it convex?

2. Show that it can be described as a single quadratic inequality of the form q( x ) = x > Ax + 2b> x + c ≤ 0, for matrix A = A> ∈ R2,2 , b ∈ R2 and c ∈ R which you will determine.

3. What is the convex hull of this set?

Solution 8.1 1. The set is not convex: (0, 1) and (−2, 0) both belong to the set, but the midpoint (−1, 1/2) does not.

2. Define q( x ) = x2 ( x2 − x1 − 1); we have q( x ) ≤ 0 if and only if ( x2 − x1 − 1) ≤ 0 and x2 ≥ 0, or ( x2 − x1 − 1) ≥ 0 and x2 ≤ 0. We can write q( x ) = x > Ax + 2b> x + c, where A=

"

0 −1/2

−1/2 1

#

, b=

"

0 −1/2

#

, c = 0.

3. The convex hull of the set is the whole space, R2 .

Exercise 8.2 (Closed functions and sets) Show that the indicator function IX of a convex set X is convex. Show that this function is closed whenever X is a closed set.

Solution 8.2 (Closed functions and sets) The indicator function of a convex set X is an extended-valued function defined as ( 0 if x ∈ X IX ( x ) = +∞ otherwise.

Clearly, the domain of this function is X , which is a convex set.

Hence, for any x, y ∈ X and λ ∈ [0, 1], we have that IX (λx + (1 − λ)y) = 0 ≤ λIX ( x ) + (1 − λ) IX (y) = 0, which shows that IX is convex.

(8.11)

69 Suppose next that X is a closed set. Let α ∈ R. If α ≥ 0, the sublevel set Sα = { x ∈ Rn : IX ( x ) ≤ α} coincides with X , which is a closed set. If α < 0, then Sα = ∅, which is also closed. Thus, all sublevel sets of IX are closed, which proves that IX is a closed function.

Exercise 8.3 (Convexity of functions) 1. For x, y both positive scalars, show that ye x/y = max α( x + y) − yα log α. α >0

Use the above result to prove that the function f defined as ( ye x/y if x > 0, y > 0, f ( x, y) = +∞ otherwise, is convex.

2. Show that for r ≥ 1, the function f r : Rm + → R, with values !r m

f r (v) =

∑ v1/r j

j =1

is concave. Hint: show that the Hessian of − f takes the form κdiag (y) − zz> for appropriate vectors y ≥ 0, z ≥ 0, and scalar κ ≥ 0, and use Schur complements19 to prove that the Hessian is positive semi-definite.

Solution 8.3 1. For x, y both positive scalars, the function g : α → α( x + y) − yα log α, with domain R++ , is convex, and differentiable. Its minimum is obtained by setting the derivative to zero, which leads to x + y = y(log α + 1) =⇒ α = e x/y .

The minimizer is in the interior of the domain. Plugging its value in the expression of the function we obtain the desired result. That result proves that f is convex, since it is the point-wise maximum (over α > 0) of affine functions. . r 2. Let V = ∑ j v1/r j , so that f ( v ) = V . We have ∂V 1 1 −1 (v) = vir , i = 1, . . . , n, ∂vi r and

1 −1 ∂f ∂V (v) = rV r−1 (v) = V r−1 vir , i = 1, . . . , n. ∂vi ∂vi

19

See Section 4.4.7.

70 Then, for j 6= i: 1 −1 r − 1 r−2 1r −1 1r −1 ∂2 f ∂V ( v ) = ( r − 1 ) V r −2 (v)vir = V vi v j , ∂vi ∂v j ∂v j r

while when i = j, there is an additional term: ∂2 f (v) ∂v2i

= =

1 −2 r − 1 r−2 2r −2 1 V vi + ( − 1)V r−1 vir r r 2

1 −2 r − 1 r −2 r −2 r V vi − V · vi r

We can write the Hessian as

∇2 f (v) = (r − 1)V r−2 zz> − V · diag (y) , 1 −1

where z = (v1r

1 −1

, . . . , vnr

1 −2

), and y = (v1r

1 −2

, . . . , vnr

).

Due to Schur complements, and from diag (y) 0, the condition ∇2 f (v) 0 is equivalent to n 1 z2i = ∑ vir = V, y i =1 i =1 i n

V ≥ z> diag (y)−1 z = ∑ which holds.

Exercise 8.4 (Some simple optimization problems) Solve the following optimization problems. Make sure to determine an optimal primal solution.

1. Show that, for given scalars α, β, β2 . = f (α, β) = min αd + d d >0

(

−∞ √ 2| β | α

2. Show that for an arbitrary vector z ∈ Rm , z2 1 m kzk1 = min ∑ di + i di d >0 2 i =1

if α ≤ 0 otherwise.

!

.

(8.12)

3. Show that for an arbitrary vector z ∈ Rm , we have z2i : d > 0, d i =1 i m

kzk21 = min ∑ d

m

∑ di = 1.

i =1

Solution 8.4 1. The result obtains with d → +∞ when α ≤ 0. When α > 0, we take the derivative with respect to d, and obtain a unique minimizer in the domain d∗ = β/α > 0; the result then follows.

71 2. The result is a direct consequence of the previous part.

3. We can apply strong duality to the problem, since it is convex and strictly feasible. Dualizing the equality constraint, and using the first part: m z2i + ν ( ∑ d i − 1) d i =1 i i =1 m

= min max ∑

p∗

d >0

ν

m z2i + ν ( ∑ d i − 1) ν d >0 i =1 d i i =1 ! n √ max −ν + 2 ν ∑ |zi | m

= max min ∑ = =

ν >0

i =1

kzk21 .

The result shows that the function g β is concave, as the point-wise minimum of affine functions.

Exercise 8.5 (Minimizing a sum of logarithms) Consider the following problem: p∗ = maxn x ∈R

s.t.:

∑in=1 αi ln xi x ≥ 0,

1> x = c,

where c > 0 and αi > 0, i = 1, . . . , n. Problems of this form arise, for instance, in maximum-likelihood estimation of the transition probabilities of a discrete-time Markov chain. Determine in closed-form a minimizer, and show that the optimal objective value of this problem is n

p∗ = α ln(c/α) + ∑ αi ln αi , i =1

. where α = ∑in=1 αi .

Solution 8.5 Let us consider the equivalent problem p∗ = minn x ∈R

s.t.:

∑in=1 −αi ln xi x ≥ 0,

1> x = c.

Since the objective is strictly decreasing over x ≥ 0 and 1> x is nondecreasing over x ≥ 0, we can replace the equality constraint by an inequality one, thus we consider the problem p∗ = minn x ∈R

s.t.:

∑in=1 −αi ln xi x ≥ 0,

1> x ≤ c.

72 The partial Lagrangian for this problem is n

L( x, µ) = =

∑ αi ln 1/xi + µ(1> x − c)

i =1 n

∑ (αi ln 1/xi + µxi ) − µc,

i =1

and, for µ ≥ 0, n

g(µ)

= min L( x, µ) = −µc + ∑ min (αi ln 1/xi + µxi ) x ≥0

i =1

x i ≥0

n

= −µc + ∑ (αi ln(µ/αi ) + αi ) i =1

n

n

i =1

i =1

= −µc + ln µ ∑ αi + ∑ αi (1 − ln αi ), the minimium with respect to xi ≥ 0 in the previous expression being attained at the unique point xi = αi /µ ≥ 0. The dual is thus d∗ = maxµ≥0 g(µ), and strong duality holds, since the primal problem is strictly feasible. The optimal dual solution is easily obtained as µ∗ =

∑in=1 αi , c

from which we obtain the optimal primal solution as xi∗ =

αi α =c n i , µ∗ ∑ i =1 α i

i = 1, . . . , n.

The expression for the optimal objective value follows by substituting this optimal solution back into the objective.

Exercise 8.6 (Monotonicity and locality) Consider the optimization problems (no assumption of convexity here) p1∗

. =

p2∗

. =

∗ p13

. =

∗ p23

. =

min f 0 ( x )

x ∈X1

min f 0 ( x )

x ∈X2

min

f0 (x)

min

f 0 ( x ),

x ∈X1 ∩X3 x ∈X2 ∩X3

where X1 ⊆ X2 .

1. Prove that p1∗ ≥ p2∗ (i.e., enlarging the feasible set cannot worsen the optimal objective).

2. Prove that, if p1∗ = p2∗ , then it holds that ∗ p13 = p1∗

⇒

∗ p23 = p2∗ .

73 3. Assume that all problems above attain unique optimal solutions.

Prove that, under such hypothesis, if p1∗ = p2∗ , then it holds that ∗ p23 = p2∗

⇒

∗ p13 = p1∗ .

Solution 8.6 (Monotonicity and locality) 1. The first point is obvious, since any optimal point of the first problem is feasible for the second problem.

2. Clearly, X1 ⊆ X2 implies that X1 ∩ X3 ⊆ X2 ∩ X3 , therefore, by ∗ ≥ p∗ . But X ∩ X ⊆ X the first point, it must be p2∗ = p1∗ = p13 2 3 2 23 ∗ ≥ p∗ = p∗ (the last equality follows since we implies that p23 2 13 ∗ = p∗ ). It must therefore be are assuming that p1∗ = p2∗ and p13 1 ∗ = p∗ , as desired. p23 13 ∗ , x ∗ , x ∗ , the unique points attaining p∗ , p∗ , p∗ , and 3. Call x1∗ , x13 2 23 2 1 13 ∗ , respectively. Since p∗ = p∗ , uniqueness of the optimal solup23 2 1 tions implies that it must be x1∗ = x2∗ (for if x1∗ 6= x2∗ , then p2∗ would ∗ = p∗ implies that be attained at two optimal solutions). Also, p23 2 ∗ ∗ ∗ ∗ ∗ x23 = x2 , hence x1 = x2 = x23 . This means in particular that x1∗ belongs to X1 ∩ X3 , thus ∗ ∗ p1∗ = f 0 ( x1∗ ) ≥ f 0 ( x13 ) = p13 . ∗ ≥ p∗ = On the other hand, X1 ∩ X3 ⊆ X2 ∩ X3 implies that p13 23 ∗ , as desired. p2∗ = p1∗ , hence it must hold that p1∗ = p13

Exercise 8.7 (Some matrix norms) Let X = [ x1 , . . . , xm ] ∈ Rn,m , and p ∈ [1, +∞]. We consider the problem . φ p ( X ) = max k X > uk p : u> u = 1. u

If the data is centered, that is, X1 = 0, the above amounts of finding a direction of largest “deviation” from the origin, where deviation is measured using the l p -norm.

1. Is φ p a (matrix) norm?

2. Solve the problem for p = 2. Find an optimal u.

3. Solve the problem for p = ∞. Find an optimal u.

4. Show that φ p ( X ) = max k Xvk2 : kvkq ≤ 1, v

where 1/p + 1/q = 1. (Hence, φ p ( X ) depends only on X > X).

Hint: you can use the fact that the norm dual to the l p -norm is the

74 lq -norm and vice-versa, in the sense that, for any scalars p ≥ 1, q ≥ 1 with 1/p + 1/q = 1, we have max

v: kvkq ≤1

u> v = kuk p .

Solution 8.7 1. For every u, the function X → k X > uk p is convex, hence φ p is.

Further, φ p is positively homogeneous, since φ p (αX ) = αφ p ( X ).

Together with convexity this ensures that φ p satisfies the triangle inequality. Finally, if φ p ( X ) = 0 then for every u, u 6= 0, k X > uk p = 0, which implies X > u = 0 for every u; thus, X = 0.

2. When p = 2, we have φ2 ( X )2

=

max

u : u > u =1

u> X > Xu

= λmax ( XX > ) = σmax ( X )2 , where σmax denotes the largest singular value. The above is due to the variational characterization of eigenvalues, see Theorem 4.3.

We obtain that φ2 ( X ) = σmax ( X ). An optimal u is given by a right singular vector corresponding to the largest singular value (see Corollary 5.1).

3. For p = +∞, we have φp (X)

=

max

max | xi> u|

max

max

u : u > u =1 1≤ i ≤ m

=

1≤ i ≤ m u : u > u =1

| xi> u|

max k xi k2 .

=

1≤ i ≤ m

4. We have, using the hint given: φp (X)

= = =

max

max

v> X > u

max

max

v> X > u

max

k Xvk2 ,

u : u > u =1 v : k v k q ≤1 v : k v k q ≤1 u : u > u =1 v : k v k q ≤1

as claimed.

Exercise 8.8 (Norms of matrices with non-negative entries) Let X ∈ n,m R+ be a matrix with non-negative entries, and p, r ∈ [1, +∞], with p ≥ r. We consider the problem φ p,r ( X ) = max k Xvkr : kvk p ≤ 1. v

75 1. Show that the function f X : Rm + → R, with values !r n

m

f X (u) = ∑

∑ Xij u j

i =1

1/p

j =1

is concave when p ≥ r.

2. Use the previous result to formulate an efficiently solvable convex problem that has φ p,r ( X )r as optimal value.

Solution 8.8 1. It suffices to prove the result for n = 1. Using the change of p variable v j = X1j u j , j = 1, . . . , m, we can further assume X1j = 1 for every j = 1, . . . , m. We are led to prove convexity of the function f : Rm + → R, with values !r m

∑ vj

f (v) = −

1/p

.

j =1

The convexity condition is obvious if p ≥ r = 1. Let us assume r > 1.

The function f is twice differentiable, so we can use the Hessian condition. Define m

S = ∑ vj

1/p

.

j =1

We have

∂f r 1/p−1 ( u ) = − S r −1 v i , i = 1, . . . , m. ∂vi p

Further, denoting by δij the Kronecker symbol (equal to 1 if i = j, 0 otherwise) r ( p − 1) r−1 1/p−1 r (r − 1) r−2 1/p−1 1/p−1 ∂2 f S vi δij − S vi vj , i, j = 1, . . . , m. = p2 p2 ∂u2i We can write the Hessian as

∇2 f ( u ) = where yi =

r ( p − 1 ) r −2 > S diag y − zz , ( ) p2

1/p−1 S( p−1) 1/p−2 , zi = vi , i = 1, . . . , m. r −1 v i

Noting that yi = 0 implies zi = 0, we can reduce the problem to the case when y > 0 (component-wise). Precisely, we can always re-order the variables so that the zero elements in y, z come first.

We then have !

0 0 > diag (y) − zz = , 0 diag (ỹ) − z̃z̃>

76 where ỹ, z̃ contains the non-zero components of y, z. This proves that it suffices to show the result when y, z are positive componentwise.

Using Schur complements, the condition diag (y) zz> becomes equivalent to ! m m z2 r−1 r−1 1/p i vi 1≥∑ = = , ∑ y S ( p − 1 ) p −1 i =1 i =1 i The above holds, due to p ≥ r > 1. p

2. With the change of variable u j = v j , j = 1, . . . , m, we obtain the convex formulation m

max f X (u) : u ≥ 0, u

∑ u j ≤ 1.

i =1

Exercise 8.9 (Magnitude least-squares) For given n-vectors a1 , . . . , am , we consider the problem 2 m p∗ = min ∑ | ai> x | − 1 . x

i =1

1. Is the problem convex? If so, can you formulate it as an ordinary least-squares problem? An LP? A QP? A QCQP? An SOCP? None of the above? Justify your answers precisely.

2. Show that the optimal value p∗ depends only on the matrix K = A> A, where A = [ a1 , . . . , am ] is the n × m matrix of data points (that is, if two different matrices A1 , A2 satisfy A1> A1 = A2> A2 , then the corresponding optimal values are the same).

Solution 8.9 1. The problem is not convex. Indeed, assume that m = n = 1, and set a1 = 1. The objective function is the function f with values f ( x ) = (| x | − 1)2 , which is not convex, since f (0) = 1 >

1 ( f (1) + f (−1)) = 0.

2

2. The fundamental theorem of linear algebra20 states that any vector x ∈ Rn can be written as x = Ay + z, for some y, z with A> z = 0. We observe that the variable z does not play any role in the objective; in other words, we can restrict our search for x to those

20

See Section 3.2.4.

77 that lie in the span of A. That is, the objective can be written as a function of y only: m

∑ |ai> ( Ay)| − 1

i =1

2

2 m = ∑ |k> , i y| − 1 i =1

. where k i = A> ai is the i-th column of A> A. Exercise 8.10 (Eigenvalues and optimization) Given an n × n symmetric matrix Q, define w1 = arg min x > Qx, and µ1 = min x > Qx, k x k2 =1

k x k2 =1

and for k = 1, 2, . . . , n − 1: wk+1 = arg min x > Qx k x k2 =1

µk+1 = min x > Qx k x k2 =1

such that wi> x = 0, i = 1, . . . , k,

such that wi> x = 0, i = 1, . . . , k.

Using optimization principles and theory: 1. Show that µ1 ≤ µ2 ≤ · · · ≤ µn .

2. Show that the vectors w1 , . . . , wn are linearly independent, and form an orthonormal basis of Rn .

3. Show how µ1 can be interpreted as a Lagrange multiplier, and that µ1 is the smallest eigenvalue of Q. 4. Show how µ2 , . . . , µn can also be interpreted as Lagrange multipliers. Hint: show that µk+1 is the smallest eigenvalue of Wk> QWk , where Wk = [wk+1 , . . . , wn ].

Solution 8.10 1. The result follows from the basic fact21 that the optimization problems yielding optimal values µk all have the same objective, while their feasible sets are nested (included in each other).

2. By construction the n vectors wi , i = 1, . . . , n are unit-norm and mutually orthogonal, hence they form an orthonormal basis.

3. We have from the variational Theorem 4.3 that µ1 is the smallest eigenvalue of Q, and that this optimal value is attained for w1 equal to the corresponding normalized eigenvector of Q. The Lagrangian of problem minx:x> x=1 x > Qx is

L( x, µ) = x > Qx + µ(1 − x > x ) = x > ( Q − µI ) x + µ

21

See point 1. in Exercise 8.6.

78 and the dual function is22

Use the result for the minimum of a quadratic function, given in Section 9.1.

22

g(µ) = inf L( x, µ) = x

(

µ if Q − µI 0 −∞ otherwise.

The dual problem amounts thus to maximizing g(µ) = µ subject to Q − µI 0, which means that µ ≤ mini=1,...,n λi ( Q) (here, λi ( Q) are the eigenvalues of Q, arranged in the usual decreasing order), hence the optimal µ is µ∗ = λn ( Q) = µ1 . This shows that strong duality holds between the primal and dual problems, and that µ1 can be interpreted as the optimal dual multiplier.

4. For k ∈ {1, . . . , n − 1}, consider the problem defining wk+1 , µk+1 : µ k +1

=

min

x : k x k2 =1

x > Qx : (wi> x )2 = 0, 1 ≤ i ≤ k.

Here, x is constrained to belong to the subspace V orthogonal to w1 , . . . , wk (that is, since w1 , . . . , wn form an orthonormal set, V = span(wk+1 , . . . , wn )) and, by the fundamental theorem of linear algebra, this subspace has dimension n − k. From the minimax principle stated in Corollary 4.1, it follows that µk+1 = λn−k ( Q), i.e., it is equal to the (n − k)-largest eigenvalue of Q. Thus, µk+1 is the (k + 1)-smallest eigenvalue of Q, and this optimal value is attained for x = wk+1 . Since x ∈ V , we let x = Wk z, and rewrite the problem in the variable z µ k +1

=

min

z : k z k2 =1

z> Wk QWk z,

where Wk QWk = diag (µk+1 , . . . , µn ). Applying the result of the previous point, we have that strong duality holds for this problem, and that µk+1 is the optimal Lagrange multiplier.

Exercise 8.11 (Block norm penalty) In this exercise we partition vectors x ∈ Rn into p blocks x = ( x1 , . . . , x p ), with xi ∈ Rni , n1 + . . . + n p = n. Define the function ρ : Rn → R with values p

ρ ( x ) = ∑ k x i k2 . i =1

1. Prove that ρ is a norm. .

2. Find a simple expression for the “dual norm,” ρ∗ ( x ) =

sup z> x. z: ρ(z)=1

3. What is the dual of the dual norm?

4. For a scalar λ ≥ 0, matrix A ∈ Rm,n and vector y ∈ Rm , we consider the optimization problem . p∗ (λ) = min k Ax − yk2 + λρ( x ). x

79 Explain the practical effect of a high value of λ on the solution.

5. For the problem above, show that λ > σmax ( Ai ) implies that we can set xi = 0 at optimum. Here, Ai ∈ Rm,ni corresponds to the i-th block of columns in A, and σmax refers to the largest singular value.

Solution 8.11 1. The function ρ must pass three tests: (a) positive homogeneity: ρ(αx ) = |α|ρ( x ) for every α ∈ R; (b) convexity; (c) positive definiteness: ρ( x ) = 0 if and only if x = 0.

Together, the first two conditions imply the triangle inequality:

∀ x, y ∈ Rn : ρ( x + y) ≤ ρ( x ) + ρ(y).

Clearly ρ satisfies the conditions.

2. The dual norm has values, for y ∈ Rn : ρ∗ (y)

= max y T x : ρ( x ) ≤ 1 x

p

= max min y x + λ 1 − ∑ k xi k2 T

x

λ ≥0

i =1 p

= min max y x + λ 1 − ∑ k xi k2 T

λ ≥0

x

i =1

!

!

= min λ : λ ≥ kyi k2 , i = 1, . . . , p =

λ ≥0

max kyi k2 ,

1≤ i ≤ p

with y = (y1 , . . . , y p ), yi ∈ Rni .

In the third line, we have used Slater’s condition for convex programs to obtain strong duality; in the fourth,the fact that, for any vector y, T

max y x − λk x k2 = x

(

0 +∞

if kyk2 ≤ λ otherwise.

3. The dual of the dual norm is always itself; indeed, ρ is closed (its

80 epigraph is closed). This can be checked directly: ρ∗∗ (y)

= max y T x : ρ∗ (y) ≤ 1 x

= max y T x : kyi k2 ≤ 1, i = 1, . . . , p x

p

= max ∑ yiT xi : kyi k2 ≤ 1 i = 1, . . . , p x

i =1

n

=

∑ k x i k2 ,

i =1

where we have used the fact that the problem in the third line is decomposable.

4. The practical effect is that when λ is high enough, we will see many blocks of x equal to zero. This is not merely the same effect as if we’d use the `1 -norm: the latter would just encourage elements (possibly scattered across different blocks) to be zero.

5. The dual of the problem is as follows. p∗ (λ)

= min max u T (y − Ax ) + v T x : kuk2 ≤ 1, ρ∗ (v) ≤ λ x

u,v

= max u T y : A T u = v, kuk2 ≤ 1, ρ∗ (v) ≤ λ u,v

= max u T y : kuk2 ≤ 1, ρ∗ ( A T u) ≤ λ u

= max u T y : kuk2 ≤ 1, k AiT uk2 ≤ λ, i = 1, . . . , p. u

In the second line, we have used Sion’s minimax theorem23 .

Let i ∈ {1, . . . , p}. If σmax ( Ai ) < λ, then

23

Theorem 8.8.

∀ u with kuk2 ≤ 1 : k AiT uk2 < λ.

This implies that the constraint k AiT uk2 ≤ λ cannot be active, at any optimum in the dual. Hence, p∗ is the value of an optimization problem where we have set Ai to zero, or alternatively, xi to zero24 .

This means that we can set xi = 0 at optimum.

See the related discussion of removing inactive constraints in a convex problem (Proposition 8.1).

24

81

9.

Linear, Quadratic and Geometric Models

Exercise 9.1 (Formulating problems as LPs or QPs) Formulate the problem . p∗j = min f j ( x ), x

for different functions f j , j = 1, . . . , 5, with values given in Table 9.2, as QPs or LPs, or, if you cannot, explain why. In our formulations, we always use x ∈ Rn as the variable, and assume that A ∈ Rm,n , y ∈ Rm , and k ∈ {1, . . . , m} are given. If you obtain an LP or QP formulation, make sure to put the problem in standard form, stating precisely what the variables, objective and constraints are. Hint: for the last one, see Example 9.10.

f1 (x) f2 (x) f3 (x) f4 (x) f5 (x)

= = = = =

k Ax − yk∞ + k x k1 k Ax − yk22 + k x k1 k Ax − yk22 − k x k1 k Ax − yk22 + k x k21 ∑ik=1 | Ax − y|[i] + k x k22

Table 9.2: Table of the values of different functions f . |z|[i] denotes the element in a vector z that has the i-th largest magnitude.

Solution 9.1 1. For p1∗ , we gave the LP formulation n

p1∗ = min t + ∑ zi : x,t,z

i =1

zi ≥ xi ≥ −zi , i = 1, . . . , n t ≥ ( Ax − y)i ≥ −t, i = 1, . . . , m.

2. Likewise, for p2∗ , we obtain the QP n

p2∗ = min t2 + ∑ zi : x,t,z

i =1

zi ≥ xi ≥ −zi , i = 1, . . . , n t ≥ ( Ax − y)i ≥ −t, i = 1, . . . , m.

3. For p3∗ , the problem is not convex.

4. For p4∗ , we have the QP !2 n

∑ zi

p4∗ = min t2 + x,t,z

:

i =1

zi ≥ xi ≥ −zi , i = 1, . . . , n t ≥ ( Ax − y)i ≥ −t, i = 1, . . . , m.

5. For p5∗ , we use the fact that the sum of the k largest elements of a vecctor z ∈ Rm can be expressed as25 k

∑ z [i ]

i =1

m

= min ks + ∑ max(0, zi − s) s

i =1 m

= min ks + ∑ ui : ui ≥ 0, ui ≥ zi − s, i = 1, . . . , m. s

i =1

We obtain the QP m

p5∗ = min ks + ∑ ui + x > x : x,s,z,u

i =1

zi ≥ xi ≥ −zi , i = 1, . . . , n ui ≥ zi − s, ui ≥ 0, i = 1, . . . , m, zi ≥ ( Ax − y)i ≥ −zi , i = 1, . . . , m.

25

See Section 9.10.

82 Exercise 9.2 (A slalom problem) A two-dimensional skier must slalom down a slope, by going through n parallel gates of known position ( xi , yi ), and of width ci , i = 1, . . . , n. The initial position ( x0 , y0 ) is given, as well as the final one, ( xn+1 , yn+1 ). Here, the x-axis represents the direction down the slope, from left to right.

1. Find the path that minimizes the total length of the path. Your answer should come in the form of an optimization problem.

2. Try solving the problem numerically, with the data given in Table 9.3.

N =5

Figure 9.9: Slalom problem with n = 5 obstacles. “Uphill” (resp. “downhill”) is on the N left (resp. right) side. � 2 minimize (zk − zk−1 ) initial and fiMiddle path is dashed, k=1 nal positions are not shown. subject to |z k − yk | ≤ σk , i = 1, . . . , N

Solution 9.2 1. Assume that ( xi , zi ) is the crossing point of gate i, the path length minimization problem is thus ! ! n +1 xi x i −1 min − ∑ z zi z i −1 i =1



−2 4

2  −2   i P =  0  

x−2i 0 4 8 12 16 20 24

1 2 3 4 5 6

2

subject to

1 T T 2x Px + q x + r

P

yi − ci /2 ≤ zi ≤ yi + ci /2, for i = 1, . . . , n z 0 = y 0 , z n +1 = y n +1 ,

which is equivalent to

  

−2

 yi ci     4 4 N/A −2   −2 4 −2 5 3 −2 2 4 2 6 2 5 1 7 2 4 N/A m

n +1

∑ ti

min z

i =1

subject to

Table 9.3: cise 9.2.

|fi | ≤ F

fi

k+1

=

zk + tk żk +

k+1

=

żk +

2

with the convention c0 = cn+1 = 0. Hence, the problem is an SOCP.

x = [0 4 8 12 16 20 24]’; y = [4 5 4 6 5 7 4]’; c = [0 3 2 2 1 2 0]’; n = 5; cvx_begin variable z(n+2); variable t(n+1); minimize sum(t) subject to z <= y + c/2 z >= y - c/2

+x

V ẋ = V żi

1 z̈ = m f (t)

yi − ci /2 ! ≤ zi ≤ yi +!ci /2, for i = 0, . . . , n + 1 z x i −1 xi ≤ ti , for i = 1, . . . , n + 1. ż − z i −1 zi

2. A CVX code for the problem is

Problemi data for Exer-

1 1 1 fk t2 = zk + (xk+1 − xk )żk + (xk+1 − xk )2 fk 2m k V 2mV 2

fk (xk+1 − xk ) mV 1 V (xN − x0 )

83 for i=1:n+1 norm([x(i+1); z(i+1)] - [x(i); z(i)]) <= t(i) end cvx_end z = [4.0000 4.3749 4.7498 5.1248 5.5000 6.000 5.000];

Exercise 9.3 (Minimum distance to a line segment) The line segment linking two points p, q ∈ Rn (with p 6= q) is the set L = {λp + (1 − λ ) q : 0 ≤ λ ≤ 1}.

1. Show that the minimum distance D∗ from a point a ∈ Rn to the line segment L can be written as a QP in one variable min kλc + dk22 : 0 ≤ λ ≤ 1, λ

for appropriate vectors c, d, which you will determine. Explain why we can always assume a = 0.

2. Prove that the minimum distance is given by26  (q> ( p−q))2 >   if p> q ≤ min(q> q, p> p),  q q − k p−qk22 2 D∗ = q> q if p> q > q> q,    p> p if p> q > p> p.

3. Interpret the result geometrically.

Solution 9.3 1. We can write

kλp + (1 − λ)q − ak2 = kλc + dk2 ,

. where c = p − q 6= 0, d = q − a. We can always translate p, q to p − a, q − a, hence we can assume without loss of generality that a = 0.

2. We have D∗ = kck2 · min kλc̃ + d˜k2 0≤ λ ≤1

. with c̃ = c/kck2 , d˜ = d/kck2 . The problem writes, after squaring, D∗2 = kck22 · min λ2 + 2λc̃> d˜ + d˜> d.˜ λ∈[0,1]

.

The optimizer is either λ0 = −c̃> d˜ (if λ0 ∈ [0, 1]), or 0 (if λ0 < 0), or 1 (if λ0 > 1). Hence  > > 2  if − 1 ≤ c̃> d˜ ≤ 0,  d˜ d˜ − (c̃ d˜) 2 2 > ˜ ˜ D∗ = k c k 2 · d d if c̃> d˜ > 0,   (c̃ + d˜)> (c̃ + d˜) if c̃> d˜ < −1,

Notice that the conditions expressing D∗2 are mutually exclusive, since | p > q | ≤ k p k2 k q k2 .

26

84 or, equivalently,  > > 2 2   d d − ( c d ) / k c k2 D∗2 = d> d   (c + d)> (c + d)

if − kck22 ≤ c> d ≤ 0, if c> d > 0, if c> d < −kck22 .

Further, for c = p − q, d = q, we have that c> d = p> q − q> q, kck22 = p> p + q> q − 2p> q, hence  (q> ( p−q))2 >   if p> q ≤ min(q> q, p> p),  q q − k p−qk22 (9.13) D∗2 = q> q if p> q > q> q,    p> p > > if p q > p p, which is the expression we wished to prove.

3. Consider the line through p and q L = { x : x = q + µc̃},

c̃ =

p−q . k p − q k2

The projection of a = 0 onto this line is given by (see Section 2.3.2.1) z∗ = q + µ∗ c̃,

µ∗ = −c̃> q,

and the optimal squared distance is kz∗ k22 = (c̃> q)2 . If µ∗ ∈ [0, k p − qk2 ], then z∗ is also the projection of a = 0 onto the segment S. The condition µ∗ ∈ [0, k p − qk2 ] is indeed equivalent to p> q ≤ min(q> q, p> p), which is the first condition in (9.13). When z∗ is outside the segment, that is, when it falls to the left of q (for µ∗ < 0) or to the right of p (for µ∗ > k p − qk2 ), then the optimal projection onto the segment is either q or p, respectively, see Figure 9.10.

Exercise 9.4 (Univariate LASSO) Consider the problem . 1 min f ( x ) = k ax − yk22 + λ| x |, 2 x ∈R where λ ≥ 0, a ∈ Rm , y ∈ Rm are given, and x ∈ R is a scalar variable.

Assume that y 6= 0 and a 6= 0, (since otherwise the optimal solution of this problem is simply x = 0). Prove that the optimal solution of this problem is ( 0 if | a> y| ≤ λ ∗ x = xls − sgn( xls ) k aλk2 if | a> y| > λ, 2

where

. a> y xls = k ak22

Figure 9.10: The three cases that may arise when projecting a point onto a segment.

85 corresponds to the solution of the problem for λ = 0. Verify that this solution can be expressed more compactly as x ∗ = sthrλ/k ak2 ( xls ), 2 where sthr is the soft threshold function defined in (12.64).

Solution 9.4 (Univariate LASSO) The subdifferential of the objective is 1 ∂ f ( x ) = ∂k ax − yk22 + λ∂| x |, 2 where ∂k ax − yk22 = x k ak22 − a> y and ∂| x | =

(

sgn( x ) { v : | v | ≤ 1}

if x 6= 0 if x = 0.

We first check under what conditions 0 is contained in the subdifferential of f at x = 0, that is n o x ∗ = 0 is optimal ⇔ 0 ∈ ∂ f (0) = − a> y + λv, |v| ≤ 1 .

Since the term λv may take any value in the interval [−λ, λ], it follows that the above condition is satisfied if and only if | a> y| ≤ λ.

Let next | a> y| > λ. Then, x ∗ 6= 0 and x is optimal if and only if x k ak22 − a> y + λ sgn( x ) = 0, i.e., for x=

a> y λ λ − sgn( x ) = xls − sgn( x ). k ak22 k ak22 k ak22

But since | a> y| > λ the sign of x is not affected by the second term in the above expression, that is sgn( x ) = sgn( xls ), and therefore the optimal solution is given by x ∗ = xls −

λ sgn( xls ), k ak22

which concludes the proof.

Exercise 9.5 (An optimal breakfast) We are given a set of n = 3 types of food, each of which has the nutritional characteristics described in Table 9.4. Find the optimal composition (amount of servings per each food) of a breakfast having minimum cost, number of calories between 2000 and 2250, amount of Vitamin between 5000 and 10000, and sugar level no larger than 1000, assuming that the maximum number of servings is 10.

Solution 9.5 (An optimal breakfast) This is a simple LP problem, which is soved via the following CVX code.

86 Food Corn Milk Bread

Cost 0.15 0.25 0.05

Vitamin 107 500 0

Sugar 45 40 60

Calories 70 121 65

% data D = [.15 107 45 70; .25 500 40 121; 0.05 0 60 65]; cvx_begin variable x(3) cost = D(:,1)’*x; vitamin = D(:,2)’*x; sugar = D(:,3)’*x; calories = D(:,4)’*x; % minimize ( cost ) subject to calories <= 2250; calories >= 2000; vitamin >= 5000; vitamin

<= 10000;

sugar <= 1000; x >= 0; x <= 10; cvx_end

Exercise 9.6 (An LP with wide matrix) Consider the LP p∗ = min c> x : l ≤ Ax ≤ u, x

where A ∈ Rm,n , c ∈ Rn , and l, u ∈ Rm , with l ≤ u. We assume that A is wide, and full rank, that is: m ≤ n, m = rank( A). We are going to develop a closed-form solution to the LP. 1. Explain why the problem is always feasible.

2. Assume that c 6∈ R( A> ). Using the result of Exercise 6.2, show that p∗ = −∞. Hint: set x = x0 + tr, where x0 is feasible, r is such that Ar = 0, c> r > 0, and let t → −∞.

3. Now assume that there exist d ∈ Rm such that c = A> d. Using the fundamental theorem of linear algebra (see Section 3.2.4), any vector x can be written as x = A> y + z for some pair (y, z) with Az = 0. Use this fact, and the result of the previous part, to express the problem in terms of the variable y only.

4. Reduce further the problem to one of the form min d> v : l ≤ v ≤ u. v

Table 9.4: Food costs and nutritional values per serving.

87 Make sure to justify any change of variable you may need. Write the solution to the above in closed form. MAake sure to express the solution steps of the method clearly.

Solution 9.6 1. The problem is always feasible because the range of A is the whole space Rm .

2. We first prove the existence of r such that c> r > 0, Ar = 0 when c 6∈ Range( A). There are two possible techniques.

A first one is based entirely on linear algebra. Due to the fundamental theorem of linear algebra, c can be written as c = A> d + r, with Ar = 0. Our assumption c 6∈ Range( A) implies r 6= 0. Since A is full row rank, AA> 0, hence we can compute y, r:

. d = ( AA> )−1 Ac, r = c − A> d = Pc, with P = I − A> ( AA> )−1 A.

Using the SVD of A, or by direct analysis, it is easy to see that the symmetric matrix P satisfies P2 = P. (P is actually the projection operator on Range( A)⊥ = Null( A).) We obtain c> r = c> Pc = c> P2 c = r > r > 0.

Another method for proving the existence of r is LP duality. Consider the LP . φ = max c> r : Ar = 0. r

The problem is feasible, hence, due to LP strong duality, φ has the same value as the dual: ( 0 if c = A> y, > φ = min max (c − A y)r = y r +∞ otherwise.

Since c 6∈ Range( A), we must have φ = +∞, and there exist a r such that c> r > 0.

To finish the proof, we let x (t) = x0 + tr, with x0 feasible, and t ∈ R. For every t, x (t) is feasible, and c> x (t) = c> x0 + tc> r.

Letting t → −∞, we obtain p∗ = −∞.

3. Now assume that c = A> d for some d ∈ Rm . The problem writes p∗ = min d> Ax : l ≤ Ax ≤ u. x

We see that the variable x appears only via Ax. Letting v = Ax, we obtain the desired result. Note that once an optimal v is found, any x such that Ax = v is optimal; the existence of x is guaranteed since A is full row rank. Of course, there will be possibly many choices for x. The minimum-norm solution is x ∗ = A> ( AA> )−1 v.

88 4. Let d = ( AA> )−1 Ac, the unique solution to a least-squares problem with a full rank matrix: d = arg min k A> ξ − ck2 . ξ

The condition c 6∈ Range( A) is equivalent to k A> d − ck2 > 0.

When this is the case, we proceed by solving p∗ = min d> v : l ≤ v ≤ u, v

(9.14)

which can be solved in time linear in m: it simply requires finding the sign of di and choose vi accordingly. Precisely, one choice for v is    li if di > 0, vi∗ = (9.15) ui if di < 0,   0 if d = 0. i

The above takes into account the case when li or ui are infinite; a zero in f indicates degrees of freedom. The above is the least-norm solution (when both l, u are finite).

The algorithm is as follows. (a) Compute d = ( AA> )−1 A> c = arg minξ k A> ξ − ck. (b) Set v∗ according to (9.15). (c) Set x ∗ = A> ( AA> )−1 v∗ .

Note that the algorithm can be made more efficient if a full SVD of A is available. The following is entirely expressed in terms of the pseudo-inverse of A, A† . (a) Form the SVD of A: A = U (S, 0)V > with S diagonal positivedefinite, U, V unitary of size m, n respectively. (b) Compute f = U (S−1 , 0)V > c = A† c. (c) Set v∗ according to (9.15). (d) Set x ∗ = V (S−1 , 0)> U > v∗ = ( A† )> v∗ .

A similar approach can be taken based on the QR decomposition of A. Exercise 9.7 (Median versus average) For a given vector v ∈ Rn , the average can be found as the solution to the optimization problem min kv − x1k22 , x ∈R

(9.16)

89 where 1 is the vector of ones in Rn . Similarly, it turns out that the median (any value x such that there is an equal number of values in v above or below x) can be found via min kv − x1k1 . x ∈R

(9.17)

We consider a robust version of the average problem (9.16): min x

max

u : kuk∞ ≤λ

kv + u − x1k22 ,

(9.18)

in which we assume that the components of v can be independently perturbed by a vector u whose magnitude is bounded by a given number λ ≥ 0.

1. Is the robust problem (9.18) convex? Justify your answer precisely, based on expression (9.18), and without further manipulation.

2. Show that problem (9.18) can be expressed as n

min ∑ (|vi − x | + λ)2 . x ∈R

i =1

3. Express the problem as a QP. State precisely the variables, and constraints if any.

4. Show that when λ is large, the solution set approaches that of the median problem (9.17).

5. It is often said that the median is a more robust notion of “middle” value than the average, when noise is present in v. Based on the previous part, justify this statement.

Solution 9.7 1. The robust problem is convex, since the objective function is the pointwise maximum (over u) of convex functions, x → kv + u − x1k22 .

2. For a given vector z ∈ Rn , we have max

u : kuk∞ ≤λ

kz + uk22

n

=

( z i + η )2 ∑ η max : |η |≤λ

i =1 n

=

∑ (|zi | + λ)2 ,

i =1

the last line resulting from

∀ η, |η | ≤ λ : |zi + η | ≤ |zi | + λ, with upper bound attained with η = λsgn(zi ).

90 3. A QP formulation is n

min ∑ (ti + λ)2 : ti ≥ ±(vi − x ), i = 1, . . . , n. x,t

i =1

4. The objective function takes the form n

∑ (|vi − x| + λ)2 = λ2 + 2λkv − x1k1 + kv − x1k22 ,

i =1

The corresponding optimization problem has the same minimizers as the problem min kv − x1k1 + x

1 kv − x1k22 , λ

When λ is large, the minimizer will tend to minimize the first term only, which implies the desired result.

5. The median problem can be interpreted as a robust version of the average problem, when the uncertainty is large.

Exercise 9.8 (Convexity and concavity of optimal value of an LP) Consider the linear programming problem . p∗ = min c> x : Ax ≤ b, x

where c ∈ Rn , A ∈ Rm,n , b ∈ Rm . Prove the following statements, or provide a counter-example.

1. The objective function p∗ is a concave function of c.

2. The objective function p∗ is a convex function of b (you may assume that the problem is feasible).

3. The objective function p∗ is a concave function of A. Solution 9.8 1. True, since p∗ is the point-wise minimum of linear (hence, concave) functions (indexed by x), f x : c → c T x. That is: p∗ = min f x (c), x ∈X

where X = { x : Ax ≤ b}. (When the latter set is empty, the problem is infeasible, and p∗ is the function with constant value +∞.)

91 2. True. Consider the function ( f ( x, b) =

cT x +∞

if Ax ≤ b, otherwise

We have p∗ (b) = min f ( x, b), x

therefore if f is convex (with respect to the pair ( x, b)), then p∗ (b) is. The former fact comes from the epigraph characterization of convexity: the condition f ( x, b) ≤ t, for t ∈ R, is equivalent to c T x ≤ t, Ax ≤ b, which are jointly convex conditions on ( x, b, t).

3. False. Consider the problem with n = 1, m = 1, b = 1, c = 1, and with A < 0. We have p∗ = −1/A if A < 0, −∞ otherwise. This function is not concave on its domain.

Exercise 9.9 (Variational formula for the dominant eigenvalue) Recall from Exercise 3.11 that a positive matrix A > 0 has a dominant eigenvalue λ = ρ( A) > 0, and corresponding left eigenvector w > 0 and right eigenvector v > 0 (i.e., w> A = λw> , Av = λv) which belong to the probability simplex S = { x ∈ Rn : x ≥ 0, 1> x = 1}.

In this exercise, we shall prove that the dominant eigenvalue has an optimization-based characterization, similar in spirit to the “variational” characterization of the eigenvalues of symmetric matrices.

Define the function f : S → R++ with values a> x . f ( x ) = min i , i =1,...,n xi

for x ∈ S,

where ai> is the i-th row of A, and we let

ai> x . xi = + ∞ if xi = 0.

1. Prove that, for all x ∈ S and A > 0, it holds that Ax ≥ f ( x ) x ≥ 0.

2. Prove that f ( x ) ≤ λ,

∀ x ∈ S.

3. Show that f (v) = λ, and hence conclude that λ = max f ( x ), x ∈S

which is known as the Collatz-Wielandt formula for the dominant eigenvalue of a positive matrix. This formula actually holds more generally for nonnegative matrices27 , but you are not asked to prove this fact.

For a nonnegative matrix A ≥ 0 an extension of the results stated in Exercise 3.11 for positive matrices holds.

More precisely, if A ≥ 0, then λ = ρ( A) ≥ 0 is still an eigenvalue of A, with a corresponding eigenvector v ≥ 0 (the difference here being that λ could be zero, and not simple, and that v may not be strictly positive).

The stronger results of λ > 0 and simple, and v > 0 are recovered under the additional assumption that A ≥ 0 is primitive, that is there exist an integer k such that Ak > 0 (PerronFrobenius theorem).

27

92 Solution 9.9 (Variational formula for the dominant eigenvalue) 1. Since ai> > 0 and 0 6= x ≥ 0, we have that ai> x > 0 for all x ∈ S, hence f ( x ) > 0, and f ( x ) x ≥ 0. Also, for all i such that xi 6= 0, ! >x a> x a j ai> x = i xi ≥ min xi = f ( x ) xi , xi xj j and for xi = 0 clearly ai> x ≥ f ( x ) xi = 0, thus ai> x ≥ f ( x ) xi holds for all i, hence Ax ≥ f ( x ) x, which, together with the previous point, proves the first claim Ax ≥ f ( x ) x ≥ 0.

2. Multiply the previous inequality on the left by w> , to obtain λw> x = w> Ax ≥ f ( x )w> x, which, since w> x > 0, implies that f ( x ) ≤ λ for all x ∈ S, as desired.

3. v > 0 belongs to S, and

[ Av]i λ [ v ]i = min = λ. i =1,...,n [ v ]i i =1,...,n [ v ]i

f (v) = min

So, f ( x ) ≤ λ for all x ∈ S, and f (v) = λ, which proves that the maximum of f ( x ) over S is attained at x = v, and it is equal to λ.

Exercise 9.10 (LS with uncertain A matrix) Consider a linear least squares problem where the matrix involved is random. Precisely, the residual vector is of the form A(δ) x − b, where the m × n A matrix is affected by stochastic uncertainty. In particular, assume that p

A(δ) = A0 + ∑ Ai δi , i =1

where δi , i = 1, . . . , p are i.i.d. random variables with zero mean and variance σi2 . The standard least-squares objective function k A(δ) x − bk22 is now random, since it depends on δ. We seek to determine x such that the expected value (with respect to the random variable δ) of k A(δ) x − bk22 is minimized. Is such a problem convex? If yes, to which class does it belong to (LP, LS, QP, etc.)?

Solution 9.10 (LS with uncertain A matrix) We have that p

A(δ) x = A0 x + ∑ Ai xδi , i =1

93 and, since δi are i.i.d. and have zero mean, E{ A ( δ ) x }

var{ A(δ) x }

=

A0 x

= E{ x > A ( δ ) > A ( δ ) x } p

p

p

i =1

i =1 j =1

= E{ x > A0> A0 x + 2 ∑ x > A0> Ai xδi + ∑ ∑ x > Ai> A j xδi δj } p

= x > A0> A0 x + ∑ σi2 x > Ai> Ai x i =1 p

= x > ( A0> A0 + ∑ σi2 Ai> Ai ) x. i =1

Now, the expected objective is E{k A(δ) x − bk22 }

= E{ x > A(δ)> A(δ) x + 2b> A(δ) x + b> b} = E{ x > A(δ)> A(δ) x } + 2b> A0 x + b> b p

= x > ( A0> A0 + ∑ σi2 Ai> Ai ) x + 2b> A0 x + b> b. i =1

Minimizing E{k A(δ) x − bk22 } is thus a convex quadratic optimization problem.

94

10.

Second-Order Cone and Robust Models

Exercise 10.1 (Squaring SOCP constraints) When considering a second-order cone constraint, a temptation might be to square it in order to obtain a classical convex quadratic constraint. This might not always work. Consider the constraint x1 + 2x2 ≥ k x k2 , and its squared counterpart:

( x1 + 2x2 )2 ≥ k x k22 .

Is the set defined by the second inequality convex? Discuss.

Solution 10.1 (Squaring SOCP constraints) It is indeed possible to square a SOC constraint

k Ax + bk2 ≤ c> x + d, provided that one takes care of adding the implicit constraint that c> x + d ≥ 0. The SOC constraint is thus equivalent to

k Ax + bk22

0

≤ ( c > x + d )2

≤ c> x + d.

In the exercise, the set defined by

( x1 + 2x2 )2 ≥ k x k22 can be expressed as x2 (4x1 − 3x2 ) ≤ 0 and it is depicted in Figure 10.11.

This set is clearly nonconvex. Instead, the intersection with the above set with the halfspace x1 + 2x2 ≥ 0 is convex and identical to the set defined by the original SOC. Notice however that certain solvers for convex optimization (such as CVX) may not accept a constraint of the form k Ax + bk22 − (c> x + d)2 ≤ 0 since, as shown above, the difference of two convex quadratic functions may well be nonconvex.

Exercise 10.2 (A complicated function) We would like to minimize the function f : R3 → R, with values: f (x)

= max x1 + x2 − min min( x1 + 2, x2 + 2x1 − 5), x3 − 6), ( x1 − x3 )2 + 2x22 , 1 − x1

with the constraint k x k∞ < 1. Explain precisely how to formulate the problem as an SOCP in standard form.

Figure 10.11: Squared SOC region.

95 Solution 10.2 (A complicated function) We first write the problem in epigraphic form as minx,t t subject to f ( x ) ≤ t. Since f is the max of two component functions f ( x ) = max( f 1 ( x ), f 2 ( x )), the constraint is equivalent to f i ( x ) ≤ t, for i = 1, 2. Further, f1 (x) ≤ t

f2 (x) ≤ t (since | x1 | < 1)

⇔

x1 + x2 − t ≤ min(min( x1 + 2, x2 + 2x1 − 5), x3 − 6)

⇔

x1 + x2 − t ≤ x1 + 2, x1 + x2 − t ≤ x2 + 2x1 − 5, x1 + x2 − t ≤ x3 − 6; ( x1 − x3 )2 + 2x22 ≤t 1 − x1

⇔

⇔

x1 + x2 − t ≤ min( x1 + 2, x2 + 2x1 − 5), x1 + x2 − t ≤ x3 − 6

⇔ ( x1 − x3 )2 + 2x22 ≤ t(1 − x1 )

The latter constraint is a hyperbolic one, and can be written as   " # x1 1 0 −1   √ k Ax k22 ≤ t(1 − x1 ), A = , x =  x2  , 0 2 0 x3

which is equivalent, for 1 − x1 > 0, to the SOC constraint (see Section 10.1.1.1) # " 2Ax ≤ 1 − x1 + t.

1 − x2 − t 2

Exercise 10.3 (A minimum time path problem) Consider Figure 10.12, in which a point in 0 must move to reach point p = [4 2.5]> , crossing three layers of fluids having different densities.

Figure 10.12: A minimum-time path problem.

In the first layer, the point can travel at a maximum speed v1 , while in the second layer and third layers it may travel at a lower maximum speeds, respectively v2 = v1 /η2 , and v3 = v1 /η3 , with η2 , η3 > 1. Assume v1 = 1, η2 = 1.5, η3 = 1.2. You have to determine what is the fastest (i.e., minimum time) path from 0 to p. Hint: you may

96 use path leg lengths `1 , `2 , `3 as variables, and observe that, in this problem, equality constraints of the type `i = “something” can be equivalently substituted by inequality constraints `i ≥ ”something” (explain why).

Solution 10.3 (A minimum time path problem) Let us denote with x1 , x2 , x3 the horizontal coordinates of points p1 , p2 and p, respectively, and with y1 , y2 , y3 the corresponding vertical coordinates (which are given: y1 = 1, y2 = 2, y3 = 2.5). Define as h1 , h2 , h3 the lengths of the horizontal projections of the three legs, that is h1 = x1 ,

h2 = x2 − x1 ,

h3 = x3 − x2 .

We shall write the problem using `1 , `2 , `3 , and h1 , h2 , h3 as variables.

Note that by Pythagoras theorem it must hold that q `1 = h21 + y21 = k(h1 , 1)k2 q h22 + (y2 − y1 )2 = k(h2 , 1)k2 `2 = q `3 = h23 + (y3 − y2 )2 = k(h3 , 0.5)k2 .

The total travel time is

T=

`1 ` ` + 2 + 2. v1 v2 v2

We hence set up our optimization problem as follows: min

`1 `2 `2 v1 + v2 + v2

s.t.:

`1 = k(h1 , 1)k2

h,`

`2 = k(h2 , 1)k2

`3 = k(h3 , 0.5)k2

h1 + h2 + h3 = 4, where the last constraint imposes that the horizontal coordinate of point p is equal to 4, as indicated by the figure. This formulation of the problem is not convex, due to the presence of nonlinear equality constraints. However, since we are minimizing a positive linear combination of the `i , we can substitute = with ≥ in the constraints, obtaining min

`1 `2 `2 v1 + v2 + v2

s.t.:

`1 ≥ k(h1 , 1)k2

h,`

`2 ≥ k(h2 , 1)k2

`3 ≥ k(h3 , 0.5)k2

h1 + h2 + h3 = 4.

97 This problem is a SOCP, and equality will hold at optimum. Hence, this problem is equivalent to the original one.

The following CVX code solves the problem with the given numerical data. % minimum-time path exercise p=[4 2.5]’; % target point tk=[1 1 .5]; % layers thickness v1 = 1; nu2=1.5; nu3=1.2; v = [v1 v1/nu2 v1/nu3]’; % cvx_begin variable leg(3) % lenghts of paths variable h(3) % orizontal lengths of path legs time = leg’*(1./v); minimize (time) subject to leg(1) >= norm([h(1) tk(1)]); leg(2) >= norm([h(2) tk(2)]); leg(3) >= norm([h(3) tk(3)]); sum(h) == 4; cvx_end %% plot results px = cumsum([0 h’]); py = cumsum([0 tk]); plot(px,py,’k’); hold on plot(px,py,’ko’); hold off grid on

Exercise 10.4 (k-Ellipses) Consider k points x1 , . . . , xk in R2 . For a given positive number d, we define the k-ellipse with radius d as the set of points x ∈ R2 such that the sum of the distances from x to the points xi is equal to d.

1. How do k-ellipses look like when k = 1 or k = 2? Hint: for k = 2, show that you can assume x1 = − x2 = p, k pk2 = 1, and describe the set in a orthonormal basis of Rn such that p is the first unit vector.

2. Express the problem of computing the geometric median, which is

98 the point that minimizes the sum of the distances to the points xi , i = 1, . . . , k, as an SOCP in standard form.

3. Write a code with input X = ( x1 , . . . , xk ) ∈ R2,k and d > 0 that plots the corresponding k-ellipse.

Solution 10.4 1. For k = 1, we obtain a circle of radius d and center x1 . For k = 2, the set is an ellipse28 .

Indeed, without loss of generality we can assume x1 = − x2 = p, so that the set has now 0 as center of symmetry. Let x ∈ E , with .

E = { x : k x − p k2 + k x + p k2 ≤ d }

We can always assume without loss of generality that k pk2 = 1, and d > 2 (otherwise E = ∅). We now change basis: we set p to be the first unit vector, and find Q = [u2 , . . . , un ] such that [ p, Q] is unitary; in particular Q> Q = In−1 . Then, any x ∈ Rn can be written as x = tp + Qq, with q ∈ Rn−1 , such that q> p = 0, and t = p> x.

The condition x ∈ E writes q q (t − 1)2 + r2 + (t + 1)2 + r2 ≤ d, (10.19) . where r = q> Q> Qq = q> q. The above is equivalent to d ≥ p (t + 1)2 + r2 , and 2

q 2 2 2 2 ( t − 1) + r ≤ d − ( t + 1) + r q 2 2 2 = d + (t + 1) + r − 2d (t + 1)2 + r2 ,

which becomes q 2d (t + 1)2 + r2 ≤ d2 + (t + 1)2 − (t − 1)2 = d2 + 4t The above in turn implies

4d2 ((t + 1)2 + r2 ) ≤ (d2 + 4t)2 , or, after some algebra, and upon replacing r = q> q: q > q + (1 −

4 2 d2 − 1. )t ≤ 2 4 d

p Note that the above condition does imply that d ≥ (t + 1)2 + r2 , so it is equivalent to the original condition (10.19).

The above is an ellipse in (t, q) space, hence the set E is also an ellipse (rotated via the unitary transformation [ p, Q]).

For n = 2, you may remember from High School the fact that an ellipse can be described as the set of points such that the sum of the distances to a pair of points, called the foci, is a constant.

28

99 2. The problem k

min ∑ k x − xi k2 x

i =1

can be written as the SOCP k

min ∑ ti : ti ≥ k x − xi k2 , i = 1, . . . , k. x,t

i =1

3. We first determine the center of the k-ellipse, which by symmetry is the average point x̂ = ( x1 + . . . + xk )/k. Then for every θ ∈ [0, 2π ], we solve the problem of maximizing the distance away from the center in the direction determined by the angle θ, with the constraint to be inside the k-ellipse: k

max r : r

∑ kx̂ + ru(θ ) − xi k2 ≤ d.

i =1

where u(θ ) = (cos(θ, sin(θ ))). The above is an SOCP. In practice, we would discretize the variable θ and obtain a finite set of points on the k-ellipse.

Exercise 10.5 (A portfolio design problem) The returns on n = 4 assets are described by a Gaussian (Normal) random vector r ∈ Rn , having the following expected value r̂ and covariance matrix Σ:     0.0064 0.0008 −0.0011 0 0.12  0.0008 0.0025  0.10  0 0      r̂ =  . , Σ =   −0.0011  0.07  0 0.0004 0  0 0 0 0 0.03

The last (fourth) asset corresponds to a risk-free investment. An investor wants to design a portfolio mix with weights x ∈ Rn (each weight xi is nonnegative, and the sum of the weights is one) so to obtain the best possible expected return r̂ > x, while guaranteeing that: (i) no single asset weights more than 40%; (ii) the risk-free assets should not weight more than 20%; (iii) no asset should weight less than 5%; (iv) the probability of experiencing a return lower than q = −3% should be no larger than e = 10−4 . What is the maximal achievable expected return, under the above constraints?

Solution 10.5 (A portfolio design problem) All constraints in this problem are simple linear constraints, except for the one involving the short-fall probability requirement. First observe that Prob{r > x < q} ≤ e

⇔ Prob{r > x ≥ q} ≥ 1 − e

⇔ Prob{−r > x ≤ −q} ≥ 1 − e.

100 Following the developments in Example 10.4 in the book, we see that the latter constraint is equivalent to a SOC constraint of the form Φ−1 (1 − e)kΣ1/2 x k2 ≤ r̂ > x − q, where Φ−1 is the inverse standard Normal cumulative distribution function. The following CVX code then solves the problem. The maximal achievable expected return is 0.0964 (i.e., 9.6%). Notice that, in Matlab, the command norminv(p,mu,sig) returns the inverse cumulative distribution function for the Normal distribution with mean mu and standard deviation sig, evaluated at the values in p. % Portfolio exercise r = [0.12 0.10 0.07 0.03]’; S = [0.0064 -0.0011

0.0008 0

-0.0011 0; 0.0008

0.0004

0.0025

0

0;...

0; 0 0 0 0];

Ss=sqrtm(S); n=length(r); ep = 1e-4; % bound of risk of loss q q = -0.03; % loss % cvx_begin cvx_solver sdpt3 variable x(n) maximize ( r’*x ) subject to x >= 0 ; sum(x) == 1; x >= 0.05; x <= 0.4; x(4) <= 0.2; norminv(1-ep,0,1)*norm(Ss*x) <= r’*x - q; cvx_end

Exercise 10.6 (A trust-region problem) A version of the so-called (convex) trust-region problem amounts to finding the minimum of a convex quadratic function over an Euclidean ball, that is min

1 > > 2 x Hx + c x + d

s.t.:

x > x ≤ r2 ,

x

where H 0, and r > 0 is the given radius of the ball. Prove that the optimal solution to this problem is unique and it is given by x (λ∗ ) = −( H + λ∗ I )−1 c,

101 where λ∗ = 0 if k H −1 ck2 ≤ r, or otherwise λ∗ is the unique value such that k( H + λ∗ I )−1 ck2 = r.

Solution 10.6 (A trust-region problem) The Lagrangian of this problem can be written as λ 1 > x Hx + c> x + d + ( x > x − r2 ) 2 2 1 > λ > x ( H + λI ) x + c x + d − r2 .

2 2

L( x, λ) = =

The Lagrangian is strongly convex, hence it has a unique minimizer x ∗ (λ) = −( H + λI )−1 c.

Since strong duality holds, the optimal solution of the problem is x ∗ (λ∗ ), where λ∗ is the optimal solution of the dual problem max g(λ), λ ≥0

with g(λ)

1 λ . = − c> ( H + λI )−1 c + d − r2 .

2 2

Let H = UΛU > be a spectral factorization of H, where U is orthogonal, and Λ = diag (λ1 , . . . , λn ) 0. We can write ( H + λI ) = U (Λ + λI )U > , hence g(λ) = d −

c̃2i λ 2 1 n r − ∑ , 2 2 i =1 λ i + λ

. where c̃i are the entries of c̃ = U > c, and it holds that . dg g0 (λ) = dλ

c̃2i 1 1 n = − r2 + ∑ 2 2 i =1 ( λ i + λ )2 1 1 = − r2 + k( H + λI )−1 ck22 .

2 2

Notice that the term k( H + λI )−1 ck2 is strictly decreasing over λ ≥ 0.

Hence, if g0 (0) ≤ 0 then g(λ) < 0 for all λ > 0, which means that g(λ) is decreasing for positive λ, hence the maximum is at the boundary point where λ = 0. If otherwise g0 (0) > 0, then the concave function g has a maximum over λ > 0, at the point where the derivative g0 is zero, that is where

k( H + λI )−1 ck22 = r2 , which is what we needed to prove. The value of λ satisfying this equation can be found numerically via any univariate search technique. For instance, one may use Newton method, starting with some

102 initial λ > 0, and iteratively updating λ ← λ−

g0 (λ) , g00 (λ)

where g00 (λ) is the second derivative n c̃2i . d2 g = − . g00 (λ) = ∑ dλ2 ( λ i + λ )3 i =1

Exercise 10.7 (Univariate square-root LASSO) Consider the problem . min f ( x ) = k ax − yk2 + λ| x |, x ∈R

where λ ≥ 0, a ∈ Rm , y ∈ Rm are given, and x ∈ R is a scalar variable. This is a univariate version of the square-root LASSO problem introduced in Example 8.23. Assume that y 6= 0 and a 6= 0, (since otherwise the optimal solution of this problem is simply x = 0). Prove that the optimal solution of this problem is ∗

x =

where

  0

 xls − sgn( xls ) λ 2 k a k2

r

k ak22 kyk22 −( a> y)2 k ak22 −λ2

if | a> y| ≤ λkyk2

if | a> y| > λkyk2 ,

. a> y xls = . k ak22

Solution 10.7 (Univariate square-root LASSO) The problem is convex but nonsmooth, hence we write the optimality conditions in terms of the subdifferential of the objective: 0 ∈ ∂ f ( x ) = ∂k ax − yk2 + λ∂| x |, where

and

 >  a ( ax − y) ∂k ax − yk2 = k ax − yk2  { a > g : k g k2 ≤ 1} ∂| x | =

(

sgn( x ) { v : | v | ≤ 1}

if ax − y 6= 0 if ax − y = 0,

if x 6= 0 if x = 0.

We first check under what conditions 0 is contained in the subdifferential of f at x = 0, that is ( ) a> y ∗ x = 0 is optimal ⇔ 0 ∈ ∂ f (0) = + λv, |v| ≤ 1 . k y k2

103 Since the term λv may take any value in the interval [−λ, λ], it follows that the above condition is satisfied if and only if | a> y|/kyk2 ≤ λ, which proves the part regarding optimality of x = 0. Also, since by the Cauchy-Schwartz inequality it holds that

| a > y | ≤ k a k2 k y k2 , it is clear that k ak2 ≤ λ implies | a> y| ≤ λkyk2 , hence the optimal solution is certainly zero when k ak2 ≤ λ.

Consider next the case when the optimal solution is nonzero, i.e., when | a> y| > λkyk2 , thus k ak2 > λ. We initially assume for simplicity that a and y are not collinear, so that ax − y 6= 0 for all x; later we show that the derived solution is still valid if this assumption is lifted. With this assumption, and since x 6= 0, we have that x is optimal

⇔

0 = ∂ f (x) =

a> ( ax − y) + λ sgn( x ), k ax − yk2

that is, since k ax − yk2 6= 0, for a> ( ax − y) = −λk ax − yk2 sgn( x ).

(10.20)

All solution to this equation are also solutions of the squared equation (10.21) ( a> ax − a> y)2 = λ2 k ax − yk22 , which is a quadratic equation in x, equivalent to:

k ak22 (k ak22 − λ2 ) x2 − 2a> y(k ak22 − λ2 ) x + ( a> y)2 − λ2 kyk22 = 0.

The roots of this equation are in s x± = xls ±

2 − xls

( a> y)2 − λ2 kyk22 . k ak22 (k ak22 − λ2 )

Observe that the term under the square root is nonnegative, since

( a> y)2 − λ2 kyk22 . 2 δ = xls − k ak22 (k ak22 − λ2 )

= =

( a> y)2 ( a> y)2 − λ2 kyk22 − k a k4 k ak22 (k ak22 − λ2 ) λ2 k ak22 kyk22 − ( a> y)2 · , k ak22 k ak22 (k ak22 − λ2 )

where, under the current conditions, k ak22 − λ2 > 0, and k ak22 kyk22 − ( a> y)2 ≥ 0, by the Cauchy-Schwartz inequality. Further, δ ≥ 0 is 2 , since the condition | a> y | > λ k y k smaller in magnitude than xls √ 2 2 − δ > 0. It follows that the sign of x = x ± δ is implies that xls ± ls √ the same sign of xls (since adding ± δ to xls cannot change its sign).

104 Then, plugging x ← x± into equation (10.20), we have the left-hand side √ √ k ak22 x± − a> y = k ak22 ( xls ± δ) − a> y = ± δ and the right-hand side

−λk ax± − yk2 sgn( x± ) = −λk ax± − yk2 sgn( xls ).

Thus, sign consistency is obtained by choosing the solution with “+” when xls is negative, and with “-” when xls is positive. In conclusion, the unique solution to eq. (10.20) is given by s k ak22 kyk22 − ( a> y)2 λ ∗ x = xls − sgn( xls ) , (10.22) k ak22 k ak22 − λ2 which is the expression we wished to prove.

It only remains to be proved that the above expression is still valid also when y and a are collinear. In this case, since k ak22 kyk22 = ( a> y)2 , eq. (10.22) gives x ∗ = xls , and we have that ax ∗ − y = 0. Let us check that this solution is indeed optimal. The subdifferential of f at x ∗ 6= 0 such that ax ∗ − y = 0 is ∂ f ( x ∗ ) = { a> g + λ sgn( x ∗ ), k gk2 ≤ 1}, and we see that 0 ∈ ∂ f ( x ∗ ) if k ak2 ≥ λ, which is indeed the condition under which the expression (10.22) for x ∗ holds.

Exercise 10.8 (Proving convexity via duality) Consider the function f : Rn++ → R, with values n

f ( x ) = 2 max t − ∑ t

i =1

q

x i + t2 .

1. Explain why the problem that defines f is a convex optimization problem (in variable t). Formulate it as an SOCP. 2. Is f convex?

3. Show that the function g : Rn++ → R, with values n

1 − y i =1 i

g(y) = ∑

1 n

∑ yi

i =1

is convex. Hint: for a given y ∈ Rn++ , show that g(y) = max − x T y − f ( x ). x >0

Make sure to justify any use of strong duality.

105 Solution 10.8 1. The objective function of the problem defining f is concave in t, hence that problem is convex. We can write it as the SOCP " √ # n xi , i = 1, . . . , n. f ( x ) = 2 max t − ∑ zi : zi ≥ t,z t i =1 2

2. Yes, since f is the pointwise maximum (over t) of the convex funcx tions with domain R+ n

ft : x → t − ∑

i =1

q

x i + t2 .

3. We have, for a given y ∈ Rn++ : max − x T y − f ( x ) x >0

n

= max min − x y − 2t + 2 ∑ x >0

>

t

i =1

q

x i + t2

n q = min −2t + max ∑ 2 xi + t2 − xi yi , x >0

t

i =1

n

= min −2t + ∑ max t

i =1

x i >0

q 2 x i + t2 − x i y i ,

where we have used strong duality for the prblem defining f , as allowed by the Slater’s condition. Next, we observe that for every scalars y > 0, t > 0, we have max 2 x >0

p

1 1 1 x + t2 − yx = 2 − y( 2 − t2 ) = + yt2 . y y y

Hence, max − x T y − f ( x ) x >0

= min −2t + ∑

= min −2t + t2

∑ yi

n

t

i =1

t

1 + y i t2 yi !

n

i =1

n

1 y i =1 i

+∑

n

1 1 − n y ∑ i i =1 y i i =1

= min ∑ t

=

g ( y ).

This shows that g is convex, as the pointwise maximum of affine functions. Note that the convexity of g is not immediately obvious.

106 Exercise 10.9 (Robust sphere enclosure) Let Bi , i = 1, . . . , m, be m given Euclidean balls in Rn , with centers xi , and radii ρi ≥ 0. We wish to find a ball B of minimum radius that contains all the Bi , i = 1, . . . , m. Explain how to cast this problem into a known convex optimization format.

Solution 10.9 (Robust sphere enclosure) Let c ∈ Rn and r ≥ 0 denote the center and radius of the enclosing ball B, respectively. We express the given balls Bi as Bi = { x : x = xi + δi , kδi k2 ≤ ρi },

i = 1, . . . , m.

We have that Bi ⊆ B if and only if max k x − ck2 ≤ r. x ∈ Bi

But max k x − ck2 = max k xi − c + δi k2 = k xi − ck2 + ρi . x ∈ Bi

kδi k2 ≤ρi

The problem is then cast as the following SOCP min c,r

s.t.:

r

k xi − ck2 + ρi ≤ r, i = 1, . . . , m.

107

11.

Semidefinite Models

Exercise 11.1 (Minimum distance to a line segment revisited) In this exercise, we revisit Exercise 9.3, and approach it using the S -procedure of Section 11.3.3.1.

1. Show that the minimum distance from the line segment L to the origin is above a given number R ≥ 0 if and only if

kλ( p − q) + qk22 ≥ R2 whenever λ(1 − λ) ≥ 0.

2. Apply the S -procedure, and prove that the above is in turn equivalent to the LMI in τ ≥ 0 " # k p − qk22 + τ q> ( p − q) − τ/2 0. q> ( p − q) − τ/2 q > q − R2 3. Using the Schur complement rule29 , show that the above is consistent with the result given in Exercise 9.3.

Solution 11.1 1. The minimum distance is bounded below by a number R ≥ 0 if and only if

kλ( p − q) + qk22 ≥ R2 whenever λ ∈ [0, 1].

The desired result follows from the equivalence between λ ∈ [0, 1], and the quadratic inequality λ(1 − λ) ≥ 0.

2. The S -procedure states that the above condition is equivalent to the existence of a scalar τ ≥ 0 such that

∀ λ : kλ( p − q) + qk22 ≥ R2 + τλ(1 − λ).

The above can be written as " #> " λ k p − qk22 + τ ∀λ : > 1 q ( p − q) − τ/2

q> ( p − q) − τ/2 q > q − R2

#"

λ 1

#

≥ 0,

which in turn is equivalent to the desired result.

3. The squared minimal distance is the largest R such that the linear matrix inequality (in R2 , τ) holds: " # k p − qk22 + τ q> ( p − q) − τ/2 2 2 D∗ = max R : 0. q> ( p − q) − τ/2 q > q − R2 R2 ,τ

29

See Theorem 4.9.

108 Using Schur complements: D∗2

= max q> q − τ ≥0

(q> ( p − q) − τ/2)2 τ + k p − qk22

= q> q − min f (τ, α, β), τ ≥0

where α = q> ( p − q), β = k p − qk2 , and f (τ, α, β) =

(α − τ/2)2 . τ + β2

Setting the gradient to zero, we obtain that τ/2 − α = 1 or τ = 2α. τ + β2 The first equality is the same as τ = −2(α + β2 ). We have obtained that the optimal τ is one of the three values: 0, 2α, or −2(α + β2 ).

If α ≥ 0, that is, p> q ≥ q> q, then −2(α + β2 ) ≤ 0 cannot be optimal. If we set τ ∗ = 2α ≥ 0 we attain the value 0 for f , which is the global minimum in that case; the corresponding squared minimum distance is then D∗2 = q> q.

Now assume that α < 0, that is, p> q < q> q, then the unconstrained minimizer is −2(α + β2 ). Two cases can occur.

If α ≤ − β2 (≤ 0), that is, p> p ≥ p> q, the unconstrained minimizer −2(α + β2 ) is feasible for the problem, and thus, optimal. The corresponding optimal value for f is −2α − β2 , and the squared minimum distance is then D∗2 = q> q + 2q> ( p − q) + k p − qk22 = p> p.

Otherwise, that is, if 0 ≥ α ≥ − β2 , that is, then the unconstrained minimizer is not feasible, and τ ∗ = 0, resulting in the optimal value for f α2 /β2 , and a minimal distance D∗2 = q> q − We have obtained  (q> ( p−q))2 >    q q − k p−qk22 D∗2 = q> q    p> p

(q> ( p − q))2 . k p − qk22

if p> q ≤ min(q> q, p> p), if p> q > q> q, if p> q > p> p.

109 Exercise 11.2 (A variation on principal component analysis) Let X = [ x1 , . . . , xm ] ∈ Rn,m . For p = 1, 2, we consider the problem m . φ p ( X ) = max ∑ | xi> u| p : u> u = 1. u

(11.23)

i =1

If the data is centered, the case p = 1 amounts of finding a direction of largest “deviation” from the origin, where deviation is measured using the `1 -norm; arguably, this is less sensitive to outliers than the case p = 2, which corresponds to principal component analysis.

1. Find an expression for φ2 , in terms of the singular values of X. 2. Show that the problem, for p = 1, can be approximated via an SDP, as φ1 ( X ) ≤ ψ1 ( X ), where m . ψ1 ( X ) = max ∑ U

i =1

q

xi> Uxi : U 0, trace U = 1.

Is ψ1 a norm?

3. Formulate a dual to the above expression. Does strong duality hold? Hint: introduce new variables zi = xi> Uxi , i = 1, . . . , m, and dualize the corresponding constraints.

4. Use the identity (8.12) to approximate, via weak duality, the problem (11.23). How does your bound compare with ψ1 ?

5. Show that ψ1 ( X )2 = min trace D : D diagonal, D 0, D X > X. D

Hint: scale the variables in the dual problem and optimize over the scaling. That is, set D = α D̄, with λmax ( X D̃ −1 X > ) = 1 and α > 0, and optimize over α. Then argue that we can replace the equality constraint on D̃ by a convex inequality, and use Schur complements to handle that corresponding inequality.

6. Show that φ1 ( X ) =

max

v : k v k ∞ ≤1

k Xvk2 .

Is the maximum always attained with a vector v such that |vi | = 1 for every i? Hint: use the fact that

k z k1 =

max

v : k v k ∞ ≤1

z> v.

110 7. A result by Yu. Nesterov30 shows that for any symmetric matrix Q ∈ Rm,m , the problem p∗ =

max

v : k v k ∞ ≤1

v> Qv

can be approximated within π/2 relative value via SDP. Precisely, (2/π )d∗ ≤ p∗ ≤ d∗ , where d∗ = min trace D : D diagonal, D Q. D

(11.24)

Use this result to show that r 2 ψ ( X ) ≤ φ1 ( X ) ≤ ψ1 ( X ). π 1 That is, the SDP approximation is within ≈ 80% of the true value, irrespective of problem data.

8. Discuss the respective complexity of the problems of computing φ2 and ψ1 (you can use the fact that, for a given m × m symmetric matrix Q, the SDP (11.24) can be solved in O(m3 )).

Solution 11.2 1. When p = 2, m

φ2 ( X )

= max ∑ ( xi> u)2 : u> u = 1 u

i =1

= max k X > uk22 : u> u = 1 u

= λmax ( XX > ) = σmax ( X )2 .

2. We have m q

φ1 ( X ) = max ∑ U

i =1

xi> Uxi : U 0, trace U = 1, rank(U ) = 1,

where we have used the variable U = uu> . The expression obtains upon dropping the rank constraint on the positive semi-definite matrix U. Since we relax (that is, drop) constraints in a maximization problem, we obtain an upper bound.

The function ψ1 is convex, since x → kU 1/2 x k2 is, for every U 0. It is positively homogeneous, and positive-definite (that is, ψ1 ( X ) = 0 implies X = 0). Hence, it is a norm.

Yu. Nesterov, “Quality of semidefinite relaxation for nonconvex quadratic optimization,” discussion paper, CORE, 1997.

30

111 3. We can express ψ1 ( X ) as m √

ψ1 ( X ) = max ∑

U,z i =1

U 0, trace U = 1, z ≥ 0, zi = xi> Uxi , i = 1, . . . , n.

zi :

The above problem is convex and strictly feasible, hence strong duality holds: denoting by U the set of n × n positive semi-definite matrices with unit trace, ψ1 ( X )

=

max

U ∈U , z≥0

= min y

min ∑

m √

zi + yi ( xi> Uxi − zi )

∑

zi + yi ( xi> Uxi − zi )

y

i =1 m √

max

U ∈U , z≥0 i =1 m

1 min max ∑ + trace U 4y y≥0 U ∈U i i =1

=

m

1 min ∑ + λmax 4yi y ≥0 i =1

=

m

∑

m

∑

yi xi xi>

i =1

yi xi xi>

i =1

!

!

,

where we have used the fact that, for a given η ∈ R: max ξ ≥0

p

ξ − ηξ =

(

1 4η

if η ≥ 0,

+∞ otherwise,

and that for any symmetric matrix Y: max trace(YU ) = λmax (Y ).

U ∈U

4. Use the identity (8.12), we obtain φ1 ( X )

=

1 max min u : u > u =1 d >0 2

( xi> u)2 d + ∑ i di i =1

≥

1 min max > d >0 u : u u =1 2

( x > u )2 ∑ di + idi i =1

=

1 m 1 min ∑ di + max u> 2 u : u > u =1 d >0 2 i =1

m

m

m

! !

1 ∑ di xi xi> i =1

!

u

= ψ1 ( X ), where we have used the the change of variable di = 1/(2yi ), i = 1, . . . , m, to prove the last equality.

5. We have, denoting by D the set of m × m positive-definite diagonal

112 matrices: ψ1 ( X )

= = = =

1 min trace D + λmax ( XD −1 X > ) 2 D∈D 1 1 min α trace D + : D̃ ∈ D , λmax ( XD −1 X > ) = 1 2 α,D̃ α p min trace D̃ : D̃ ∈ D , λmax ( X D̃ −1 X > ) = 1 D̃ p min trace D̃ : D̃ ∈ D , λmax ( X D̃ −1 X > ) ≤ 1, D̃

where in the last line we have exploited the fact that we can always scale D̃ in the last problem, and improve the objective while making the inequality an equality. Now that inequality is equivalent to I X D̃ −1 X > , which, since D̃ 0, is equivalent, via Schur complements, to " # I X 0, X > D̃ and, using Schur complements again, to D̃ X > X. 6. Using the hint: m

ψ1 ( X )

= = =

∑ vi (xi> u)

max

max

max

max ( Xv)> u

u : u > u =1 v k v k ∞ ≤1 i =1 v k v k ∞ ≤1 u : u > u =1

max

v k v k ∞ ≤1

k Xvk2 .

The maximum is always attained as claimed, since we are maximizing a convex function over a polytope, so that the maximum is always attained at the vertices.

7. We have, according to part 6, φ1 ( X )2 =

max

v : k v k ∞ ≤1

v> Qv,

where Q = X > X. The corresponding bound from Nesterov is φ1 ( X )2 ≥ min trace D : D diagonal, D X > X, D

which is precisely the expression we have found in part 5.

8. For p = 2 the complexity is in O(m3 + nm2 ) (to account for the cost of forming X > X), the same as ψ1 ( X ).

113 Exercise 11.3 (Robust Principal Component Analysis) The following problem is known as Robust Principal Component Analysis31 :

31

See Section 13.5.4.

. p∗ = min k A − X k∗ + λk X k1 X

where k · k∗ stands for the nuclear norm32 , and k · k1 here denotes the sum of the absolute values of the elements of a matrix. The interpretation is the following: A is a given data matrix and we would like to decompose it as a sum of a low rank matrix and a sparse matrix. The nuclear norm and `1 norm penalties are respective convex heuristics for these two properties. At optimum, X ∗ will be the sparse component and A − X ∗ will be the low rank component such that their sum gives A.

The nuclear norm is the sum of the singular values of the matrix; see Section 11.4.1.4 and Section 5.2.2.

32

1. Find a dual for this problem. Hint: we have, for any matrix W:

kW k∗ = max trace W > Y : kY k2 ≤ 1, Y

where k · k2 is the largest singular value norm.

2. Transform the primal or dual problem into a known programming class (i.e. LP, SOCP, SDP etc.). Determine the number of variables and constraints. Hint: we have

kY k2 ≤ 1 ⇐⇒ I − YY > 0, where I is the identity matrix.

3. Using the dual, show that when λ > 1, the optimal solution is the zero matrix. Hint: if Y ∗ is the optimal dual variable, the complementary slackness condition states that |Yij∗ | < λ implies Xij∗ = 0 at optimum.

Solution 11.3 1. We can write, thanks to the hint: p∗ = min max trace Y > ( A − X ) + trace( Z > X ) : kY k ≤ 1, k Z k∞ ≤ λ, X

Y,Z

where k · k∞ is the largest magnitude of the entries of the matrix argument. Applying Sion’s minimax theorem33 , we obtain p∗

= max Y,Z

min trace Y > ( A − X ) + trace( Z > X ) X

= max trace Y > A : kY k ≤ 1, kY k∞ ≤ λ.

Y

33

See Theorem 8.8.

: kY k ≤ 1, k Z k∞ ≤ λ

114 2. The condition kY k∞ ≤ λ is equivalent to n(n + 1) ordinary inequalities: −λ ≤ Yij ≤ λ, 1 ≤ i ≤ j ≤ n.

The condition kY k ≤ 1 is equivalent to I − YY > 0, which, using Shcur complements, writes as the linear matrix inequality in Y: " # I Y 0.

Y> I Hence the problem can be written as an SDP: " # I Y > max trace Y A : 0, Y Y> I −λ ≤ Yij ≤ λ, 1 ≤ i ≤ j ≤ n.

3. The complementarity condition can be written as Xij∗ (λ − |Yij∗ |) = 0, 1 ≤ i, j ≤ n.

Hence if kY ∗ k∞ < λ then X ∗ = 0. Now the condition is in turn satisfied if ∀ Y, kY k ≤ 1 : kY k∞ < λ.

Thus, X ∗ = 0 if maxi,j φij < λ, where

φi,j = max Yij : kY k ≤ 1.

Y

Note that Yij = trace A> Y, where A = e j ei> , where ek denotes the k-th unit vector in Rn . Note that the singular values of A are all zero, except the largest, which is 1.

Applying the hint to part 1, we obtain φij = k Ak∗ = 1. This concludes the proof.

Exercise 11.4 (Boolean least-squares) Consider the following problem, known as Boolean Least Squares: φ = min k Ax − bk22 : xi ∈ {−1, 1}, i = 1, . . . , n. x

Here, the variable is x ∈ Rn , where A ∈ Rm,n and b ∈ Rm are given.

This is a basic problem arising, for instance, in digital communications. A brute force solution is to check all 2n possible values of x, which is usually impractical.

1. Show that the problem is equivalent to φ = min trace( A> AX ) − 2b> Ax + b> b X,x

s.t.:

X = xx > , Xii = 1,

i = 1, . . . , n,

in the variables X = X > ∈ Rn,n and x ∈ Rn .

115 2. The constraint X = xx > , i.e., the set of rank-1 matrices is not convex, therefore the problem is still hard. However, an efficient approximation can be obtained by relaxing this constraint to X xx > , as discussed in Section 11.3.3, obtaining φ ≥ φsdp = min trace( A> AX ) − 2b> Ax + b> b X " # X x s.t.: 0, x> 1 Xii = 1,

i = 1, . . . , n.

The relaxation produces a lower-bound to the original problem.

Once that is done, an approximate solution to the original problem can be obtained by rounding the solution: xsdp = sgn( x ∗ ), where x ∗ is the optimal solution of the semidefinite relaxation.

3. Another approximation method is to relax the non-convex constraints xi ∈ {−1, 1} to convex interval constraints −1 ≤ xi ≤ 1 for all i, which can be written k x k∞ ≤ 1. Therefore a different lower bound is given by: . φ ≥ φint = min k Ax − bk22 : k x k∞ ≤ 1.

Once that problem is solved, we can round the solution by xint = sgn ( x ∗ ) and compare the original objective value k Axint − bk22 .

4. Which one of φsdp and φint produces the closest approximation to φ? Justify carefully your answer.

5. Use now 100 independent realizations with normally distributed data, A ∈ R10,10 (independent entries with mean zero) and b ∈ R10 (independent entries with mean 1). Plot and compare the histograms of k Axsdp − bk22 of part 2, k Axint − bk22 of part 3, and the objective corresponding to a naïve method k Axls − bk22 , where

xls = sgn ( A> A)−1 A> b is the rounded ordinary Least Squares solution. Briefly discuss accuracy and computation time (in seconds) of the three methods.

6. Assume that, for some problem instance, the optimal solution ( x, X ) found via the SDP approximation is such that x belongs to the original non-convex constraint set { x : xi ∈ {−1, 1}, i = 1, . . . , n}. What can you say about the SDP approximation in that case?

Solution 11.4

116 1. Expanding the square and replacing Boolean constraints with xi2 = 1 we get φ = min x > A> Ax − 2b> Ax + b> b : xi2 = 1, i = 1, . . . , n. x

Using the property trace UV = trace VU and introducing a new variable X, we obtain the desired expression: φ = min trace( A> AX ) − 2b> Ax + b> b : X = xx > , Xii = 1, i = 1, . . . , n.

X,x

2. Since

"

X x>

x 1

#

0

we obtain in particular Xi,i ≥ xi2 ≥ 0, i = 1, . . . , n. Therefore φ I NT is a relaxation of φSDP . Hence, φSDP ≥ φ I NT .

3. A CVX code is below. for iter = 1:100 m = 10; n = 10; A = randn(m,n); b = 1*ones(m,1) + randn(m,1); cvx_begin SDP variable X(n,n) symmetric variable x(n) minimize(trace(A’*A*X)-2*b’*A*x+b’*b) subject to [X,x;x’,1]>=0; for i=1:n X(i,i) == 1; end cvx_end xs = sign(x); cvx_quiet(true); cvx_begin variable x(n) minimize(norm(A*x-b,2)) subject to norm(x,Inf)<=1; cvx_end xi = sign(x);

117

xn = sign(pinv(A)*b); sdp(iter) = norm(A*xs-b); naive(iter) = norm(A*xn-b); linf(iter) = norm(A*xi-b); end

As seen in Fig. 11.13, on average, SDP out-performs INT and Least Squares in estimation and approximation quality. The computation time is shortest for Least Squares, while INT and SDP are similar.

4. In that case, the optimal solution of the original problem is the same as the optimal solution of the relaxation.

Exercise 11.5 (Auto-regressive process model) We consider a process described by difference equation y(t + 2) = α1 (t)y(t + 1) + α2 (t)y(t) + α3 (t)u(t), t = 0, 1, 2, . . . where the u(t) ∈ R is the input, y(t) ∈ R the output, and the coefficient vector α(t) ∈ R3 is time-varying. We seek to compute bounds on the vector α(t) that are (a) independent of t, (b) consistent with some given historical data.

The specific problem we consider is: given the values of u(t) and y(t) over a time period 1 ≤ t ≤ T, find the smallest ellipsoid E in R3 such that, for every t, 1 ≤ t ≤ T, the equation above is satisfied for some α(t) ∈ E .

1. What is a geometrical interpretation of the problem, in the space of α’s?

2. Formulate the problem as a semidefinite program. You are free to choose the parametrization, as well as the measure of the size of E that you find most convenient.

3. Assume we restrict our search to spheres instead of ellipsoids.

Show that the problem can be reduced to a linear program.

4. In the previous setting, α(t) is allowed to vary with time arbitrarily fast, which may be unrealistic. Assume that impose a bound on the variation of α(t), such as kα(t + 1) − α(t)k2 ≤ β, where β > 0 is given. How would you solve the problem with this added restriction?

Figure 11.13: Histogram comparing the performance of algorithms to approximate Boolean Least Squares.

118 Solution 11.5 1. For each t, denote by L(t) the line in α-space: n o L(t) = α ∈ R3 : y(t + 2) = α1 y(t + 1) + α2 y(t) + α3 u(t) .

Geometrically, we are seeking the ellipsoid with minimal “size” that intersects every one of the above lines.

2. Let us parametrize the ellipsoid as

E = {c + Ru : kuk2 ≤ 1} , where c ∈ R3 and R ∈ R3×3 , R 6= 0. For given a ∈ R3 , a 6= 0, . b ∈ R, the ellipsoid above intersects the line L = {α : a> α = b} if and only if .

1 ≥ p∗ = min kuk2 : a> (c + Ru) = b. u

The above is a simple minimum-norm problem, with optimal value p∗ =

|b − a> c| . k R > a k2

The intersection condition reduces to a> RR> a ≥ (b − a> c)2 , Note that the condition can be written as a linear matrix inequality . in P = RR> and c: " # a> Pa b − a> c 0. b − a> c 1 Applying this result to our problem, and choosing the trace of P as measure of size (this corresponds to the sum of the squared semi-axis lengths of E ), we obtain the problem # " bt − a > a> t Pat t c min trace P : 0, t = 1, . . . , T − 2, P,c bt − a > 1 t c where, for t = 1, . . . , T − 2:   y ( t + 1)   a t =  y ( t )  , bt = y ( t + 2 ) . u(t) The above is an SDP.

119 3. When the ellipsoid must be a sphere, P is restricted to be of the form r2 I3 , with r ≥ 0. In that case, the problem reduces to min r : k at k2 r ≥ |bt − a> t c |, t = 1, . . . , T − 2, r,c

which is a linear program, as claimed.

4. The issue is to formulate the conditions, keeping the variables αt .

For every α, the condition α ∈ E , that is, α = c + Ru holds for some u, kuk2 ≤ 1, if and only if P (α − c)(α − c)> . Using this, we express the problem as min

P,c,(αt )tT=−11

trace P :

k"αt+1 − αt k2 ≤ β, a> t#αt = bt , P αt − c 0, t = 1, . . . , T − 2. (αt − c)> 1

Exercise 11.6 (Non-negativity of polynomials) A second-degree polynomial with values p( x ) = y0 + y1 x + y2 x2 is non-negative everywhere if and only if

∀x :

"

x 1

#> "

y0 y1 /2

y1 /2 y2

#"

x 1

#

≥ 0,

which in turn can be written as an LMI in y = (y0 , y1 , y2 ): " # y0 y1 /2 0. y1 /2 y2 In this exercise, you show a more general result, which applies to any polynomial of even degree 2k (polynomials of odd degree can’t be non-negative everywhere). To simplify, we only examine the case k = 2, that is, fourth-degree polynomials; the method employed here can be generalized to k > 2.

1. Show that a fourth-degree polynomial p is non-negative everywhere if and only if it is a sum of squares, that is, it can be written as 4

p ( x ) = ∑ q i ( x )2 , i =1

where qi ’s are polynomials of degree at most two. Hint: show that p is non-negative everywhere if and only if it is of the form

p( x ) = p0 ( x − a1 )2 + b12 ( x − a2 )2 + b22 ,

for some appropriate real numbers ai , bi , i = 1, 2, and some p0 ≥ 0.

120 2. Using the previous part, show that if a fourth-degree polynomial is a sum of squares, then it can be written as   1 h i   p( x ) = 1 x x2 Q  x  .

(11.25) 2 x for some positive-semidefinite matrix Q.

3. Show the converse: if a positive semi-definite matrix Q satisfies condition (11.25) for every x, then p is a sum of squares. Hint: use a factorization of Q of the form Q = AA> , for some appropriate matrix A. 4. Show that a fourth-degree polynomial p( x ) = y0 + y1 x + y2 x2 + y3 x3 + y4 x4 is non-negative everywhere if and only if there exist a 5 × 5 matrix Q such that Q 0, yl −1 =

∑ Qij , l = 1, . . . , 5.

i + j=l

Hint: equate the coefficients of the powers of x in the left and right sides of equation (11.25).

Solution 11.6 1. A fourth-degree polynomial that is everywhere non-negative must have no real root; since the polynomial has real coefficients, the roots must come in complex conjugate pairs. Hence the roots must be of the form a1 ± bi , i = 1, 2. We obtain the expression for p( x ) given in the hint; the leading coefficient must be non-negative, as seen from the case x → +∞. The result ensues by expanding the expression.

2. Since every polynomial qi is of degree 2 at most, it can be written as h i qi ( x ) = 1 x x 2 qi , for some appropriate vector qi ∈ R3 . We obtain 4

p( x )

= =

∑ q i ( x )2

i =1 4 h

∑( 1 x

x2

h

i

i =1

=

1

x

x2

i



q i )2

 1   Q x , x2

121 where

4

Q = ∑ qi qi> 0. i =1

3. Assume that Q satisfies (11.25), and Q 0. We can write Q = AA> for some 3 × 3 matrix A. Denote by qi , i = 1, 2, 3 the i-th column of Q: we have A = [q1 , q2 , q3 ], and 4

Q = AA> = ∑ qi qi> . i =1

This shows that p( x ) =

where qi ( x ) =

h

1

h

1

x

x2

x

x2

i

i



 1 4   Q  x  = ∑ q i ( x )2 , i =1 x2

qi , i = 1, 2, 3.

4. Let p( x ) = y0 + y1 x + y2 x2 + y3 x3 + y4 x4 . By identifying the coefficients of terms x l , l = 0, . . . , 4, in the condition   1 h i   ∀ x : p( x ) = 1 x x2 Q  x  , x2 we obtain

y l −1 =

∑ Qij , l = 1, . . . , 5.

i + j=l

The result follows.

Exercise 11.7 (Sum of top eigenvalues) For X ∈ Sn , and i ∈ {1, . . . , n}, we denote by λi ( X ) the i-th largest eigenvalue of X. For k ∈ {1, . . . , n}, we define the function f k : Sn → R with values k

f k ( X ) = ∑ λ i ( X ). i =1

This function is an intermediate between the largest eigenvalue (obtained with k = 1) and the trace (obtained with k = n).

1. Show that for every t ∈ R, we have f k ( X ) ≤ t if and only if there exist Z ∈ Sn and s ∈ R such that t − ks − trace( Z ) ≥ 0, Z 0, Z − X + sI 0.

Hint: for the sufficiency part, think about the interlacing property34 .

34

See Eq. (4.6)

122 2. Show that f k is convex. Is it a norm?

3. How would you generalize these results to the function that assigns the sum of top k singular values to a general rectangular m × n matrix, with k ≤ min(m, n)? Hint: for X ∈ Rm,n , consider the symmetric matrix " # 0 X .

X̃ = .

X> 0 Solution 11.7 1. First, the sufficiency part: assume that the stated conditions hold with some matrix Z ∈ Sn and scalar s. We then have X Z + sI. From the interlacing property of eigenvalues, this implies that λi ≤ λi ( Z + sI ), i = 1, . . . , n, where λi (·) is the i-th largest eigenvalue of its matrix argument.

Summing, we obtain k

k

i =1

i =1

f k ( X ) ≤ ∑ λi ( Z + sI ) = ks + ∑ λi ( Z ) ≤ ks + trace( Z ) ≤ t, as needed.

For the necessary part, assume that f k ( X ) ≤ t. Let U ∈ Rn,n be a unitary matrix such that U > XU = Λ is diagonal, with the eigenvalues ordered in decreasing fashion: λ1 ≥ . . . λn . Let Z = UYU > , with Y = diag (λ1 − s, . . . , λk−1 − s, 0, . . . , 0) , s = λk .

We have t − ks − trace(Y ) ≥ 0, Y 0, Y − Λ + sI 0.

Using congruence transformations35 , we obtain that the pair ( Z, s) satisfies the conditions.

2. The condition f k ( X ) ≤ t is a linear matrix inequality in (t, s, Z, X ); thus the epigraph, as the projection (onto the (t, X )-space) of a convex set, is itself convex. Thus, the function f k is convex. The function is also a norm: it satisfies the scaling property ( f k (αX ) = α f k ( X ) for every α ≥ 0). Since it is convex, it satisfies the triangle inquality. Indeed, for every X, Y ∈ Sn , 1 1 f k ( ( X + Y )) ≤ ( f k ( X ) + f k (Y )).

2 2 Using the scaling property, and multiplying by α = 2, we get the triangle inequality.

35

See Section 4.4.3.

123 3. For X ∈ Rm,n , let us examine the eigenvalues of the symmetric matrix X̃ given in the hint. A vector (u, v) ∈ Rm+n is an eigenvector of X̃ associated with the eigenvalue λ if and only if λu = Xv, λv = X > u, which implies that XX > u = λ2 u, X > Xv = λ2 v.

Thus, λ2 is an eigenvalue of XX > , and |λ| a singular value of X. Conversely, if σ is a singular value of X, then there exist u, v such that σu = Xv, σv = X > u.

Changing u in −u, we see that

−σ(−u) = Xv, −σv = X > (−u).

This means that the eigenvalues of the augmented matrix X̃ are ±σi ( X ), with σi the singular values of X. The sum of the top k singular values is thus the same as the sum of the top k eigenvalues of the augmented matrix. We obtain that the sum of the top k largest singular values of X is bounded above by t if and only if there exist Z ∈ Sm+n and s ∈ R such that t − ks − trace( Z ) ≥ 0, Z 0, Z − X̃ + sI 0.

124

12.

Introduction to Algorithms

Exercise 12.1 (Successive projections for linear inequalities) Consider a system of linear inequalities Ax ≤ b, with A ∈ Rm,n , where ai> , i = 1, . . . , m, denote the rows of A, which are assumed, without loss of generality, to be nonzero. Each inequality ai> x ≤ bi can be normalized by dividing both terms by k ai k2 , hence we shall further assume without loss of generality that k ai k2 = 1, i = 1, . . . , m.

Consider now the case when the polyhedron described by these . inequalities, P = { x : Ax ≤ b} is nonempty, that is, there exist at least a point x̄ ∈ P . In order to find a feasible point (i.e., a point in P ), we propose the following simple algorithm. Let k denote the iteration number and initialize the algorithm with any initial point xk = x0 at k = 0. If ai> xk ≤ bi holds for all i = 1, . . . , m, then we found the desired point, hence we return xk , and finish. If instead . there exist ik such that ai> xk > bik , then we set sk = ai> xk − bik , we k

k

update36 the current point as

x k +1 = x k − s k a i k , and we iterate the whole process.

1. Give a simple geometric interpretation of this algorithm.

2. Prove that this algorithm either finds a feasible solution in a finite number of iterations, or it produces a sequence of solutions { xk } that converges asymptotically (i.e., for k → ∞) to a feasible solution (if one exists).

3. The problem of finding a feasible solution for linear inequalities can be also put in relation with the minimization of the nonsmooth function f 0 ( x ) = maxi=1,...,m ( ai> xk − bi ). Develop a subgradienttype algorithm for this version of the problem, discuss hypotheses that need be assumed to guarantee convergence, and clarify the relations and similarities with the previous algorithm.

Solution 12.1 (Successive projections for linear inequalities) 1. The iterate xk − sk aik simply takes point xk (who violates the ik -th . constraint), and projects it back onto the hyperplane Hk = { x : ai> xk − bik = 0}. Indeed, sk = ai> xk − bik > 0 is the distance from k k xk to the hyperplane, and aik is the unit norm vector orthogonal to the hyperplane, hence xk+1 is the projection of xk onto the hyperplane. The algorithm then simply checks if the current point violates some constraint and, in the positive case, chooses any constraint that is violated and projects xk on it. A particular choice

This algorithm is a version of the socalled Agmon-Motzkin-Shoenberg relaxation method for linear inequalities, which dates back to 1953.

36

125 would be to select the constraint with the maximal violation (i.e., the index ik such that sk is maximal), in which case the algorithm becomes closely connected to the variation mentioned in point 3. of the exercise.

2. If at some finite iteration k the current point xk satisfies all inequalities, then the algorithm exits with a feasible solution. Otherwise, suppose the algorithm runs indefinitely, and let x̄ ∈ P (such a feasible point is supposed to exist). We have that

k xk+1 − x̄ k22

= k( xk − x̄ ) − sk aik k22

= k xk − x̄ k22 + s2k k aik k22 − 2sk ai>k ( xk − x̄ )

= k xk − x̄ k22 + s2k − 2sk ( ai>k xk − bik ) − ( ai>k x̄ − bik )

≤ k xk − x̄ k22 + s2k − 2sk ( ai>k xk − bik ),

where the last expression follows from the fact that, since x̄ is feasible by definition, it holds that ai> x̄ − bik ≤ 0. Finally, noting k . that sk = ai> xk − bik , from the last expression we obtain k

k xk+1 − x̄ k22 ≤ k xk − x̄ k22 − s2k , which shows that the distance of the current point from any fixed feasible point decreases at each iteration. Applying this inequality recursively from 0 onward, we obtain that k

k xk − x̄ k22 ≤ k x0 − x̄ k22 − ∑ s2k . i =0

Now, for k → ∞, the summation must tend to a constant (for otherwise the right-hand side of the above expression would become negative and unbounded below), and this implies that sk must tend to zero. Since sk represents the distance from xk to a violated constraint, we conclude that the algorithm will produce a sequence of solutions xk that tends to a feasible solution as k → ∞.

3. Let us assume that the finite minimum value p∗ of f 0 is attained at some point x ∗ (this is for instance guaranteed if the polyhedron P is assumed to be bounded). A subgradient of f 0 at xk is given by gk = aik , where ik is the value of index i that attains the maximum in maxi=1,...,m ( ai> xk − bi ), i.e., such that f 0 ( xk ) = ai> xk − bik . We can k thus minimize f 0 by applying the projected subgradient method described in Section 12.4.1 (only, there is nothing to project here, since the problem is unconstrained), which takes the form x k +1 = x k − s k a i k ,

126 where now sk must be a non-summable and diminishing sequence ∗ the lowest objective value (e.g., sk = 1/(k + 1)). Denoting with f 0,k ∗ → obtained up to iteration k of this algorithms, we have that f 0,k ∗ ∗ p . In practice, if at some k we find f 0,k ≤ 0, then we exit and ∗ return a feasible solution xk . If, otherwise, the sequence of f 0,k ∗ seemingly converges to a positive p , then we conclude that the problem is infeasible (P is empty).

Exercise 12.2 (Conditional gradient method) Consider a constrained minimization problem p∗ = min f 0 ( x ),

(12.26)

x ∈X

where f 0 is convex and smooth and X ⊆ Rn is convex and compact.

Clearly, a projected gradient or proximal gradient algorithm could be applied to this problem, if the projection onto X is easy to compute.

When this is not the case, the following alternative algorithm has been proposed37 .

Initialize the iterations with some x0 ∈ X , and set k = 0. Deter. mine the gradient gk = ∇ f 0 ( xk ) and solve

Versions of this algorithm are known as the Franke-Wolfe algorithm, which was developed in 1956 for quadratic f 0 , or as the LevitinPolyak conditional gradient algorithm (1966).

37

zk = arg min gkT x x ∈X

Then, update the current point as x k + 1 = ( 1 − γk ) x k + γk z k , where γk ∈ [0, 1], and, in particular, we choose γk =

2 , k+2

k = 0, 1, . . .

Assume that f 0 has a Lipschitz continuous gradient with Lipschitz constant38 L, and that k x − yk2 ≤ R for every x, y ∈ X . In this exercise, you shall prove that 2LR2 . δk = f 0 ( xk ) − p∗ ≤ , k = 1, 2, . . . k+2

38

As defined in Section 12.1.1.

39

See Lemma 12.1.

(12.27)

1. Using the inequality f 0 ( x ) − f 0 ( xk ) ≤ ∇ f 0 ( xk ) T ( x − xk ) +

L k x − xk k22 , 2

which holds for any convex f 0 with Lipschitz continuous gradient39 , prove that f 0 ( xk+1 ) ≤ f 0 ( xk ) + γk ∇ f 0 ( xk ) T (zk − xk ) + γk2

LR2 .

2

Hint: write the inequality condition above, for x = xk+1 .

127 2. Show that the following recursion holds for δk : δk+1 ≤ (1 − γk )δk + γk2 C,

k = 0, 1, . . . ,

2 . for C = LR 2 . Hint: use the optimality condition for zk , and the convexity inequality f 0 ( x ∗ ) ≥ f 0 ( xk ) + ∇ f 0 ( xk ) T ( x ∗ − xk ).

3. Prove by induction on k the desired result (12.27).

Solution 12.2 (Conditional gradient method) 1. Using inequality (12.5), we have that f 0 ( x k +1 )

≤ = ≤

≤

L kx − xk k22 2 k +1 L f 0 ( xk ) + γk ∇> f 0 ( xk )(zk − xk ) + γk kzk − xk k22 2 LR2 f 0 ( xk ) + γk ∇> f 0 ( xk )(zk − xk ) + γk2 2 ! since, ∀ x ∈ X, ∇> f 0 ( xk )zk ≤ ∇> f 0 ( xk ) x f 0 ( xk ) + ∇> f 0 ( xk )( xk+1 − xk ) +

f 0 ( xk ) + γk ∇> f 0 ( xk )( x ∗ − xk ) + γk2

LR2 , 2

which proves the first point in the exercise.

2. Continuing the previous chain of inequalities, and considering that, due to convexity inequality, it holds ∀y that f 0 (y) ≥ f 0 ( x ) + ∇> f 0 ( x )(y − x ), we have that f 0 ( x k +1 )

≤

f 0 ( xk ) + γk ( f 0 ( x ∗ ) − f 0 ( xk )) + γk2

LR2 .

2

Subtracting p∗ = f 0 ( x ∗ ) from both sides of this inequality, we obtain f 0 ( xk+1 ) − p∗ ≤ (1 − γk )( f 0 ( xk ) − p∗ ) + γk2

LR2 , 2

which we rewrite more compactly as δk+1 ≤ (1 − γk )δk + γk2 C,

k = 0, 1, . . . ,

(12.28)

2 . . where δk = ( f 0 ( xk ) − p∗ ), C = LR 2 .

3. We next prove40 that δk ≤

This approach was proposed by M. Jaggi, Ph.D. Thesis, ETH, Zürich, 2011.

40

4C , k+2

for k = 1, 2, . . . .

(12.29)

128 using induction on k. First, evaluating (12.28) for k = 0, and recalling that γk = k+2 2 , we have that δ1 ≤ C ≤

4 C, 3

which shows that (12.29) is true for k = 1. Assume next (12.29) is true for generic k, and evaluate δk+1 :

≤ (1 − γk )δk + γk2 C 4C + γk2 C (by the inductive hyp.) ≤ (1 − γk ) k+2

2 4C 4C 1 = 1− + k+2 k+2 k+2k+2

1 4C 1− = k+2 k+2 4C (k + 2) − 1 4C (k + 2) − 1 + 1 = ≤ k + 2 ( k + 2) k + 2 ( k + 2) + 1 4C k + 2 4C = = , k+2k+3 k+3 δk+1

which concludes the inductive proof.

Exercise 12.3 (Bisection method) The bisection method applies to onedimensional convex problems41 of the form

See an application tion 11.4.1.3.

41

in

Sec-

min f ( x ) : xl ≤ x ≤ xu x

where xl < xu are both finite, and f : R → R is convex. The algorithm is initialized with the upper and lower bounds on x: x = xl , x = xu , and the initial x is set as the midpoint x=

x+x .

2

Then, the algorithm updates the bounds as follows: a sub-gradient g of f at x is evaluated; if g < 0, we set x = x; otherwise42 , we set x = x. Then the midpoint x is recomputed, and the process is iterated until convergence.

1. Show that the bisection method locates a solution x ∗ within accuracy e in at most log2 ( xu − xl )/e − 1 steps.

2. Propose a variant of the bisection method for solving the unconstrained problem minx f ( x ), for convex f .

3. Write a code to solve the problem with the specific class of functions f : R → R, with values

n 1 2 f ( x ) = ∑ max A x + Bij x + Cij , 2 ij i =1 1≤ j ≤ m

Actually, if g = 0 then the algorithm may stop and return x as an optimal solution.

42

129 where A, B, C are given n × m matrices, with every element of A non-negative.

Solution 12.3 (A bisection method) 1. Let g( x ) be a subgradient of f at x. By the subgradient inequality it holds for all y that f (y) ≥ f ( x ) + g( x )(y − x ).

If g( x ) < 0, then all points y such that y < x are such that f (y) > f ( x ), thus it is useless to look for minimizers on the left of the current point, and the algorithm indeed sets x = x. If otherwise g( x ) > 0, then all points y such that y > x are such that f (y) > f ( x ), thus it is useless to look for minimizers on the right of the current point, and the algorithm indeed sets x = x.

Let `k denote the length of the search interval at iteration k of the algorithm, being `0 = xu − xl the initial length, let xk denote the candidate solution at iteration k, and let x ∗ be the optimal solution.

At each iteration k = 0, 1, . . ., it holds that | xk − x ∗ | ≤ 12 `k . Further, the length of the search interval is halved at each iteration, i.e., `k+1 = 21 `k , therefore, `k = 21k `0 , and

| xk − x ∗ | ≤

1 `0 .

2k +1

For given accuracy e > 0 it thus holds that

| xk − x ∗ | ≤ e whenever k ≥ log2

`0 − 1. e

2. Initialize the algorithm with some x ∈ R, and evaluate a subgradient g = g( x ). If g = 0 we terminate the algorithm, since x is optimal. If g < 0 we set xl = x and increase x ← 2sgn( x) x; if g > 0 we set xu = x and decrease x ← 2−sgn( x) x. We iterate until both xl and xu are set, or otherwise declare that f is unbounded below.

Once xl and xu are obtained, we switch to the standard bisection algorithm described in the previous point.

3. This function is the sum of maxima of convex functions (since Aij ≥ 0), hence it is convex. A subgradient g of f at x can be found using the max rule (see Section 8.2.3.1) as n

g = ∑ Aij∗ x + Bij∗ , i =1

130 where j∗ is any index attaining the maximum in max

1

1≤ j ≤ m 2

Aij x2 + Bij x + Cij .

The problem can then be solved via the approach described in point 2. of this exercise.

Exercise 12.4 (KKT conditions) Consider the optimization problem43 n

minx∈Rn ∑

i =1

2 1 2 di xi + ri xi

s.t.: a> x = 1, xi ∈ [−1, 1], i = 1, . . . , n, where a 6= 0 and d > 0.

1. Verify if strong duality holds for this problem, and write down the KKT optimality conditions.

2. Use the KKT conditions and/or the Lagrangian to come up with the fastest algorithm you can to solve this optimization problem.

3. Analyze the running time complexity of your algorithm. Does the empirical performance of your method agree with your analysis?

Solution 12.4 (KKT conditions) 1. Write the interval constraints on xi as xi2 ≤ 1, and define D = diag (d1 , . . . , dn ), Λ = diag (λ), λ = (λ1 , . . . , λn ), r = (r1 , . . . , rn ).

The problem is a standard convex quadratic problem of the form min x

1 > > 2 x Dx + r x

s.t.:

a> x = 1 xi2 ≤ 1

i = 1, . . . , n.

Strong duality holds for this problem, due to satisfaction of the Slater’s conditions (see Proposition 8.7). The Lagrangian of the problem is !

1 > 1 n > L( x, λ, µ) = x ( D + Λ) x + (r + µa) x − µ + ∑ λi 2 2 i =1 The KKT necessary and sufficient conditions for optimality of x ∗ and of λ∗ , µ∗ require primal feasibility, dual feasiblity (i.e., λ∗ ≥ 0), complementary slackness λi∗ (( xi∗ )2 − 1) = 0, and Lagrangian stationarity:

∇ x L( x, λ∗ , µ∗ ) = ( D + Λ∗ ) x + (r + µ∗ a) = 0.

43

Problem due to Suvrit Sra (2013).

131 That is, since D is strictly positive, x ∗ = −( D + Λ∗ )−1 (r + µ∗ a), i.e., since D and Λ are diagonal, r + µ ∗ ai xi∗ = − i , di + λi∗

i = 1, . . . , n.

(12.30)

Notice that the problem is essentially separable in the variables xi , except for the coupling constraint a> x = 1.

2. The dual function is g(λ, µ)

= inf L( x, λ, µ) x

=

1 − (r + µa)> ( D + Λ)−1 (r + µa) − 2

= −µ −

1 n µ + ∑ λi 2 i =1

!

1 n (ri + µai )2 + λi , 2 i∑ di + λi =1

and the dual problem amounts to maximising g(λ, µ) w.r.t. µ and λ ≥ 0. Let us find the maximum of g(λ, µ) w.r.t. λ, for fixed µ.

The problem is separable, and it amounts to finding max − λ i ≥0

(ri + µai )2 − λi . di + λi

The unconstrained optimal point is at λi = |ri + µai | − di . When this quantity is nonnegative, it is also the optimal solution of the constrained problem, otherwise the optimal solution is at the boundary, i.e., λi = 0. We thus have that ( |ri + µai | − di if |ri + µai | − di ≥ 0 (12.31) λi∗ (µ) = 0 otherwise.

Substituting (12.31) into (12.30), we can express xi∗ (µ) as a function of the scalar dual variable µ only: ( −sgn(ri + µai ) if |ri + µai | − di ≥ 0 i = 1, . . . , n. xi∗ (µ) = −(ri + µai )/di otherwise.

(12.32) The optimal µ is the value which makes the constraint ∑i ai xi∗ (µ) = 1 satisfied. We can find such value by bisection over µ. Since the gradient of g(λ∗ , µ) with respect to µ is simply given by a> xi∗ (µ) − 1 (see Section 8.5.9), we increase µ when the gradient is positive, and decrease it when it is negative. In practice, we initialize two values µl < 0, µr > 0 for which we know a priori that µ∗ ∈ [µl , µr ], and execute the following bisection algorithm44

44

See also Exercise 12.3.

132 (a) Set µ = (µr + µl )/2 (b) Evaluate h = a> x ∗ (µ) − 1

(c) If h > 0, let µl = µ, else let µr = µ (d) If |µr − µl | ≤ e, or h = 0, exit and return µ, else goto (a).

This algorithm returns a value of µ that is within e from the true optimum µ∗ in a number of iterations upper bounded by dlog2 (`/e) − 1e, where ` is the length of the initial localization interval.

3. The numerical simulations confirm precisely this iteration bound.

The following Matlab script implements the proposed bisection algorithm and compares it with the solution obtained via CVX. % Exercise on KKT conditions n=10; % some problem data d=rand(n,1); r=randn(n,1); a = randn(n,1); %% solve with CVX cvx_begin quiet Dsqrt=diag(sqrt(d)); variable xc(n) dx = Dsqrt*xc; minimize (0.5*(dx’*dx)+r’*xc) subject to a’*xc == 1; abs(xc) <= 1; cvx_end %% solve by bisection mu_l=-100; mu_r=100; ell = mu_r-mu_l; ep=1e-6; k=1; while 1 mu = (mu_l+mu_r)/2; y = r+mu*a; yind = find(abs(y)-d >= 0); x = -y./d; x(yind) = -sign(y(yind));

133 h=a’*x-1; if h>0 mu_l=mu; else mu_r=mu; end; if abs(mu_r-mu_l)<=ep | h==0 break; end; k=k+1; end;

Exercise 12.5 (Sparse Gaussian graphical models) We consider the following problem in a symmetric n × n matrix variable X max log det X − trace(SX ) − λk X k1 : X 0 X

where S 0 is an empirical covariance matrix, k X k1 denotes the sum of the absolute values of the elements of the positive definite matrix X, and λ > 0 encourages the sparsity in the solution X. The problem arises when fitting a multivariate Gaussian graphical model to data45 . The `1 -norm penalty encourages the random variables in the model to become conditionally independent.

1. Show that the dual of the problem takes the form min − log det(S + U ) : |Uij | ≤ λ U

2. We employ a block-coordinate descent method to solve the dual.

Show that if we optimize over one column and row of U at a time, we obtain a subproblem of the form min x > Qx : k x − x0 k∞ ≤ 1, x

where Q 0 and x0 ∈ Rn−1 are given. Make sure to provide the expression of Q, x0 as functions of the initial data, and the index of the row/column that is to be updated.

3. Show how you can solve the constrained QP problem above using following methods. Make sure to state precisely the algorithm’s steps.

• Coordinate descent.

• Dual coordinate ascent.

• Projected subgradient.

45

See Section 13.5.5.

134 • Projected subgradient method for the dual.

• Interior-point method (any flavor will do).

Compare the performance (e.g., theoretical complexity, running time/convergence time on synthetic data) of these methods.

4. Solve the problem (using block-coordinate descent with 5 updates of each row/column, each step requiring the solution of the QP above) for a data file of your choice. Experiment with different values of λ, report on the graphical model obtained.

Solution 12.5 (Sparse Gaussian graphical models) TBD Exercise 12.6 (Polynomial fitting with derivative bounds) In Section 13.2, we examined the problem of fitting a polynomial of degree d through m data points (ui , yi ) ∈ R2 , i = 1, . . . , m. Without loss of generality, we assume that the input satisfies |ui | ≤ 1, i = 1, . . . , m. We parametrize a polynomial of degree d via its coefficients: p w ( u ) = w0 + w1 u + . . . + w d u d , where w ∈ Rd+1 . The problem can be written as min kΦ> w − yk22 , w

where the matrix Φ has columns φi = (1, ui , . . . , uid ), i = 1, . . . , m. As detailed in Section 13.2.3, in practice it is desirable to encourage polynomials that are not too rapidly varying over the interval of interest.

To that end, we modify the above problem as follows: min kΦ> w − yk22 + λb(w), w

(12.33)

where λ > 0 is a regularization parameter, and b(w) is a bound on the size of the derivative of the polynomial over [−1, 1]: b(w) = max

u : |u|≤1

d pw (u) . du

1. Is the penalty function b convex? Is it a norm?

2. Explain how to compute a subgradient of b at a point w.

3. Use your result to code a subgradient method for solving problem (12.33).

Solution 12.6 (Polynomial fitting with derivative bounds)

135 1. We have that d pw (u) = w1 + 2w2 u + · · · + dwd ud−1 = w̃> C φ̃(u), du where C = diag (1, 2, . . . , d) ,

w̃> = [w1 · · · wd ],

φ̃(u)> = [1 u · · · ud−1 ].

Thus, d pw (u) = w̃> C φ̃(u) = max(w̃> C φ̃(u), −w̃> C φ̃(u)), du hence b(w) = b(w̃) = max

u:|u|≤1

d pw (u) = max max(w̃> C φ̃(u), −w̃> C φ̃(u)) du u:|u|≤1

is the max of linear functions of w, hence it is convex. b(w) is not a norm on Rd+1 , since it does not depend on w0 . However, it is indeed a norm on Rd , since it satisfied the three axioms defining a norm (see Section 2.2.1.2). Specifically, if w̃ = 0 ∈ Rd , then clearly b(w̃) = 0. Conversely, if b(w̃) = 0 then by definition max |w̃> C φ̃(u)| = 0,

|u|≤1

that is the polynomial in u, w̃> C φ̃(u) is zero for all u: |u| ≤ 1, which implies that this polynomial must be the zero polynomial, that is w̃ = 0. Further, it is easily verified that b(αw̃) = |α|b(w̃).

Finally, the triangle inequality holds, since for any w̃(1) ∈ Rd , w̃(2) ∈ Rd we have b(w̃(1) + w̃(2) )

= max |(w̃(1) + w̃(2) )> C φ̃(u)| |u|≤1

= max |w̃(1)> C φ̃(u) + w̃(1)> C φ̃(u)| |u|≤1

≤ max |w̃(1)> C φ̃(u)| + |w̃(1)> C φ̃(u)| |u|≤1

≤ max |w̃(1)> C φ̃(u)| + max |w̃(2)> C φ̃(v)| |u|≤1

= b(w

(1)

) + b(w

(2)

|v|≤1

).

2. Let us find a value u∗ of u that achieves the optimum in b(w̃) = max | p0w (u)| = max max( p0w (u), − p0w (u)), |u|≤1

|u|≤1

. where p0w (u) = w̃> C φ̃(u). To this end, consider the derivative of p0w (u) w.r.t. u p00w (u) = w̃> C φ̃0 (u) = 2w2 + 6w3 u + · · · + d(d − 1)wd ud−2 .

136 p00w (u) is a polynomial of degree at most d − 2. We define a set of points R containing all the real roots of p00w (u) = 0 lying in the interval [−1, 1], plus the extremes of this interval (thus, R contains at most d real numbers). The optimum we seek is achieved at one of the values in R, hence u∗ = arg min | p0w (u)|, u∈R

which is readily computed by evaluating | p0w (u)| at all the points in R. Then, applying the rule of max for the subgradients46 , we have that a subgradient for b at w is given by gb (w) = (0, sgn(w̃> C φ̃(u∗ ))C φ̃(u∗ )).

46

See Section 8.2.3.1.

47

See Section 12.4.1.

(12.34)

3. A subgradient for the objective function in (12.33) at w is given by g(w) = 2Φ(Φ> w − y) + λgb (w), where gb (w) is given by (12.34). A subgradient algorithm for solving problem (12.33) is thus specified as follows47 : initialize with k = 0 and some w(0) ∈ Rd+1 , and iterate until convergence

w

gk

=

g ( w(k) )

sk ( k +1)

=

γ/(k + 1)

=

w ( k ) − s k gk

k

← k + 1,

where γ > 0 is an algorithm parameter, e.g., γ = 0.1.

Exercise 12.7 (Methods for LASSO) Consider the LASSO problem, discussed in Section 9.6.2: min x

1 k Ax − yk22 + λk x k1 , 2

Compare the following algorithms. Try to write your code in a way that minimizes computational requirements; you may find the result in Exercise 9.4 useful.

1. A coordinate-descent method.

2. A sub-gradient method, as in Section 12.4.1.

3. A fast first-order algorithm, as in Section 12.3.4.

Solution 12.7 (Methods for LASSO) Exercise seems doable and OK.

137 Exercise 12.8 (Nonnegative terms that sum to one) Let xi , i = 1, . . . , n, be given real numbers, which we assume without loss of generality to be ordered as x1 ≤ x2 ≤ · · · ≤ xn , and consider the scalar equation in variable ν that we encountered in Section 12.3.3.3: f (ν) = 1,

. n where f (ν) = ∑ max( xi − ν, 0). i =1

1. Show that f is continuous and strictly decreasing for ν ≤ xn .

2. Show that a solution ν∗ to this equation exists, it is unique, and it must belong to the interval [ x1 − 1/n, xn ].

3. This scalar equation could be easily solved for ν using, e.g., the bisection method. Describe a simpler, “closed-form” method for finding the optimal ν.

Solution 12.8 (Nonnegative terms that sum to one) Let us denote with sk the sum of the k-th largest elements in x = ( x1 , . . . , xn ), that is, since x is ordered, sk = ∑in=n−k+1 xi , thus s1 = xn , sn = ∑in=1 xi .

1. If ν ≥ xn , then xi − ν ≤ 0 for all i, hence f (ν) = 0. If ν ≤ x1 , then xi − ν ≥ 0 for all i, hence f (ν) = sn − nν. Therefore, f is linear and decreasing at rate n for ν ≤ x1 , and it is identically zero for ν ≥ xn . Further, if ν ∈ ( xk , xk+1 ], k = 1, . . . , n − 1, then we readily see that f (ν) = sn−k − (n − k )ν, hence it is linear and decreasing with rate (n − k) in each of these intervals. The limits on the left and on the right at each break-point xk coincide (since sn−(k−1) − (n − (k − 1)) xk = sn−(k−1) − (n − k) xk + xk = sn−k − (n − k) xk ), thus f is piece-wise linear, continuous, and strictly decreasing for ν ≤ xn .

2. From the previous point we have that f ( x1 − 1/n) = sn − n( x1 − 1/n) = sn − nx1 + 1, thus

s 1 f ( x1 − 1/n) n = − x1 + . n n n Since the average of the points sn /n is no smaller than the minf ( x1 −1/n) ≥ 1/n and hence imum x1 of the points, we have that n f ( x1 − 1/n) ≥ 1. Since we have also that f ( xn ) = 0, and since f is continuous and strictly decreasing, we conclude that there exist a unique point ν in the interval [ x1 − 1/n, xn ] for which f (ν) = 1.

3. Evaluate the function at the following n points: . . . f 0 = f ( x1 − 1/n), f 1 = f ( x1 ), . . . , f n−1 = f ( xn−1 ).

138 We have f 0 = sn − nx1 + 1, and f k = sn−k − (n − k) xk , for k = 1, . . . , n − 1. Since f is decreasing, we have that f 0 > f 1 ≥ f 2 ≥ · · · ≥ f n−1 ≥ 0. Then, find the largest index k ∈ {0, . . . , n − 1} such that f k ≥ 1, and call it k∗ . The optimal ν is ν∗ =

sn−k∗ − 1 . n − k∗

Exercise 12.9 (Eliminating linear equality constraints) We consider a problem with linear equality constraints min f 0 ( x ) : Ax = b, x

where A ∈ Rm,n , with A full row rank: rank A = m ≤ n, and where we assume that the objective function f 0 is decomposable, that is n

f 0 ( x ) = ∑ h i ( x i ), i =1

with each hi a convex, twice differentiable function. This problem can be addressed via different approaches, as detailed in Section 12.2.6.

1. Use the constraint elimination approach of Section 12.2.6.1, and consider the function f˜0 defined in 12.33. Express the Hessian of f˜0 in terms of that of f 0 .

2. Compare the computational effort48 required to solve the problem using the Newton method via the constraint elimination technique, versus using the feasible update Newton method of Section 12.2.6.3, assuming that m n.

Solution 12.9 (Eliminating linear equality constraints) 1. Since f 0 is separable, we have that 

n

∂h1 ∂x1 ( x1 )

 ∂hi ( xi ) =   ∂xi i =1

 .. , .  ∂hn ∂xn ( xn )

∂2 h1

∂2 h n

∇ f0 (x) = ∑ and 2



∇ f 0 ( x ) = diag

∂x12

2

( x1 ), . . . ,

∂xn2

2

!

( xn ) .

Let N ∈ Rn,n−m contain by columns a basis for the nullspace of A, then all x satisfying the equality constraints are written as x = x̄ + Nz

48

See the related Exercise 7.4.

139 where x̄ ∈ Rn is some fixed solution of Ax = b, and z ∈ Rn−m is a new free variable. The minimization problem becomes now unconstrained in the variable z: min f˜0 (z), z

where f˜0 (z) = f 0 ( x̄ + Nz). The Hessian of f˜0 is obtained using the result in point 3. of Exercise 3.1, i.e.,

∇2 f˜0 (z) = N > ∇2 f 0 ( x (z)) N. We observe that the Hessian of f˜0 is in general a full symmetric matrix, while the Hessian of f 0 is diagonal.

2. The standard Newton method applied at f˜0 (z) is based on the basic iteration z k +1 = z k + s k h k , where hk solves the full symmetric system of dimension n − m

[∇2 f˜0 (zk )]hk = −∇ f˜0 (zk ).

The feasible update Newton method iterates instead directly on x x k +1 = x k + s k h k , where hk solves the augmented system of dimension m + n " #" # " # ∇2 f 0 ( x k ) A > hk −∇ f 0 ( xk ) = , A 0 λ 0 where ∇2 f 0 ( xk ) is diagonal. The solution of this latter system can be expressed as hk = −[∇2 f 0 ( xk )]−1 (∇ f 0 ( xk ) + A> λ), where λ = −( A[∇2 f 0 ( xk )]−1 A> )−1 A[∇2 f 0 ( xk )]−1 ∇ f 0 ( xk ) This approach can be computationally more convenient than the first one when m n, since it only requires inversion of the m × m matrix A[∇2 f 0 ( xk )]−1 A> (the Hessian is readily inverted, since it is diagonal).

140

13.

Learning from Data

Exercise 13.1 (SVD for text analysis) Assume you are given a data set in the form of a n × m term-by-document matrix X corresponding to a large collection of news articles. Precisely, the (i, j) entry in X is the frequency of the word i in the document j. We would like to visualize this data set on a two-dimensional plot. Explain how you would do to do the following (describe your steps carefully in terms of the SVD of an appropriately centered version of X).

1. Plot the different news sources as points in word space, with maximal variance of the points.

2. Plot the different words as points in news-source space, with maximal variance of the points.

Solution 13.1 1. Denote by d j the j-th column of A, which corresponds to particular news source. Thus A = [d1 , . . . , dn ]. The SVD of A can be written as r

A = ∑ σl ul v> l , l =1

where r is the rank of A (r ≤ min(m, n)), and the singular values are ordered, so that σ1 ≥ . . . ≥ σr .

Since the j-th news source is d j = Ae j ∈ Rm with e j the j-th unit vector in Rn , we have r

r

l =1

l =1

d j = Ae j = ∑ σl ul v> l e j = ∑ αl ul , . where αl = σl v> l e j , l = 1, . . . , r. Since the ul ’s form an orthonormal basis, the numbers αl are simply the components of d j along ul .

That is, αl = d> j ul for every l. We can use this for plotting any news source d ∈ Rm , even if it is not any one of the columns of A, using the numbers (d> u1 , d> u2 ).

The particular projection on (u1 , u2 ) is of maximal variance, in the sense that u1 is a direction that captures the largest variance, if we assume the data to be centered (d1 + . . . + dn = 0). Indeed, if we consider the columns of A to be data points then the associated covariance matrix is the m × m matrix: Σ=

1 n 1 d j d> AA> . ∑ j = n j =1 n

141 We can write nΣ = AA> = US2 U > . The direction of maximal variance is the solution to max

u : k u k2 ≤1

u> Σu,

a solution of which is u1 , the first column of U. Similarly u2 is the direction of maximal variance once we have projected the data on the hyperplane orthogonal to u2 .

2. This second question simply amounts to replace A with its transpose. In fact, if A = USV > is the SVD of A, then that of A> is A> = VS> U > , with S> still diagonal, so the roles of U, V are simply reversed.

To summarize: to plot a news source vector d ∈ Rm in word space, we simply project that vector on the first two columns of U, u1 , u2 , that is, we use the coordinates (d> u1 , d> u2 ). To plot a word vector w ∈ Rn in news source space, we use (w> v1 , w> v2 ).

Exercise 13.2 (Learning a factor model) We are given a data matrix X = [ x (1) , . . . , x (m) ], with x (i) ∈ Rn , i = 1, . . . , m. We assume that the data is centered: x (1) + . . . + x (m) = 0. An (empirical) estimate of the covariance matrix is49 Σ=

1 m (i) (i)> x x . m i∑ =1

In practice, one often finds that the above estimate of the covariance matrix is noisy. One way to remove noise is to approximate the covariance matrix as Σ ≈ λI + FF > , where F is a n × k matrix, containing the so-called “factor loadings,” with k n the number of factors, and λ ≥ 0 is the “idiosyncratic noise” variance. The stochastic model that corresponds to this is setup is x = F f + σe, where x is the (random) vector of centered observations, ( f , e) is a random variable with zero mean and unit covariance matrix, and σ = √ λ is the standard deviation of the idiosyncratic noise component σe.

The interpretation of the stochastic model is that the observations are a combination of a small number k of factors, plus a noise part that affects each dimension independently.

To fit F, λ to data, we seek to solve min kΣ − λI − FF > k F .

F,λ≥0

(13.35)

49

See Example 4.2.

142 1. Assume λ is known and less than λk (the k-th largest eigenvalue of the empirical covariance matrix Σ). Express an optimal F as a function of λ, which we denote F (λ). In other words: you are asked to solve for F, with fixed λ.

2. Show that the error E(λ) = kΣ − λI − F (λ) F (λ)> k F , with F (λ) the matrix you found in the previous part, can be written as p

∑ ( λ i − λ )2 .

E ( λ )2 =

i = k +1

Find a closed-form expression for the optimal λ that minimizes the error, and summarize your solution to the estimation problem (13.35).

3. Assume that we wish to estimate the risk (as measured by variance) involved in a specific direction in data space. Recall from Example 4.2 that, given a unit-norm n-vector w, the variance along direction w is w> Σw. Show that the rank-k approximation to Σ results in an under-estimate of the directional risk, as compared with using Σ. How about the approximation based on the factor model above? Discuss.

Solution 13.2 1. Let Σ = UΛU > be an eigenvalue decomposition of Σ, with U unitary, and Λ = diag (λ1 , . . . , λn ). The eigenvalue decomposition of Σ − λI is then Σ − λI = U (Λ − λI )U > . The best rank-k approximation to that matrix is then . k Σ̂k = ∑ (λi − λ)ui ui> . i =1

Since λ ≤ λk , we see that the above is positive semi-definite. It can be written as F (λ) F (λ)> , with h √ i √ F (λ) = λ1 − λu1 · · · λk − λuk .

2. From the previous part, it follows that E ( λ )2

= kΣ − λI − F (λ) F (λ)> k2F n

=

∑

i = k +1 p

=

2

(λi − λ)ui ui>

∑ ( λ i − λ )2 ,

i = k +1

F

143 as claimed. For the optimal λ, the first order conditions give us λ∗ =

p

1 λi . n − k i=∑ k +1

We do check that λ∗ ≤ λk ; hence λ∗ is indeed optimal for the problem (13.35).

The solution approach to that problem is: (a) Form the sample covariance matrix. (b) Set λ∗ to be the average of the n − k smallest eigenvalues. . (c) Find the corresponding factor loading matrix F∗ = F (λ∗ ).

3. For any direction w ∈ Rn , with Σ̂k the rank-k estimate: ! w> (Σ − Σ̂k )w = w>

n

∑ λi ui ui>

i = k +1

w ≥ 0.

Using the other estimate, denoting by (λ∗ , F∗ ) the optimal values found before: ! w> (Σ − (λ∗ I + F∗ F∗> ))w

= w>

n

∑ (λi − λ∗ )ui ui>

w

i = k +1

n

=

∑ (λi − λ∗ )(w> ui )2 .

i = k +1

The sign of the error depends on w. For example, with w = ek+1 , the error is non-negative, and the factor model under-estimates the risk along w; for w = un , the opposite is true.

Exercise 13.3 (Movement prediction for a time-series) We have a historical data set containing the values of a times series r (1), . . . , r ( T ).

Our goal is to predict if the time-series is going up or down. The basic idea is to use a prediction based on the sign of the output of an auto-regressive model that uses n past data values (here, n is fixed).

That is, the prediction at time t of the sign of the value r (t + 1) − r (t) is of the form ŷw,b (t) = sgn (w1 r (t) + . . . + wn r (t − n + 1) + b) , In the above, w ∈ Rn is our classifier coefficient, b is a bias term, and n T determines how far back into the past we use the data to make the prediction.

1. As a first attempt, we would like to solve the problem T −1

min ∑ (ŷw,b (t) − y(t))2 , w,b

t=n

144 where y(t) = sgn(r (t + 1) − r (t)). In other words, we are trying to match, in a least-squares sense, the prediction made by the classifier on the training set, with the observed truth. Can we solve the above with convex optimization? If not, why?

2. Explain how you would set up the problem and train a classifier using convex optimization. Make sure to define precisely the learning procedure, the variables in the resulting optimization problem, and how you would find the optimal variables to make a prediction.

Solution 13.3 1. The problem is not convex in (w, b).

2. We can use a support vector machine for example. For t ∈ {1, . . . , T − 1}, we set yt ∈ {−1, 1} to be the sign of y(t) = sgn(r (t + 1) − r (t)), and we define the “feature vector” at time t to be xt = (r (t), . . . , r (t − n + 1)), t = n, . . . , T − 1.

We try to find w, b so that, on the available data, the sign predictions match. If that is so, then w, b must satisfy y(t)(w> xt + b) ≥ 0, t = n, . . . , T − 1.

The next steps are entirely similar to those taken in Section 13.3.1.

Once an optimal pair (w, b) is found, our prediction at time T is ŷ( T ) = sgn (w1 r (t) + . . . + wn r (t − n + 1) + b) .

Exercise 13.4 (A variant of PCA) Return to the variant of PCA examined in Exercise 11.2. Using a (possibly synthetic) data set of your choice, compare the classical PCA and the variant examined here, especially in terms of its sensitivity to outliers. Make sure to establish an evaluation protocol that is as rigorous as possible. Discuss your results.

Solution 13.4 To be completed later. This exercise should be correct.

Exercise 13.5 (Squared vs. non-squared penalties) Consider the problems .

P(λ) : p(λ) = min f ( x ) + λk x k, x 1 .

Q(µ) : q(µ) = min f ( x ) + µk x k2 , x 2

145 where f is a convex function, k · k is an arbitrary vector norm, and λ > 0, µ > 0 are parameters. Assume that for every choice of these parameters, the corresponding problems have a unique solution.

In general, the solutions for the above problems for fixed λ and µ do not coincide. This exercise shows that we can scan the solutions to the first problem, and get the set of solutions to the second, and vice-versa.

1. Show that both p, q are concave functions, and q̃ with values q̃(µ) = q(1/µ) is convex, on the domain R+ .

2. Show that p(λ) = min q(µ) + µ >0

λ2 λ2 , q(µ) = max p(λ) − , 2µ 2µ λ >0

For the second expression, you may assume that dom f has nonempty interior.

3. Deduce from the first part that the path of solutions coincide. That is, if we solve the first problem for every λ > 0, for any µ > 0 the optimal point we thus find will be optimal for the second problem; and vice-versa. It will convenient to denote by x ∗ (λ) (resp. z∗ (µ)) the (unique) solution to P(λ) (resp. Q(µ)).

4. State and prove a similar result concerning a third function . r (κ ) : r (κ ) = min f ( x ) : k x k ≤ κ. x

5. What can you say if we remove the uniqueness assumption?

Solution 13.5 1. p, q are concave, since they are point-wise minima of affine functions of their argument. We have 1 . q̃(µ) = min g( x, µ) where g( x, µ) = f ( x ) + k x k2 . x 2µ The function g is jointly convex with respect to ( x, µ) on the convex domain µ > 0, x ∈ Rn . Indeed, thanks to the Schur complement Lemma, the condition g( x, µ) ≤ t for µ > 0 can be written as a linear matrix inequality in ( x, µ, t): 2µ x

x> tI

!

0.

146 2. The first expression comes from the fact that, for every γ, λ ∈ R+ : µγ2 λ2 + = γλ, 2 2µ µ >0

min

with µ = λ/γ at optimum. Hence: min q(µ) + µ >0

λ2 2µ

= = =

1 λ2 min f ( x ) + µk x k2 + 2 2µ x,µ>0 min f ( x ) + λk x k x

p ( λ ).

At optimum, we have µk x k = λ.

For the second expression, we use the fact that for every γ ≥ 0, µ > 0: λ2 µ max λγ − = γ2 , 2µ 2 λ ≥0 with λ = γµ at optimum. We have then max p(λ) − λ ≥0

λ2 2µ

= max min f ( x ) + λk x k −

λ2 2µ

= min max f ( x ) + λk x k −

λ2 2µ

x

λ ≥0 x

λ ≥0

1 = min f ( x ) + µk x k2 x 2 = q ( µ ).

(13.36)

In the third line, we use strong duality. This is allowed if dom f has non-empty interior. Again, at optimum we have µk x k = λ.

3. We first prove that for any µ > 0, there exists a value of λ such that the (unique) solution to P(λ), x ? (λ), is optimal for Q(µ). We now show that one such value is given by any maximizer λ? (µ) for the problem λ2 max p(λ) − .

2µ λ ≥0

Indeed, defining y? = x ? (λ? (µ)), at optimum of the above problem we have µ k y ? k = λ ? ( µ ).

Thus: q(µ)

λ ? ( µ )2 2µ

=

p(λ? (µ)) −

=

f (y? ) + λ? (µ)ky? k −

=

f (y? ) +

µ ? 2 ky k .

2

λ ? ( µ )2 2µ

147 This shows that y? is optimal for Q(µ).

Likewise, we now show that any λ ≥ 0, there exists a value of µ such that the (unique) solution to Q(µ), z? (µ), is optimal for Q(µ). We now show that one such value is given by any maximizer µ? (λ) > 0 for the problem min q(µ) + µ >0

λ2 .

2µ

Indeed, defining y? = z? (µ? (λ)), at optimum of the above problem we have µ? (λ)ky? k = λ.

Thus: p(λ)

= q(µ? (λ)) + = =

λ2 2µ? (λ)

λ2 µ? (λ) ? 2 ky k − ?

2 2µ (λ) f ( y ? ) + λ k y ? k.

f (y? ) +

This shows that y? is optimal for P(λ).

4. Similar statements can be made, as strong duality holds. Precisely, we have for κ > 0 r (κ )

= min max f ( x ) + λ(k x k − κ ) x

λ ≥0

= max min f ( x ) + λ(k x k − κ ) λ ≥0

x

= max p(λ) − λκ. λ ≥0

Here, strong duality can be invoked, since the problem R(κ ) is strictly feasible. Likewise, since r (κ ) = min f ( x ) : x

1 1 k x k2 ≤ κ 2 , 2 2

applying strong duality we obtain r (κ )

µ (k x k2 − κ 2 ) x 2 µ ≥0 µ max min f ( x ) + (k x k2 − κ 2 ) x 2 µ ≥0

= min max f ( x ) + =

= max q(µ) − µ ≥0

µκ 2 .

2

5. If we remove the uniqueness assumption, a similar statement can be made. The solution to Q(µ) is unique for any µ > 0, but that

148 of P(λ) may not be. However, we can show that for any λ > 0, there exist a µ > 0 such that one of the optimal points for P(λ) is optimal for Q(µ).

Exercise 13.6 (Cardinality-penalized least-squares) We consider the problem . φ(k) = min k X > w − yk22 + ρ2 kwk22 + λ card(w), w

where X ∈ Rn,m , y ∈ Rm , ρ > 0 is a regularization parameter, and λ ≥ 0 allows to control the cardinality (number of non-zeros) in the solution. This in turn allows better interpretability of the results. The above problem is hard to solve in general. In the sequel, we denote by ai> , i = 1, . . . , n the i-th row of X, which corresponds to a particular “feature” (that is, dimension of the variable w).

1. First assume that no cardinality penalty is present, that is, λ = 0.

Show that ! −1 1 n > > y. φ (0) = y I + 2 ∑ ai ai ρ i =1 2. Now consider the case λ > 0. Show that φ(λ) = min

u∈{0,1}n

y

>

1 n Im + 2 ∑ ui ai ai> ρ i =1

! −1

n

y + λ ∑ ui . i =1

3. A natural relaxation to the problem obtains upon replacing the constraints u ∈ {0, 1}n with interval ones: u ∈ [0, 1]n . Show that the resulting lower bound φ(λ) ≥ φ(λ) is the optimal value of the convex problem

( ai> v)2 − λ)+ . ρ2 i =1 n

φ(λ)

= max 2y> v − v T v − ∑ ( v

How would you recover a sub-optimal sparsity pattern from a solution v∗ to the above problem?

4. Express the above problem as an SOCP. 5. Form a dual to the SOCP, and show that it can be reduced to the expression

n ρxi > 2 φ(λ) = k X w − yk2 + 2λ ∑ B √ , λ i =1 where B is the (convex) reverse Hüber function: for ξ ∈ R, 

 |ξ | if |ξ | ≤ 1 2 ξ . 1 2+1 B(ξ ) = min z + = ξ  2 0≤ z ≤1 z otherwise.

2

149 Again, how would you recover a sub-optimal sparsity pattern from a solution w∗ to the above problem?

6. A classical way to handle cardinality penalties is to replace them with the `1 -norm. How does the above approach compare with the `1 -norm relaxation one? Discuss.

Solution 13.6 1. We first find an expression for the regularized LS problem, which corresponds to λ = 0: φ (0)

= min k X > w − bk22 + ρ2 kwk22 x

= max τ : ∀ w, τ ≤ k X > w − bk22 + ρ2 w> w. τ

The constraint holds for every w if and only if w −1

∀w :

!>

XX > + ρ2 In ( Xy)>

Xy > y y−τ

!

w −1

!

≥ 0.

In turn the above is equivalent to XX > + ρ2 In y> X >

Xy > y y−τ

!

0.

Using Schur complements, and a matrix inversion formula, we obtain 1 φ(0) = y> y − y> X > ( XX > + ρ2 In )−1 Xy = y> ( 2 X > X + Im )−1 y. ρ 2. We have φ(λ) = min k X > D (u)z − yk22 + ρ2 z> D (u)z + λ1> u : u ∈ {0, 1}n , u,z

where D (u) = diag (u), and using D (u)2 = D (u). In fact, the problem above is equivalent to min k X > D (u)z − yk22 + ρ2 z> z + λ1> u : u ∈ {0, 1}n , u,x

since the components of z corresponding to the zero components of u do not appear in the first term in the objective, and are driven down to zero due to the second term. We can use the expression we found above for the regularized LS problem, in order to eliminate z. This simply entails replacing X with D (u) X in the expression of φ(0) above. We obtain the Boolean formulation: φ(λ) = min

u∈{0,1}n

λ1> u + y> A(u)−1 y,

150 where

A(u) =

1 1 n 1 > X D (u)2 X + Im = 2 X > D (u) X + Im = Im + 2 ∑ ui ai ai> , 2 ρ ρ ρ i =1

with ai> ∈ Rm is the i-th row of X. Alternatively the problem reads φ(λ) = min

u∈{0,1}n

max λ1> u + 2y> v − v> A(u)v, v

with v = A(u)−1 y at optimum.

3. A natural relaxation involves replacing the Boolean constraints with interval ones. This amounts to exchange the min and the max in the above expression. We obtain the convex lower bound φ(λ) ≥ φ(λ), where φ(λ)

= max v

min

u∈{0,1}n

>

>

λ1 u + 2y v − v

1 n Im + 2 ∑ ui ai ai> ρ i =1

>

( ai> v)2 − λ)+ . ρ2 i =1 n

!

v

= max 2y> v − v> v − ∑ ( v

A sub-optimal sparsity pattern can be inferred from the solution to the above problem, setting ui = 0 if λρ2 ≥ ( ai> v)2 , and ui = 1 otherwise.

4. We can express the problem as an SOCP with rotated secondorder cone constraints. Using the fact that, for every vector z and scalars r, t: !

2z > ≤ r + t, z z ≤ rt, r ≥ 0, t ≥ 0 ⇐⇒ r−t 2

we obtain φ(λ)

= max 2y> v − v> v − p> 1 : p ≥ 0, pi ≥ v,p

1 > 2 ( a v) − λ, i = 1, . . . , n ρ2 i !

2ρ−1 ai> v pi + λ − 1

= max 2y> v − v> v − p> 1 : p ≥ 0, v,p

2

≤ pi + λ + 1, i = 1, . . . , n.

5. We proceed with the dual to the above SOCP: φ(λ)

= =

n max min 2y> v − v> v − p> 1 + ∑ γi ( pi + λ − 1) − 2ρ−1 αi ai> v − β i ( pi + λ + 1)

v,p≥0

α,β,γ

α,β,γ

v,p≥0

i =1

s.t. k(αi , β i )k2 ≤ γi , i = 1, . . . , n min max 2(y − ρ−1 X > α)> v − v> v + p> ((γ − β − 1) + 1> (λ(γ − β) − (γ + β)) s.t.

k(αi , β i )k2 ≤ γi , i = 1, . . . , n.

151 . .

Setting w = ρ−1 α, s = γ + β, z = γ − β, and eliminating v = b − X > w, we obtain φ(λ)

= =

=

min max k X > w − yk22 + p> (z − 1) + 1> (λz + s) w,s,z

p ≥0

s.t. min

k(2ρwi , si − zi )k2 ≤ si + zi , i = 1, . . . , n k X > w − bk22 + 1> (λz + s)

s.t.

0 ≤ zi ≤ 1, si ≥ 0, ρ2 xi2 ≤ si zi ,! i = 1, . . . , n n ρ2 wi2 : 0 ≤ z ≤ 1. k X > w − bk22 + ∑ λzi + zi i =1

w,s,z

min w,z

We obtain that our relaxation is in the form φ(λ) = k X

>

w − bk22 + 2λ

n

ρx ∑ B( √λi ). i =1

exact berhu l1 6

5

4

3

2λ

2

λ

We can extract a sparsity pattern from the above, setting v = y − X > w, and ui = 0 if λρ2 ≥ ( ai> v)2 , and ui = 1 otherwise.

1

0 −4

√ −3

−2

−1

0

1

2

λ/ρ 3

4

We observe that we have the following bound:

√ ρξ ρ2 ξ 2 + λ1(ξ ) ≥ 2λB( √ ) ≥ 2ρ λ|ξ |, λ where 1(·) is zero when its argument is zero, and 1 otherwise.

This is illustrated in Fig. 13.14.

Hence

√ . φ(λ) ≥ φ(λ) ≥ ψ(λ) = min k X > w − bk22 + 2ρ λk x k1 , x

which is the classical approximation. Note that the penalty in the classical approximation is not the same as that on the l0 norm.

Our approach provides a natural guideline for how to threshold the result when solving for the relaxation, namely that we should √ zero out elements of x with magnitude less than λ/ρ.

Figure 13.14: The relaxation consists in replacing the non-convex penalty (red) with its convex envelope (blue), which corresponds to the reverse Huber function. The classical `1 penalty (green) is a lower bound only if we choose the penalty parameter to be √ 2ρ λ.

152

14.

Computational Finance

Exercise 14.1 (Diversification) You have $12,000 to invest at the beginning of the year, and three different funds from which to choose.

The municipal bond fund has a 7% yearly return, the local bank’s Certificates of Deposit (CDs) have an 8% return, and a high-risk account has an expected (hoped-for) 12% return. To minimize risk, you decide not to invest any more than $2,000 in the high-risk account.

For tax reasons, you need to invest at least three times as much in the municipal bonds as in the bank CDs. Denote by x, y, z be the amounts (in thousands) invested in bonds, CDs, and high-risk account, respectively. Assuming the year-end yields are as expected, what are the optimal investment amounts for each fund?

I took this out, too simplistic Exercise 14.2 (Portfolio optimization problems) We consider a single-period optimization problem involving n assets, and a decision vector x ∈ Rn which contains our position in each asset. Determine which of the following objectives or constraints can be modeled using convex optimization.

1. The level of risk (measured by portfolio variance) is equal to a given target t (the covariance matrix is assumed to be known).

2. The level of risk (measured by portfolio variance) is below a given target t.

3. The Sharpe ratio (defined as the ratio of portfolio return to portfolio standard deviation) is above a target t ≥ 0. Here both the expected return vector and the covariance matrix are assumed to be known.

4. Assuming that the return vector follows a known Gaussian distribution, ensure that the probability of the portfolio return being less than a target t is less than 3%.

5. Assume that the return vector r ∈ Rn can take three values r (i) , i = 1, 2, 3. Enforce the following constraint: the smallest portfolio return under the three scenarios is above a target level t.

6. Under similar assumptions as in part 5: the average of the smallest two portfolio returns is above a target level t. Hint: use new variables si = x > r (i) , i = 1, 2, 3, and consider the function s → s[2] + s[3] , where for k = 1, 2, 3, s[k] denotes the k-th largest element in s.

153 7. The transaction cost (under a linear transaction cost model, and with initial position xinit = 0) is below a certain target.

8. The number of transactions from the initial position xinit = 0 to the optimal position x is below a certain target.

9. The absolute value of the difference between the expected portfolio return and a target return t is less than a given small number e (here, the expected return vector r̂ is assumed to be known).

10. The expected portfolio return is either above a certain value tup , or below another value tlow .

Solution 14.1 1. The constraint reads x > Cx = t. This is not a convex constraint, since it is an equality involving a non-linear function.

2. The constraint reads x > Cx ≤ t. This is a convex constraint, since C is positive semi-definite, hence the function f ( x ) = x > Cx − t is convex. √ 3. The Sharpe ratio is r̂ > x/ x > Cx. The constraint writes p r̂ > x ≥ t x > Cx.

This is a second-order cone constraint. Indeed, since C is positive semi-definite, there exist R ∈ Rn,n such that C = R> R. Then the constraint above can be written as: the Euclidean norm of a vector affine in x is less than some affine function of x: r̂ > x ≥ tk Rx k2 .

Since t ≥ 0, this constraint is convex.

4. As seen in Prop. 10.1, the constraint can be expressed as p r̂ > x ≥ κ x > Cx

where κ > 0 is a function of e. Similar to the previous part, this constraint is a (convex) second-order cone constraint.

5. This is convex, as it can be represented as three affine constraint x > r (i) ≥ t, i = 1, 2, 3.

6. Here we introduce new variables si , i = 1, 2, 3 and add the affine equality constraints si = x > r (i) , i = 1, 2, 3. It remains to examine if the constraint on s ∈ R3 1 ( s + s [3] ) ≥ t 2 [2]

154 is convex, where for k = 1, 2, 3, s[k] denotes the k-th largest element in s. The above constraint is indeed convex, since the function on the LHS is concave. To see this, we write s[2] + s[3] = min(s1 + s2 , s2 + s3 , s3 + s1 ).

Hence the function s → s[2] + s[3] is the point-wise minimum of linear functions, thus it is concave.

7. This is the convex constraint k x k1 ≤ t.

8. This is a non-convex constraint.

9. This is the convex constraint |r̂ > x − t| ≤ e.

10. Here we introduce new variable s, and add the linear equality constraint s = r̂ > x. The constraint can be written as the quadratic constraint on s, x

(tup − s)(s − tlow ) ≤ 0, s = r̂ > x.

To check that this is not convex, we observe that the first constraint can be written as the non-convex constraint q(s) ≥ 0, where q(s) = s2 + (a linear function of s). (In x-space, the constraint above represents the outside of a slab.)

Exercise 14.3 (Median risk) We consider a single-period portfolio optimization problem with n assets. We use past samples, consisting of single-period return vectors r1 , . . . , r N , where rt ∈ Rn contains the returns of the assets from period t − 1 to period t. We denote with . r̂ = (1/N )(r1 + . . . + r N ) the vector of sample averages; it is an estimate of the expected return, based on the past samples.

As a measure of risk, we use the following quantity. Denote by ρt ( x ) the return at time t (if we had held the position x at that time).

Our risk measure is . 1 N R1 ( x ) = |ρt ( x ) − ρ̂( x )|, N t∑ =1 where ρ̂( x ) is the portfolio’s sample average return.

1. Show that R1 ( x ) = k R> x k1 , with R a n × N matrix that you will determine. Is the risk measure R1 convex?

2. Show how to minimize the risk measure R1 , subject to the condition that the sample average of the portfolio return is greater than a target µ, using linear programming. Make sure to put the problem in standard form, and define precisely the variables and constraints.

155 3. Comment on the qualitative difference between the resulting portfolio, and one that would use the more classical, variance-based risk measure, given by . 1 N R2 ( x ) = (ρt ( x ) − ρ̂( x ))2 .

N t∑ =1 Solution 14.2 (Median risk) 1. We have ρt ( x ) = rt> x, t = 1, . . . , N, ρ̂( x ) = r̂ > x.

Hence,

R1 ( x ) =

1 N |(rt − r̂ )> x | = k R> x k1 , N t∑ =1

where R is the n × N matrix with t-th colum equal to (1/N )(rt − r̂ ), t = 1, . . . , N. The risk measure is a convex function of x, as the composition of the `1 -norm with an affine transformation x → R> x.

2. The problem writes min k R> x k1 : r̂ > x ≥ µ. x

We can put the above problem in standard LP format: N

min ∑ zt : x,z

t =1

− Nzt ≤ (rt − r̂ )> x ≤ Nz − t, t = 1, . . . , N, r̂ > x ≥ µ.

3. In practice the resulting portfolio would have most of the deviations ρt ( x ) − ρ̂( x ) very small in magnitude, and tolerate a few that are potentially very large. In constrast, the variance-based measure would tend to produce a deviation vector that is more evenly distributed.

Exercise 14.4 (Portfolio optimization with factor models – 1) 1. Consider the following portfolio optimization problem p∗ = min x

s.t.:

x > Σx r̂ > x ≥ µ,

where r̂ ∈ Rn is the expected return vector, Σ ∈ Sn , Σ 0 is the return covariance matrix, and µ is a target level of expected

156 portfolio return. Assume that the random return vector r follows a simplified factor model of the form r = F ( f + fˆ),

. r̂ = F fˆ,

where F ∈ Rn,k , k n, is a factor loading matrix, fˆ ∈ Rk is given, and f ∈ Rk is such that E{ f } = 0 and E{ f f > } = I. The above optimization problem is a convex quadratic problem that involves n decision variables. Explain how to cast this problem into an equivalent form that involves only k decision variables. Interpret the reduced problem geometrically. Find a closed-form solution to the problem.

2. Consider the following variation on the previous problem p∗ = min x

s.t.:

x > Σx − γr̂ > x x ≥ 0,

where γ > 0 is a tradeoff parameters that weigths the relevance in the objective of the risk term and of the return term. Due to the presence of the constraint x ≥ 0, this problem does not admit, in general, a closed-form solution.

Assume that r is specified according to a factor model of the form r = F ( f + fˆ) + e where F, f and fˆ are as in the previous point, and e is an idiosyncratic noise term, which is uncorrelated with f (i.e., E{ f e> } = 0) . and such that E{e} = 0 and E{ee> } = D2 = {d21 , . . . , d2n } 0.

Suppose we wish to solve the problem using a logarithmic barrier method of the type discussed in Section 12.3.1. Explain how to exploit the factor structure of the returns to improve the numerical performance of the algorithm. Hint: with the addition of suitable slack variables, the Hessian of the objective (plus barrier) can be made diagonal.

Solution 14.3 (Portfolio optimization with factor model – 1) 1. Since r = r̂ + F f , and E{ f } = 0, E{ f > f } = I, we have that Σ = var{r } = FF > . Thus the original optimization problem is p∗ = min

x > FF > x

x

s.t.: fˆ> F > x ≥ µ.

By defining a new variable z = F > x, this problem reduces to p∗ = min z

z> z

s.t.: fˆ> z ≥ µ.

157 This problem has k n variables. Moreover, the problem has the geometric interpretation of finding the projection of the origin onto the halfspace { fˆ> z ≥ µ}, which has the unique optimal solution µ ˆ z∗ = f. k fˆk22 All optimal solutions to the original problem solve F > x = z∗ , and are thus given by x ∗ = ( F > )† z∗ + v,

v ∈ N ( F > ).

If F is full rank, then we may write ( F > )† = F ( F > F )−1 , and x∗ =

µ F ( F > F )−1 fˆ + v, k fˆk22

v ∈ N ( F > ).

2. For the proposed factor model, we have that Σ = var{r } = D2 + FF > , and the problem objective becomes f 0 ( x ) = x > D2 x + x > FF > x − γr̂ > x.

The composite objective in a barrier method for this problem would be of the form t f 0 ( x ) + φ ( x ),

φ( x ) = − ∑ ln xi , i

where the Hessian of φ is ∇2 φ( x ) = diag x1−2 , . . . , xn−2 . However, the Hessian of f 0 is Σ, which is not diagonal, in general. .

Let us introduce slack variables z = F > x. The problem now becomes p∗ = min x,z

s.t.:

x > D2 x + z> z − γr̂ > x x ≥ 0,

F > x = z.

We can solve this problem again via a barrier method, using feasible updates50 in the Newton iterations to deal with the equality constraint F > x = z. The advantage is that the Hessian of the objective function is now

∇2 f 0 ( x, z) = diag D2 , I ,

50

See Section 12.2.6.3.

158 which is diagonal, thus readily invertible. The Newton step ∆ is then found by solving the KKT system of the form " #" # " # H F̃ ∆ −g = , F̃ > 0 η 0 where H and ξ are, respectively, the Hessian (which is diagonal) and the gradient of the composite objective function at ( x, z), and .

F̃ > = [ F > − I ] ∈ Rk,n+k . From this system, we obtain η by solving

( F̃ > H −1 F̃ )η = − F̃ > H −1 ξ, where the matrix on the right-hand side has dimension k × k (hence, much smaller than n × n), and then obtain ∆ as ∆ = − H −1 (ξ + F̃η ).

Exercise 14.5 (Portfolio optimization with factor models – 2) Consider again the problem and setup of in point 2 of Exercise 14.4. Let . z = F > x, and verify that the probem can be rewritten as p∗ = min

x ≥0,z

s.t.:

x > D2 x + z> z − γr̂ > x F > x = z.

Consider the Lagrangian

L( x, z, λ) = x > D2 x + z> z − γr̂ > x + λ> (z − F > x ) and the dual function . g(λ) = min L( x, z, λ). x ≥0,z

Strong duality holds, since the primal problem is convex and strictly feasible, thus p∗ = d∗ = maxλ g(λ).

1. Find a closed-form expression for the dual function g(λ).

2. Express the primal optimal solution x ∗ in terms of the dual optimal variable λ∗ .

3. Determine a subgradient of − g(λ).

Solution 14.4 (Portfolio optimization with factor model – 2) 1. The Lagrangian is strongly convex in ( x, z), and it is decomposable as

k n L( x, z, λ) = ∑ z2i + λi zi + ∑ d2i xi2 − (γr̂i + [ Fλ]i ) xi , i =1

i =1

159 thus g(λ)

= =

min L( x, z, λ)

x ≥0,z

k n min ∑ z2i + λi zi + ∑ d2i xi2 − (γr̂i + [ Fλ]i ) xi

x ≥0,z

i =1

i =1

n = min ∑ z2i + λi zi + min ∑ d2i xi2 − (γr̂i + [ Fλ]i ) xi k

z

i =1

k

=

∑ min zi

i =1

x ≥0

i =1

n

z2i + λi zi + ∑ min i =1

x i ≥0

d2i xi2 − (γr̂i + [ Fλ]i ) xi .

The minimum of z2i + λi zi is equal to −λ2i /4 (attained at zi = −λi /2). Also, the unique minimizer of minξ ≥0 αξ 2 − βξ is ξ = [ β/(2α)]+ , where [·]+ denotes the projection of the argument onto R+ ; hence

min x i ≥0

1 d2i xi2 − (γr̂i + [ Fλ]i ) xi = − 2 [γr̂i + [ Fλ]i ]2+ , 4di

which is attained at the unique optimal point xi =

1 [γr̂i + [ Fλ]i ]+ .

2d2i

(14.37)

Thus, we have that g(λ)

= −

1 k 2 1 n [γr̂i + [ Fλ]i ]2+ λi − ∑ 4 i∑ 4 i =1 d2i =1

1 1 = − kλk22 − k D −1 [γr̂ + Fλ]+ k22 .

4 4 2. If λ∗ is an optimal solution to the dual problem maxλ g(λ), then an optimal solution to the primal problem is recovered applying (14.37), as 1 x ∗ = D −2 [γr̂ + Fλ∗ ]+ .

2 .

3. Let us compute the subdifferential of h = − g: ∂h(λ)

and

=

1 1 n ∂[γr̂i + [ Fλ]i ]2+ λ+ ∑ 2 4 i =1 d2i

=

1 1 n 2[γr̂i + [ Fλ]i ]+ ∂[γr̂i + [ Fλ]i , ]+ λ+ ∑ 2 4 i =1 d2i

   fi ∂[γr̂i + [ Fλ]i , ]+ = 0   α f , α ∈ [0, 1] i

if γr̂i + f i> λ > 0 if γr̂i + f i> λ < 0 if γr̂i + f i> λ = 0

160 A subgradient of h(λ) is thus given by 1 1 n 1 λ + ∑ 2 [γr̂i + f i> λ]+ f i .

2 2 i =1 d i Exercise 14.6 (Kelly’s betting strategy) A gambler has a starting capital W0 and repeatedly bets his whole available capital on a game where with probability p ∈ [0, 1] he wins the stake, and with probability 1 − p he looses it. His wealth Wk after k bets is a random variable: ( 2k W0 with prob. pk Wk = 0 with prob. 1 − pk 1. Determine the expected wealth of the gambler after k bets. Determine the probability with which the gambler eventually runs broke at some k.

2. The results of the previous point should have convinced you that the described one is a ruinous gambling strategy. Suppose now that the gambler gets more cautious, and decides to bet, at each step, only a fraction x of his capital. Denoting with w and ` the (random) number of times where the gambler wins and looses a bet, respectively, we have that his wealth at time k is given by Wk = (1 + x )w (1 − x )` W0 , where x ∈ [0, 1] is the betting fraction, and w + ` = k. Define the exponential rate of growth of the gambler capital as G = lim

1

k→∞ k

log2

Wk .

W0

(a) Determine an expression for the exponential rate of growth G as a function of x. Is this function concave? (b) Find the value of x ∈ [0, 1] that maximizes the exponential rate of growth G. Betting according to this optimal fraction is known as the optimal Kelly’s gambling strategy51 .

3. Consider a more general situation, in which an investor can invest a fraction of his capital on an investment opportunity that may have different payoffs, with different probabilities. Specifically, if W0 x dollars are invested, then the wealth after the outcome of the investment is W = (1 + rx )W0 , where r denotes the return of the investment, which is assumed to be a discrete random variable taking values r1 , . . . , rm with respective probabilities p1 , . . . , pm (pi ≥ 0, ri ≥ −1, for i = 1, . . . , m, and ∑i pi = 1).

After J.L. Kelly, who introduced it in 1956.

51

161 The exponential rate of growth G introduced in point 2 of this exercise is nothing but the expected value of the log-gain of the investment, that is G = E{log(W/W0 )} = E{log(1 + rx )}.

The particular case considered in point 2 corresponds to taking m = 2 (two possible investment outcomes), with r1 = 1, r2 = −1, p1 = p, p2 = 1 − p. (a) Find an explicit expression for G as a function of x ∈ [0, 1].

(b) Devise a simple computational scheme for finding the optimal investment fraction x that maximizes G.

Solution 14.5 (Kelly’s betting strategy) 1. The expected value of Wk is E{Wk } = 2k W0 pk + 0(1 − pk ) = (2p)k W0 , so, if p > 0.5, the expected wealth grows at geometric rate. However, such average growth is rather illusory in practice, since the probability of running broke at iteration k also tends to one at geometric rate: Prob{Wk = 0} = 1 − pk , hence limk→∞ Wk = 0 with probability one (whenever p < 1).

2. (a) We have G

= = =

1 W 1 log2 k = lim (w log2 (1 + x ) + ` log2 (1 − x )) W0 k→∞ k k→∞ k ` w log2 (1 + x ) + lim log2 (1 − x ) lim k→∞ k k→∞ k p log2 (1 + x ) + (1 − p) log2 (1 − x ). lim

This function is a positive combination of two concave functions, hence it is concave. (b) Imposing that the derivative of G w.r.t. x is zero, we obtain that x = 2p − 1. If this quantity is positive, it is also the optimal solution we sought, otherwise the optimal solution is zero, that is ( 2p − 1 if p > 0.5 x∗ = 0 otherwise. and the corresponding optimal growth rate is G ∗ = 1 + p log2 p + (1 − p) log2 (1 − p), if p > 0.5, and G ∗ = 0 otherwise.

162 3. (a) We have that m

G = E{log(1 + rx )} = ∑ pi log(1 + ri x ) i =1

(b) We need to solve the univariate optimization problem m

max ∑ pi log(1 + ri x ) x

i =1

s.t.: x ∈ [0, 1].

The objective function is concave, with derivative given by m

pi ri .

1 + ri x i =1

g( x ) = ∑

The optimal fraction can thus be computed via bisection over the interval [0, 1], see Exercise 12.3.

Exercise 14.7 (Multi-period investments) We consider a multi-stage, single-asset investment decision problem over n periods. For any given time period i = 1, . . . , n, we denote by yi the predicted return, σi the associated variance, and ui the dollar position invested. Assuming our initial position is u0 = w, the investment problem is

n +1 . φ(w) = max ∑ yi ui − λσi2 u2i − c|ui − ui−1 | : u0 = w, un+1 = 0, u

i =1

where the first term represents profit, the second, risk, and the third, approximate transaction costs. Here, c > 0 is the unit transaction cost, and λ > 0 a risk-return trade-off parameter. (We assume λ = 1 without loss of generality.)

1. Find a dual for this problem.

2. Show that φ is concave, and find a sub-gradient of −φ at w. If φ is differentiable at w, what is its gradient at w?

3. What is the sensitivity issue of φ with respect to the initial position w? Precisely, provide a tight upper bound on |φ(w + e) − φ(w)| for arbitrary e > 0, and with y, σ, c fixed. You may assume φ is differentiable for any u ∈ [w, w + e].

Solution 14.6 1. The problem is equivalent to n +1

max

u : u n +1 =0

subject to

∑ yi ui − σi2 u2i − cti

i =1

ti ≥ ui − ui−1 , i = 1, . . . , n + 1 ti ≥ −ui + ui−1 , i = 1, . . . , n + 1,

163 The Lagrangian is n +1 n

L(u, t, λ, µ) = ∑

i =1

yi ui − σi2 u2i − cti + λi (ti − ui + ui−1 ) + µi (ti + ui − ui−1 )

o n n = (λ1 − µ1 )w + ∑ (yi − λi + λi+1 + µi − µi+1 )ui − σi2 u2i

o

i =1

n +1

+ ∑ ( λi + µi − ci ) ti . i =1

The dual function is max L(u, t, λ, µ) u,t

n o n = (λ1 − µ1 )w + ∑ max (yi − λi + λi+1 + µi − µi+1 )ui − σi2 u2i i =1

n +1

ui

+ ∑ max(λi + µi − ci )ti ti i =1 n  ( y i − λ i + λ i +1 + µ i − µ i +1 )2  (λ − µ )w + 1 1 ∑ 4σi2 = i =1   +∞

if λi + µi − c = 0, otherwise.

Eliminating the variable µ, we obtain the dual problem as: n

(yi − 2λi + 2λi+1 )2 .

4σi2 i =1

(2λ1 − c)w + ∑

min

0≤ λ ≤ c

Let v = 2λ − c1, the dual problem reduces to: n

( y i − v i + v i +1 )2 : kvk∞ ≤ c.

4σi2 i =1

φ(w) = min v1 w + ∑ v

2. The function φ is concave. The formula for sub-gradient of maxima of functions52 shows that for any w, a sub-gradient of −φ at w is v1∗ (w), where v∗ (w) is an optimal dual variable. If φ is differentiable at w, then the sub-gradient at w is unique, and equal to the gradient. Hence, φ0 (w) = v1∗ (w) for points w such that φ is differentiable at w.

3. Assuming that φ is differentiable in [w, w + e], let u ∈ [w, w + e], and let v1∗ (u) be a corresponding gradient of φ at u. We then have φ(w + e)

= φ(w) +

Z w+e

= φ(w) + e

w

Z 1 0

φ0 (u) du

φ0 (w + te) dt.

With the bound |φ0 (u)| = |v1∗ (u)| ≤ c:

|φ(w + e) − φ(w)| = e

Z 1 0

φ0 (w + te) dt ≤ ec.

52

See Section 8.2.3.1.

164 Exercise 14.8 (Personal finance problem) Consider the following personal finance problem. You are to be paid for a consulting job, for a total of C = $30, 000, over the next six months. You plan to use this payment to cover some past credit card debt, which amounts to be D = $7000. The credit card’s APR (annual interest rate) is r1 = 15.95%. You have the following items to consider: • At the beginning of each month, you can transfer any portion of the credit card debt to another card with a lower APR of r2 = 2.9%.

This transaction costs r3 = 0.2% of the total amount transferred.

You cannot borrow any more from either credit cards; only transfer of debt from card 1 to 2 is allowed.

• The employer allows you to choose the schedule of payments: you can distribute the payments over a maximum of six months. For liquidity reasons, the employer limits any month’s pay to 4/3 × (C/6).

• You are paid a base salary of B = $70, 000 per annum. You cannot use the base salary to pay off the credit card debt; however it affects how much tax you pay (see next).

• The first three months are the last three months of the current fiscal year and the last three months are the first three months of the next fiscal year. So if you choose to be paid a lot in the current fiscal year (first three months of consulting), the tax costs are high; they are lower if you choose to distribute the payments over several periods. The precise tax due depends on your gross annual total income G, which is your base salary, plus any extra income. The marginal tax rate schedule is given in Table 14.5.

• The risk-free rate (interest rate from savings) is zero.

• Time line of events: all events occur at the beginning of each month, i.e. at the beginning of each month, you are paid the chosen amount, and immediately you decide how much of each credit card to pay off, and transfer any debt from card 1 to card 2. Any outstanding debt accumulates interest at the end of the current month.

• Your objective is to maximize the total wealth at the end of the two fiscal years whilst paying off all credit card debt.

1. Formulate the decision-making problem as an optimization problem. Make sure to define the variables and constraints precisely.

165

Total gross income G $0 ≤ G ≤ $80, 000 $80, 000 ≤ G

Marginal tax rate 10% 28%

Table 14.5: Marginal tax rate schedule.

Total tax 10% × G 28% × G plus $8000 = 10% × $80, 000

To describe the tax, use the following constraint: Ti = 0.1 min( Gi , α) + 0.28 max( Gi − α, 0)

(14.38)

where Ti is the total tax paid, Gi is the total gross income in years i = 1, 2 and α = 80, 000 is the tax threshold parameter.

2. Is the problem a linear program? Explain.

3. Under what conditions on α and Gi can the tax constraint (14.38) be replaced by the following set of constraints? Is it the case for our problem? Can you replace (14.38) by (14.39) in your problem?

Explain.

Ti

= 0.1d1,i + 0.28d2,i

d2,i

≥ Gi − α

d2,i d1,i d1,i

(14.39)

≥ 0

≥ Gi − d2,i

≥ d2,i − α

4. Is the new problem formulation, with (14.39), convex? Justify your answer.

5. Solve the problem using your favorite solver. Write down the optimal schedules for receiving payments and paying off/transferring credit card debt, and the optimal total wealth at the end of two years. What is your total wealth W? 6. Compute an optimal W for α ∈ [70k, 90k] and plot α vs. W in this range. Can you explain the plot?

Solution 14.7 1. The variables are, for i = 1, . . . , 6: xi : amount of pay received at the beginning of month; yi,j : amount of credit card j = 1, 2 paid off at the beginning of month i = 1, . . . , 6; ti : amount of transfer from credit card 2 to 1 at the beginning of month.

We also introduce the following redundant variables, for i = 1, . . . , 6, j, k = 1, 2: wi : amount remaining at the end of month after paying credit card debt/transfer; zi,j : amount owed to credit card j at the

166 end of of month i; Tk : total tax paid in year k; Gk total gross income in years k; W: total wealth at the end of two fiscal years.

The optimization problem is max W s. t.

Month 1 balance equations : w1 = x1 − y1,1 − r3 t1 z1,1 = (1 + r1 /12)( D − y1,1 − t1 ) z1,2 = (1 + r2 /12)t1 Month 2 ≤ k ≤ 6 balance equations : wk = wk−1 + xk − yk,1 − yk,2 − r3 tk zk,1 = (1 + r1 /12)(zk−1,1 − yk,1 − tk ) zk,2 = (1 + r2 /12)(zk−1,2 − yk,2 + tk ) Pay off all credit card debt in 6 months : z6,1 = 0 z6,2 = 0 Tax Calculation G1 = B + x1 + x2 + x3 G2 = B + x4 + x5 + x6 Ti = 0.1 min( Gi , α) + 0.28 max( Gi − α, 0)

i = 1, 2

Total wealth at end of 2yrs W = 2B + w6 − T1 − T2 Variable constraints ∑6i=1 xi = C xi ≤ 4/3 × C/6 W, t, x, y, z, w, T, G ≥ 0

i = 1, . . . , 6

2. No, because the tax equality constraints Ti = . . . is not affine in variables Gi .

3. Consider solving the problem with constraints (14.39) instead of (14.38).

As we maximize W = 2B + w6 − T1 − T2 , the dummy variable d2,i is forced to be satisfy d2,i = max( Gi − α, 0), i = 1, 2.

Now if d2,i = 0 ≥ Gi − α (income is in the low tax bracket), then (14.39) forces d1,i ≥ Gi & d1,i ≥ −α, and maximization forces d1,i = Gi . Now if d2,i = Gi − α ≥ 0 (income is in the high tax bracket), then (14.39) forces d1,i ≥ α & d1,i ≥ Gi − 2α, and maximization forces d1,i = max(α, Gi − 2α). For the equivalence of

167 this new formulation to the original problem, we require that: α ≥ Gi − 2α =⇒ 3α ≥ Gi at optimality. In the given problem, the gross annual income in any year cannot exceed B + C = $100, 000, whereas α = $80, 000. So for the given problem parameters, we can safely replace the tax constraints (14.38) by (14.39).

4. The new problem is convex, in fact, a linear program, as constraints are affine inequalities.

5. For α = 80, 000, the results are (rounding the solution to 2 decimals): W∗ x∗ y∗:,1 y∗:,2 t:

= 144, 198.52 = [6666.67 = [6666.00 = [0 = [334.05

4506.23 0 334.81 0

4397.92 0 0 0

4732.92 0 0 0

4760.70 0 0 0

4936.25] 0] 0] 0]

Note the optimal decision may not be unique.

6. As seen in Fig. (14.15), the optimal schedule is unaffected as α increases beyond a threshold: when α is large enough, only the low tax bracket plays an effect in the optimal schedule.

Exercise 14.9 (Transaction costs and market impact) We consider the following portfolio optimization problem max r̂ > x − λx > Cx − c · T ( x − x0 ) : x ≥ 0, x ∈ X , x

(14.40)

where C is the empirical covariance matrix, λ > 0 is a risk parameter, and r̂ is the time-average return for each asset for the given period.

Here, the constraint set X is determined by the following conditions: • No shorting is allowed.

• There is a budget constraint x1 + . . . + xn = 1.

In the above, the function T represents transaction costs and market impact, c ≥ 0 is a parameter that controls the size of these costs, while x0 ∈ Rn is the vector of initial positions. The function T has the form n

T ( x ) = ∑ B M ( x ), i =1

where the function B M that is piece-wise linear for small x, and quadratic for large x; that way we seek to capture the fact that transaction costs are dominant for smaller trades, while market impact kicks in for larger ones. Precisely, we define B M to be the so-called

Figure 14.15: Optimal wealth as a function of income α.

168 “reverse Hüber” function with cut-off parameter M: for a scalar z, the function value is   |z| if |z| ≤ M, .

2 + M2 B M (z) = z  otherwise.

2M

The scalar M > 0 describes where the transition from a linearly shaped to a quadratically shaped penalty takes place.

1. Show that B M can be expressed as the solution to an optimization problem: B M (z) = min v + w + v,w

w2 : |z| ≤ v + w, v ≤ M, w ≥ 0.

2M

Explain why the above representation proves that B M is convex.

2. Show that, for given x ∈ Rn : T ( x ) = min 1> (v + w) + w,v

1 > w w : 2M

v ≤ M1, w ≥ 0, | x − x0 | ≤ v + w,

where, in the above, v, w are now n-dimensional vector variables, 1 is the vector of ones, and the inequalities are component-wise.

3. Formulate the optimization problem (14.40) in convex format. Does the problem fall into one of the categories (LP, QP, SOCP, etc) seen in Chapter 8?

4. Draw the efficient frontier of the portfolio corresponding to M = .01, .05, .1, 1, 5, with c = 5.10−4 . Comment on the qualitative differences between the optimal portfolio for two different values of M = .01, 1.

Solution 14.8 1. We can eliminate the variable v, and get p∗

w2 . = min v + w + : |z| ≤ v + w, v ≤ M, w ≥ 0 v,w 2M 1 = |z| + min w2 , 2M w≥γ

where γ = max(0, |z| − M ). If |z| ≤ M, then γ = 0, and we obtain p∗ = |z| +

1 min w2 = |z|.

2M w≥0

169 If |z| ≥ M, then p∗

= |z| +

1 min w2 2M w≥|z|− M

(|z| − M)2 2M z2 + M 2 .

2M

= |z| + =

The representation proves convexity of B M , since the constraints are convex in z. Indeed, the epigraph condition B M (z) ≤ t, for t ∈ R, is equivalent to the conditions v+w+

w2 ≤ t, |z| ≤ v + w, v ≤ M, w ≥ 0 2M

and those are jointly convex in (z, t, v, w).

2. Denoting z = x − x0 , we have n

n

T ( x ) = ∑ B M (zi ) = min ∑ vi + wi + v,w

i =1

i =1

wi2 : 2M

|zi | ≤ vi + wi , vi ≤ M, wi ≥ 0.

The above can indeed be written in vector form, as T ( x ) = min 1> (v + w) + w,v

1 > w w : 2M

v ≤ M1, w ≥ 0, | x − x0 | ≤ v + w.

In the above, v, w are now n-dimensional vector variables, 1 is the vector of ones, and the inequalities are component-wise.

3. The optimization problem in convex format is given by

1 > maximize r̂ > x − λx > Cx − c · 1> (v + w) + w w 2M subject to 1> x = 1, | x − x0 | ≤ v + w, x ≥ 0, v ≤ M1, w ≥ 0.

This optimization problem is a QP. 4. The efficient frontier is shown in Fig. 14.16.

5. The optimal portfolio for M = 1 strictly dominates the one for M = 0.1, i.e., it gives higher average return for all values of the standard deviation. Moreover, while the average return for the optimal portfolio for M = 1 increases with standard deviation, the average return for the optimal portfolio for M = 0.1 actually decreases with standard deviation.

Figure 14.16: Efficient frontier with a reverse Hüber function cost model.

170 Exercise 14.10 (Optimal portfolio execution) This exercise deals with an optimal portfolio execution problem, where we seek to optimally liquidating a portfolio given as a list of n asset names and initial number of shares in each asset. The problem is stated over a given time horizon T, and shares are to be traded at fixed times t = 1, . . . , T. In practice, the dimension of the problem may range from n = 20 to n = 6000.

The initial list of shares is given by a vector x0 ∈ Rn , and the final target is to liquidate our portfolio. The initial position is given by a price vector p ∈ Rn , and a vector s that gives the side of each asset (1 to indicate long, −1 to indicate short). We denote by w = p ◦ s the so-called price weight vector, where ◦ denotes the component-wise product53 .

Our decision variable is the execution schedule, a n × T matrix X, with Xit the amount of shares (in hundreds, say) of asset i to be sold at time t. We will not account for discretization effects and treat X as a real-valued matrix. For t = 1, . . . , T, we denote by xt ∈ Rn the t-th column of X; xt encapsulates to all the trading that takes place at period t.

In our problem, X is constrained via upper and lower bounds: we express this as X l ≤ X ≤ X u , where inequalities are understood component-wise, and X l , X u are given n × T matrices (for example, a no short selling condition is enforced with X l = 0). These upper and lower bounds can be used to make sure we attain our target at time t = T: we simply assume that the last columns of X l , X u are both equal to the target vector, which is zero in the case we seek to fully liquidate the portfolio.

We may have additional linear equality or inequality constraints.

For example we may enforce upper and lower bounds on the trading: .

0 ≤ yt = xt−1 − xt ≤ yut , t = 1, . . . , T, where Y = [y1 , . . . , y T ] ∈ Rn,T will be referred to as the trading matrix, and Y u = [y1u , . . . , yuT ] is a given (non-negative) n × T matrix that bounds the elements of Y from above. The lower bound ensures that trading decreases over time; the second constraint can be used to enforce a maximum participation rate, as specified by the user.

We will denote by X ⊆ Rn,T our feasible set, that is, the set of n × T matrices X = [ x1 , . . . , x T ] that satisfy the constraints above, including the upper and lower bounds on X. We also want to enforce a dollar neutral strategy at each time step.

This requires to have the same dollar position both in long and short.

This can be expressed with the conditions w> xt = 0, t = 1, . . . , T, where w = p ◦ s ∈ Rm contains the price weight of each asset. We

For two n-vectors u, v, the notation u ◦ v denotes the vector with components ui vi , i = 1, . . . , n.

53

171 can write the dollar-neutral constraint compactly as X > w = 0.

Our objective function involves three terms, referred to as impact, risk, and alpha respectively. The impact function is modeled as T

n

I ( X ) = ∑ ∑ Vti ( Xti − Xt−1,i )2 , t =1 i =1

where V = [v1 , . . . , v T ] is a n × T matrix of non-negative numbers that model the impact of transactions (the matrix V has to be estimated with historical data, but we consider it to be fully known here). In the above, the n-vector of initial conditions x0 = ( X0,i )1≤i≤n is given.

The risk function has the form T

R ( X ) = ∑ ( w ◦ x t ) > Σ ( w ◦ x t ), t =1

where ◦ is the component-wise product, w = p ◦ s is the price weight vector, and Σ is a positive semi-definite matrix the describes the daily market risk. In this problem, we assume that Σ has a “diagonal-pluslow-rank” structure, corresponding to a factor model. Specifically, Σ = D2 + FF > , where D is a n × n, diagonal positive-definite matrix, and F is a n × k “factor loading” matrix, with k ≈ 10 − 100 the number of factors in the model (typically, k n). We can write the risk function as T

2 R( X ) = ∑ xt> ( Dw + Fw Fw> ) xt , t =1

. . where Dw = diag (w) D is diagonal, positive definite, and Fw = diag (w) F. Finally, the alpha function accounts for views on the asset return themselves, and is a linear function of X, which we write as T

C ( X ) = ∑ c> t xt , t =1

where C = [c1 , . . . , c T ] ∈ Rn,T is a given matrix that depends on α ∈ Rn , which contains our return predictions for the day. Precisely, ct = αt ◦ p, where p ∈ Rn is the price vector, and αt is a vector of predicted returns.

1. Summarize the problem data, and their sizes.

2. Write the portfolio execution problem as a QP. Make sure to define precisely the variables, objective and constraints.

3. Explain how to take advantage of the factor model to speed up computation. Hint: look at Exercise 12.9.

172 Solution 14.9 1. Let us first summarize our problem data: • p ∈ Rn : initial price of each asset.

• s ∈ Rn : side of each asset (1 to indicate long, −1 to indicate short) • w ∈ Rn : price weight vector, computed as w = p ◦ s, so that |w| = p.

• Covariance matrix Σ = D2 + FF > , with

– F ∈ Rn,k , factor loading matrix, and Fw = diag (w) F ∈ Rn×k , its weighted counterpart; – D ∈ Rn,n , positive-definite diagonal matrix in the factor co. variance model, and Dw = diag (w) D, its weighted counterpart.

• C = [c1 , . . . , c T ] ∈ Rn×T , a matrix that accounts for return prediction.

• V ∈ Rn,T + , a non-negative matrix that contains market impact parameters.

• x0 , xtarget , initial and final list of shares. Since a dollar-neutral strategy is sought, we assume that w> x0 = w> xtarget are both zero.

• X u , X l and Y u , Y l : n × T matrices that contain upper and lower bounds on the schedule and trading matrices.

2. We formulate the portfolio execution as follows: Decision variables. We define two sets of decision variables given by: xt , yt ∈ Rn , t = 1, . . . , T. Here, xt encapsulates to all the portfolio execution that takes place at period t, and yt encapsulates the change in the portfolio between time t − 1 and t.

Constraints. The upper and lower bounds on xt are given by: xtl ≤ xt ≤ xtu , t = 1, . . . , T. The yt variables are defined by the following constraints: yt = xt − xt−1 , t = 1, . . . , T.

173 Upper and lower bounds on the trading are enforced as follows: 0 ≤ yt ≤ yut , t = 1, . . . , T. The dollar neutral strategy is enforced by: w> xt = 0, t = 1, . . . , T. Objective function. The objective function is given by T

∑ y>t diag (vt ) yt + xt> ( Dw2 + Fw Fw> )xt − c>t xt .

t =1

The above problem has a convex quadratic function that is minimized over a convex set defined by linear constraints. Hence, it has the required QP format.

3. We can take advantage of the factor model to speed up computation as follows. For t = 1, · · · , T, we can define new variables zt = Fw> xt . Here zt ∈ Rk , where k n. With this in place, the objective function becomes >

∑ (y>t diag (vt ) yt + xt> Dw2 xt + z>t zt − c>t xt .

t =1

We observe that the above function of ( xt , yt , zt )tt=1 is decomposable; that is, its Hessian is diagonal. As explained in Exercise 12.9, this leads to great speed-ups. Precisely, the computational time will grows linearly with the number of assets n, as opposed to a cubic growth.

174

15.

Control Problems

Exercise 15.1 (Stability and eigenvalues) Prove that the continuoustime LTI system (15.20) is asymptotically stable (or stable, for short) if and only if all the eigenvalues of the A matrix, λi ( A), i = 1, . . . , n, have (strictly) negative real parts.

Prove that the discrete-time LTI system (15.28) is stable if and only if all the eigenvalues of the A matrix, λi ( A), i = 1, . . . , n, have moduli (strictly) smaller than one.

Hint: use the expression x (t) = e At x0 for the free response of the continuous-time system, and the expression x (k) = Ak x0 for the free response of the discrete-time system. You may derive your proof under the assumption that A is diagonalizable.

Solution 15.1 (Stability and eigenvalues) The continuous-time LTI system (15.20) is, by definition, asymptotically stable if and only if its free response (i.e., the state evolution obtained for input u(t) = 0 for all t ≥ 0) goes to zero as t → ∞, for any initial condition x0 . Since we assume that A is diagonalizable, we can write A = UΛU −1 (where Λ is diagonal and contains the eigenvalues λi of A, and U contains by columns the eigenvectors ui ), and apply the result of Example 3.8, writing the matrix exponential as e At = UeΛt U −1 . We thus have that

ke At k2

= kUeΛt U −1 k2 ≤ kU k2 · kU −1 k2 · keΛt k2 = κ keΛt k2 = κ max eσi t i =1,...,n

= κeσmax t , where κ > 0 is a constant, σi is the real part of λi , and σmax is the maximum real part of the eigenvalues. If σi < 0 for all i = 1, . . . , n, then clearly σmax < 0, and ke At k2 → 0 for t → ∞54 . Now, since the free evolution of the state is given by x (t) = e At x0 , we have that

k x (t)k2

= ke At x0 k2 ≤ ke At k2 · k x0 k2 ≤ κ k x0 k2 eσmax t ,

thus k x (t)k2 → 0 for t → ∞, which proves that if all the eigenvalues of A have negative real parts, then the system is asymptotically stable. We can also prove the converse, that is if the system is asymptotically stable, then all eigenvalues must have negative real parts.

We actually prove the following equivalent statement: if there exist

Although we proved this fact only for diagonalizable A, this assumption can be removed, and the result actually holds for all A. 54

175 σi ≥ 0, then the system is not asympotically stable. To prove this fact, it suffices to choose the initial condition x0 = Uei to see that, for this initial condition,

k x (t)k2 = kUeΛt U −1 x0 k2 = kUeΛt ei k2 = kui eλi t k2 = kui k2 eσi t , thus k x (t)k2 doesn’t go to zero as t → ∞ (since σi ≥ 0).

The corresponding result for the discrete-time LTI system (15.28) can be proved analogously, by considering the free evolution x (k ) = Ak x0 = UΛk U −1 x0 .

Exercise 15.2 (Signal norms) A continuous-time signal w(t) is a function mapping time t ∈ R to values w(t) in either Cm or Rm . The energy content of a signal w(t) is defined as .

E(w) = kwk22 =

Z ∞

−∞

kw(t)k22 dt,

where kwk2 is the 2-norm of the signal. The class of finite-energy signal contains signals for which the above 2-norm is finite.

Periodic signals typically have infinite energy. For a signal with period T, we define its power content as . 1 P(w) = T

Z t0 + T t0

kw(t)k22 dt.

1. Evaluate the energy of the harmonic signal w(t) = ve ωt , v ∈ Rm , and of the causal exponential signal w(t) = veat , for a < 0, t ≥ 0 (w(t) = 0 for t < 0).

2. Evaluate the power of the harmonic signal w(t) = ve ωt and of the sinusoidal signal w(t) = v sin(ωt).

Solution 15.2 (Signal norms) 1. For a complex vector signal w(t), we have that

kw(t)k22 = w? (t)w(t), where ? denotes the transpose conjugate. Thus, for the harmonic signal w(t) = ve ωt we have kw(t)k22 = kvk22 , and the energy is E(w) =

Z ∞

−∞

kw(t)k22 dt = kvk22

Z ∞

−∞

dt = ∞,

hence the harmonic signal has infinite energy.

For the causal exponential signal w(t) = veat , a < 0, we have instead Z ∞ kvk22 E(w) = kvk22 e2at dt = .

2| a | 0

176 2. The harmonic signal w(t) = ve ωt has period T = 2π/ω; its power is given by . 1 P(w) = T

Z t0 + T t0

kw(t)k22 dt = kvk22

1 2π/ω

Z 2π/ω 0

dt = kvk22 .

The power of the sinusoidal signal w(t) = v sin(ωt) is instead .

P(w) = kvk22

1 2π/ω

Z 2π/ω 0

sin2 (ωt)dt = kvk22

π . ω

Exercise 15.3 (Energy upper bound on the system’s state evolution) Consider a continuous-time LTI system ẋ (t) = Ax (t), t ≥ 0, with no input (such a system is said to be autonomous), and output y(t) = Cx. We wish to evaluate the energy contained in the system’s output, as measured by the index .

J ( x0 ) =

Z ∞ 0

. where Q = C > C 0.

y(t)> y(t)dt =

Z ∞ 0

x (t)> Qx (t)dt,

1. Show that if the system is stable, then J ( x0 ) < ∞, for any given x0 .

2. Show that if the system is stable and there exist a matrix P 0 such that A> P + PA + Q 0,

then it holds that J ( x0 ) ≤ x0> Px0 . Hint: consider the quadratic form V ( x (t)) = x (t)> Px (t), and evaluate its derivative with respect to time.

3. Explain how to compute a minimal upper bound on the state energy, for the given initial conditions.

Solution 15.3 (Energy upper bound on the system’s state evolution) 1. Using an argument analogous to the one introduced in the solution of Exericse 15.1, one can prove that ky(t)k2 ≤ ck x0 k2 eσmax t , where σmax is the maximum real part of the eigenvalues λi of A. If the system is stable, then Re (λi ) < 0 for all i, hence ky(t)k2 is upper bounded by a decreasing exponential, and thus it is integrable over 0 to ∞, hence J ( x0 ) < ∞, for any given x0 . .

2. Define V ( x ) = x > Px. Evaluating V along the trajectory of the system we have V ( x (t)) = x (t)> Px (t), and V̇ ( x (t))

= =

dV ( x (t)) dx (t) = ∇> = 2x (t)> P ẋ (t) xV· dt dt 2x (t)> PAx (t) = x (t)> PAx (t) + x (t)> A> Px (t)

= x (t)> ( PA + A> P) x (t).

177 Assume now there exist P 0 such that PA + A> P + Q 0.

(15.41)

multiplying this inequality on the left by x (t)> and on the right by x (t), we have x (t)> ( PA + A> P) x (t) + x (t)> Qx (t) ≤ 0.

Recognizing that the first term in the above inequality is equal to V̇ ( x (t)), we rewrite it as x (t)> Qx (t) ≤ −V̇ ( x (t)).

Integrating from 0 to ∞, we have J ( x0 )

Z ∞

=

0

x (t)> Qx (t)dt ≤ −

Z ∞ 0

V̇ ( x (t))dt

= V ( x0 ) − V ( x (∞)) = V ( x0 ) = x0> Px0 , where we have used the hypothesis of stability, which implies that x (∞) = 0.

3. From the previous point, any P 0 that satisfies (15.41) provides an upper bound on the state energy: J ( x0 ) ≤ x0> Px0 . We can thus find the minimal upper bound by minimizing x0> Px0 over P 0 that satisfy (15.41), which amounts to solving the following SDP min

x0> Px0

s.t.:

P 0

P

PA + A> P + Q 0.

Exercise 15.4 (System gain) The gain of a system is the maximum energy amplification from the input signal to output. Any input signal u(t) having finite energy is mapped by a stable system to an output signal y(t) which also has finite energy. Parseval’s identity relates the energy of a signal w(t) in the time domain with the energy of the same signal in the Fourier domain (see Remark 15.1), that is .

E(w) = kwk22 =

Z ∞

−∞

kw(t)k22 dt =

1 2π

Z ∞

−∞

. kŴ (ω )k22 dω = kŴ k22

The energy gain of system (15.26) defined as . energy gain =

kyk22 .

2 u(t):kuk2 <∞,u6=0 k u k2 sup

178 1. Using the above information, prove that, for a stable system, energy gain ≤ sup k H ( ω )k22 , ω ≥0

where k H ( ω )k2 is the spectral norm of the transfer matrix of system (15.26), evaluated at s = ω. The (square root of the) energy gain of the system is also known as the H∞ -norm, and it is denoted by k H k∞ .

Hint: use Parseval’s identity and then suitably bound a certain integral. Notice that equality actually holds in the previous formula, but you are not asked to prove this.

2. Assume that system (15.26) is stable, x (0) = 0, and D = 0. Prove that if there exist P 0 such that "

A> P + PA + C > C B> P

PB − γ2 I

#

0

(15.42)

then it holds that

k H k∞ ≤ γ.

Then, devise a computational scheme that provides you with the lowest possible upper bound γ∗ on the energy gain of the system.

Hint: define a quadratic function V ( x ) = x > Px, and observe that the derivative in time of V, along the trajectories of system (15.26) is dV ( x ) = x > P ẋ + ẋ > Px. dt Then, show that the LMI condition (15.42) is equivalent to the condition that dV ( x ) + kyk2 − γ2 kuk2 ≤ 0, dt

∀ x, u satisfying (15.26),

and that this implies in turn that k H k∞ ≤ γ.

Solution 15.4 (System gain) 1. We recall that, for a stable system, the Fourier transform Ŷ (ω ) of the outpuy y(t) is given by55 Ŷ (ω ) = H ( ω )Û (ω ), where U (ω ) is the Fourier transform of the input u(t) (which exists, since u is assumed to be of finite energy). Using Parseval’s identity, we

55

See Remark 15.1.

179 rewrite the energy gain as energy gain

= ≤ ≤

R∞

2 −∞ k H ( ω )Û ( ω )k2 dω R∞ 2 kÛ k22 <∞,Û 6=0 −∞ kU ( ω )k2 dω R∞ 2 2 −∞ k H ( ω )k2 kÛ (ω )k2 dω R∞ sup 2 kÛ k22 <∞,Û 6=0 −∞ kU ( ω )k2 dω R∞ supω k H ( ω )k22 −∞ kÛ (ω )k22 dω R∞ sup 2 kÛ k22 <∞,Û 6=0 −∞ kU ( ω )k2 dω

sup

= sup k H ( ω )k22 . ω

Further, since H ( ω ) is the Fourier transform of real signals (the impulse response of the system), its norm is centrally symmetric with respect to ω, hence the supremum above can be restricted to the nonnegative ω, thus proving the claim that energy gain = supω ≥0 k H ( ω )k22 .

2. Following the hint, we let V ( x (t)) = x (t)> Px (t), and evaluate V̇ ( x (t))

. =

=

dV ( x (t)) = x (t)> P ẋ (t) + ẋ (t)> Px (t) dt x (t)> P( Ax (t) + Bu(t)) + ( Ax (t) + Bu(t))> Px (t)

= x (t)> ( PA + A> P) x (t) + 2x (t)> PBu(t).

Next, multiplying Eq. (15.42) on the right by ( x, u) and on the left by its transpose, and recalling that y(t) = Cx (t), we have 0

≥

"

x (t) u(t)

#> "

A> P + PA + C > C B> P

PB − γ2 I

#"

x (t) u(t)

#

= x (t)> ( PA + A> P) x (t) + 2x (t)> PBu(t) + ky(t)k22 − γ2 ku(t)k22

= V̇ ( x (t)) + ky(t)k22 − γ2 ku(t)k22 .

Integrating over t from 0 to ∞, we obtain that Z ∞ 0

ky(t)k22 dt − γ2

Z ∞ 0

ku(t)k22 dt ≤ −

Z ∞ 0

V̇ ( x (t))dt

= V ( x (0)) − V ( x (∞)) = −V ( x (∞)) ≤ 0,

where we have used the fact that x (0) = 0 and that V ( x ) ≥ 0 for all x. We thus have that, for all u with finite energy, the current hypotheses imply that

kyk22 ≤ γ2 kuk22 ,

180 hence

k y k2 ≤ γ, k u k2

∀u 6= 0, kuk2 < ∞,

which implies that k H k∞ ≤ γ, as desired.

The best upper bound γ∗ on k H k∞ can thus be obtained by solving an SDP problem in variables ( P, γ2 ), where one minimizes γ2 , subject to P 0 and to the LMI (15.42). It could actually be proved (we did not do this) that this optimal upper bound is tight, i.e., γ∗ is equal to k H k∞ .

Exercise 15.5 (Extended superstable matrices) A matrix A ∈ Rn,n is said to be continuous-time extended superstable56 (which we denote with A ∈ Ec ) if there exist d ∈ Rn such that

∑ |aij |d j < −aii di , di > 0,

i = 1, . . . , n.

j 6 =i

Similarly, a matrix A ∈ Rn,n is said to be discrete-time extended superstable (which we denote with A ∈ Ed ) if there exist d ∈ Rn such that n

∑ |aij |d j < di , di > 0,

i = 1, . . . , n.

j =1

If A ∈ Ec , then all its eigenvalues have real parts smaller than zero, hence the corresponding continuous-time LTI system ẋ = Ax is stable. Similarly, if A ∈ Ed , then all its eigenvalues have moduli smaller than one, hence the corresponding discrete-time LTI system x (k + 1) = Ax (k) is stable. Extended superstability thus provides a sufficient condition for stability, which has the advantage of being checkable via feasibility of a set of linear inequalities.

1. Given a continuous-time system ẋ = Ax + Bu, with x ∈ Rn , u ∈ Rm , describe your approach for efficiently designing a statefeedback control law of the form u = −Kx, such that the controlled system is extended superstable.

2. Given a discrete-time system x (k + 1) = Ax (k) + Bu(k), assume that matrix A is affected by interval uncertainty, that is aij = âij + δij ,

i, j = 1, . . . , n,

where âij is the given nominal entry, and δij is an uncertainty term, which is only known to be bounded in amplitude as |δij | ≤ ρrij , for given rij ≥ 0. Define the radius of extended superstability as the largest value ρ∗ of ρ ≥ 0 such that A is extended superstable for all the admissible uncertainties. Describe a computational approach for determining such ρ∗ .

See B.T. Polyak, “Extended superstability in control theory,” Automation and Remote Control, 2004.

56

181 Solution 15.5 (Extended superstable matrices) 1. The controlled system is described by the linear differential equation ẋ = ( A − BK ) x.

This system is continuous-time extended superstable, if there exist d ∈ Rn , d > 0, such that n

∑ |( A − BK)ij |d j < −( A − BK)ii di ,

i = 1, . . . , n.

j =1

Since d j ≥ 0, we can absorb it inside the absolute value on the left, obtaining n

∑ |aij d j − ( BK)ij d j | < −aii di + ( BK)ii di ,

i = 1, . . . , n.

j =1

Further, since ( BK )ij = ∑m p=1 bip k pj , we may define new variables . z pj = k pj d j , p = 1, . . . , m, j = 1, . . . , n, and write the above condition as n

m

m

j =1

p =1

p =1

∑ |aij d j − ∑ bip z pj | < −aii di + ∑ bip z pi ,

i = 1, . . . , n,

which is a set of linear inequality conditions on the entries of d . and Z = Kdiag (d). Choosing a small e > 0, we can determine a suitable feedback matrix K by first solving the following linear program for variables d and Z min

trace( Z )

s.t.:

d ≥ e1

d,Z

m ∑nj=1 | aij d j − ∑m p=1 bip z pj | ≤ − aii di + ∑ p=1 bip z pi − e,

i = 1, . . . , n,

and then retrieving K as K = Z (diag (d))−1 . Notice that the choice of the objective trace( Z ) is immaterial in the above problem, since we are actually just required to find a feasible design K. 2. Since the uncertainty on the entries of A is independent and bound in intervals, we have max | aij | = | âij | + ρrij .

|δij |≤ρrij

The uncertain matrix is robustly discrete-time extended superstable if if there exist d ∈ Rn , d > 0, such that n

max

∑ |aij |d j < di ,

|δij |≤ρrij j=1

i = 1, . . . , n,

182 which is equivalent to n

∑ (|âij | + ρrij )d j < di ,

i = 1, . . . , n,

j =1

and this is a set of linear inequality conditions on d. To compute the largest ρ ≥ 0 such that the matrix is robustly superstable, we may proceed via a bisection approach: fix some small e > 0, let ρl = 0, and suppose we know in advance that ρ∗ > 0 and ρ∗ < ρr (some large value)57 .

Let ρ = (ρr + ρl )/2, and check feasibility (e.g., by solving an LP with zero objective) of the linear inequalities n

∑ (|âij | + ρrij )d j ≤ di − e,

i = 1, . . . , n.

j =1

If feasible, let ρl ← ρ and repeat, otherwise let ρr ← ρ and repeat.

This algorithm will converge at a geometric rate to the optimal value ρ∗ .

The first condition can be easily checked by verifying feasibility at ρ = 0. The initial ρr can be found by progressively increasing ρ until unfeasibility is detected.

57

183

16.

Engineering Design

Exercise 16.1 (Network congestion control) A network of n = 6 peer-to-peer computers is shown in Figure 16.17. Each computer can upload or download data at a certain rate on the connection links shown in the figure.

Let b+ ∈ R8 be the vector containing the packet transmission rates on the links numbered in the figure, and let b− ∈ R8 be the vector containing the packet transmission rates on the reverse links, where it must hold that b+ ≥ 0 and b− ≥ 0. Define an arc-node incidence matrix for this network   1 0 1 1 0 0 0 0   0 0 0 0 0 0   −1 1   0 0 −1 1 0 0 0  .  0 , A=  0 −1 −1 0 0 −1 −1 0      0 0 0 −1 1 0 1   0 0 0 0 0 0 0 1 −1

. . and let A+ = max( A, 0) (the positive part of A), A− = min( A, 0) (the negative part of A). Then, the total output (upload) rate at the nodes is given by vupl = A+ b+ − A− b− , and the total input (download) rate at the nodes is given by vdwl = A+ b− − A− b+ . The net outflow at nodes is hence given by vnet = vupl − vdwl = Ab+ − Ab− , and the flow balance equations require that [vnet ]i = f i , where f i = 0 if computer i is not generating or sinking packets (it just passes on the received packets, i.e., it is acting as a relay station), f i > 0 if computer i is generating packets, or f i < 0 if it is sinking packets at an assigned rate f i .

Each computer can download data at a maximum rate of v̄dwl = 20 Mbit/s and upload data at a maximum rate of v̄upl = 10 Mbit/s (these limits refer to the total download or upload rates of a computer, through all its connections). The level of congestion of each connection is defined as − c j = max(0, (b+ j + b j − 4)),

j = 1, . . . , 8.

Assume that node 1 must transmit packets to node 5 at a rate f 1 = 9 Mbit/s, and that node 2 must transmit packets to node 6 at a rate f 2 = 8 Mbit/s. Find the rate on all links such that the average congestion level of the network is minimized.

4

1

3 1

5

3

5

6 8

4 2

7

2

Figure 16.17: A small network.

6

184 Solution 16.1 As stated in the exercise, let A+ b+ − A− b−

vupl

=

vdwl

=

vnet

= vupl − vdwl = Ab+ − Ab−

A+ b− − A− b+

be, respectively, the total output (upload) rate at the nodes, the total input (download) rate at the nodes, and the net outflow at nodes.

The requirements of a maximum download rate of v̄dwl = 20 Mbit/s and maximum upload rate v̄upl = 10 Mbit/s are imposed as linear constraints on the rate vectors b+ , b− A+ b− − A− b+ ≤ 20 · 1,

A+ b+ − A− b− ≤ 10 · 1.

The node flows f i are specified as follows: f 1 = 9, f 5 = −9 (node 1 transmits to node 5 at rate 9), f 2 = 8, f 6 = −8 (node 2 transmits to node 6 at rate 8), and f i = 0 for all other nodes. The objective to be minimized is the average congestion level, which is proportional to 8

8

i =1

i =1

J = ∑ ci = ∑ max(0, bi+ + bi− − 4), so the problem is written as min

∑8i=1 max(0, bi+ + bi− − 4)

s.t.:

A+ b− − A− b+ ≤ 20 · 1

b+ ,b−

A+ b+ − A− b− ≤ 10 · 1 Ab+ − Ab− = f

b+ ≥ 0, b− ≥ 0.

This problems can be recast in the form of an LP, by epigraphic reformulation: min

∑8i=1 zi

s.t.:

z ≥ 0, b+ + b− − 4 · 1 ≤ z

b+ ,b− ,z

A+ b− − A− b+ ≤ 20 · 1 A+ b+ − A− b− ≤ 10 · 1 Ab+ − Ab− = f

b+ ≥ 0, b− ≥ 0.

The exercise can be solved numerically via the following CVX code.

We obtained optimal average congestion level of 1.3750, and rates b+

= [0 7 3 7 7 0 0 2.9233]

b−

= [1 0 0 0 0 4.7187 5.2813 0.2047].

185 %% CVX code for network congestion exercise A=[1 -1 0 0 0 0;0 1 0 -1 0 0;1 0 0 -1 0 0;...

1 0 -1 0 0 0;0 0 1 0 -1 0;0 0 0 -1 1 0;...

0 0 0 -1 0 1;0 0 0 0 1 -1]’; [n,p]=size(A); Ap = max(A,0); Am = min(A,0); vmax_dnl=20; vmax_upl=10; f=zeros(n,1); f(1) = 9; f(5)=-f(1); f(2) = 8; f(6)=-f(2); % cvx_begin variables bp(p) bm(p) z(p) v_upl = Ap*bp - Am*bm; v_dnl = Ap*bm - Am*bp; v_net = v_upl - v_dnl; minimize ( sum(z) ) subject to bp >= 0; bm >= 0; v_net == f; % flow balance v_upl <= vmax_upl; v_dnl <= vmax_dnl; bp+bm-4 <= z; 0 <= z; cvx_end

Exercise 16.2 (Design of a water reservoir) We need to design a water reservoir for water and energy storage, as depicted in Figure 16.18.

The concrete basement has square section of side length b1 and height h0 , while the reservoir itself has square section of side length b2 and height h. Some useful data is reported in Table 16.6.

The critical load limit Ncr of the basement should withstand at least twice the weight of water. The structural specification h0 /b12 ≤ 35 should hold. The form factor of the reservoir should be such that 1 ≤ b2 /h ≤ 2. The total height of the structure should be no larger than 30 m. The total weight of the structure (basement plus reservoir full of water) should not exceed 9.8 × 105 N. The problem is to find the dimensions b1 , b2 , h0 , h such that the potential energy Pw of the stored water is maximal (assume Pw = (ρw hb22 )h0 ). Explain if and how the problem can be modeled as a convex optimization problem

Figure 16.18: A water reservoir on concrete basement.

186 and, in the positive case, find the optimal design.

Value 9.8 30 × 109 10 × 103 25 × 103 b14 /12 2 π JE/(2h0 )2

Quantity g E ρw ρb J Ncr

Units m/s2 N/m2 N/m3 N/m3 m4 N

Description gravity acceleration basement long. elasticity modulus specific weight of water specific weight of basement basement moment of inertia basement critical load limit

Solution 16.2 (Design of a water reservoir) All specs in this problem can be formulated in the form of monomial inequalities. The problem can thus be posed as a GP, as shown in the following CVX code. Using the provided numerical data, we obtained an optimal design with b1 = 0.78,

b2 = 4.7,

h0 = 21.38,

h = 2.95.

%% water reservoir design exercise ro_w = 10e3; % N/m^3 specific weigths ro_s = 25e3; Y=30e9; % elastic modulus in N/m^2 g = 9.8; Wlimit = 1e5*g; % max weight cvx_begin gp variables h0 h b1 b2 m_w = ro_w*b2^2*h; % weight of water m_s = ro_s*h0*b1^2; % weigth of basement E = m_w*h0; % energy maximize (E) subject to % structural limit J = b1^4/12; %basement moment of inertia Ncr = pi^2*J*Y/(2*h0)^2; % structural limit Ncr >= 2*m_w; % form factor of reservoir b2 >= h; b2 <= 2*h; % total heigth h0+h <=30; % h0 <= 35*b1^2;

Table 16.6: Data for reservoir problem.

187 m_w+m_s <= Wlimit; cvx_end

Exercise 16.3 (Wire sizing in circuit design) Interconnects in modern electronic chips can be modeled as conductive surface areas deposed on a substrate. A “wire” can thus be thought as a sequence of rectangular segments, as shown in Figure 16.19.

We assume that the lengths of these segments are fixed, while the widths xi need be sized according to the criteria explained next. A common approach is to model the wire as the cascade connection of RC stages, where, for each stage, Si = 1/Ri , Ci are, respectively, the conductance and the capacitance of the i-th segment, see Figure 16.20.

The values of Si , Ci are proportional to the surface area of the wire segment, hence, since the lengths `i are assumed known and fixed, they are affine functions of the widths, i.e., (0)

Si = Si ( xi ) = σi (0)

+ σi xi ,

(0)

Ci = Ci ( xi ) = ci

+ ci xi ,

(0)

where σi , σi , ci , ci are given positive constants. For the three-segment wire model illustrated in the figures, one can write the following set of dynamic equations that describe the evolution in time of the node voltages vi (t), i = 1, . . . , 3:       S1 S1 0 0 C1 C2 C3       0  v ( t ) +  0  u ( t ).  0 C2 C3  v̇(t) = −  −S2 S2 0 0 − S3 S3 0 0 C3 These equations are actually expressed in a more useful form if we introduce a change of variables   1 0 0   v(t) = Qz(t), Q =  1 1 0  , 1 1 1 from which we obtain

where

 S1   C( x )ż(t) = −S( x )z(t) +  0  u(t), 0



C1 + C2 + C3 .  C( x ) =  C2 + C3 C3



C2 + C3 C2 + C3 C3

 C3  C3  , C3

.

S( x ) = diag (S1 , S2 , S3 ) .

Clearly, C( x ), S( x ) are symmetric matrices whose entries depend affinely on the decision variable x = ( x1 , x2 , x3 ). Further, one may

Figure 16.19: A wire is represented as a sequence of rectangular surfaces on a substrate. Lenghts `i are fixed, and the widths xi of the segments are the decision variables. This example has three wire segments.

Figure 16.20: RC model of a threesegment wire.

188 observe that C( x ) is nonsingular whenever x ≥ 0 (as it is physically the case in our problem), hence the evolution of z(t) is represented by (we next assume u(t) = 0, i.e., we consider only the free-response time evolution of the system) ż(t) = −C( x )−1 S( x )z(t).

The dominant time constant of the circuit is defined as τ=

1 , λmin (C( x )−1 S( x ))

and it provides a measure of the “speed” of the circuit (the smaller τ, the faster is the response of the circuit).

Describe a computationally efficient method for sizing the wire so to minimize the total area occupied by the wire, while guaranteeing that the dominant time constant does not exceed an assigned level η > 0.

Solution 16.3 (Wire sizing in circuit design) The request that τ ≤ η is written as 1 ≤ η, τ= λmin (C( x )−1 S( x )) which holds if and only if

λmin (C( x )−1 S( x )) ≥

1 . η

This condition is in turn equivalent to λi (C( x )−1 S( x )) ≥

1 , η

∀i,

where λi denotes the i-th eigenvalue of the symmetric matrix C( x )−1 S( x ).

This latter condition can be expressed in matrix inequality form as

C( x )−1 S( x )

1 I, η

which is finally written as an LMI in the variable x:

S( x )

1 C( x ). η

The wire-sizing problem can thus be cast (and hence efficiently solved) in SDP format as follows: min

∑3i=1 `i xi

s.t.:

x≥0

x

S( x ) η1 C( x ).