ℓ0 and ℓ1-balls. In this exercise, we consider the...

$ \ell_0$ and $\ell_1$-balls. In this exercise, we consider the relationship between $\ell_0$ and $\ell_1$-balls, and prove a containment property related to the success of $\ell_1$-relaxation. For an integer $r \in\{1, \ldots, p\}$, consider the following two subsets: $$ \begin{aligned} & \mathbb{L}_0(r):=\mathbb{B}_2(1) \cap \mathbb{B}_0(r)=\left\{\theta \in \mathbb{R}^p \mid\|\theta\|_2 \leq 1, \text { and }\|\theta\|_0 \leq r\right\} \\ & \mathbb{L}_1(r):=\mathbb{B}_2(1) \cap \mathbb{B}_1(\sqrt{r})=\left\{\theta \in \mathbb{R}^p \mid\|\theta\|_2 \leq 1, \text { and }\|\theta\|_1 \leq \sqrt{r}\right\} \end{aligned} $$ Let $\overline{\text { conv }}$ denote the closure of the convex hull (when applied to a set). (a) Prove that $\overline{\text { conv }}\left(\mathbb{L}_0(r)\right) \subseteq \mathbb{L}_1(r)$. (b) Prove that $\mathbb{L}_1(r) \subseteq 2 \overline{\text { conv }}\left(\mathbb{L}_0(r)\right)$.

$ \ell_0$ and $\ell_1$-balls. In this exercise, we consider the relationship between $\ell_0$ and $\ell_1$-balls, and prove a containment property related to the success of $\ell_1$-relaxation. For an integer $r \in\{1, \ldots, p\}$, consider the following two subsets:

$$
\begin{aligned}
& \mathbb{L}_0(r):=\mathbb{B}_2(1) \cap \mathbb{B}_0(r)=\left\{\theta \in \mathbb{R}^p \mid\|\theta\|_2 \leq 1, \text { and }\|\theta\|_0 \leq r\right\} \\
& \mathbb{L}_1(r):=\mathbb{B}_2(1) \cap \mathbb{B}_1(\sqrt{r})=\left\{\theta \in \mathbb{R}^p \mid\|\theta\|_2 \leq 1, \text { and }\|\theta\|_1 \leq \sqrt{r}\right\}
\end{aligned}
$$

Let $\overline{\text { conv }}$ denote the closure of the convex hull (when applied to a set).
(a) Prove that $\overline{\text { conv }}\left(\mathbb{L}_0(r)\right) \subseteq \mathbb{L}_1(r)$.
(b) Prove that $\mathbb{L}_1(r) \subseteq 2 \overline{\text { conv }}\left(\mathbb{L}_0(r)\right)$.

Key Concepts

?0 Norm and Sparsity

The ?0 norm, though not a true norm, is used as a measure of sparsity by counting the number of nonzero components in a vector. It is central in applications such as sparse recovery and compressed sensing, where one seeks solutions with many zeros. Its combinatorial nature, however, makes optimization problems involving the ?0 norm generally NP-hard, motivating the study of relaxations and alternative formulations.

?1 Norm and Convex Relaxation

The ?1 norm is the sum of the absolute values of vector components and serves as a convex surrogate for the ?0 norm in optimization problems. It promotes sparsity while preserving the convexity of the problem, which is essential for tractable optimization. The ?1 norm therefore is widely used in regularization techniques and sparse estimation, as it often encourages solutions that have many zero entries.

?2 Ball and Norm Constraints

The ?2 ball refers to the set of vectors whose Euclidean norm is bounded by a specified radius. In many problems, imposing an ?2 constraint, such as ???? ? 1, controls the overall energy or magnitude of the vector. This normalization is common in various fields to ensure well-posedness and to reduce sensitivity to scale, and it often interacts with other norm-based constraints like those involving ?0 or ?1 norms.

Convex Hull and Closure

The convex hull of a set is the smallest convex set containing that set, and taking its closure ensures that it is a closed set. In the context of optimization, convex hulls are used to approximate non-convex sets by convex ones, facilitating the use of convex optimization techniques. This concept is crucial when deriving relationships and containment properties between sets defined through non-convex measures (like the ?0 norm) and their convex relaxations (like the ?1 norm).

Containment Relationships Between Norm Balls

Understanding the containment relationships between sets defined by different norm constraints, such as those involving ?0 and ?1 balls, is key to analyzing the success of convex relaxations. Specifically, proving that the convex hull of a non-convex ?0 ball is contained in or approximated by an ?1 ball (or vice versa) provides theoretical guarantees about how well the convex approximation mimics the behavior of the original non-convex set.

Transcript

Please give Ace some feedback