Design of Asymptotically Optimal Improper Constellations With Hexagonal Packing

This paper addresses the problem of designing asymptotically optimal improper constellations with a given circularity coefficient (correlation coefficient between the constellation and its complex conjugate). The designed constellations are optimal in the sense that, at high signal-to-noise-ratio (SNR) and for a large number of symbols, yield the lowest probability of error under an average power constraint for additive white Gaussian noise channels. As the number of symbols grows, the optimal constellation is the intersection of the hexagonal lattice with an ellipse whose eccentricity determines the circularity coefficient. Based on this asymptotic result, we propose an algorithm to design finite improper constellations. The proposed constellations provide significant SNR gains with respect to previous improper designs, which were generated through a widely linear transformation of a standard $M$ -ary quadrature amplitude modulation constellation. As an application example, we study the use of these improper constellations by a secondary user in an underlay cognitive radio network.


I. INTRODUCTION
T He statistical analysis of improper complex-valued ran- dom signals has a long story.Probably, Wooding [1] was among the first to fully characterize the second-order statistical properties of Gaussian random vectors.Later, Brown and Crane introduced the complementary variance, i.e., the correlation of a complex signal and its complex conjugate, and the concept of widely linear transformation (WLT) in [2].This topic received renewed interest in the nineties with the fundamental studies by Picinbono and co-workers [3]- [5], as well as with works by Neeser and Massey in [6].
Improper signals are common in communications since many important digital modulation schemes yield improper complex baseband signals.Examples are binary phase shift keying (BPSK) or Gaussian minimum shift keying (GMSK) [7].The design of widely linear receivers that exploit the impropriety of the received signal has been addressed in [8], [9].
More recent works consider the deliberate transmission of improper signals as a novel way to handle interference in multiuser systems.For instance, the use of improper input signals has been shown to provide benefits in several interferencelimited wireless networks, such as interference channels [10]- [12], Z-interference channels [13], underlay cognitive radio (UCR) systems [14], [15], and relay channels [16], [17].
Recent results for the two-user single-input single-output (SISO) interference channel show that the superiority of improper signaling only holds for pure transmission strategies without time-sharing [18].However, the time-sharing strategies discussed in [18] require the power to be averaged over several transmit strategies, which might be impractical or even infeasible for wireless systems.Therefore, the development of new improper signaling schemes for interference-limited networks, as well as the analysis of their benefits, are topics of great interest to researchers.
From an information-theoretic standpoint proper Gaussian distributed signals are optimal, i.e., they are maximum-entropy signals under a given power constraint (see Theorem 4.4 in [19]), and they maximize the mutual information under circularly symmetric additive white Gaussian noise (AWGN) channels [20].In addition, Theorem 4.8 in [19] proves that, if the correlation coefficient between a complex random signal and its conjugate (the so-called circularity coefficient) is fixed, then the distribution that maximizes the differential entropy is an improper Gaussian.In this sense, (improper) Gaussian signals are also optimal when we restrict the analysis to improper signals.Nevertheless, the unbounded peak-to-average power ratio of Gaussian signals makes them unsuitable for practical communication systems.It is therefore relevant to investigate the design of improper discrete constellations with limited peak-to-average power ratios that may be used in practice instead of the ideal Gaussian ones.
Several works have considered the optimization of improper signaling schemes with finite discrete modulations.In [21] the authors consider the K-user interference channel where all users transmit conventional proper modulation formats and find optimal widely linear precoders and decoders to minimize the symbol error rate.Widely linear transceivers have been also used to exploit the impropriety of BPSK or GMSK signals in [8], [9], [22], to mention a few representative works.All these works, however, use standard modulation formats and have no control over the circularity coefficient of the transmitted constellation.
The design of discrete constellations with a prescribed circularity coefficient was addressed for the first time in [23].The improper constellations in [23] are generated through a WLT of a standard M-ary quadrature amplitude modulation (M-QAM) constellation.The WLT is optimized to maximize the minimum Euclidean distance between signal points of the improper constellation, while preserving the signal energy.However, the improper constellations described in [23] are not optimal for two reasons.In the first place, they have a shaping loss of 1.53 dB with respect to the improper Gaussian signaling [23], which coincides with the shaping loss of proper square M-QAM modulations [24].In the second place, they cannot guarantee an optimal (hexagonal) packing of the signal points of the improper constellation.To solve these two limitations, in this paper we address the design of improper constellations with hexagonal packing and an elliptical envelope that minimize the symbol error rate (SER).Asymptotically, as the number of constellation points M → ∞, the optimal constellation is the intersection of the hexagonal lattice with an ellipse whose eccentricity determines the circularity coefficient.From this result, we propose an algorithm to design improper signal constellations for finite M. Using maximum likelihood (ML) decoding, the proposed constellations provide significant gains in signal-to-noise-ratio (SNR) with respect to the WLT-based designs of [23], thus closing the gap with the ideal improper Gaussian signals.
As an illustrative application, this paper considers a UCR scenario where a secondary user (SU) is allowed to access the primary user (PU) channel as long as a rate requirement for the PU is satisfied.Exploiting the results in [14], the SU knows when to transmit improper signals and with which circularity coefficient.Then, the optimal improper constellation for a given spectral efficiency can easily be designed and used for transmission.Depending on the level of interference, the optimal improper constellation could provide a SER reduction of several orders of magnitude compared to the use of proper constellations, which must be transmitted with less power in order to meet the PU rate requirement.
The rest of the paper is organized as follows.Section II describes the system model, motivates the problem, and discusses previous work regarding the design of improper constellations.Section III briefly reviews the concepts of shaping, coding and power gains, which will be used as figures of merit to assess the quality of the designed constellations.The proposed method to design improper signal constellations, carved from the hexagonal lattice, is described in detail in Section IV.The SER performance of the designed constellations is evaluated, for both AWGN and UCR channels, by means of Monte Carlo simulations in Section V. Finally, the main conclusions are summarized in Section VI.

A. Notation
Scalar values are denoted in lowercase letters x.Scalar random variables are denoted by uppercase letters X. Vectors and matrices are denoted by lowercase boldface letters, x, and uppercase boldface letters, X, respectively.For complex variables (deterministic or random), we will use the following representations: real and imaginary parts, x = x I + j x Q , in magnitude and phase, x = |x|e j∠x being |x| = x 2 I + x 2 Q and ∠x = arctan The complex conjugate is symbolized by an asterisk, i.e., x * = x I − j x Q = |x|e −j∠x .E {X } stands for the expected value or the mean value of X. x ∼ CN(0, R) indicates that x is a complex circular Gaussian random vector of zero mean and covariance matrix R. Z notes the set of integers and R stands for the set of real numbers.Moreover, the notations Z n and R n represent the Cartesian product of n copies of Z and R, respectively, i.e., Z n = (z 0 , z 1 , . . ., z n−1 ) T z 0 , z 1 , . . ., z n−1 ∈ Z and R n = (r 0 , r 1 , . . ., r n−1 ) T r 0 , r 1 , . . ., r n−1 ∈ R .

II. BACKGROUND AND PROBLEM STATEMENT
We start this section by briefly reviewing some preliminaries about improper random variables that will be used throughout the paper.Then, we present the system model and state the problem.Finally, we also describe a suboptimal design recently proposed in [23] that will be used for comparison.

A. Improper Random Variables
The next definitions [6], [7], characterize an improper random variable, which is assumed to be zero mean.Definition 1.A complex-valued random variable X is said to be circular (or circularly symmetric) if e jθ X has the same probability distribution as X for all real θ.Definition 2. A complex-valued zero-mean random variable X is said to be proper if X is uncorrelated with its complex conjugate, i.e., E X 2 = 0. Otherwise it is called improper.
A random variable X that is circular is also proper, but the converse is not necessarily true.However, for complex Gaussian random variables, propriety implies circularity, so both terms can be used interchangeably.Definition 3. The complementary variance of a zero-mean complex random variable X is defined as σ2 X = E X 2 .Furthermore, σ 2 X and σ2 X are a valid pair of variance and complementary variance if and only if σ 2 X ≥ 0 and | σ2 X | ≤ σ 2 X .Definition 4. The circularity coefficient of a zero-mean complex random variable X, which measures the degree of impropriety, is defined as The circularity coefficient satisfies 0 ≤ κ ≤ 1.If κ = 0, then X is proper, otherwise is improper.If κ = 1 we call X maximally improper.
Any improper random variable V can be generated from a proper random variable X by a WLT: where the coefficients h 1 and h 2 are chosen to achieve the desired variance and complementary variance.The interested reader is referred to [7] for a comprehensive treatment of the topic of improper and non-circular signals and their processing.

B. System Model and Problem Statement
Let us consider a complex AWGN channel where the noise N ∼ CN(0, 1) is circular and Gaussian, V is the zero-mean and unit variance transmitted signal, and the channel power gain g is a deterministic value, which we take as real without loss of generality and represents the SNR.
In this paper, we consider the design of improper finite constellations with a prescribed circularity coefficient, κ, in AWGN channels.As we will see in Section V-C, one scenario where this problem arises is underlay cognitive radio, where the AWGN channel (3) models the link for a secondary (unlicensed) user.
Notice that, for an improper input V uncorrelated with the noise N (and also with N * ), the channel output is also improper with variance E |Y | 2 = 1 + g, and circularity coefficient κ y = κ g/(1 + g).
Differently from the widely-studied proper constellations, such as standard M-QAM, optimal design of their improper counterparts still remains an open problem.In this work, we address the design of improper two-dimensional (2D) constellations with asymptotically optimal packing and envelope shaping properties.We denote a 2D constellation as a complex random variable V = V I + jV Q taking values in a M-ary alphabet V = v i = v I i + jv Q i , i = 0, 1, . . ., M − 1 .Alternatively, we will also use the real composite notation for the constellation signal points: T .We will assume that the signaling is equiprobable over the signal set, that is, the symbols are taken with probabilities 1/M.This assumption is in general suboptimal in terms of mutual information [25], but it is common in most practical implementations.The equiprobability assumption lets us focus on the selection of the set of signal points.In the sequel, we will use the term constellation for both the complex random variable V and the M-ary alphabet V.
The designed constellations are optimal in the sense that, at high signal-to-noise-ratio and for an asymptotically large number of symbols, yield the lowest probability of error under an average power constraint for AWGN channels.Since for any discrete constellation the SER, as well as the mutual information or the minimum mean-squared error (MMSE), all have an asymptotic behavior proportional to Q √ gd/2 , where , the problem is equivalent to finding the 2D signal set of size M that, for a given channel power gain g, maximizes d.

C. Widely Linear Improper Constellations
Recently, a family of improper discrete constellations, which are generated by a WLT of a standard proper M-QAM constellation, has been proposed in [23].More specifically, starting from a standard unit energy M-QAM constellation X = X I + j X Q , the improper constellation with a prescribed circularity coefficient κ is generated as where h 1 = 0.5(1 + α) and h 2 = 0.5(1 − α)e jφ , where α = (1 − κ 2 ), and the optimal phase φ * ∈ [0, π/2] is chosen to maximize the minimum distance between constellation points.For 0 ≤ κ ≤ 0.5 the optimal phase in (4) for an arbitrary M is φ * = π/2.On the other hand, when 0.5 ≤ κ ≤ 2 √ 5 the optimal phase satisfies the nonlinear equation whose solution is straightforwardly obtained as Meanwhile, for M = 4 it is possible to calculate φ * for 0.5 ≤ κ ≤ 1 using ( 6), the φ * for M > 4 and 2 √ 5 < κ ≤ 1 must be found by an iterative algorithm that explores all pairwise distances between constellation points.
The widely linear transformation (4) can be expressed in matrix form as The WLT constellation generated as ( 7) is easy to obtain and allows us to derive linear MMSE-like receivers (or to apply successive cancelation techniques).Nonetheless, even if φ * is used, the produced improper constellation is not optimal from an error rate point of view.In this paper, we propose another improper constellation with lower probability of error in AWGN channels, as well as better shaping, coding, and power gains.These concepts are briefly reviewed in the next section.

III. SHAPING, CODING, AND POWER GAINS A. 2D lattices
A signal constellation is commonly defined as a set of points of a repetitive grid or lattice, denoted here as Λ, that are within a selected boundary.An n-dimensional lattice is defined as a discrete subset of R n that has a group structure under common vector addition [27], [28].
In this work, we focus on 2-dimensional (2D) lattices which are of interest to define M-QAM constellations.Two typical examples are the square lattice The minimum distance between any two lattice points for both Z 2 and H 2 equals one, i.e., d min Z 2 = d min (H 2 ) = 1.In general, however, we will consider arbitrary scaled versions of the aforementioned lattices with minimum distance denoted as d = d min (Λ).It is well known that the optimal 2D lattice for both coding and quantization purposes (i.e., considering the packing efficiency and covering efficiency of the lattice) is the hexagonal lattice [28].Therefore, in this paper we focus on this lattice to design the improper constellations.For higher-dimensional lattices, the so-called Voronoi constellations, based on partitions of N-dimensional lattices are known to achieve good shaping gains [29].
The Voronoi region of a 2D lattice point x is the set of all points in R 2 that are closer to x than any other lattice point [27].Due to the repetitive structure of the lattice, the Voronoi regions of its points are alike and, since they are disjoint, induce a partition or tiling of R 2 .The volume (or surface in R 2 ) of the Voronoi region defines the fundamental volume of a lattice, denoted as Vol (Λ).For instance, the Voronoi region for Z 2 is a square of side d = 1: Vol Z 2 = d 2 = 1; and for 2 .The result of a lattice, Λ, shifted (or translated) by a vector c Λ, denoted by Λ + c, is not another lattice because the resultant grid does not include point 0 and, consequently, the group structure is lost.Nonetheless, the translated lattice preserves the periodic structure and thus the geometrical properties of Λ.In this paper, we consider 2D constellations (square constellations) and from

B. Shaping and Coding Gains
Since there exists one lattice point per fundamental volume, its reciprocal is the number of lattice points per unit volume and, when properly scaled, is a measure of the packing of the lattice points.Specifically, the coding gain of a 2D lattice is defined as [30] γ c (Λ) ≈ 1.1547.A signal constellation may be shaped by choosing the points of Λ or Λ + c that are within some region R, i.e.: C (Λ, R) = (Λ + c) ∩ R. R is commonly referred as the envelope of the constellation C (Λ, R) (or simply C).A proper selection of the shifting vector c and of the constellation envelope R provides certain desirable properties to C, such as: zero mean, number of signals that is a power of 2, or increased power efficiency.
The shaping gain of a 2D region R is defined as [30] γ s (R) where Vol (R) is the volume of the region R and ε (R) denotes the average energy (or the non-centered second order moment) of a uniformly distributed random variable x over the region R: The factor 6 that appears in the denominator of (9) produces a unit value shaping gain for an √ M d-side square region R that is the envelope of a standard square M-QAM constellation.
Given an M-ary discrete constellation C with sufficiently large M, ε (R) approaches the average energy of C, ε (C), which is known as the continuous approximation Neglecting the constellation borders effect, it is also possible to approximately compute M by means of the quotient between the volume of R and the fundamental volume:

C. Constellation Figure of Merit and Power Gain
The aggregate effect of the coding and shaping gains of a constellation C is measured by the Constellation Figure of Merit [27], which for a 2D constellation is defined as For large M, we can use the approximations given in ( 11) and ( 12) to write CFM (C) as which can be finally expressed in terms of the coding and shaping gains as where CFM 0 = 6 M is the approximate value for large M of the CFM of a square M-QAM constellation CFM (M-QAM) = 6 M−1 .The CFM of a unit energy constellation equals d 2 , thus, the asymptotic behavior of the error probability is proportional to Q g CFM (C)/2 .As a consequence, it is possible to define the asymptotic power gain of constellation C 1 over constellation C 2 , both with M symbols, as the quotient Example 1.Let C 1 be a M-QAM constellation carved from the hexagonal lattice with a circular envelope for which and γ s (R 2 ) = π 3 ; and let C 2 be a square M-QAM constellation whose coding and shaping gains are, respectively, γ c Z 2 = 1 and γ s (R 1 ) = 1, then, it follows that: Example 2. Let C 1 be a square M-QAM constellation whose minimum distance is noted as d; and let C 2 be the optimal WLT of C 1 , as described in [23], with circularity coefficient (see (6) for φ * ).

A. Asymptotic case
In the proper case, it is well-known that as ).This result has been proven in [31] and follows from standard results in [32].
In the improper case, a similar result holds, and the optimum constellation is the intersection of the hexagonal lattice with an ellipse, R = E, that determines the prescribed circularity coefficient κ, i.e., C(H 2 , E).The optimal improper constellation preserves the coding gain of its proper counterpart, because both are carved from the hexagonal lattice, and it only suffers from a reduction on the shaping gain that is necessary to achieve κ.Specifically, we have To determine the ellipse E we have the following proposition.
T be a continuous random variable uniformly distributed over an elliptic domain E ⊂ R 2 whose semi-major axis and semi-minor axes are denoted as a and b, respectively.Then, the circularity coefficient of V = Proof.See Appendix A.
Note that the major axis of E may form an arbitrary angle θ with the horizontal axis.In the sequel, we will use E 0 when θ = 0 (horizontal ellipse).Note also that when a = b the ellipse degenerates on a circumference and the proper case κ = 0 is recovered.On the other hand, the maximally improper case κ = 1 is achieved when b = 0 and then E becomes a linear segment of length 2a.Since maximally improper constellations reduce trivially to rectilinear M-PAM constellations, which can easily be designed, in the sequel we consider the design of improper constellations with κ ∈ [0, 1).
Considering the expression of φ * given in Subsection II-C, it is possible to obtain an analytical expression of the asymptotic power gain of the optimal improper constellation over the WLT of a square M-QAM for κ ∈ 0, 2 √ 5 . Recalling the result of Example 1 and considering (17), it follows that the asymptotic power gain of the optimal improper constellation V (carved from the hexagonal lattice bounded by an elliptic envelope with circularity coefficient κ) over a square M-QAM is In addition, using the result of Example 2 and (19), the asymptotic power gain of V over the optimal WLT of the square M-QAM constellation, W, may be approximated by Fig. 1 plots, in logarithmic units, the power gain of V over W.
The minimum value of the asymptotic power gain g V ,W that appears at κ = 0.5 is because for such circularity coefficient .
the lattice from which both improper constellations are carved is hexagonal and thus, their coding gain is 2/ √ 3 and they only differ on their envelope shape that yields a power gain of π/3 (0.2 dB).
It is interesting to note that due to the optimal packing properties of the hexagonal lattice the optimal improper constellation V asymptotically outperforms the WLT of a square M-QAM for any circularity coefficient κ.Moreover, (19) shows that V even outperforms the square M-QAM for κ ∈ [0, 0.5622].

B. Constellation Design for Finite M
For large M, the volume of the ellipse E, Vol(E), can be approximated by the sum of the volumes of the Voronoi regions of the M signals, that is From ( 18) and ( 21) it is straightforward to obtain the semiaxes of the ellipse as For large M, C(H 2 , E 0 ) with E 0 being a horizontal ellipse with semi-axes given by ( 22) and ( 23) is the optimal improper constellation with prescribed circularity coefficient κ.Nevertheless, the accuracy of the continuous approximation, used to derive ( 22) and ( 23), degrades as M decreases and, thus, the following problems may appear: • There is no guarantee that E 0 with semi-axes given by ( 22) and ( 23) will enclose exactly M signal points of the hexagonal lattice.• Even if E 0 encloses exactly M signal points, the actual circularity coefficient of the constellation may differ from the prescribed κ.
The proposed algorithm to produce improper constellations solves these issues in two stages.During the first stage, a purely hexagonal M-ary constellation V is found by enclosing exactly M hexagonal lattice points within a rotated version of E 0 .Since the circularity coefficient of V, κ V , might not be the targeted one κ, the second stage proposes a simple widely linear transformation that attains the prescribed κ with a small distortion of the hexagonal lattice.These two stages and the final procedure are described in detail throughout the next paragraphs.
1) Stage I. Finding a pure hexagonal constellation V: The elliptic domain E 0 defined by ( 22) and ( 23) and whose semi-major axis is aligned with the I axis may not enclose M points of the translated hexagonal lattice.The idea to solve this problem is to find a rotated version of E 0 that contains exactly M lattice points.We use the lattice shifting vector c = 1  2 , 0 T because the translated lattice guarantees zero-mean symmetric constellations with an even number of constellation points.This simplifies this stage and guarantees asymptotical optimality.Due to symmetry, the rotation angle θ may be confined within the interval 0, π 2 .Specifically, the proposed method (sketched in Algorithm 1) finds all L candidate constellations, V i , whose envelope is E 0 rotated by θ[i], i = 1, 2, . . ., L. To explore all the possible constellations that a counterclockwise rotated version of the ellipse may produce, it is necessary to carefully select the ellipse rotation angles.For each lattice point that might be part of a candidate constellation, v n ∈ H 2 + 1 2 , 0 T v n ≤ a , we define an input and an output angle.The input angle of v n , noted as θ n in , is the smallest non-negative rotation angle for which v n lays within the rotated ellipse.An initially considered v n is disregarded if θ n in > π 2 because it cannot belong to any rotated ellipse.In addition, for each v n it is possible to find an interval of rotation angles [θ n in , θ n in + ∆θ] for which v n is within the rotated ellipse, then we define ∆θ n = max {∆θ} and the output angle of v n as θ n out = θ n in + ∆θ n .If θ n in = 0 and θ n out > π 2 then v n lays within all the rotated versions of the ellipse.Then, the set of considered rotation angles ({θ[i], i = 1, 2, . . ., L}) can be formed by the set of sorted distinct input and output angles for all eligible lattice points.From all the M-ary candidate constellations, Algorithm 1 selects the one, V, that maximizes the minimum constellation distance d.
2) Stage II.Applying a WLT over V: Algorithm 1 provides a constellation V carved from the hexagonal lattice and whose circularity coefficient, κ V , approximates the targeted κ.In this second stage of the method, the pure hexagonal constellation V is widely linear transformed if κ V is not sufficiently close to κ (|κ V − κ| > ). is a parameter that measures the maximum tolerable error on the obtained circularity coefficient.The WLT Algorithm 1: Design of a hexagonal M-ary constellation with circularity coefficient κ V ≈ κ.
Input: M and κ; Output: V, d, and κ V ; set ; of unit energy M-PAM*/; define E 0 (set a, as in (22), and b, as in ( 23)); define θ n in and θ n out for each with the desired circularity coefficient κ.A large value of can be used to avoid the WLT of V, hence preserving the optimal packing properties of the hexagonal lattice.A small value of should be used when it is necessary to obtain, with high accuracy, a constellation with a prescribed circularity coefficient.
Based on the assumption that κ V will be not too far from the desired κ and that σ 2 V = κ V (note that in Algorithm 1 the phase of the complementary variance of the energy normalized output V is set to zero), in Appendix B we derive the following WLT for this problem where φ = 0 for κ > κ V , and φ = π for κ < κ V ; and . Notice that h 1 and h 2 are both real scalars for the proposed WLT.
The proposed WLT requires no optimization and slightly distorts the hexagonal lattice.To show the latter statement, let v m = v I m + jv Q m and v n = v I n + jv Q n be two arbitrary points of the original constellation V with squared distance Applying the proposed WLT we get the transformed points whose squared distance can be expressed as Taking into account the definition of h 1 and h 2 in ( 24) and ( 25), respectively, it is easy to get Since we are assuming that κ V is close to κ, the first term in the right hand side of ( 28) is typically much smaller than the second one.This, in turn, means that the squared distances are only slightly modified by the proposed WLT and, therefore, the quasi-hexagonal constellation V is a barely distorted version of the starting hexagonal constellation V.
Figures 2 and 3 show a graphical comparison between the hexagonal, V, and the quasi-hexagonal, V , 16-ary and 64-ary constellations for a target κ = 0.4.In both cases, the distortion of the constellation provoked by the WLT is negligible and the power efficiency loss is minor.The power gain of the 16-ary hexagonal constellation over its quasi-hexagonal counterpart is 0.22 dB.The same result for the 64-ary constellation is just 0.01 dB.In general, for larger values of M the distortion and the power efficiency loss with respect to the hexagonal constellation diminishes because κ and κ V are usually closer, as shown in the example.
3) Final algorithm: Stages I and II combined yield either a quasi-hexagonal or a hexagonal constellation with a circularity coefficient that matches, within the maximum tolerance , the target κ.This combination may be summarized as follows.First, Algorithm 1 is applied to obtain a purely hexagonal M-QAM constellation V with an elliptical envelope of prescribed eccentricity.The circularity coefficient of V, noted as κ V , approximates κ due to the definition of the intersecting ellipse semi-axes according to (22) and (23).If the error of this approximation is larger than the established , we apply the WLT detailed in Stage II.
Nonetheless, the constellation V that eventually maximizes the minimum distance of V is not always obtained using an ellipse of circularity coefficient κ.The final algorithm takes this issue into account and takes advantage of the fast computation of the minimum distance of the transformed constellation V to expand the search.In Stage I, this expanded search uses several values for the target circularity coefficient around the target one κ.The result of this search is a hexagonal or quasi-hexagonal constellation that, achieving a circularity   coefficient within the prescribed precision , provides the largest minimum distance among all explored constellations.Algorithm 2 sketches the final procedure.
From a computational viewpoint, an efficient implementation of Stage I has complexity O (L log (L)), with L being the number of distinct input/output angles, whereas the complexity of Stage II is O(M).

V. SIMULATION RESULTS
In the sequel, we will note a general square M-QAM constellation as X and its optimal WLT, described in [23], as W. In addition, we will write V for a general improper M-QAM constellation obtained by means of the procedure Algorithm 2: Design of an improper M-ary constellation with circularity coefficient κ.
Input: M and κ; Output: V and d; set ; define the search interval around κ, denoted as κ I ; use Algorithm 1 to obtain V i , d i , and set h 1 , as in (24), and h 2 , as in (25); /*Output quasi-hex.constellation*/ end end end described in this paper.Finally, we are using = 10 −8 as the maximum error on the obtained circularity coefficient.

A. A first example
Fig. 4a shows an improper constellation with M = 256 signal points designed with the proposed algorithm.The quasihexagonal lattice provides a nearly optimal packing, while the elliptic boundary optimally shapes the constellation for a prescribed circularity coefficient κ = 0.95.For comparison, the constellation designed with the optimal WLT of the 256-QAM is shown in Fig. 4b.In this case, the minimum distances of each constellations are: d min (W) = 0.0860 and d min (V ) = 0.0942 (note that d min (X) = 0.1534).Thus, the power gain of the quasi-hexagonal constellation over the optimal WLT of the 256-QAM is about 0.8 dB.

B. SER evaluation
In this subsection, we study the SER performance of the proposed improper constellations, V , by means of 10 000 simulations each one considering the decoding of 10 000 symbols.The receiver applies the optimal maximum likelihood decoder for this problem, which amounts to finding the closest lattice point to a given noisy observation [33], [34].
We compare the performance of V with the proper square M-QAM constellation, X, and its improper counterpart obtained by means of the WLT, described in [23], with both ML (WLT-ML) and MMSE (WLT-MMSE) decoding.Fig. 5 shows the results for improper constellations with M = 16 and κ = 0.8.For high SNRs the proposed quasi-hexagonal design provides an improvement of about 0.5 dB with respect to the WLT design using ML decoding, meanwhile the asymptotic result (see Fig. 1) is about 0.78 dB.The performance of the MMSE decoder for the WLT constellation degrades significantly in comparison to the optimal ML decoding due to noise enhancement.When the circularity coefficient is reduced to κ = 0.2 we obtain the curves shown in Fig. 6.Interestingly, in this case the proposed improper constellation even outperforms the proper 16-QAM constellation.Particularly in this case, the power gain is about 0.3 dB.
In the second example, we design constellations with M = 64 symbols.The results for κ = 0.8 and κ = 0.2 are shown in Figs.7 and 8, respectively.For high SNRs, the proposed quasi-hexagonal design for κ = 0.8 provides an improvement of about 0.6 dB with respect to the WLT design using ML decoding which is closer to the asymptotic result than for M = 16.For κ = 0.2 the proposed constellation provides a power gain about 0.5 dB over the proper 64-QAM constellation that also gets closer to the asymptotic result (0.74 dB) than for M = 16.In general, the power gain of V over W gets closer N ∼ CN (0, 1) Fig. 9. UCR channel model.The top link is the PU, whereas the bottom link is the SU, which may transmit improper signals.
decreased signal level and is thus implicit in the power gain g in (30).
In the UCR paradigm the unlicensed SU, represented by the bottom link in Fig. 9 and modeled in (30), is allowed to coexist with the PU (top link represented by ( 29)) as long as it ensures a minimum transmission rate for the PU.That is, the transmit power and signaling scheme of the SU must be selected to control the interference level at the primary receiver such that the PU's rate requirement is guaranteed.
In [14], it was shown that for improper Gaussian signaling improves the SU rate, while ensuring the PU a prescribed rate R, if If this condition is satisfied, the SU can transmit at a higher rate by increasing its transmit power and its circularity coefficient according to the expressions provided in [14].That is, the transmit power is increased over the transmit power used by the proper signaling scheme.Since improper Gaussian signals maximize the differential entropy among all random signals with the same κ, the rate of the PU, R, is still ensured if the SU uses an improper discrete modulation scheme instead of improper Gaussian signaling.The prescribed PU rate constraint has been chosen as α log 2 (1 + g p ), where 0 ≤ α ≤ 1 is called the loading factor and the second term is the channel capacity.
We consider a UCR system with α = 0.25 and equal channel power gains g p = g = 30 dB.The SU transmits a 64-QAM constellation.Fig. 10 shows the resulting SER achieved by the SU using the following three constellations: the conventional proper (or square) 64-QAM, the improper WLT design [23] with ML detection, and the proposed design.We observe that the improper modulation schemes provide significant SER reduction over the conventional QAM constellation.Furthermore, the proposed design outperforms the WLT-based design from [23], as suggested by the evaluation of Subsection V-B.This practical example also shows that the results and conclusions drawn from Gaussian signaling carry over to more realistic discrete modulation schemes.a given circularity coefficient under a given power constraint in AWGN channels.At high SNR the proposed method is asymptotically (as the number of symbols grows to infinity) optimal.The proposed algorithm for M-ary constellations consists of two stages.In the first stage we find a rotated ellipse that intersects the hexagonal lattice enclosing exactly M lattice points.The circularity coefficient of the constellation designed in this way approximates the desired one.To exactly achieve the prescribed circularity coefficient, in the second stage of the algorithm we apply a widely linear transformation with a slight distortion of the hexagonal lattice.A secondary user of an underlay CR network transmitting the designed constellations may achieve a significant SER reduction in comparison to the use of proper M-QAM constellations or previously proposed improper constellations.

APPENDIX A PROOF OF PROPOSITION 1
Let us define a continuous random variable v = V I ,V Q T that is uniformly distributed over an elliptic domain E 0 ⊂ R 2 whose semi-major axis is noted as a and whose semi-minor axis is noted as b.Without loss of generality we also assume that the semi-major axis of the ellipse is aligned with horizontal axis I.Under these assumptions, the circularity coefficient κ of a complex random variable constructed as V = V I + jV Q is defined as follows where f V (v) notes the probability density function of the random variable V at point v = v I + jv Q .Since V is uniformly distributed over its domain of definition, then f V (v) = 1 Vol(E 0 ) , where Vol (E 0 ) = abπ is the volume (surface, in this bidimensional case) of E 0 .As a consequence, equation (32) takes the form In order to solve the integrals that appear in (33) it is very convenient to use a variable change to convert the elliptic domain expressed in Cartesian coordinates, v I and v Q , into a unit circle expressed in polar coordinates, ρ and θ.In this manner, if we note as g v I , v Q a general scalar function depending on v = v I + jv Q then is is possible to write where ab is the Jacobian determinant of the vector function v I , v Q = F 1 x I , x Q = ax I , bx Q that transforms the unit circle into ellipse E 0 , and ρ is the Jacobian determinant of the vector function x I , x Q = F 2 (ρ, θ) = (ρ cos θ, ρ sin θ) that transforms polar coordinates into Cartesian ones.
Moreover, it is straightforward to express g v I , v Q = v 2 in polar coordinates where it has been considered that cos 2 θ = 1+cos 2θ (39)
VI. CONCLUSIONSMotivated by a problem in UCR networks, this paper has addressed the design of improper discrete constellations with

Fig. 10 .
Fig. 10.Comparison of the SER achieved by an SU in an UCR scenario using different 64-QAM constellation designs.The loading factor is α = 0.25 and the SNR is 30 dB.