Audio Engineering Society
Convention Paper 9255 Presented at the 138th Convention 2015 May 7–10 Warsaw, Poland This paper was peer-reviewed as a complete manuscript for presentation at this Convention. This paper is available in the AES E-Library, http://www.aes.org/e-lib All rights reserved. Reproduction of this paper, or any portion thereof, is not permitted without direct permission from the Journal of the Audio Engineering Society.
Practical Considerations of Time-Varying Feedback Delay Networks Sebastian J. Schlecht1 , Emanu¨el A.P. Habets1 1
International Audio Laboratories Erlangen, A joint institution of the University of Erlangen-Nuremberg and Fraunhofer IIS, Erlangen, Germany
Correspondence should be addressed to Sebastian J. Schlecht (
[email protected]) ABSTRACT Feedback delay networks (FDNs) can be efficiently used to generate parametric artificial reverberation. Recently, the authors proposed a novel approach to time-varying FDNs by introducing a time-varying feedback matrix. The formulation of the time-varying feedback matrix was given in the complex eigenvalue domain, whereas this contribution specifies the requirements for real valued time-domain processing. In addition, the computational costs of different time-varying feedback matrices, which depend on the matrix type and modulation function, are discussed. In a performance evaluation, the proposed orthogonal matrix modulation is compared to a direct interpolation of the matrix entries.
1.
INTRODUCTION
Parametric artificial reverberation algorithms are commonly designed as linear time-invariant filters. The feedback delay network (FDN) is a general structure that has been the superior choice in recent years to generate artificial reverberation [1]. To further account for physical acoustic changes [2], to reduce the computational requirements [3] or to enhance the perceived quality [4], various techniques have been developed to incorporate temporal variations into artificial reverberation. In the context of reverberation enhancement systems, which are formally
related to FDNs [5], time-variation has been successfully employed to improve the system’s stability and coloration [6–9]. In [4], the authors proposed the use of time-varying feedback matrices in FDNs, and demonstrated that it provides an attractive alternative to time-varying delay lines or allpass filters. This contribution investigates practical considerations in parameterization and computational efficiency of the implementation of time-varying feedback matrices. The theory of time-varying feedback matrices in FDNs has been outlined by the authors in [4]. The
Schlecht AND Habets
Practical Time-Varying FDN
unitary matrices that were proposed in this work ensure the unconditional stability even under arbitrary time-variation, which in general cannot be ensured for other time-variation FDN methods. To avoid audible parameter transitions, a technique to incrementally evolve the feedback matrix was presented. Hereby, a formulation of the feedback matrix in the eigenvalue domain proved useful. This contribution is dedicated to the investigation of the practical considerations one has to make in the implementation of a time-varying feedback matrix. In Section 2, we give a brief summary on the theory of time-varying FDNs as given in [4]. In Section 3, firstly, linear modulation functions are discussed for even time-variation. Secondly, the additional requirements on the general theory for time-domain processing, namely real valued as opposed to complex valued signals are presented. Thirdly, we give an entirely real parameterization of the orthogonal matrix space, which is needed for real valued processing. Fourthly, the developed real valued theory is applied to facilitate continuous change in the echo density of the FDN’s impulse response. In Section 4, the computational cost of the matrix modulation is discussed and an efficient matrix class and variation process is presented. In Section 5, the performance of the proposed orthogonal feedback matrix modulation is compared to a more straightforward method, where feedback matrices evolve directly on the matrix entries. 2.
TIME-VARYING FDN
2.1. General Structure The time-varying FDN is given in the time domain by the relation y(n) = cT s(n) + d x(n) s(n + m) = A(n) s(n) + b x(n),
(1)
where x(n) and y(n) are the input and output values at time instance n, respectively. The transpose operation is indicated by (·)T . The FDN dimension N is equal to the number of delay lines and to emphasize the dimension, we occasionally write N -FDN. The matrix A(n) ∈ CN ×N at time sample n is the time-varying feedback matrix, b and c are N × 1 column vectors for input and output gains and d is the
A(n)
Figure 1: Time-varying feedback delay network structure with time-varying feedback matrix as proposed in [4]. direct signal gain. In time domain processing, all coefficients are commonly chosen as real values, which is discussed in more detail in Section 3. The lengths of the N delay lines in samples are given by the vector m = [m1 , . . . , mN ]. The vector s(n) ∈ CN ×1 denotes the delay-line outputs at time n. The vector argument notation s(n + m) abbreviates the vector [s1 (n + m1 ), . . . , sN (n + mN )]. 2.2. Lossless FDN To design FDNs, it is common practice to first design the system to be lossless [10, 11], i.e. all energy fed into the FDN remains unattenuated in the feedback loop. More precisely, an FDN is lossless if all system poles lie on the unit circle and are simple [11]. Starting from a lossless FDN, a smooth frequency-dependent reverberation time can be ensured by extending every delay element with a frequency-dependent attenuation [12]. It has been shown that a unitary feedback matrix A, i.e. AH A = I, where (·)H denotes the complex conjugate transpose operation and I is the identity matrix, is a sufficient condition for the FDN to be lossless [11]. It can be also shown that there are non-unitary feedback matrices constituting a lossless FDN. Nonetheless, unitary feedback matrices are the main focus of the present work as they exhibit favorable properties in the context of time-variation [4]. The theory of losslessness and stability in particular of time-varying recursive filters is more subtle than it
AES 138th Convention, Warsaw, Poland, 2015 May 7–10 Page 2 of 9
Schlecht AND Habets
Practical Time-Varying FDN
2
2
2
2
4
4
4
4
6
6
6
6
8
8
-1
4
-0.5
6 0
8
8
0.5
2 1
-1
-0.5
(a) n = 1
6 0
0.5
2 1
-1
-1
0.5 0 -0.5 -1
0
Real Part
(e) n = 1
-0.5
1
6 0
8
0.5
2 1
-1
0
-0.5
1
Real Part
(f) n = 2
6 0
8
0.5
1
1
0.5 0 -0.5 -1
-1
4
(d) n = 20
1
Imaginary Part
Imaginary Part
0 -0.5
4
(c) n = 3
1
0.5
-1
8
8
(b) n = 2
1
Imaginary Part
4
Imaginary Part
2
0.5 0 -0.5 -1
-1
0
1
Real Part
(g) n = 3
-1
0
1
Real Part
(h) n = 20
Figure 2: Evolution of time-varying feedback matrix A(n) in a 8-FDN. The upper row (a)-(d) shows the matrix A(n) at time instance n. The lower row (e)-(h) shows the position of the eigenvalues of A(n) on the unit circle. is for time-invariant filters [13]. For example, timevarying filters can be unstable even if they are stable at each time instance. The authors have shown that a time-varying FDN is both stable and lossless as long as the feedback matrix is always unitary [4].
To investigate the Frobenius difference of unitary matrices, a parametrization of this matrix class, which reveals the underlying properties, have to be found. It turns out that the matrix eigendecomposition gives a straightforward explanation of the Frobenius difference [4]. Also later in Sections 3 and 4, we see that the eigen-decomposition proves to be the main tool of analysis.
2.3. Continuous Feedback Matrix Variation Large changes of filter coefficients across time can cause audible clicks and have to be avoided therefore. Hence, to achieve a continuous update of the feedback matrix, every matrix entry has to evolve in small steps. To measure the amount of change in the feedback matrix entries, we use an upper bound for the change of every matrix entry in two consecutive matrices given by kA(n + 1) − A(n)kF . Hereby, the qPFrobenius norm k·kF is defined as kAkF = N 2 i,j=1 |aij | . The closer two different matrices are the smaller is the Frobenius norm of the difference matrix.
The eigen-decomposition of a unitary time-variant feedback matrix is given by A(n) = UH (n) Λ(n) U(n),
(2)
where U(n) is a unitary matrix, Λ(n) is a diagonal matrix of unimodular eigenvalues, i.e., |λi (n)| = 1 ∀i. It is impractical to vary U(n) over time as it is just another unitary matrix variation. Instead, only the eigenvalues Λ(n) will be time-variant: A(n) = UH Λ(n) U.
AES 138th Convention, Warsaw, Poland, 2015 May 7–10 Page 3 of 9
(3)
Schlecht AND Habets
Practical Time-Varying FDN
kA(n + 1) − A(n)kF
= UH (Λ(n + 1) − Λ(n)) U F v uN uX 2 |λ (n + 1) − λ (n)| , =t i
i
(4a) (4b)
30
Amplitude [lin] with offset
We examine the continuity of two consecutive unitary matrices with the eigen-decomposition given in (3):
25 20 15 10
i=1
5 0
Figure 2 illustrates the continuous evolution of the feedback matrix and its eigenvalues over time. Whereas a single step like (a) to (b) or (b) to (c) are hardly perceivable, after more iterations, as shown in (d), the feedback matrix has evolved considerably. Figure 3 shows the matrix evolution per entry and the corresponding eigenvalues of a 4-FDN. It is interesting to note, that although there are only two regular modulation functions, the resulting evolution of the matrix entries is much more complex. Every entry has a unique modulation amplitude and phase, which makes the overall audible result less obviously modulated. 3. PRACTICAL CONSIDERATIONS ON FEEDBACK MATRIX MODULATION 3.1. The Modulation Functions For A(n) to be unitary, λi (n) have to be unimodular, hence there is a representation λi (n) = exp(−ıπφi (n)) for real valued functions φi (n). We refer to φi (n) as the modulation functions. Although from a stability point of view, there is no restriction on the evolution across time of φi (n), practically it is favorable for the eigenvalues to evolve in small and regularly sized steps. If we choose |φi (n + 1) − φi (n)| = const, in other words φi (n) having constant change, the Frobenius difference between two matrices in (4b) is constant
500
1000
1500
2000
Time [samples]
(a) Evolution of Matrix Entries Angle [radiant]
where λi (n) are the diagonal entries of Λ(n). The step from (4a) to (4b) results from the unitary invariance of the Frobenius norm [14]. The closer the eigenvalues λi (n + 1) are to λi (n), the closer are the matrix entries of the two consecutive feedback matrices.
0
:
0 -: 0
500
1000
1500
2000
Time [samples]
(b) Evolution of Matrix Eigenvalues
Figure 3: Evolution of time-varying feedback matrix A(n) in a 4-FDN. Figure 3(a) shows the evolution of the matrix entries. For better visualization, the 16 matrix entries are displaced by 2 in the ydirection. Figure 3(b) shows the corresponding angles of the eigenvalues. Here, we plotted only half of the eigenvalues as the other half is the antisymmetric to the first half. as well, because v uN uX t e−ıπφi (n+1) − e−ıπφi (n) 2 i=1
v uN uX e−ıπφi (n+1) 1 − e−ıπ(φi (n)−φi (n+1)) 2 =t i=1
v uN uX 1 − e−ıπ(φi (n)−φi (n+1)) 2 = const. =t
(5)
i=1
Therefore piecewise linearity is a natural choice for the modulation functions to ensure a regular change over the modulation period. The most simple choice is the entirely linear φi (n) =
AES 138th Convention, Warsaw, Poland, 2015 May 7–10 Page 4 of 9
Schlecht AND Habets
Practical Time-Varying FDN
n/ρi , where ρi is the modulation period in samples. µ fs The modulation period is defined as ρi = F,i 2 , where µF,i is the ith modulation frequency in Hz and fs is the sampling rate. It is interesting to note that these linear modulation functions create a feedback matrix evolution equivalent to the recursive update: A(n + 1) = A(n)R,
(6)
where A(n) is as given in (3) and the update matrix is R = UH diag( ρ11 , . . . , ρ1N )U. Figure 3 gives an example of the linear modulation functions plotted with an angular wrap of the eigenvalues at [−π, π]. A more flexible modulation function ing a triangle wave: n n φi (n) = 2µA,i − + ρi ρi
is obtained us1 2
,
(7)
where fs is the sampling rate and b·c indicates the floor function. The shape of the triangle wave is determined by the modulation frequency µF,i and the modulation amplitude µA,i , which can be chosen individually for each φi (n). Because of the circular nature of the eigenvalue evolution, only |µA | ≤ 1 are meaningful as a modulation amplitude. From an informal listening test we can report that the effect of the modulation on the overall reverberation scales directly with the modulation amplitude. Low modulation frequencies under 0.2 Hz are difficult to perceive as modulation, whereas modulation frequencies higher than 2 Hz introduce obvious periodic temporal patterns and detuning of musical tones. 3.2. Requirements for Real Matrices Time-domain processing of audio signals with the FDN is typically restricted to real valued filter coefficients. Therefore, also the time-variant feedback matrix has to be real at all time instances and consequently orthogonal. In the following, section we discuss the requirements needed. First of all, it can be shown that A(n) is real if and only if every real eigenvalue λi has a real eigenvector ui and all non-real eigenvalues and eigenvectors come in pairs λj , λk = λj ∗ and uj , uk = uj ∗ . ∗ Hereby, (·) denotes the complex conjugate operation. This can be seen by realizing that U−1 Λ U ∗ and (U−1 Λ U) = U−∗ Λ∗ U∗ are identical up to
column and row order, which has no influence on the eigen-decomposition representation. As a consequence, real eigenvectors pair necessarily with real eigenvalues. However, these real eigenvalues are unimodular so that they are either +1 or −1 and therefore cannot modulate continuously and are forced to remain constant. For this reason, we call them here static eigenvalues. In the context of modulation, static eigenvalues are naturally undesirable as they diminish the effect of the modulation. Please note that matrices with an odd dimension N require at least one static eigenvalue as it cannot come in a pair. For a given matrix U = [u1 , . . . , uN ] of unitary eigenvectors ui , which are either real or come in complex conjugate pairs, to ensure a real feedback matrix the modulation functions have to be chosen so that φi (n) = −φj (n) φi (n) ∈ Z
for ui = uj ∗ for ui ∈ RN .
(8)
3.3. Parameterization of Orthogonal Matrices To construct an orthogonal matrix with no static eigenvalues, we have to find a unitary matrix that has no real column or row. To simplify this task, we give an entirely real parameterization of the orthogonal matrices without static eigenvalues. We start with a unique transformation between orthogonal matrices Q without the eigenvalue −1, and therefore orthogonal matrices with det Q = +1, and skew-symmetric matrices S, i.e. ST = −S. The correspondence is given by the Cayley transformation, so that Q = (I − S)(I + S)
−1 −1
S = (I − Q)(I + Q)
(9) .
(10)
It can be shown that Q in (9) is orthogonal if and only if S is skew-symmetric and vice-versa in (10). Given the eigenvalue decomposition S = U−1 ΛU, it follows that −1
Q = U−1 (I − Λ)(I + Λ)
U.
(11)
Consequently, if λ is an eigenvalue of S then (1 − λ)/(1 + λ) is an eigenvalue of Q. Hence, −1 cannot
AES 138th Convention, Warsaw, Poland, 2015 May 7–10 Page 5 of 9
Practical Time-Varying FDN
Skew-symmetric matrices can be generated from any real matrix R by S = 12 (R − RT ). The eigenvalues of skew-symmetric matrices, and therefore also the orthogonal matrix, can be specified directly via a real parameterization:
0.5
0.5
0.4
0.4
0.3
0.3
Level [lin]
be an eigenvalue of Q and if S is of full-rank then 1 is also not an eigenvalue of Q. It follows that the full-rank skew-symmetric matrices parameterize the orthogonal matrices without static eigenvalues.
Level [lin]
Schlecht AND Habets
0.2 0.1 0
0.2 0.1 0
−0.1
−0.1
−0.2
−0.2
0
2000
4000
6000
0
Time [samples]
Hence, (12) parametrizes all orthogonal matrices with prescribed eigenvalues entirely defined in the real domain. 3.4. Continuous Change of Echo Density In this section, we want to give a different application for the continuously parametrized feedback matrix apart from time-variation. The echo density of the reverberation tail produced by an FDN depends on the feedback matrix [15]. The least amount of echo density is produced by an identity matrix, which is virtually equal to a set of parallel comb filters. The maximum amount of echo density can be achieved with a matrix, where all entries have the same magnitude, e.g. Hadamard matrices. Here, the density of the P matrix is quanN tified by the absolute norm |A| = i,j=1 |aij |, the higher the norm, the more dense the matrix is. Every matrix can be transformed continuously towards the identity matrix I by moving all its eigenvalues towards +1 on the unit circle. Following (8), we can ensure orthogonality at every step. Given the unitary feedback matrix A(k) and its eigenvalue decomposition A(k) = UH Λ(k)U with modulation functions φi (k) = k, |k| scales the density of the matrix: A(0) is the identity matrix and |A(k)| increases for |k| > 0. Figure 4 shows the resulting echo densities for a 4-FDN with different echo density coefficients k. It is important to note that the scaling of the echo density becomes arbitrary for higher k as the eigenvalues start to approach the +1 position again.
0.5
0.5
0.4
0.4
0.3
0.3
0.2 0.1 0
0.1 0 −0.1
−0.2
−0.2 2000
4000
6000
0.2
−0.1
0
4000
(b) k = 1/4
Level [lin]
Level [lin]
(a) k = 1/8
S = TT diag(Λ1 , Λ2 , . . . ΛN , 0, . . . , 0)T, (12) 0 λi where Λi = −λi 0 and T is any orthogonal matrix. It can be shown that the ±ıλi in (12) are actually the eigenvalues of S. The subsequent zeros, if any present, correspond to the eigenvalues +1 of Q.
2000
Time [samples]
6000
Time [samples]
(c) k = 1/3
0
2000
4000
6000
Time [samples]
(d) k = 1/2
Figure 4: Echo density of the reverberation tail depending on the feedback matrix A(k) with varying echo density coefficients k. 4. EFFICIENT IMPLEMENTATION The FDN processing with the time-variant feedback matrix involves two steps: update of the feedback matrix and multiplication of the signal vector with the feedback matrix. First, we discuss the direct update of the feedback matrix by multiplication with another orthogonal matrix as shown in (6). The processing then is implemented unaltered as in (1). The computational cost per sample for the matrix-vector multiplication of (1) is O(N 2 ), for the matrix-matrix multiplication in (6) is O(N 3 ). Additionally, the iterative update in (6) forbids efficiency improvements from block processing. Instead, the computation is performed more efficiently in the eigenvalue domain. From (3), we see that the update of the feedback matrix is only dependent on the evolution of the eigenvalues in Λ(n). The update of the eigenvalues can be computed from the modulation functions φi (n) via the exponential functions. On standard CPUs, the exponential func-
AES 138th Convention, Warsaw, Poland, 2015 May 7–10 Page 6 of 9
Schlecht AND Habets
Practical Time-Varying FDN
(a) No Modulation
(b) Orthogonal Modulation
(c) Linear Interpolation Modulation
Figure 5: Spectrogram of the reverberation tail of a snare drum hit with different modulation types. tion is realized by tenth or even hundreds of multiplications. Alternatively, φi (n) can be restricted to only have linear updates like it is in the triangle wave case (7). The eigenvalue update is then λi (n + 1) = pλi (n) with p± = exp(−ı2µA ±1 ρi ). Additionally, a counter has to decide which direction (±) of the eigenvalue update should be performed. Because the eigenvalues come in complex conjugate pairs only half of the eigenvalue updates are necessary. The complex multiplication consist of four real multiplications. Therefore, the eigenvalue update needs 2N multiplications in total. The signal processing with the eigenvalue decomposition representation of the feedback matrix in (3) involves two complex matrix-vector multiplications and one complex vector-vector operation. Therefore, we expect to have O(N 2 ) of complex multiplications. Compared to time-invariant state update, the number of multiplications are more than 8 times higher, but lower than the recursive update of the feedback matrix for N > 8. Further, as Λ is a diagonal matrix, it can be summarized into a single matrix for a whole block of consecutive samples allowing (3) to be implemented in blockwise processing and therefore make use of CPU specific advantages. An even more efficient implementation can be achieved by the use of circulant lossless matrices, a subclass of the unitary matrices. Circulant matrices C = [cij ] are defined as cij = γi−j and γi = γi+N
for any given values γi . Lossless circulant matrices are also characterized by the fact that the eigenvector matrix U is a DFT matrix of size N × N [11]. Instead of performing a matrix based transform, an FFT algorithm is applied to compute the backward and forward transform to the eigenvalue domain which brings the cost down to O(N log N ). Compared to the time-invariant version, only the eigenvalue update of N multiplications are added as additional costs. It is important to note that the DFT matrix has two real column vectors for even N , and one real column vector for odd N . Consequently, the maximum number of non-static eigenvalues for even N is lower by two when using circulant matrices. Still, for larger N , the use of circulant matrices with linear modulation functions gives a highly efficient algorithm for time-varying feedback matrices. 5. PERFORMANCE EVALUATION In this section, we give a performance evaluation of the proposed feedback matrix variation in two ways: Firstly, the proposed algorithm is compared to a more straightforward method, which employs linear interpolation directly on each matrix entry. Secondly, we present an example of the benefits of time-varying matrices on the smoothness of the frequency response of the reverberation tail. Given two orthogonal matrices A and A0 , the direct
AES 138th Convention, Warsaw, Poland, 2015 May 7–10 Page 7 of 9
Schlecht AND Habets
Practical Time-Varying FDN
linear interpolation of these two matrices is given by Aα = αA − βA0 ,
(13)
where 0 ≤ α ≤ 1 and β = 1 − α. We show that all eigenvalues of Aα lie within the unit circle and therefore that it constitutes a stable system. Let λ be an eigenvalue of Aα and v the corresponding eigenvector. Then: |λv| = |αAv + βA0 v| p = (α|Av|)2 + 2αβ
