On Existential and Constructive Neural Complexity

ON EXISTENTIAL AND CONSTRUCTIVE NEURAL COMPLEXITY RESULTS Valeriu Beiu Washington State University, School of EE&CS 102 Spokane Street (EME), P.O. Box 642752, Pullman, WA 99164-2752, USA ABSTRACT In this paper we will review major circuit complexity results for networks of perceptrons. In the first part we will present many theoretical results, while the second part is much more practical, as comparing nine different constructive solutions for a particular but important case: the addition of two binary numbers.

fication can be justified. For answering that, the precise four-dimensional neuron model of Hodgkin and Huxley has been used, and the threshold model has been tested on a spike train generated by the Hodgkin-Huxley model with a stochastic input. The result was that the threshold model correctly predicts nearly 90% of the spikes, justifying the description of a neuron as a TLG [15].

KEY WORDS Neural / circuit complexity, perceptrons, threshold logic, addition.

In the last decade the impetus of VLSI technology has made neurocomputer design a really lively topic. Hundreds of designs have been build, while several are available as commercial products [16]. Still, TLGs (or their variations) have rarely been used in practice (for a review of their VLSI implementations see [17, 18]). The earliest examples are Mark I Perceptron, built by Rosenblatt (1957-1958), and Widrow’s memistor (1960). Variations of TLGs have been used in: • the MIPS R2010 floating point co-processor in 1988 [19] (the gates were in fact discovered in 1973 [20]); • the SUN Sparc V9 in 1995 [21, 22]; and • the Itanium 2 microprocessor in 2002 [23–26].

1. Introduction Neural networks (NNs) are made of neurons composed of a cell body and many outgrowths: the axon and the dendrites. Formally, a network is a graph having several input nodes, and some (at least one) output nodes. If a synaptic weight is associated with each edge, and each node i computes the weighted sum of its inputs to which an activation function is applied: fi (xi,1, …, xi,k)

= σi (∑ j = 1 wj xi, j + θi), ∆

(1)

the network is a NN, with the synaptic weights wj ∈ IR, θi ∈ IR known as the threshold, ∆ ∈ IN the fan-in, and σi an activation function. If the underlying graph is acyclic, the NN does not have feedback, and can be layered. This is the well-known multilayer feedforward NN. Two cost functions have been used to characterize NNs: (i) depth (i.e., number of edges on the longest input-to-output path, or number of layers); and (ii) size (i.e., number of neurons). If the activation function is the sign, the node is called a perceptron, or threshold logic gate (TLG), and the network is a threshold logic circuit (TLC). The early reviews on TLGs [1-13], have been followed by one recent chapter [14]. The general belief that a neuron is a TLG, which fires when some variable reaches a threshold, can be questionable as to whether such a drastic simpliThis material is based on research partly sponsored by the Air Force Research Laboratory under agreement number F29601-02-2-0299. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the author and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Air Force Research Laboratory or the U.S. Government.

For VLSI implementations the area of the connections counts, and the area of one neuron can be related to its associated weights [27]. That is why cost functions which can be linked to a VLSI implementation by the assumptions one makes on how the area scales with weights and thresholds have been taken into account [28, 29]. It is worth emphasizing that it is anyhow desirable to limit the range of values [30, 31], because: (i) the maximum value of the fanin [32]; and (ii) the maximal ratio between the largest and the smallest weight cannot grow over a certain (technological) limit [33–35]. The paper will start by reviewing general complexity results for TLCs. The main problems concern the reduction of size in small (constant-)depth, while also reducing the weights, and possibly the fan-ins. These will be followed by comparing nine constructive solutions for the addition of two binary numbers. Such constructive results are interesting for CMOS implementations [36–40], as well as for future quantum implementations where TLGs are a perfect fit [41]. Currently, the devices which could get central stage positions in the future are: single electron tunneling (SET) devices, and resonant tunneling devices (RTDs). SET technology combines large integration and ultra low power dis-

sipation. RTDs are already operating at room temperature, and appear to hold the most promise as a short/medium term solution. They have already been used for implementing TLGs [42]. All of these are reasons why a clear understanding of the known depth-size-weights/fan-ins tradeoffs could prove useful for VLSI designers, and have stimulated the writing of this review paper.

2. Neural Complexity Results From the very beginning, there have been two different ways of approaching TLC: theoretical and practical. The theoretical one falls under the well-established computational circuit theory. This line of research has concentrated on: • determining the approximation capabilities of NN; • finding tight bounds. In this paper, only the second aspect (i.e., finding tight bounds) will discussed. The problem can be stated as: the quest to find the smallest NN which can realize an arbitrary function given a set of m vectors in n dimensions, and many mathematical results have been obtained [43–48]. Two classes of constraints could be thought of: • some arise from physical constraints, and include time constants, energy limitations, volumes, geometrical relations, and bandwidth capacities; • others are logical constraints, like computability constraints, and complexity constraints, giving upper and/or lower bounds on some specific resource. The theoretical results relating to physical constraints could (and should) be useful for VLSI designeds. From the theoretical point of view [49, 50], there is considerable justification that shallow (i.e., with relatively few layers) TLCs are computationally more powerful than Boolean circuits (BCs). TLCs are more powerful than BCs as a Boolean gate (BG) can compute only one function whereas a TLG with fan−in = ∆ can compute up to the or2

d er of 2 α∆ fu nction s b y vary ing the weights, with 2

1 / 2 ≤ α ≤ 1 [46]. The exact bound is 2 ∆ − ∆log∆ + O (∆) [51] (see [52]). An important result which clearly separates TLCs and BCs is due to Yao [53, 54] (see also [55, 56]). It states that in order to compute a highly oscillating Boolean function (BF) like PARITY [57, 58] in constant depth, at n k BGs with unbounded fan-in are needed. In conleast 2 c √ trast, a depth-2 TLC for PARITY has linear size. The particular case of BFs has been intensively studied [43, 48, 59]. The first lower bound on the size of a TLC for ‘almost all’ n-ary BFs (f : IB n→ IB, where IB = {0, 1}) was given by Nechiporuk [44]: size

≥ 2 (2 n/n)1 / 2.

(2)

A very tight upper bound for the same case size

≤ 2 (2 n/n)1 / 2 ×{1 + Ω [(2 n/n)1 / 2]},

(3)

was proven in depth = 4 [47]. An existence lower bound of Ω (2 n / 3) for arbitrary BFs was detailed in [60, 61]. A lot of work has been devoted to find minimum size and/or minimum (constant-)depth TLCs [62–68]. Several of the results obtained have been gathered in TABLE I [44, 47, 54, 61, 69–82]. For the nomenclature used see [83–86]). Three complexity classes for sigmoid feedforward NNs are also included [77–79, 82, 87–89]: • NN k is the class of functions computed by polynomially sized NNs with weights and thresholds determined to b bits of accuracy, fan-in ∆ and depth h, and  satisfying log∆ = O (√ logn ), b log∆ = O (logn), and h log∆ = O (log kn); • NN∆k is the class of functions computed by polynomially sized NNs having linear fan-in and logarithmic accuracy, i.e., ∆ = O (n) and b = O (logn); • NN∆,k ε is the class of functions computed by polynomially sized NNs satisfying log∆ = O (log1−ε n), and accuracy b = O (log1−ε n), for ε a small positive constant. Such theoretical results are of interest for designers, only if they are constructive. Even more, the aspects of interest to VLSI designers are those closely related to the limitations imposed by a given technology, namely the limited fan-in and the limited range of weights. The precision of the weights is linked to very practical noise margins and power dissipation [17]. That is why theoretical research has also focused on weights, or more precisely on the capabilities of TLG with restricted weights [90–97]. In [46] it has been proven that any TLG could be implemented with integer weights, and also that the number of bits per weight is O (∆log∆). The bound is tight as Håstad [98] has presented a function having weights up to 2 (∆ / log∆) / 2 − ∆, i.e., requiring Ω (∆log∆) bits per weight. All these results have been obtained for very small constant-depth.

3. Constructive Complexity Results Particular classes of functions allow for simpler implementations, i.e., small(er) weights and fan-ins, shallow depth and polynomial size [29, 99]. That is why symmetric functions [43, 44, 46, 48, 57–59] as well as basic arithmetic operations [61, 65, 67, 68, 70, 71, 73, 74, 80, 84, 85, 89, 100–104] have been among the potential candidates. Addition has been considered as a challenging function [83, 105–107]. Obviously, beside minimum depth or minimum size solutions, solutions having small(er) fan-ins and small(er) weights are of interest [83, 108–110]. The additio n o f two n- bit bin ary numbers, an augend X = xn−1xn−2 … x1x0 and an addend Y = yn−1yn−2 … y1y0, is de-

TABLE I CIRCUIT COMPLEXITY RESULTS (CHRONOLOGICAL ORDER) Author(s)

Year Ref. Result(s)

Remark(s) n

≥ 2 (2 / n)

1/2

Nechiporuk

1964

[44] size

Lupanov

1973

Yao

1989

[47] size ≤ 2 (2 n/ n) 1 / 2 × {1 + Ω [(2 n/ n) 1 / 2]} depth = 4 [54] TC 0 ≠ NC 1 (conjecture)

Allender

1989

Immermann & Landau 1989 Bruck Siu et al.

1990 1991

AC 0

≠ TC 0

0

= NC1

[70] TC

1991

Bruck & Smolensky

1992

1992

^ ∈ LT4 ^ ∈ LT5 ^ ∈ LT6

X

⊂ PT1

⊂ PL∞

⊂ PT1

⊂ MAJ2

0

⊄ PL1

AC 0

⊄ PL∞

AC 0

⊄ MA J2 ^ ⊂ LT3 ^ ⊆ LT2d + 1

depth size ^ ∈ LT3 ^ ∈ LT4

MAX (x1, ..., xn) SORT (x1, ..., xn), MUL (x,y) 1992

[77] NC

k

⊂ NN

Partly constructive.

Lower bound on the size of TLCs implementing BFs. fan-in unbounded. Existence proof. Existence proofs.

PL1

[76] LT1

≤ nc

weights ≤ n c fan-in unbounded.

= Ω (2 n / 3)

LTd

Shawe-Taylor et al.

size

= const.

Upper bound on the size of TLCs implementing BFs. fan-in unbounded. Existence proof.

[75] PL1 AC

Siu & Bruck

Existence proofs. depth

MUL (x1, …, xn) [61] size = O (2 n / 2) depth = 3 size

There are BFs for which depth-(k−1) TLCs of unbounded fan-in

Conjecture.

(conjecture)

[72] LT1 ⊂ PT1 ⊂ LT2 [74] Xmodp, X nmodp, c Xmodp MUL (x,y) DIV (x, y), X , c

Upper bound for the size of a TLC for “almost all” n-ary BFs.

(i.e., TC 0 circuits) require exponential size, depth-k Boolean circuits of unbounded fan-in. Any BF computable by a polynomial size constant-depth logic circuit with unbounded fan-in (i.e., AC0) is also computable by a depth-3 TLC of superpolynomial size (fan-in unbounded).

[69] size = n O (logn) depth = 3

n

Siu et al.

Lower bound on the size of a TLC for “almost all” n-ary BFs.

k

= const. ≤ nc

weights ≤ n c fan-in unbounded.

k

⊆ AC

Existence proofs. Partly constructive.

 LTlog n ⊂ NN k∆ NN k∆, ε ⊂  ^ k+1  LTlog n ∩ ∩ k+1 k+2 MAJ log n ⊂ MAJ log n k+ε

Beiu et al.

1994

[78]

Beiu et al.

1994

[79] NN k

⊂ NN∆k, ε

[81] LTd

⊂ MAJd+1

Existence proof. Such a simulation is possible even if the depth d grows with the number of variables n.

k [82] NN∆

k+1  ⊂  NC ^ k+1 k+1 MAJ log n ⊂ LTlog n 

Constructive proofs (based on TLG carry save addition).

Goldmann & Karpinski 1994 Beiu & Taylor

1996

Constructive proofs (based on trees of TLG adders).

 NC k+1

⊂ 

k k+2  NN ∆ ⊂ NC

fined as the unsigned sum of the addend added to the augend S = snsn−1 … s1s0. An established method is: si = xi ⊕ yi ⊕ ci−1

(4)

ci = (xi∧yi) ∨ (xi∧ci−1) ∨ (yi∧ci−1)

(5)

where c−1 = 0 and sn = cn for 0 ≤ i ≤ n−1 [59, 83, 105, 111, 112]. The ci are the “carry” bits, and addition reduces to computing the carries [59, 83, 113]. The carry chain can be computed based on an associative operator o:

Constructive proofs (based on trees of Boolean gate adders).

(g, p) o (g′, p′)

= [g ∨ (p∧g′), p∧p′]

(6)

(Gi, Pi)

= (gi, pi) o (Gi−1, Pi−1)

(7)

for 2 ≤ i ≤ n. The sum bits si can also be computed as _ = (xi∧yi∧ci−1) ∨ [ci ∧ (xi∨yi∨ci−1)] (8) si which is a linearly separable function having weights (1,1,1,2). This solution was discovered in 1969 [114] (see also [115]), with variations in [116, 117]. The equation was later used in 1997 [110], and rediscovered in 1999 [118].

3.1. Boolean Logic Many of the known adders have been built using BGs of (un)bounded fan-ins. It has been proven that a depth-2 circuit of AND-OR logic gates for ADDITION must have exponential size [61]. Some authors [119, 120] have formulated the problem of minimizing the latency in carry-skip and block carry-lookahead adders [121–123], as multidimensional dynamic programming. Others [124] have investigated implementations based on spanning trees. Still, most of the effort has been devoted to practical implementations [119, 124–127]. Interesting results using fan-ins larger than two have also been reported. Such extensions of [112, 128] have been presented by Ong and Atkins [129], and later by Ngai and Irwin [130], while similar extensions for a hybrid prefix algorithm were detailed in [122]. Han et al. [122] detail two implementation enhancements for reducing the area: a folding method, and the use of hierarchical leaf cells. Their very thorough analysis has led to the conclusion that for all the operand length n ≤ 1024, the optimal fan-in for the hybrid prefix algorithm is either 3 or 4. They mentioned that increasing the fan-in affects the time performance of the circuitry in three different ways: • the depth decreases from O (logn) to O (logn/log∆); • the processing delay of each element increases (due to the need to implement more complex logic); • the delay is also increased due to the larger fan-out capacitance (which grows as ∆2). Obviously, TLGs are interesting candidates for complex logic gates, as potentially maintaining (or even reducing) the delay of the processing elements for large fan-ins.

3.2. Threshold Logic An existence depth-2 solution of polynomial size was presented in 1990 [106]. It was followed in 1991 by a constructive depth-2 majority gate circuit (majority gates are TLGs with weights of −1, 0, or +1) of size O (n 4) [107]. Two other constructions for addition based on AND-OR logic gates have been detailed in 1991 [61]. Because AND and OR can be simulated by TLGs, they have been considered as: • a depth-3 TLC of size O (n 2), or more precisely n 2/ 2 + 7n / 2 − 1, was presented in Theorem 7 (p. 1410); • a depth-7 TLC of size O (nlogn) was constructed in four steps in Lemma 4 (p. 1408). Going for a lower depth (from 7 to 3) increases the size complexity from O (nlogn) to O (n2). Based on [66], the authors concluded that “a substantial reduction in size is not possible without increasing the depth beyond 3.” As we shall see, it did not take too long to constructively prove better.

In ’93-94, the depth-7 construction from [61] was reduced to depth-5, while still having the same O (nlogn) size complexity [109, 131, 132]. This solution is based on a class of linearly separable functions with fan-in = ∆, IF∆: f∆ (g∆/2−1, p∆/2−1, …, g0, p0) = ∨ ∆j =/ 02 − 1 [gj ∧ (∧ ∆k =/ 2j +− 11 pk)], where by convention ∧ iα=−α1 pi = 1. The input variables are pair-dependent, i.e., in each of the ∆ / 2 pairs of two input variables: (g∆/2−1, p∆/2−1), …, (g0, p0) one variable is “dominant,” i.e., when a dominant variable is 1, the other variable forming the pair will also be 1. Because the BFs from Step 3 and Step 4 of Lemma 4 from [61] are IF∆ functions, the depth-7 construction can be shrunk to depth-5 by replacing the AND-OR gates from the intermediate layers with TLGs. By mid 1994, a linear size solution of 7n was obtained in depth-4 [89, 109]. This was a substantial reduction in size from O (nlogn) to linear, but not yet in depth-3. The solution was in fact a class of constructive solutions based on IF∆. By varying the fan-in ∆ the solutions span depths from 4 to 3 + logn, while the sizes vary from 7n to 2nlogn + 5n. This class of adders was used to prove inclusions amongst many circuit complexity classes, including three neural network complexity classes [82]. Many variations are possible depending on the way the sum bits si are implemented: classical AND-OR solutions, or using the last layer from [61]. If equation (8) already known since 1969 [114] is used in conjunction with the class of solutions detailed in [82, 89, 109, 132], the depth is reduced by one and the size by 2n (with respect to the case when the last layer would be implemented by a classical AND-OR solution). This led to the first depth-3 solution having linear size 5n, substantially reducing the size over [61]. One disadvantage is the exponential growth of the weights. Decreasing the fan-in from 2n to 4, the depth increases from 3 to 2 + logn, and the size from 5n to 3n + 2nlogn, while the weights decrease from 2 n to 4. In 1995-1996, three other depth-3 solutions were presented [133–135]. A linear size solution 1) of linear size n  is achieved in depth-3. The weights are 6n + 2 n / √ slightly smaller 2 √n , but still exponential. Another solution having polynomially bounded weights O (n k) was also detailed in [135]. It has O (n 2/ klogn) size in depth-3, for any 1 ≤ k ≤ n / logn. By varying k, the sizes cover the interval from 2n 2/ logn + 8n to 10n. For linear size solutions (i.e., those approaching 10n), the weights are exponential again: n n / (clogn) = 2 n / c. Finally, in 1999, three other solutions have been detailed. 1

In this paper x is the ceiling of x, and x is the floor of x; all the logarithms are taken to base 2 (except explicitly mentioned otherwise).

TABLE II CONSTRUCTIVE THRESHOLD LOGIC CIRCUITS FOR ADDITION (CHRONOLOGICAL ORDER) Author(s) Siu & Bruck Alon & Bruck

Year Ref. 1990 1991

Delay (#layers)

[106] 2

Size (#gates) n

c

Max weight

Max fan-in

Remark(s)

—

—

Existence solution.

[107] 2

O (n )

{−1, 0, +1}

n

(n + 7n − 2) / 2

{−1, 0, +1}

2n

{−1, 0, +1}

2n

4

2

Siu et al.

1991

[61]

7

O (nlogn)

Beiu et al.

1994

[109] 5

O (nlogn)

2

Beiu

1994

[89]

5n + 2n logn / (log∆ − 1)

2∆/2

3

3 + logn / (log∆ − 1) 4

7n

3+ 1994 1969

logn log (logwmax)

[89] 2 + logn / (log∆ −1) [114] 3 4

Vassiliadis et al. 1996

n

Constructive TLC.

∆

Constructive class of TLCs. Also in Beiu et al. 1994 [132], and in Beiu and Taylor 1996 [82].

d−3

n √

2nlogn log (logwmax)

wmax

2 logwmax

3n + 2n logn / (log∆ − 1)

2∆/2

∆

n

2n

√n

n 2√

5n +

5n

2 2

d−2 √ n

2dn − n

2

logn 2+ log (logwmax)

2nlogn 3n + log (logwmax)

wmax

n  6n + 2 n / √

2

2n [n / (mlogn) − 1] + 13n

AND-OR

2n

2

d

3

Constructive solutions.

2

7n

[135] 3

Constructive TLC.

2n

d−3 √ n

2dn − n

d

Beiu using Betts

2

n

4

2

d−2

n √

Constructive class of TLCs, obtained by implementing the last layer from [82, 89, 109, 132] using the solution presented in [114].

2 logwmax

√n 

n  + 3 2 n / √

Constructive TLC.

O (n m)

max {2mlogn, 4 [n / (mlogn) + 2]}

Class of TLC for 1 ≤ m ≤ n / logn.

Yeh et al.

1999

[136] (logn+2) / log∆ + 1.44logk

O [2nlog(k + 1) / ∆]

{−1, 0, +1}

∆

Constructive TLC for any positive k.

Ramos & Bohórquez

1999

[118] 2

2n

2n

2n + 1

Constructive limited fan-in class of TLCs based on using the equation from [114] together with a modification of the classic carry lookahead method.

n  4n − 2 n / √

3

1999

2

3

4

n  6n − 2 n / √

d

2n(d − 1) − 2(d−3)

1+ Beiu

n / √n 

logn log (logwmax)

[138] 1 + logn / (log∆ − 1)

d−1

nd− 2  √

d−1

wmax

2 logwmax + 1

n∆ / 2 + 2n logn / (log∆ − 1)

2 0.7∆

∆

n + 2n n + 4n n√

2

n + 6n n√

d

n

• Yeh et al. [136] (see also [137]) present a majority gate circuit having depth (logn + 2) / log∆ + 1.44logk and size O [2nlog(k+1) / ∆] for any positive integer k. • Beiu [138] improves on [82, 89, 109, 132] by lowering the depth to 2 in n 2 + 2n size, with exponential weights. A new depth-3 solution having size is also presented. This

n 2√

3 1.4 √ n

2 1.4

n + 2n(d − 1) √ 2nlogn 0.7 ) log (logwmax

n d−2  √  +1

2n

2

d−1

0.7nlogwmax +

1.4n 1.4√ n

3

4

n 1.4√

d−1

2nlogn + O (n) log (logwmax)

3

logn 0.7 ) log (logwmax

3

n  + 1 2 n / √ 2

2

1+

n  + 1 2 n / √

d−2  n 2 √

2

n + 4n and weights 2 n√

2

3 n / √n 

3

n 2√

d−1

n √

Constructive limited fan-in class of TLCs. Modified first and second layer (for more details see [38– 40]).

d−1

n √

2

2

wmax

1.4 logwmax

new class of solutions is based on functions with no restrictions on the input variables. The weights (of these functions) are the Fibonacci numbers Fibm = Fibm − 1 + Fibm − 2 extended to negative values using Binet’s formula Fib− m = (−1) m+1Fibm being bounded by 2 0.7∆.

• Ramos and Bohórquez [118] solution has its roots in [135]. They rediscovered the equation for si presented in [114–117] and [110], obtaining a depth-2 linear size 2n solution with exponentially bounded weights 2 n, and a depth-3 solution of size 4n having weights bounded by 2 √n (these result directly from [135] using [114]). The novelty comes from improving the depth-3 so lutio n b y r ed ucing the size to n , and by extending this last solution to 4n − 2 n / √ arbitrary limited weights. The complexity results of this class of solutions with respect to size and depth are the same as those of solutions already obtained in 1994 [89, 109, 132], exhibiting smaller constants for size, and slightly larger constants for weights. For an easier comparison, all these nine solutions are presented in a compact form in chronological order in TABLE II. They exhibit interesting depth-size tradeoffs which are clearly related to the range of weights and fan-ins.

solutions have been reported later: [118] and [138]; • in depth-d, a 2nd − n size solution was reported in [89, 109, 132], and was later improved in [118] to 2n(d−1) − 2(d−3)

Finally, interesting open questions remain (fresh resultas might appear in [139, 140]). Can we improve by allowing super-polynomial weights? What could be obtained with sub-linear weights? What could be obtained in sublogarithmic depth?

References [1]

[2]

[3]

4. Conclusions The paper has reviewed complexity results for TLCs. Circuit complexity has certain drawbacks that should be mentioned: • the extension of the known poly-size results to other functions is not at all straightforward; • even the known bounds are sometimes weak; • time (i.e., delay) is not properly considered; • all complexity results are asymptotic in nature and may not be meaningful for the practical range of a particular application. The main conclusions are that size-optimal TGCs for implementing arbitrary BFs should have sub-linear fan-ins, and that area grows exponentially with accuracy, and like the cube of the fan-in. We have also presented nine constructive solutions for addition which show interesting depth-size tradeoffs. These directly influence design parameters like fan-in and weights, hence affecting the overall area, delay and power consumption of VLSI implementations. Over the last decade, the improvements have been significant: • in depth-2, the size has been reduced from O (n 4) with constant weights [107], to O (n 2) with exponential weights [138], and finally to O (n) with exponential weights [118]; • in depth-3, the size has been reduced from O (n 2) with constant weights [ 61 ], to O (n) with exponential weights [89, 109, 132], while two other O (n) solutions with exponential weights have been reported later: [135] and [118]; • in depth-4, the size has been reported O (n) with exponential weights [89, 109, 132], while two other O (n)

−2 . nd   √

d−1

[4]

[5] [6] [7] [8] [9] [10]

[11] [12]

[13] [14]

[15]

[16]

W.S. McCulloch, and W. Pitts, “A logical calculus of the ideas immanent in nervous activity,” Bull. Math. Biophysiol., vol. 5, pp. 115-133, 1943. M.L. Minsky, Neural Nets and the Brain-Model Problem, Ph.D. dissertation, Princeton Univ., USA, 1954. F. Rosenblatt, “The perceptron: A probabilistic model for information storage and organization in the Brain,” Psych. Revue, 65, pp. 386-408, 1958. B. Widrow, and M.E. Hoff, Jr., “Adaptive switching theory,” IRE Wescon Convention Record, part 4, pp. 96-104, 1960. R.O. Winder, Threshold Logic, Ph.D. dissertation, Math. Dept., Princeton Univ., USA, 1962. C.H. Mays, Adaptive Threshold Logic, Ph.D. dissertation, Stanford Univ., USA, 1963. S. Hu, Threshold Logic, Berkeley: Univ. of California Press, 1965. D.W. Dearholt, On Threshold Logic, Ph.D. dissertation, Univ. of Washington, USA, 1965. P.M. Lewis, II, and C.L. Coates, Threshold Logic, New York: Wiley, 1967. R.O. Widrow, “Fundamentals of threshold logic,” in J.T. Tou (ed.), Applied Automata Theory, New York: Academic Press, pp. 235-318, 1968. R.O. Widrow, “The status of threshold logic,” RCA Review, vol. 30, pp. 62-84, 1969. S.L. Hurst, “An introduction to threshold logic: A survey of present theory and practice,” The Radio and Elecrtonic Engineer, pp. 339-351, 1969. D. Hampel, and R.O. Winder, “Threshold logic,” IEEE Spectrum, vol. 8, pp. 32-39, 1971. M.J. Avedillo, J.M. Quintana, and A. Rueda, “Threshold logic,” in J.G. Webster (ed.), Wiley Encyclopedia of Electrical and Electronic Enginering, New York, NY: Wiley, 1999, vol. 22, pp. 178-190. W.M. Kistler, W. Gerstner, and J.L. van Hemmen, “Reduction of the Hodgkin-Huxley ecuations to a single-variable threshold model,” Neural Computation, vol. 9, pp. 1015-1045, 1997. E. Fiesler, and R. Beale, Eds., Handbook of Neural Computation, New York, NY: Inst. of Physics, 1996.

[17]

[18]

[19]

[20]

[21]

[22] [23]

[24]

[25]

[26]

[27]

[28]

[29] [30]

[31]

V. Beiu, J.M. Quintana, and M.J. Avedillo, “VLSI implementation of threshold logic: A comprehensive survey,” IEEE Trans. Neural Networks: Special Issue on Hardware Implementations, vol. 14 (no. 5), 2003. V. Beiu, “Threshold logic implementations: The early days,” Proc. IWANN’03, Menorca (Spain), Jun. 2003 M.G. Johnson, “A symmetric CMOS NOR gate for high-speed applications,” IEEE J. Solid-State Circuits, vol. 23, pp. 1233-1236, 1988. J.B. Lerch, “Threshold gate circuits employing field-effect transistors,” U.S. Patent 3715603, Feb. 6, 1973. L.A. Lev, A. Charnas, M. Tremblay, A.R. Dalal, B.A. Frederick, C.R. Srivatsa, D. Greenhill, D.L. Wendell, D.D. Pham, E. Anderson, H.K. Hingarh, I. Razzack, J.M. Kaku, K. Shin,M.E. Levitt, M. Allen, P.A. Ferolito, R.I. Bartolotti, R.K. Yu, R.J. Melanson, S.I. Shah, S. Nguyen, S.S. Mitra, V. Reddy, V. Ganesan, and W.J. de Lange, “A 64-b microprocessor with multimedia support,” IEEE J. Solid-State Circuits, vol. 30, pp. 1227-1238, 1995. L.A. Lev, “Fast static cascode logic gate,” U.S. Patent 5438283, Aug. 1, 1995. S.D. Naffziger, G. Colon-Bonet, T. Fischer, R. Reidlinger, T.J. Sullivan, and T. Grutkowski, “The implementation of the Itanium 2 microprocessor,” IEEE J. Solid-State Circuits, vol. 37, pp. 1448-1460, 2002. S.D. Naffziger, “Feedback-induced pseudo-nMOS static (FIPNS) logic gate and method,” U.S. Patent 6466057, Oct. 15, 2002. E.S. Fetzer, M. Gibson, A. Klein, N. Calick, C. Zhu, E. Busta, and B. Mohammad, “A fully bypassed sixissue integer datapath and register file on the Itanium-2 microprocessor,” IEEE J. Solid-State Circuits, vol. 37, pp. 1433-1440, 2002. C. A. Poirier, S.D. Naffzinger, W.D. Kever, “Contention based logic gate driving a latch and driven by a pulsed clock,” U.S. Patent 6265897, Jul. 24, 2001. R.C. Williamson, “ε−entropy and the complexity of feedforward neural networks,” in R.P. Lippmann, J.E. Moody, and D.S. Touretzky (eds.), Advs. Neural Info. Proc. Sys. 3, San Mateo, CA: Morgan Kaufmann, 1991, pp. 946-952. V. Beiu, “On the circuit and VLSI complexity of threshold gate COMPARISON,” Neurocomputing, vol. 19, pp. 77-98, 1998. V. Beiu, VLSI Complexity of Discrete Neural Networks, accepted for publication. A.H. Kramer, “Array-based analog computation; principles, advantages and limitations,” Proc. MicroeNeuro’96, 1996, pp. 68-79. J. Wray, and G.G.R. Green, “Neural networks, approximation theory, and finite precision computa-

[32]

[33]

[34]

[35]

[36] [37]

[38]

[39]

[40]

[41]

[42]

[43] [44]

[45] [46] [47]

[48]

tion,” Neural Networks, vol. 8, pp. 31-37, 1995. D. Hammerstrom, “The connectivity analysis of simple associations –or– How many connections do you need,” in D.Z. Anderson (ed.), Neural Info. Proc. Sys., New York, NY: American Inst. of Physics, 1988, pp. 338-347. J. Bruck, and J.W. Goodmann, “On the power of neural networks for solving hard problems,” J. Complexity, vol. 6, pp. 129-135, 1990. M. Goldmann, J. Håstad, and A. Razborov, “Majority gates vs. general weighted threshold gates, Proc. CoCo’92, 1992, pp. 2-13. V. Beiu, and H.E. Makaruk, “Deeper sparser nets can be optimal,” Neural Proc. Lett., vol. 8, pp. 201210, 1998. H. Jeong, “Neural network implementation of a binary adder,” U.S. Patent 5016211, May 14, 1991. V. Beiu, “Adder and multiplier circuits employing logic gates having discrete, weighted inputs and methods of performing combinatorial operations therewith,” U.S. Patent 6205458, Mar. 20, 2001. V. Beiu, “Adder having reduced number of internal layers and method of operation thereof,” U.S. Patent 6438572, Aug. 20, 2002. V. Beiu, “Adder circuit employing logic gates having discrete weighted inputs and method of operation therewith,” U.S. Patent 6502120, Dec. 31, 2002. V. Beiu, “Microprocessor and a digital signal processor including adder and multiplier circuits employing logic gates having discrete and weighted inputs,” U.S. Patent 6516331, Feb. 4, 2003. K.F. Goser, C. Pacha, A. Kanstein, and M.L. Rossmann, “Aspects of system and circuits for nanoelectronics,” Proc. IEEE, vol. 85, pp. 558-573, 1997. C. Pacha, U. Auer, C. Burwick, P. Glösekötter, A. Brennemann, W. Prost, F.-J. Tegude, and K.F. Goser, “Threshold logic circuit design for parallel adders using resonant tunneling devices,” IEEE Trans. VLSI Sys., vol. 8, pp. 558-572, 2000. R.C. Minnik, “Linear-input logic,” IRE Trans. Electron. Comp., vol. 10, pp. 6-16, 1961. E.I. Nechiporuk, “The synthesis of networks from threshold elements,” Problemy Kibernetiki, vol. 11, pp. 49-62, 1964. English transl., Automation Express, vol. 7, pp. 27-32 & pp. 35-39, 1964. S. Hu, Threshold Logic, Berkeley, CA: Univ. of California Press, 1965. S. Muroga, Threshold Logic and Its Applications, New York, NY: Wiley, 1971. O.B. Lupanov, “On circuits of threshold elements,” Dokl. Akad. Nauk SSSR, vol. 202, pp. 1288-1291 (1971). English transl., Sov. Phys. Dokl., vol. 17, pp. 91-93, 1972. N.P. Red’kin, “Synthesis of threshold circuits for certain classes of Boolean functions,” Kibernetika, vol. 5, pp. 6-9, 1970. English transl., Cybernetics,

[49]

[50]

[51]

[52]

[53] [54] [55]

[56]

[57]

[58] [59] [60]

[61]

[62]

[63]

[64]

[65]

vol. 6, pp. 540-544, 1973. N. Pippenger, “The complexity of computations by networks,” IBM J. Res. & Develop., vol. 31, pp. 235243, 1987. V.P. Roychowdhury, K.-Y. Siu, A. Orlitsky, and T. Kailath, “On the circuit complexity of neural networks,” in R.P. Lippmann, J.E. Moody, and D.S. Touretzky (eds.), Advs. Neural Info. Proc. Sys. 3, San Mateo, CA: Morgan Kaufmann, 1991, pp. 953959. A.A. Irmatov, “On the number of threshold functions,” Diskretnaya Matematika (in Russian), vol. 5, pp. 40-43, 1993. V. Bohossian, Neural Logic: Theory and Implementation, Ph.D. dissertation, Caltech, USA, 1998. Available: http://www.paradise.caltech.edu/papers/ vincent_thesis.ps A.C.C. Yao, “Separating the polynomial-time hierarchy by oracles,” Proc. FOCS’85, 1985, pp. 1-10. A.C.-C. Yao, “On ACC and threshold circuits,” Proc. FOCS’89, 1989, pp. 619-627. J. Håstad, “Almost optimal lower bounds for small depth circuits,” Proc. STOC’86, 1986, pp. 6-20. Extended version in Adv. Comp. Res., vol. 5, pp. 143170, 1987. R. Smolensky, “Algebraic methods in the theory of lower bounds for Boolean circuit complexity,” Proc. STOC’87, 1987, pp. 77-82. M. Furst, J.B. Saxe, and M. Sipser, “Parity, circuits and the po lyn om ial- time hierarchy,” Proc. FOCS’81, 1981, pp. 260-270. Also in Math. Sys. Theory, vol. 17, pp. 13-27, 1984. R. Paturi, and M. Saks, “On threshold circuits for parity,” Proc. FOCS’90, 1990, vol. I, pp. 397-404. I. Parberry, Circuit Complexity and Neural Networks, Cambridge, MA: MIT Press, 1994. V.P. Roychowdhury, A. Orlitsky, and K.-Y. Siu, “Lower bounds on threshold and related circuits via communication complexity,” IEEE Trans. Info. Th., vol. 40, pp. 467-474, 1994. K.-Y. Siu, V.P. Roychowdhury, and T. Kailath, “Depth-size tradeoffs for neural computations,” IEEE Trans. Comp., vol. 40, pp. 1402-1412, 1991. A.A. Razborov, “Lower bounds for the size of circuits of bounded depth with basis {∧, ⊕},” Math. Notes Acad. Sci. USSR, vol. 41, pp. 333-338, 1987. A. Hajnal, W. Maass, P. Pudlák, M. Szegedy, and G. Turán, “Threshold circuits of bounded depth.” Proc. FOCS’87, 1987, pp. 99-110. Also in J. Comp. Sys. Sci., vol. 46, pp. 129-154, 1993. K.-Y. Siu, and J. Bruck, “On the dynamic range of linear threshold elements,” Res. Rep. RJ 7237, IBM Yorktown Heights, New York, 1990. K.-Y. Siu, Complexity Issues in Circuits for Neural Computation, Ph.D. dissertation, Stanford Univ., USA, 1991.

[66]

[67]

[68]

[69] [70]

[71]

[72]

[73]

[74]

[75]

[76]

[77]

[78]

[79]

K.-Y. Siu, V.P. Roychowdhury, and T. Kailath, “Computing with almost optimal size threshold circuits,” Tech. Rep. Info. Sys. Lab., Stanford, 1990. Also in: Proc. ISIT’91, 1991, p. 370; in S.J. Hanson, J.D. Cowan, and C.L. Giles (eds.), Advs. Neural Info. Proc. Sys. 5, 1993, pp. 19-26. T. Hofmeister, W. Hohberg, and S. Köhling, “Some notes on threshold circuits and multiplication in depth 4,” Info. Proc. Lett., vol. 39, pp. 219-226, 1991. K.-Y. Siu, and V.P. Roychowdhury, “Optimal depth neural networks for multiplication and related problems,” in S.J. Hanson, J.D. Cowan, and C.L. Giles (eds.), Advs. Neural Info. Proc. Sys. 5, 1993, pp. 5964. Extended version published as: “On optimal depth threshold circuits for multiplication and related problems,” SIAM J. Discr. Math., vol. 7, pp. 284-292, 1994. E. Allender, “A Note on the power of threshold circuits,” Proc. FOCS’89, 1989, pp. 580-584. N. Immerman, and S. Landau, “The complexity of iterated multiplication,” Proc. CoCo’89, 1989, pp. 104-111. J. Bruck, Computing with Networks of Threshold Elements, Ph.D. dissertation, Stanford Univ., USA, 1989. J. Bruck, “Harmonic analysis of polynomial threshold functions,” SIAM J. Disc. Math., vol. 3, pp. 168177, 1990. P.W. Beame, S.A. Cook and H.J. Hoover, “Log depth circuits for division and related problems,” SIAM J. Computing, vol. 15, pp. 994-1003, 1986. K.-Y. Siu, J. Bruck, and T. Kailath, “Depth efficient neural networks for division and related problems,” Res. Rep. RJ 7946 (72929), IBM Yorktown Heights, New York, 1991. Also in IEEE Trans. Info. Theory, vol. 39, pp. 946-956, 1993. J. Bruck, and R. Smolensky, “Polynomial threshold functions AC 0 functions and spectral norms,” SIAM J. Comp., vol. 21, pp. 33-42, 1992. K.-Y. Siu, and J. Bruck, “Neural computing with small weights,” in J.E. Moody, S.J. Hanson, and R.P. Lippmann (eds.), Advs. Neural Info. Proc. Sys. 4, 1992, pp. 944-949. J.S. Shawe-Taylor, M.H.G. Anthony, and W. Kern, “Classes of feedforward neural networks and their circuit complexity,” Neural Networks, vol. 5, pp. 971-977, 1992. V. Beiu, J.A. Peperstraete, J. Vandewalle, and R. Lauwereins, ”Placing feedforward neural networks among several circuit complexity classes,” Proc. WCNN’94, 1994, pp. 584-589. V. Beiu, J.A. Peperstraete, J. Vandewalle, and R. Lauwereins, “On the circuit complexity of feedforward neural networks,” Proc. ICANN’94, 1994, pp. 521-524.

[80]

[81]

[82]

[83] [84]

[85]

[86]

[87]

[88]

[89]

[90]

[91]

[92]

[93]

[94] [95]

M. Goldmann, On Threshold Circuits and Related Models of Computation, Ph.D. dissertation, Royal Institute of Technology (Kungliga Tekniska Hogskolan, KTH), Sweden, 1992. M. Goldmann, and M. Karpinski, “Simulating threshold circuits by majority circuits,” Proc. STOC’93, 1993, pp. 551-560. Also in SIAM J. Comp., vol. 27, pp. 230-246, 1998. V. Beiu, and J.G. Taylor, “On the circuit complexity of sigmoid feedforward neural networks,” Neural Networks, vol. 9, pp. 1155-1171, 1996. I. Wegener, The Complexity of Boolean Functions, Stuttgart: Wiley-Teubner, 1987. V.P. Roychowdhury, K.-Y. Siu, and A. Orlitsky, Eds., Theoretical Advances in Neural Computation and Learning, Boston, MA: Kluwer Academic, 1994. K.-Y. Siu, V.P. Roychowdhury, and T. Kailath, Discrete Neural Computation: A Theoretical Foundation, Englewood Cliffs, NJ: Prentice-Hall, 1994. C.V. Papadopoulos, and T.S. Andronikos, “Modeling the complexity of parallel and VLSI computations with Boolean circuits, Microproc. & Microsys., vol. 19, pp. 43-50, 1995. W. Maass, G. Schnitger, and E.D. Sontag, “On the computational power of sigmoid versus Boolean threshold circuits,” Proc. FOCS’91, 1991, pp. 767776. K.-Y. Siu, V.P. Roychowdhury, and T. Kailath, “On the complexity of neural networks with sigmoid units,” Proc. NNSP’92, 1992, pp. 23-28. V. Beiu, Neural Networks Using Threshold Gates: A Complexity Analysis of Their Area- and Time-Efficient VLSI Implementations, PhD dissertation, Katholieke Univ. Leuven, Belgium, 1994. J. Myhill, and W.H. Kautz, “On the size of weights required for linear-input switching functions,” IRE Trans. Electron. Comp., vol. 10, pp. 288-290, 1961. S. Muroga, “Generation of self-dual threshold functions and lower bounds on the number of threshold functions and a maximum weight in switching theory and logical design,” AIEE Special Publication S134, pp. 170-184, 1962. J.S. Denker and B.S. Wittner, “Network generality, training required, and precision required,” in D.Z. Anderson (ed.), Neural Info. Proc. Sys., New York: AIPress, 1988, pp. 219-222. M. Stevenson, R. Winter, and B. Widrow, “Sensitivity of feedforward neural networks to weight errors,” IEEE Trans. Neural Networks, vol. 1, pp. 71-80, 1990. M. Stevenson, Sensivity of Madalines to Weight Errors, Ph.D. dissertation, Stanford Univ., USA, 1991. M. Stevenson, and S. Huq, “On the capability of threshold adalines with limited-precision weights,” Neural Computation, vol. 8, pp. 1603-1610, 1996.

[96]

[97]

[98]

[99] [100]

[101]

[102]

[103]

[104]

[105] [106]

[107]

[108]

[109]

[110]

[111]

[112]

M. Stevenson, “The effects of limited-precision weights on the threshold Adaline,” IEEE Trans. Neural Networks, vol. 8, pp. 549-552, 1997. J.L. Holt, and J.-N. Hwang, “Finite precision error analysis of neural network hardware implementations,” IEEE Trans. Comput., vol. 42, pp. 281-290, 1993. J. Håstad, “On the size of weights for threshold gates,” SIAM J. Discr. Math., vol. 7, pp. 484-492, 1994. J.H. Rief, “On threshold circuits and polynomial computations,” Proc. CoCo’87, 1987, pp. 118-123. I. Wegener, “Optimal lower bounds on the depth of polynomial size threshold circuits for some arithmetic functions,” Information Proc. Lett., vol. 46, pp. 85-87, 1993. K. Siu, V.P. Roychowdhury, and T. Kailath, “Toward massively parallel design of multipliers,” J. Parallel and Distributed Computing, 24, pp. 86-93, 1995. C.-H. Yeh, and E.A. Varvarigos, “Depth-efficient threshold circuits for multiplication and symmetric functions,” Proc. CoCooN’96, 1996, pp. 231-240. C.-H. Yeh, and E.A. Varvarigos, “New efficient majority circuits for the computation of some basic arithmetic functions,” J. Comput. Information, vol. 2, pp. 114-136, 1996. S. Cotofana, and S. Vassiliadis, “Periodic symmetric functions, serial addition, and multiplication with neural networks,” IEEE Trans. Neural Networks, vol. 9, pp. 1118-1128, 1998. H. Ling, “High speed binary adder,” IBM J. Res. & Develop., vol. 25, pp. 156-166, 1981. K.-Y. Siu, and J. Bruck, “Neural computations of arithmetic functions,” Proc. IEEE, vol. 78, pp. 16691675, 1990. N. Alon, and J. Bruck, “Explicit construction of depth-2 majority circuits for comparison and addition,” Res. Rep. RJ 8300 (75661), IBM Almaden Res. Center, 1991. Also in Proc. ISIT’93, 1993, p. 433; SIAM J. Discr. Math., vol. 7, pp. 1-8, 1994; U.S. Patent 5357528, Oct. 18, 1994. K.-Y. Siu, and J. Bruck, “On the power of threshold circuits with small weights,” SIAM J. Disc. Math., vol. 4, pp. 423-435, 1991. V. Beiu, J.A. Peperstraete, J. Vandewalle, and R. Lauwereins, “Addition using constrained threshold gates,” Proc. ConTI’94, 1994, pp. 166-177. S. Cotofana, and S. Vassiliadis, “Low weight and fan-in neural networks for basic arithmetic operations,” Proc. IMACS’97, 1997, vol. IV, pp. 227-232. V.M. Khrapchenko, “Asymptotic estimation of add ition tim e of a p ar allel add er,” Problemy Kibernetiki, vol. 19, pp. 107-122, 1967. English transl., Sys. Th. Res., vol. 19, pp. 105-122, 1970. R.P. Brent, and H.T. Kung, “A regular layout for

[113] [114] [115] [116]

[117] [118]

[119]

[120]

[121]

[122]

[123]

[124]

[125]

[126]

[127]

[128]

parallel adders,” IEEE Trans. Comp., vol. 31, pp. 260-264, 1982. R.E. Ladner, and M.J. Fischer, “Parallel prefix computations,” J. ACM, vol. 27, pp. 831-838, 1980. R. Betts, “Majority logic binary adder,” U.S. Patent 3440413, Apr. 22, 1969. H.M. Eckerlin, “Fluidic threshold gate,” U.S. Patent 3643861, Feb. 22, 1972. R.O. Winder, “Binary arithmetic circuits employing threshold gates in which both the sum and the carry are obtained in one gate delay interval,” U.S. Patent 3506817, Apr. 14, 1970. J.D. Heightley, “Threshold logic three-input adder,” U.S. Patent 3769499, Oct. 30, 1973. J.F. Ramos, and A.G. Bohórquez, “Two operand binary adders with threshold logic,” IEEE Trans. Comp., vol. 48, pp. 1324-1337, 1999. B.W.Y. Wei, and C.D. Thompson, “Area-time optimal adder design,” IEEE Trans. Comp., vol. 39, pp. 666-675, 1990. P.K. Chang, M.D.F Schlag, C.D. Thomborson, and V.G. Oklobdzija, “Delay optimization of carry-skip adders and block carry-lookahead adders using multidimensional programming,” IEEE Trans. Comp., vol. 41, pp. 920-930, 1992. T. Rhyne, “Limitations on carry lookahead networks,” IEEE Trans. Comp., vol. 33, pp. 373-374, 1984. T. Han, D.A. Carlson, and S.P Levitan, “VLSI design of high-speed, low-area addition circuitry,” Proc. ICCD’87, 1987, pp. 418-422. R.W. Doran, “Variants of an improved carry lookahead adder,” IEEE Trans. Comp., vol. 37, pp. 11101113, 1988. T. Lynch, and E.E. Swartzlander, Jr., “A spanning tree carry lookahead adder,” IEEE Trans. Comp., vol. 41, pp. 931-939, 1992. R.K. Montoye, “Area-time efficient addition in charge based technology,” Proc. DAC’81, 1981, pp. 862-872. T.P. Kelliher, R.M. Owens, M.J. Irwin, and T.-T. Hwang, “ELM a fast addition algorithm discovered by a program,” IEEE Trans. Comp., vol. 41, pp. 1181-1184, 1992. N.T. Quach, and M.J. Flynn, “High-speed addition in CMOS,” IEEE Trans. Comp., vol. 41, pp. 16121615, 1992. P.M. Kogge, and H.S. Stone, “A parallel algorithm for the efficient solution of a general class of recur-

[129]

[130]

[131]

[132]

[133] [134]

[135]

[136]

[137]

[138]

[139]

[140]

rence equations,” IEEE Trans. Comp., vol. 22, pp. 783-791, 1973. S. Ong, and D.E. Atkins, “A comparison of ALU structures for VLSI technology,” Proc. ARITH 6, 1983, pp. 10-16. T.F. Ngai, and M.J. Irwin, “Regular area-efficient carry-lookahead adders,” Proc. ARITH 7, 1985, pp. 9-15. V. Beiu, J.A. Peperstraete, J. Vandewalle, and R. Lauwereins, “Comparison and threshold gate decomposition,” Proc. MicroNeuro’93, 1993, pp. 8390. V. Beiu, J.A. Peperstraete, J. Vandewalle, and R. Lauwereins, “Area-time performances of some neural computation,” Proc. SPRANN’94, 1994, pp. 664668. S.A. Cannas, “Arithmetic perceptrons,” Neural Computation, vol. 7, pp. 173-181, 1995. S. Cotofana, Addition-Related Arithmetic Operations with Threshold Logic, Ph.D. dissertation, EE Dept. Technische Universiteit Delft, The Netherlands, 1998. Available: http://ce-serv.et.tudelft.nl/ orsres/1998/cont_COTO_2120.ps.gz S. Vassiliadis, S. Cotofana, and K. Berteles, “2-1 addition and related arithmetic operations with thresho ld lo gic,” IEEE Trans. Comp., vol. 45, pp. 1062-1067, 1996. C.-H. Yeh, E.A. Varvarigos, B. Parhami, and H. Lee, “Optimal-depth threshold circuits for multiplication and related problems,” Proc. Asilomar’99, 1999, Vol. II, pp. 1331-1335. C.-H. Yeh, E.A. Varvarigos, and B. Parhami, “Optimal-depth circuits for prefix computation and addition,” Proc. Asilomar’2000, 2 00 0, Vol. II, pp. 1349-1353. V. Beiu, “Neural addition and Fibonacci numbers,” Proc. IWANN’99, LNCS 1607, 1999, Vol. II, pp. 198-207. P. Celinski, Implementation of High Performance Digital VLSI Systems Using Threshold Logic, Ph.D. dissertation (in development), Dept. E&EE, Univ. of Adelaide, Australia. Address: http://www.eleceng.adelaide.edu.au/Personal/celinski M. Padure, Threshold Logic Arithmetic Units, Ph.D. dissertation (in development), EE Dept., Technical University of Delft, The Netherlands. Address: http://ce.et.tudelft.nl/~marius/