Designs of Second-Order Associated Memory Networks with Threshold Logics: Winner-Take-All and Selective Voting

Abstract

The capacity of an order-d associative memory model is O(N^d/logN) where N is the memory size in bit. In contrast, the capacity of the Hopfield network is limited to O(N/logN). Among higher order associative memory models (d > 1), the second order memory (d = 2) has attractive properties: a relatively small implementation cost of O(N²), a small number of spurious states, and the presence of a diagonalization form. Due to these properties, it is of both practical and scientific interests to investigate efficient computational mechanisms of such network. One disadvantage of higher order associative memory is that it cannot be implemented with simple threshold neurons or McCulloch-Pitts neurons, thus a direct implementation of its computational mechanism on a biological substrate is questionable and its silicon implementation is expensive. In this paper, we propose two approximation models of a second order associative memory using threshold logics. Both are two-layered and employ eigenvalue decomposition of the correlation tensor. The first model uses a winner-takeall mechanism and the second uses a weighted voting by those with significant responses. Architectural-level designs of these memory models are presented. Extensive numerical simulations demonstrate effectiveness of the proposed models in retrieving contents with noisy probe vectors.