System description

As shown in fig. 1, the proposed system can be divided into two functional blocks. The first block, mainly realized in software, captures the EEG signals from a subject and processes them to extract the distinct features of three vowels. Each feature is converted into a series of 32-bit binary code to be used by the 32 pre-neurons to generate a spike signal. The second block is a single-layer neural network that consists of 32 pre-neurons, 192 memristive synapses and 6 post-neurons. The pre-neurons are hard-coded into a field-programmable gate array (FPGA) and 192 memristive synapses in a cross-point memristive synapse array are selected for synaptic interconnection. A leaky integrate-and-fire neuron is used as the post-neuron and decision logic determines which post-neuron fires first based on the output signals from the post-neurons. The system’s overall control signals are also generated by the FPGA. Note that, to increase the system’s recognition rate, a pairwise comparison is performed. In other words, the post-neurons are paired into three groups and each group compares two of the three vowels, i.e. /a/ vs /i/, /a/ vs /u/ and /i/ vs /u/.

Figure 1 Proposed memristive HNN system for EEG pattern recognition Schematic illustrations and images of components for a proposed electronic system with memristive synapse. It can be categorized by two approaches: EEG preprocessing and implement of memristive hardware neural network. Full size image

The system operates in two different modes, learning and testing. In learning mode, the system’s learning network adjusts the conductance of memristive synapses using spike signals generated according to the EEG signals of the pre-selected vowel. In testing mode, the subject randomly chooses and imagines saying one of the three vowels. The system analyses the subject’s EEG signal and attempts to determine which of the three vowels was chosen.

EEG analysis of imagined speech

The experimental paradigm for this study is described in fig. 2(a). The acquired EEG data are segmented according to the trial and the stimuli associated with /a/, /i/ and /u/ (Fig. 2(b), top left). The segmented data are then analysed to identify the distinct features of the three experimental conditions (/a/, /i/ and /u/).

Figure 2 EEG analysis and processing (a) Experimental paradigm for EEG study consists of four parts. (b) EEG data analysis. (c) Signal processing. Full size image

Initially, continuous raw EEG data are segmented into each condition, with artifact rejection used to counter the low signal-to-noise ratio of EEG data. To estimate the activity evoked by the speech imagery data, we averaged the EEG data over all trials for each condition, a process known as time-lock analysis. A time-frequency analysis was then conducted using the Morlet wavelet. As shown in the top right panel of fig. 2(b), the alpha band (8–12 Hz) activities of each vowel are distinct. To identify the current sources of the speech imagery EEG data, we conducted source localization using a Laplacian-weighted current density estimator. As shown on the right of the second row in fig. 2(b), the current sources of speech imagery EEG data were located close to Broca’s and Wernicke’s areas, as well as the primary and secondary cortex. Broca’s area and Wernicke’s area are closely related to speech production and perception, respectively.

Based on these results, we extracted the features needed to classify speech imagery EEG data for each vowel. First, we applied an IIR band-pass filter (Butterworth order: 5, bandwidth: 8–30 Hz) and baseline correction using pre-stimulus data to eliminate residual noise. We then used independent component analysis to eliminate artefacts and decomposed the EEG data into intrinsic mode functions (IMFs) using multivariate empirical mode decomposition (MEMD). After MEMD, the dominant alpha-band IMF was extracted based on the time-frequency analysis. To enhance the classification performance, we applied a common spatial pattern filter that maximizes the variance between groups. Finally, we binarized the extracted features for input to the hardware.

Considerations for implementing a cross-point memristive synapse array

To simplify the architecture of HNN, memristive synapses with advanced synaptic behavior should use identical pulses and a simple cross-array structure with two terminals. Advanced synaptic behavior implies not only gradual, but also symmetric responses in both potentiation and depression. The asymmetric I-V characteristics (|current (@ + 1 V)|<<|current (@ − 1 V)|) of a previous Al/PCMO memristive synapse14 are shown in fig. 3(a). The Al/PCMO memristive synapse has an inhomogeneous barrier at the interface between the Al (reactive metal) and PCMO (p-type semiconductor)15. Because the conductance of the memristive synapse must be simultaneously updated in the HNN, the asymmetric conductance response (Fig. 3(b)) from identical pulses limits the overall accuracy of recognition16. Advanced synaptic behavior also requires the conductance to vary continuously when a relatively large voltage bias is applied, but the conductance should remain constant when a smaller or no bias is applied. This characteristic is vital for the nondestructive read/write scheme that eliminates the unintended switching issue, as explained in detail below. A simple, two-terminal cross array structure is necessary for the practical implementation of high-density HNN17.

Figure 3 Considerations for a memristive synapse implementation in cross-point array (a) The asymmetric I-V characteristics of Al/PCMO structure. (b) The asymmetric conductance response of Al/PCMO memristive synapse in identical pulse scheme. (c) By optimizing nitrogen concentration during TiN x deposition. (d) Symmetric I-V characteristics of AlO x /TiN x /PCMO structure. (e) The symmetric conductance response of AlOx/TiNx/PCMO memristive synapse in identical pulse scheme. (f) A Photograph of a memristive synapse array in 8 inch wafer. (g) A SEM view image of the cross-point memristive synapse array. (h) A TEM view image of the cross-point memristive synapse array. Full size image

In this work, we have enhanced the synaptic behavior without a complicated programming scheme by engineering a PCMO-based memristive synapse. We optimized the nitrogen concentration during TiN x deposition (Fig. 3(c)) to minimize the inhomogeneous barrier between the top electrode(TE)–PCMO interface18. This allowed us to obtain symmetric responses for the I-V and conductance characteristics (Fig. 3(d)). We also developed a 200 mm wafer-scale PCMO-based memristive synapse (Fig. 3(f)) that exhibits excellent switching uniformity and analogue memory behaviour (see Supplementary Fig. S1 online). Figure 3(g) shows a typical scanning electron microscope (SEM) image of a 32 × 6 array (192 cells) of the proposed PCMO-based memristive synapse. (Although memristive synapse arrays can be fabricated up to a size of 11 kbit, our memristive HNN allows only 32 × 6 arrays because of the limited switch logic.) If the memristive synapse and CMOS circuits are integrated in a single chip, there is no critical need for high-density memristive HNN. Figure 3(h) shows a TEM image of our PCMO-based memristive synapse fabricated using the 200 mm wafer process. It consists of active Pt/AlO x /TiN x /Pr 0.7 Ca 0.3 MnO 3 /Pt (from top to bottom) devices (see methods part for details).

Memristive HNN learning

To classify feature code by using memristive HNN, we require a learning algorithm. We propose a modified learning algorithm based on a conventional and widely used supervised learning algorithm19. In conventional supervised learning, feature codes and label data are required. The firing neuron is predetermined by the label data and synaptic weights are updated by feature codes, allowing the predetermined neuron to fire. To apply this algorithm to memristive HNN, the unintended switching problem must be carefully handled.

In the learning mode(rather than testing mode), the unintended switching issue causes non-linear conductance changes of unwanted memristive synapses. Generally, in memory applications20, this problem is easily solved using the half-bias scheme. However, it is not so simple in memristive HNN, because multiple memristive synapses must be updated at the same time (see Supplementary Fig. S2 online).

To overcome this problem, the conventional half-bias scheme was modified to enable its application to memristive HNN. The proposed learning requires two operation phases for a given feature code, potentiation and depression. The spike signals applied to the top electrode (TE) and bottom electrode (BE) are determined according to label data and feature codes. Note that when the pulse amplitude between the TE and BE is in the range |V|<V R ≈ 1 V (where V R is the read voltage), the state changes are negligible. To update the memristive synapse conductance, the training voltage (|V T | > V R ) should be supplied to the target cell by applying V H (or V L ) and V L (or V H ) to the TE and BE of the cross-point array, respectively. As shown in fig. 4, the proposed learning algorithm for a single feature code can be easily explained with simple memristive HNN (4 pre-neurons & 2 post-neurons) and detail of the circuit implementation are presented in the bottom-left of fig. 5.

Figure 4 The proposed learning scheme (a) An example of the feature code and label data to train memristive synapses of top1. (b) A schematic of the 4 pre-neurons and 2 post-neurons HNN according to operation phase. Full size image

Figure 5 Circuit Implementation The circuit includes a switch array, switch control logic, 32x6 cross-point memristive synapse array and six neuron circuits. Each neuron contains OPAMP based inverting integrator and comparator. Also, a schematic of HNN circuit according to operation phase are shown in this figure. Full size image

In the potentiation phase, if the label data is ‘1’, the TE of all synapses in the first row (labelled as Top1) is connected to V L , then the BE of each synapse is either connected to V H for a feature code of ‘1’, or to V CM for ‘0’. As V H and V L are 5 V and 3 V spike signals, respectively, the synapse whose BE is connected to V H sees a 2 V potential difference across it and increases its conductance. Whereas the synapses in the first row are trained, those in the second row retain their states by applying V CM to Top2. This guarantees that the potential difference across the synapses in the second row does not exceed V R and minimizes unintended switching effect.

In the depression phase, conductance of the selected synapses reduces by applying V H and V L to the TE and BE of the associated synapses, respectively. This process is repeated until all the memristive synapses are trained for all feature codes.

Notice that the initial conductance was set to a mid-value between minimum (≈1.5 nA/V) and maximum (≈5.5 nA/V) conductance. As shown in fig. 3(e), the change in conductance for each learning pulse depends on the current state of a memristive synapse. When a memristive synapse is in its low-conductance state, pulses will result in a rapid increase in conductance. The change rate is about 0.2 ~ 0.5 nA/V for each pulse. The change rate gradually decreases to about 0.02 ~ 0.05 nA/V as the synapse approaches its high-conductance state. Finally, the conductance becomes saturated.

Memristive HNN testing and classification results

In testing mode, applied feature codes to the memristive HNN are recognized by the decision logics based on the output signals of the post-neurons. Like the learning mode, the testing mode requires two operation phases, integrating and refractory. The spike signals applied in these periods are shown in the bottom-right of fig. 4(b) and detail of circuit implementation is shown in the bottom-right of fig. 5.

A leaky integrate-and-fire neuron, including a comparator and inverting leaky integrator, is used as the post neuron21. Integrator output decreases as the current flowing through the memristive synapse accumulates on the integration capacitor during the integrating phase. If the trained conductance is large, we observe a large amplitude in the input current. Thus, the output voltage quickly decreases. As soon as the integrator output drops below a neuron’s threshold voltage (V TH = 3 V), the comparator output generates a high value (logic value of 1) and the neuron is assumed to have fired. Note that the time required for the integrators to reach V TH with a given feature code depends on the trained conductance of the memristive synapse. After the integrating phase has completed, the refractory period starts. During this phase, the charge on the integration capacitor needs to be fully discharged through the leaky path to prepare the neuron for the next feature code.

The output measured at each integrator during the testing mode is shown in fig. 6. The outputs from six integrators for the feature /a/ are shown in the first column. As expected, the integrators’ output drops almost linearly to their saturation value which depends on the conductance of the memristive synapse and the RC time constant of the integrator, at different rates after an initial reset (=4 V). Thus, for /a/, two of the six neurons generate a fire signal as the integrators’ output reaches V TH and the feature code applied to the system is finally recognized as /a/ through the decision logic. After the integration phase, all integrators’ outputs are reset to 4 V during the refractory phase. The output responses for /u/ and /i/ are also shown in fig. 6, which are similar results as for /a/.