Braille Recognition Based on a Dual-Mode Tactile Sensor With Piezoresistive and Piezoelectric Properties by the CNN-ResNet-BiLSTM Fusion Model
Braille is an essential tactile communication system for visually impaired individuals. Traditional image-based Braille recognition methods face difficulties under worn, low-light, or degraded conditions and often require high computational power. In contrast, flexible tactile sensors, which utilize the physical properties of Braille dots, enable real-time recognition without optical constraints.
This research introduces a micropyramid-structured dual-mode tactile sensor (MCP-DTS) that integrates a multiwalled carbon nanotube (MWCNT) cotton fabric piezoresistive layer with a polyvinylidene fluoride (PVDF) piezoelectric layer. Inspired by the human skin’s slow- and fast-adapting mechanoreceptors, the piezoresistive sensor detects static pressures, while the piezoelectric sensor responds to dynamic forces.
Finite element simulations were used to optimize six substrate designs. The micropyramid geometry showed superior stress transfer and displacement, boosting sensor sensitivity and responsiveness.
These sensors are separated by an ultrathin Polydimethylsiloxane (PDMS) insulating layer and assembled on an optimized micropyramid-structured PDMS substrate. The fabrication process involved soaking cotton fabric in a solution of MWCNTs to create the piezoresistive layer, attaching electrodes to polyvinylidene fluoride (PVDF) films for piezoelectric sensing, and stacking PDMS films to provide electrical isolation and mechanical stability.
The entire sensor was encapsulated in polyimide film and has a compact form factor of 10×10×3.11 mm. Performance testing over a pressure range of 0–10 N revealed that the micropyramid substrate doubled the sensitivity of the piezoelectric sensor (12.15 mV/N compared to 6.86 mV/N) and significantly improved piezoresistive sensitivity, especially at low pressures.
The dual-mode design allowed precise detection of transient and steady-state tactile signals, as confirmed by repeated pressure-holding and release cycles. Long-term stability tests conducted over 800 loading cycles showed a consistent voltage response with minimal signal distortion.
MCP-DTS technology was used to scan 25 Braille characters with an automated sliding platform that simulated finger movements. Signal analysis showed that piezoresistive and piezoelectric responses displayed distinct symmetry and waveform patterns, which offered a diverse feature space for classification.
To process this data, a fusion model combining a convolutional neural network (CNN)-residual network (ResNet)-bidirectional long short-term memory (BiLSTM) was developed. The convolutional layers extracted local features, residual connections stabilized deep training, and the bidirectional LSTM analyzed the temporal dependencies in the signals.
The model was trained on 1800 samples and validated and tested on 600 each, achieving 97.17% accuracy for classifying 25 Braille characters. This fusion approach outperformed individual sensor modalities and benchmark algorithms, including SVM, random forest, Transformer, CNN, and LSTM.
To evaluate real-world usability, the sensor was mounted on a user's index finger for natural Braille sliding gestures. Despite fluctuations in signal caused by variations in pressure and finger deformation, the trained model maintained the classification accuracy of 89.17%, demonstrating the model's robustness under practical conditions.
The integration of MCP-DTS with the CNN-ResNet-BiLSTM model represents a major step forward in tactile Braille recognition technology. The dual-mode sensor's complementary sensing capabilities and optimized microstructure yield accurate and stable tactile data, while the advanced fusion algorithm enables comprehensive feature extraction, leading to high-precision classification. It holds strong potential for supporting visually impaired individuals and advancing intelligent tactile interface technologies.