A pattern-based cognitive communication system (PBCCS) that optimizes non-periodic RF waveforms for security applications is proposed. PBCCS is a cross-layer approach that merges the channel encoding and modulation. The transmitter encodes sequences of bits into continuous signal patterns by selecting the proper symbol glossaries. The cognitive receiver preprocesses the received signal by extracting a limited set of wavelet features. The extracted features are fed into an artificial neural network (ANN) to recover the digital data carried by the distorted symbol. The PBCCS system offers a flexible management for robustness against a high noise level and increases the spectral efficiency. In this study, the spectral efficiency and robustness of a PBCCS scheme for an additive white Gaussian noise (AWGN) channel is investigated. The results show that at an SNR of -5 dB, a 3-bit glossary achieves a bit error rate (BER) of 10(-5). Also, the link spectral efficiency (LSE) of the proposed system is 2.61 bps/Hz.