A sigmoid function is a mathematical function having a characteristic “S”-shaped curve or sigmoid curve.
Long before transformers and ReLU activations took over, the sigmoid function did a lot of the heavy lifting inside early neural language models. LSTM networks — the architecture behind machine translation and language modelling through the early-to-mid 2010s — use sigmoid functions as gates: the input, forget, and output gates each squash their signal into the range (0, 1), acting like a dial for how much information should flow through the cell at every step of a sequence. That gating mechanism is what let those networks learn which words to remember and which to forget across long sentences, long before self-attention took over the job.
That’s where this site’s name comes from.