Training a GPT on Raw Sleep EEG, Ten Tokens a Second

During my DPhil I built a sleep headband and a small CNN, MorpheusNet, that stages sleep on a microcontroller. It classifies 30 second windows. It knows nothing about what happened a minute ago, and nothing about the second-scale events inside a window: spindles, K-complexes, slow waves. I wanted to see what happens if you go the other way. Treat EEG like audio, tokenize it at a fine time scale, train a language model on the tokens, and ask whether a model that only ever learned to predict the next 100 ms picks up the structure of a whole night.

This is what I did over two days, what it cost, and what came out. Code, checkpoints and all the numbers are at github.com/ali77sina/eeg-gpt.

Getting the data

Most of the time went into data, which is usually the honest answer.

I started with the Dreem Open Datasets I used in the original paper, 25 healthy nights (DOD-H) and 56 apnea patients (DOD-O). The S3 bucket they used to live in is gone. They are now on Zenodo as a 22 GB and a 36 GB zip, and my laptop had 7 GB free. The fix was to read the zip over HTTP range requests and pull one recording at a time, keep the two EEG channels and the hypnogram, delete the rest. Zenodo threw gateway timeouts every twenty minutes, so the loop retries.

Twenty-five nights is not enough to pretrain anything, so I built a corpus from every public polysomnography dataset that needs no approval: PhysioNet 2018 (994 nights), the Bitbrain open sleep dataset (128 nights recorded with both a clinical PSG and a forehead headband), Sleep-EDF (197), HMC (151), CAP (105), and the two Dreem sets. PhysioNet has an S3 mirror that is about fifty times faster than its website, which is the difference between an afternoon and a week. Everything got converted to one format: 100 Hz, bandpassed 0.5 to 40 Hz, robust-scaled per channel, plus the 30 second stage labels. That is 1,728 recordings, about 14,000 hours of nights, 8,861 single-channel streams, 48 GB. The channel names are a mess across datasets (A1 versus M1, three spellings of Fp2-F4, bare electrodes with unknown references), so each stream keeps its derivation as recorded and the model gets a tag token saying which dataset and channel it is looking at.

DOD-O was held out of training completely. I wanted one dataset the model had never touched, on a population it had never seen.

The model

A VQ-VAE tokenizer turns every 100 ms of EEG into one of 512 codes. It has 190k parameters, an EMA codebook with dead-code restarts, and a loss that mixes waveform error with a log spectrogram term so it does not smear the spindle band. On held-out data it reconstructs at about 11 dB SNR, keeps slow waves and spindles, and loses most of what is above 20 Hz. A night becomes roughly 300,000 integers.

On top of that, a plain decoder-only transformer: 8 layers, 384 wide, 15M parameters, 1024 tokens of context, which is 102 seconds. Next-token cross-entropy and nothing else. No stage labels anywhere in pretraining. Forty thousand steps at batch 64 on a rented RTX 4090, about 90 minutes and under two dollars.

Ten percent of subjects in each dataset were held out before anything was trained. Subjects with several nights stay on one side. I checked the split down to the token level after a reviewer in my head asked.

What it learned

Held-out loss ends at 7.5 bits per token. A bigram model gets 8.4 and unigram 8.7, so most of a token is unpredictable noise and the model is squeezing about one bit out of it. The same model trained on the 25 DOD-H nights alone reached 7.9, so seventy times more data bought 0.4 bits. Train and held-out loss sit on top of each other for the entire run. That is the tell that a model this size is not memorizing anyone.

The interesting part is not the loss. It is what a frozen model knows about hours when it was trained on seconds. I ran the model over every held-out night, averaged its hidden state per 30 second epoch, and fit a logistic regression from that to the sleep stage. Linear readout, no fine-tuning.

Purpose-built supervised models get somewhere between 80 and 89 on these, so a frozen 15M model with one linear layer is in the neighborhood but not at the front.

The number I care about is DOD-O, because nothing from it was in training and the population is different, apnea patients rather than healthy volunteers:

MorpheusNet, trained and tested within DOD-H with leave-one-subject-out, gets 84.3 on F3-M2 and 80.7 on F3-F4. So a model that never saw a label, or the dataset, lands two to three points behind a supervised model on its home turf, on a harder population.

Two smaller things. If you only give the readout the last one second of hidden state per epoch instead of thirty, accuracy drops from 76.5 to 73.3 percent across all datasets pooled. The stage is mostly there at the second scale. And N2 is the least predictable stage in every dataset, wake and N3 the most, which matches where the events are.

Where it fails

A readout trained on PhysioNet F3-M2 transfers to DOD-H F3-M2 at 80 percent and to HMC at 72, and then falls apart on other montages: 35 percent on Sleep-EDF Fpz-Cz, 55 on the Bitbrain forehead channel. The representation carries across datasets. The linear readout does not carry across electrode placements. Each montage needs its own few hundred labeled epochs, which is cheap but not zero.

The samples

Seed the model with 25 seconds of real held-out EEG in a given stage and let it continue. Blue is what actually followed the seed, orange is the model. It keeps the character of the stage without being told what the stage is: slow high-amplitude waves after an N3 seed, low fast activity after wake or REM. After the N2 seed it produces spindle-like 12 to 15 Hz bursts and a K-complex-shaped deflection of its own. Amplitudes are compressed and fast activity is smoothed, which is the tokenizer's doing more than the transformer's.

A free-running 60 second N2 sample stays N2-like the whole way. The spectra of real and generated continuations overlap from 0.5 to 30 Hz in every stage.

Scaling

I ran four sizes with the same data, the same 1024-token context and a fixed set of 256 held-out windows for evaluation: 3.7M, 15M, 39M and 115M parameters. The two small ones ran the full 150k steps. I stopped the two big ones part way, at 26k and 34k steps, once the answer was clear.

At step 26k, the last point all four reached, held-out loss goes 7.749, 7.627, 7.583, 7.556 bits. Every step up in size helps, and the gaps shrink roughly the way they do in language models. The 115M model at 34k steps had already passed the 15M model at 150k. Train and held-out stayed together for every size, so none of them had started memorizing when they stopped.

At 2.3B training tokens the 115M model sits at about 20 tokens per parameter, the usual compute-optimal ratio. Going bigger than that needs more nights, not more GPU time. Bigger sleep corpora exist but need data-access agreements, which is where this stopped.

What I found out afterwards

While this was running I went and read the recent literature properly. Three groups published close relatives in the last year: Hypnos from Oxford (next-token prediction on residual-quantized tokens at 1 Hz, 20,000 nights, up to 222M parameters), SleepGPT from Peking (59,000 hours, time-frequency pretraining), and Sleep2.0 from Johns Hopkins (11,000 nights, microstates found by clustering embeddings). All of them are bigger than this. The one thing this setup has that they do not is the token rate: 100 ms instead of a second, which is what you would need to forecast an event before it happens rather than describe it afterwards.

Whether that matters depends on what you want to do with it. For staging, it does not. For timing a stimulus to a slow wave, it might. I have not tested that yet.

What it cost

One Mac, a CPU pod for the downloads, two rented GPUs for a day. Under 30 dollars in compute, most of it spent on things that failed the first time: a TensorFlow-metal bug that drops a ReLU before a Concatenate layer, an open-file limit from memory-mapping 8,861 token files, community-cloud pods that copy data to each other at 2 MB/s and lose their disk when deleted. All of that is in the repo README so nobody has to find it twice.