Inside the OS Interrupt Handlers: How IRQs and NMIs Reach Your Code

Atari text screen titled IRQ SPY (VIMIRQ HOOK) with hexadecimal counters for keyboard IRQs, timer 1 IRQs, other IRQs, BRKs seen by the hook and BRKs at VBREAK

Every interrupt on an Atari 8-bit, from a key press to the vertical blank, passes through a small piece of the operating system before it reaches the code that handles it. That code decides which of a dozen RAM vectors to jump through, in what order, and with which registers saved. If you install a display list interrupt, a POKEY timer or your own keyboard handler, you are plugging into this dispatcher, and knowing how it works explains several classic bugs. This guide traces both interrupt paths through the 400/800 and XL/XE operating systems. It ends with a tested program that hooks the master IRQ vector and catches the OS losing BRK instructions.

What you'll learn: the hardware sources of NMIs and IRQs, the ROM entry points, the RAM vectors from VDSLST ($0200) to VPIRQ ($0238), the order in which the OS checks IRQ sources on each OS version, what is on the stack at each vector, how to hook VIMIRQ, and the rules for safe handlers.

What you need: MADS and an emulator for the example, and the background from Vertical Blank Interrupts and Display List Interrupts. Everything below was checked by disassembling the genuine OS-B and XL/XE Rev. 2 ROMs in atari800's monitor, and the example was run on both.

1. Two lines into the CPU

The 6502 has two interrupt inputs, and the Atari's chips share them out:

  • NMI (non-maskable): driven by ANTIC. Its sources are the display list interrupt (DLI), the vertical blank interrupt (VBI) and, on the 400 and 800 only, the RESET key. You enable them in NMIEN ($D40E): bit 7 for DLIs, bit 6 for the VBI. The status is read from NMIST ($D40F) and cleared by writing NMIRES (the same address). SEI does not stop an NMI.
  • IRQ (maskable, blocked by SEI): driven by POKEY, the PIA and, on XL/XE machines, devices on the Parallel Bus Interface. POKEY alone has eight sources, selected by IRQEN ($D20E). The 6502's BRK instruction also enters through the IRQ vector.

The two behave differently. An NMI is edge-triggered: a single event that runs the handler once. An IRQ is level-triggered: POKEY holds the IRQ line down until every pending source has been cleared, so the CPU keeps re-entering the handler until all of them are serviced. That is why the OS handler services one source per entry and simply returns.

2. The ROM entry points

The CPU fetches its handler addresses from the top of memory. We read them from the ROMs themselves:

Vector OS-B (400/800) XL/XE OS Rev. 2
NMI ($FFFA) $E791 $C018
RESET ($FFFC) $F125 (power-up) $C2AA
IRQ ($FFFE) $E6F3 $C02C

The NMI entry is short. It tests bit 7 of NMIST and, if a DLI is pending, jumps straight through VDSLST ($0200) with nothing saved: your DLI gets control with only the return address and flags on the stack. Otherwise, the 400/800 OS checks bit 5 for the RESET key and jumps to the warm start. If neither applies, it assumes a vertical blank, pushes A, X and Y, writes NMIRES and jumps through VVBLKI ($0222). The XL/XE version does the same without the RESET test, because RESET on those machines is a real CPU reset, and it adds a CLD.

The IRQ entry is a single instruction on both: JMP (VIMIRQ). VIMIRQ ($0216) is the master IRQ vector, and it normally points to the OS dispatcher ($E706 in OS-B, $C030 on XL/XE).

3. The vector table

The OS initialises these RAM vectors at power-up and again on every RESET. Default values are from the ROMs as dumped.

Address Name Source OS-B default XL/XE default
$0200 VDSLST DLI (NMI) $E790 (RTI) $C0CE (RTI)
$0202 VPRCED PIA, serial proceed line $E78F (PLA, RTI) $C0CD
$0204 VINTER PIA, serial interrupt line $E78F $C0CD
$0206 VBREAK BRK instruction $E78F $C0CD
$0208 VKEYBD POKEY key press $FFBE $FC19
$020A VSERIN POKEY serial input ready $EB0F $EB2C
$020C VSEROR POKEY serial output ready $EA90 $EAAD
$020E VSEROC POKEY serial output complete $EACF $EAEC
$0210 VTIMR1 POKEY timer 1 $E78F $C0CD
$0212 VTIMR2 POKEY timer 2 $E78F $C0CD
$0214 VTIMR4 POKEY timer 4 $E78F $C0CD
$0216 VIMIRQ every IRQ $E706 $C030
$0222 VVBLKI VBI, immediate $E7AE $C0E2
$0224 VVBLKD VBI, deferred $E905 $C28A
$0236 BRKKY BREAK key (OS-B and later) $E754 $C092
$0238 VPIRQ PBI device IRQ (XL/XE) none $C96E

What's on the stack when control reaches each vector matters:

  • VDSLST: the return address and flags only. Save what you use and end with RTI.
  • VVBLKI and VVBLKD: A, X and Y have already been pushed. End by jumping to XITVBV ($E462), which restores them.
  • Every IRQ vector except VIMIRQ: the OS has pushed A. Your handler must end with PLA then RTI. The default "ignore" routine is exactly those two instructions.
  • VIMIRQ: nothing has been pushed. Code here must preserve every register it touches.

On the original OS-A, the BREAK key was handled inside the dispatcher, with no vector. OS-B added BRKKY, and OS-B also fixed a bug that kept OS-A from ever jumping through VTIMR4.

4. The order of the checks

The dispatcher services one source per entry, and it checks them in a fixed order. In the OS-B and XL/XE ROMs, the code is table-driven:

  1. Serial input ready (IRQST bit 5) is checked first, because SIO can't afford to miss an incoming byte.
  2. On XL/XE only: a PBI device interrupt. The OS ANDs the device status at $D1FF with the mask at $0249 and jumps through VPIRQ if any bit is set.
  3. Then a table loop, in this order: serial output ready (bit 4), serial output complete (bit 3, only if enabled in POKMSK), timer 1 (bit 0), timer 2 (bit 1), timer 4 (bit 2), keyboard (bit 6) and BREAK (bit 7).
  4. If no POKEY bit was pending, it checks the PIA: the proceed line (bit 7 of PACTL, $D302) through VPRCED, then the interrupt line (bit 7 of PBCTL, $D303) through VINTER.
  5. Last of all, it looks at the B flag in the processor status that the CPU pushed. If B is set, the entry was a BRK, and it jumps through VBREAK. Otherwise it pulls A and returns.

The original OS-A checks serial output ready before serial input, but the rest of the order is the same.

To "clear the interrupt", the dispatcher writes IRQEN with that one bit off, then writes POKMSK ($10) back to IRQEN. IRQEN is write-only, and IRQST ($D20E when read) shows a pending source as a 0 bit. POKMSK is the OS's copy of what IRQEN should contain. That leads to the first rule: whenever you change IRQEN, change POKMSK too, with interrupts masked. The OS User's Manual gives exactly that pattern:

        sei
        lda POKMSK
        ora #$01        ; enable timer 1 (use AND #$FE to disable)
        sta POKMSK
        sta IRQEN
        cli

If you write IRQEN without updating POKMSK, the next IRQ of any kind restores the old mask and your source silently switches off.

5. Hooking VIMIRQ

The per-source vectors are enough for most programs. Hooking VIMIRQ itself is for code that needs to see every IRQ, such as a debugger, a profiler, or a program that wants to serve one source faster than the dispatcher's table loop. The safe way is to do your work and then jump to the old handler, so the OS still clears and dispatches the interrupt.

The example below does exactly that. Its hook looks at each IRQ before the OS does and counts it as a key press, a timer 1 tick, a BRK or something else. To have something to count, it also starts POKEY timer 1 at about 250 Hz with an empty handler, installs a VBREAK handler, and executes one BRK per frame in its main loop.

; IRQ spy: hooks VIMIRQ ahead of the OS dispatcher, adds a POKEY
; timer 1 handler and a BRK handler, and shows live counters.
POKMSK  = $10
RTCLOK  = $12
SAVMSC  = $58
VBREAK  = $0206
VTIMR1  = $0210
VIMIRQ  = $0216
AUDF1   = $D200
AUDC1   = $D201
AUDCTL  = $D208
STIMER  = $D209
IRQEN   = $D20E
IRQST   = $D20E
PTR     = $CB

        org $3000
start   sei                 ; no IRQs while vectors are half-written
        lda VIMIRQ          ; keep the OS dispatcher's address
        sta oldvim
        lda VIMIRQ+1
        sta oldvim+1
        lda #<spy
        sta VIMIRQ
        lda #>spy
        sta VIMIRQ+1
        lda #<timer1
        sta VTIMR1
        lda #>timer1
        sta VTIMR1+1
        lda #<brkhnd
        sta VBREAK
        lda #>brkhnd
        sta VBREAK+1
        lda #0
        sta AUDCTL          ; 64 kHz clock
        sta AUDC1           ; silent
        lda #$FF
        sta AUDF1           ; 256 x 28 cycles: about 250 per second
        lda POKMSK
        ora #$01            ; enable the timer 1 IRQ
        sta POKMSK
        sta IRQEN
        sta STIMER          ; start the timers
        cli
        jsr labels

main    lda RTCLOK+2        ; once per frame...
wait    cmp RTCLOK+2
        beq wait
        brk                 ; ...execute a BRK
        dta $EA             ; padding byte: RTI returns past it
        jsr show
        jmp main

; --- VIMIRQ hook: entered with nothing pushed, must keep A ---
spy     pha
        txa
        pha
        tsx                 ; stack: X, A, P, return address
        lda $0103,x         ; the P register pushed by the CPU
        and #$10            ; B flag set: this is a BRK, not an IRQ
        beq hwirq
        inc nbrks
        bne chain
        inc nbrks+1
        bne chain
hwirq   lda IRQST           ; pending sources read as 0 bits
        and #$40
        bne notkey
        inc nkey
        bne chain
notkey  lda IRQST
        and #$01
        bne other
        inc ntim
        bne chain
        inc ntim+1
        bne chain
other   inc noth
        bne chain
        inc noth+1
chain   pla
        tax
        pla
        jmp (oldvim)        ; let the OS dispatch it as usual

; --- POKEY timer 1: OS pushed A, so finish with PLA/RTI ---
timer1  pla
        rti

; --- BRK: OS pushed A as well ---
brkhnd  inc nbrk
        bne bx
        inc nbrk+1
bx      pla
        rti

; --- display ---
labels  ldx #0
l1      lda text,x
        jsr scrxy
        inx
        cpx #textlen
        bne l1
        rts
scrxy   pha                 ; store A at screen offset X
        txa
        tay
        lda SAVMSC
        sta PTR
        lda SAVMSC+1
        sta PTR+1
        pla
        sta (PTR),y
        rts

show    lda SAVMSC
        sta PTR
        lda SAVMSC+1
        sta PTR+1
        ldy #40+20
        lda nkey
        jsr hex2
        ldy #80+20
        lda ntim+1
        jsr hex2
        lda ntim
        jsr hex2
        ldy #120+20
        lda noth+1
        jsr hex2
        lda noth
        jsr hex2
        ldy #160+20
        lda nbrks+1
        jsr hex2
        lda nbrks
        jsr hex2
        ldy #200+20
        lda nbrk+1
        jsr hex2
        lda nbrk
        jsr hex2
        rts
hex2    pha
        lsr
        lsr
        lsr
        lsr
        tax
        lda hexd,x
        sta (PTR),y
        iny
        pla
        and #$0F
        tax
        lda hexd,x
        sta (PTR),y
        iny
        rts

text    dta d'IRQ SPY (VIMIRQ HOOK)                   '
        dta d'KEYBOARD IRQS:                          '
        dta d'TIMER 1 IRQS:                           '
        dta d'OTHER IRQS:                             '
        dta d'BRKS SEEN BY HOOK:                      '
        dta d'BRKS AT VBREAK:                         '
textlen = *-text
hexd    dta d'0123456789ABCDEF'
oldvim  dta a(0)
nkey    dta 0
ntim    dta a(0)
noth    dta a(0)
nbrk    dta a(0)
nbrks   dta a(0)

        run start

Assemble with mads irqspy.asm -o:irqspy.xex and run it. Points worth noticing:

  • The vectors are changed under SEI. An IRQ arriving between the two bytes of a vector would jump to a half-written address.
  • The hook saves A and X, because VIMIRQ is entered with nothing pushed, and restores both before JMP (oldvim). The OS dispatcher then starts exactly as if the hook were not there.
  • Telling BRK from an IRQ works the way the OS does it: after the two pushes, the processor status pushed by the CPU is at $0103,X, and bit 4 (B) is set only for BRK.
  • BRK is a two-byte instruction. The CPU pushes the address of BRK plus 2, so RTI skips the byte after it. The example puts a $EA there. Debuggers often use that byte as a breakpoint number.

6. What the test showed: lost BRKs

We ran the program on the genuine XL/XE Rev. 2 and OS-B ROMs in atari800 and typed a few keys. The keyboard count matched the keys pressed, the timer count came to about 249 per second (the expected 1,789,773 ÷ 7,168 cycles on NTSC), and "other" stayed at zero. But the two BRK counters disagreed. In one 12-second run on the XL OS, the hook saw 739 BRKs and VBREAK received only 724. The OS-B run lost 14 out of 661. With timer 1 left off, the two counts were identical.

The cause is the dispatch order in section 4. BRK and hardware IRQs share one vector, and the dispatcher looks at POKEY's pending bits before it looks at the B flag. If a timer interrupt is pending at the moment a BRK enters, the OS services the timer, returns with RTI, and never calls VBREAK. Because the CPU has already moved past the BRK, it is lost. The general rule is that a handler sharing the IRQ vector with BRK must service the BRK before it exits, because unlike a hardware IRQ, nothing will raise it again. The OS doesn't. If you rely on BRK for breakpoints or system calls while POKEY interrupts are active, check the B flag yourself in a VIMIRQ hook, as the spy does, and handle the BRK there.

There is a rarer case too: a BRK that starts within a few cycles of a DLI or VBI can be taken over by the NMI, so the NMI handler runs and the BRK is lost.

7. Rules for interrupt handlers

  • Keep the stack balanced. IRQ vectors other than VIMIRQ end with PLA/RTI. VBIs end with JMP XITVBV (or JMP SYSVBV for an immediate VBI that wants the OS's processing to follow). DLIs end with RTI.
  • Don't clear the I flag in a DLI, an immediate VBI or a timer 1 routine. The OS manual warns that if the NMI interrupted an IRQ handler, clearing I lets IRQs nest inside it. The stage 2 VBI clears I itself, after checking that it is safe.
  • Critical sections. The VBI checks CRITIC ($42) and the I flag of the code it interrupted. If either is set, it does only stage 1 (the frame counter, attract mode and system timer 1) and skips the shadow-register updates, timers 2–5, keyboard repeat and controller reads. SIO sets CRITIC while serial I/O is in progress.
  • Expect delays. An interrupt can only be taken between instructions, and STA WSYNC can stretch one instruction by up to a scan line. The OS keyboard handler uses WSYNC to make its click, so interrupts can be delayed while a key is being read.
  • Turning a POKEY source off safely needs SEI first, then IRQEN and POKMSK. An IRQ that was already signalled can still arrive just after the write to IRQEN.
  • Use SETVBV ($E45C) to change VVBLKI and VVBLKD, so a vertical blank can't arrive halfway through the change.

Troubleshooting

  • Crash after a few seconds of a timer IRQ: the handler forgot the PLA before RTI, or used X or Y without saving them.
  • The interrupt fires once, then never again: IRQEN was written without POKMSK, and the next keyboard IRQ restored the old mask.
  • Keys stop working: your VIMIRQ hook doesn't chain to the old vector, or it changed A before jumping.
  • BRK breakpoints are occasionally ignored: a POKEY interrupt was pending. See section 6.
  • The handler works on OS-B but not on OS-A: timer 4 is never dispatched on OS-A. Use timer 1 or 2.

Further reading

  • Atari, Inc., Operating System User's Manual, Section 6 "Interrupt Processing", with the stack-content table and the POKMSK examples.
  • Atari, Operating System Source Listing (400/800 OS Rev. A): the interrupt handler module.
  • Ian Chadwick, Mapping the Atari: locations 512–569, 53774 (IRQEN/IRQST) and 54286–54287 (NMIEN/NMIST).
  • Avery Lee, Altirra Hardware Reference Manual: section 3.4 (6502 interrupts, BRK hijacking) and section 5.7 (POKEY interrupt timing).
  • On this site: Vertical Blank Interrupts, Display List Interrupts (DLIs), Writing an Interrupt-Driven Keyboard Handler, Keeping Time: RTCLOK, System Timers and POKEY Hardware Timers, POKEY Timer Interrupts in Depth.

Credits