Signed Numbers on the 6502: Two's Complement, Overflow and Signed Comparisons

Atari text screen with four lines of hexadecimal test results at the top and, below, a trail of asterisks tracing a ball that falls and bounces three times with decreasing height

Sooner or later every Atari game needs negative numbers: a player moving left, a ball whose vertical speed changes sign when it bounces, a sine table for a wave effect. The 6502 has no separate "signed" instructions, but its ADC, SBC and flags handle negative numbers perfectly well once you know the convention, two's complement, and which flag to look at. This guide explains both, corrects a common misunderstanding about the overflow flag, and finishes with tested routines for signed comparison, sign extension and halving, used in a small bouncing-ball simulation.

What you'll learn: how two's complement works and why the same ADC serves signed and unsigned numbers, what C, V and N mean for each, how to negate 8- and 16-bit values, how to compare signed numbers correctly, sign extension, arithmetic shift right, and signed fixed-point motion.

What you need: MADS and an emulator, and the basics from Getting Started with 6502 Assembly on the Atari 8-bit. Integer Maths on the 6502: Multiply, Divide, BCD Scores and Printing Numbers covers unsigned multiplication and division. The example was assembled with MADS and run in atari800, and every result shown below was read off its screen.

1. Two's complement

A byte holds 256 patterns. Unsigned, they mean 0 to 255. Signed, the patterns from $80 to $FF are reused for −128 to −1:

Byte Unsigned Signed
$00 0 0
$7F 127 +127
$80 128 −128
$FB 251 −5
$FF 255 −1

Bit 7 is the sign, but the rest of the byte is not simply the size of the number. −5 is not $85. It is the value that gives zero when added to +5, ignoring the carry out of bit 7: $FB + $05 = $100. To negate a number, invert every bit and add one:

        lda #5
        eor #$FF        ; $FA (one's complement)
        clc
        adc #1          ; $FB = -5

Atari Roots walks through why inverting alone (one's complement) gives wrong sums, and why adding the one fixes it. The practical point is this: because negative numbers are defined so that ordinary binary addition works, the 6502 uses exactly the same ADC and SBC for signed and unsigned numbers. Only your interpretation of the result, and of the flags, changes.

2. What the flags mean

After ADC or SBC, the CPU sets every flag as if both interpretations were possible. You pick the flags that match the one you mean:

  • C (carry) is the unsigned overflow: set if an addition passed 255, or clear if a subtraction needed to borrow. For signed numbers it means nothing on its own.
  • V (overflow) is the signed overflow: set if the true signed result does not fit in −128 to +127. That happens when two numbers of the same sign are added and the result's sign comes out different.
  • N (negative) is simply bit 7 of the result, the sign, which is correct only if V is clear.

The example's first three tests show the difference (results printed as result, C, V):

Operation Result C V Meaning
−5 + 8 ($FB + $08) $03 1 0 +3: correct; the carry is irrelevant
100 + 50 ($64 + $32) $96 0 1 signed overflow: 150 doesn't fit, $96 reads as −106
−100 − 50 ($9C − $32) $6A 1 1 signed overflow again: −150 doesn't fit

A common misunderstanding

Atari Roots tells readers to use the overflow flag "rather than the carry flag to carry numbers from one byte to another" when adding signed numbers. That is not how multi-byte arithmetic works. A 16-bit signed number is simply a 16-bit two's-complement number, and its bytes are chained with the carry, exactly like unsigned numbers. V matters only after the last (highest) byte, where it tells you whether the whole result overflowed. Test 9 in the example adds $00F0 (240) and $FFE0 (−32) with an ordinary CLC, ADC, ADC sequence and gets $00D0 (208), the correct answer.

V has two other quirks. There is CLV but no "set V" instruction, although BIT sets V from bit 6 of the byte it tests. And V is changed only by ADC, SBC, BIT, CLV, PLP and RTI, never by CMP.

3. Comparing signed numbers

CMP is designed for unsigned comparison. It sets C if A ≥ the operand, as unsigned numbers. Test 6 shows the result for $FD (−3) against 2: carry set, "greater or equal", which is wrong for signed numbers. CMP also leaves V alone, so you can't fix the result afterwards.

The standard method is to do a real subtraction with SBC and look at N EOR V. If there was no overflow, N is the true sign of A − B. If there was, the sign came out inverted. So "A < B (signed)" is exactly "N ≠ V". The example's routine turns that into the carry flag:

scmp2   sec                 ; signed A < 2 ?  returns C=1 if so
        sbc #2
        bvc sc1             ; no overflow: N is the true sign
        eor #$80            ; overflow: the sign came out inverted
sc1     asl                 ; move the corrected sign bit into carry
        rts

Test 5 prints 01 for "−3 < 2". We also checked the N EOR V rule against all 65,536 pairs of signed bytes with a short script, with no exceptions. Another well-known trick is to flip bit 7 of both values with EOR #$80 and then compare them unsigned. It works because it maps −128…+127 onto 0…255 in order.

4. Widening and halving

Sign extension. To turn a signed byte into a 16-bit value, the new high byte must be $FF for a negative number and $00 for a positive one, never just 0. Test 7 turns $FD into $FFFD. Forgetting this is the classic bug when an 8-bit speed is added to a 16-bit position: moving left becomes a jump of almost 256 to the right.

Arithmetic shift right. LSR divides an unsigned number by two, but it shifts a 0 into bit 7, which turns a negative number positive. To halve a signed number, the sign must be copied in instead. The 6502 has no instruction for that, but two do the job:

        cmp #$80        ; carry = bit 7 (the sign)
        ror             ; shift right, copying the sign back into bit 7

Test 8 halves −7 ($F9) and gets −4 ($FC). The result rounds down (towards minus infinity), not towards zero, so −1 halved stays −1. The 16-bit version does CMP #$80 on the high byte, then ROR high and ROR low.

Branch offsets are signed bytes too. A branch can reach 128 bytes back or 127 bytes forward from the following instruction. While writing this example, MADS rejected two branches as "out of range" when the bounce code grew. The fix is a branch with the opposite condition over a JMP.

5. Signed motion: a bouncing ball

Games rarely need signed multiplication, but they constantly add signed velocities to positions. Fixed point makes that smooth. The example keeps the ball's height y and its vertical speed vy as 16-bit 8.8 numbers: the high byte is whole screen rows, the low byte is 1/256ths of a row. Each step:

  1. vy = vy + gravity (gravity is $0030, about 0.19 rows per step per step).
  2. y = y + vy, an ordinary 16-bit addition. When the ball rises, vy is negative and the addition makes y smaller, with no special code.
  3. At the floor, negate vy with a 16-bit 0 − vy, then remove a quarter of it, using two 16-bit arithmetic shifts and a subtraction, so each bounce is lower.

The trail of asterisks in the screenshot is the result: three bounces of decreasing height across the 40 columns.

6. The complete program

; Signed arithmetic on the 6502: test results printed as hex, then a
; bouncing-ball trail computed with a signed 8.8 fixed-point velocity.
SAVMSC  = $58
PTR     = $CB               ; screen pointer
RES     = $CD               ; scratch

        org $3000
start   lda SAVMSC
        sta PTR
        lda SAVMSC+1
        sta PTR+1

; 1. -5 + 8: signed result 3, carry set (ignore it), no overflow
        clc
        lda #$FB            ; -5
        adc #$08
        jsr flags           ; show result, C and V
; 2. 100 + 50: 150 doesn't fit in -128..127 -> V set
        clc
        lda #100
        adc #50
        jsr flags
; 3. -100 - 50: -150 doesn't fit either -> V set
        sec
        lda #-100
        sbc #50
        jsr flags
; 4. negate 5: invert all bits, add one
        lda #5
        eor #$FF
        clc
        adc #1
        jsr hexa
; 5. is -3 < 2? signed compare: subtract, then N EOR V
        lda #$FD            ; -3
        jsr scmp2           ; carry = 1 if A < 2 (signed)
        lda #0
        rol
        jsr hexa
; 6. the same with an unsigned CMP: $FD >= 2, so carry is set
        lda #$FD
        cmp #2
        lda #0
        rol
        jsr hexa
; 7. sign-extend -3 to 16 bits
        lda #$FD
        jsr sext            ; high byte in A, low byte kept in RES
        jsr hexa
        lda RES
        jsr hexa
; 8. arithmetic shift right of -7: -4 (rounds down)
        lda #$F9
        cmp #$80            ; copy the sign into carry...
        ror                 ; ...and shift it back in
        jsr hexa
; 9. 16-bit: $00F0 + (-32 = $FFE0) = $00D0; the carry chains the bytes
        clc
        lda #$F0
        adc #$E0
        sta RES
        lda #$00
        adc #$FF
        jsr hexa
        lda RES
        jsr hexa

        jsr bounce
        jmp *

scmp2   sec                 ; signed A < 2 ?  returns C=1 if so
        sbc #2
        bvc sc1             ; no overflow: N is the true sign
        eor #$80            ; overflow: the sign came out inverted
sc1     asl                 ; move the corrected sign bit into carry
        rts

sext    sta RES             ; sign-extend A: returns high byte in A
        and #$80
        beq sx1             ; positive: $00
        lda #$FF            ; negative: $FF
sx1     rts

flags   php                 ; print A, then C and V as 0/1
        jsr hexa
        plp
        php
        lda #0
        rol                 ; C into bit 0
        jsr hexa
        plp
        lda #0
        bvc fl1
        lda #1
fl1     jsr hexa
        jmp nextl
hexa    pha                 ; print A as two hex digits plus a space
        lsr
        lsr
        lsr
        lsr
        tax
        lda hexd,x
        jsr out
        pla
        and #$0F
        tax
        lda hexd,x
        jsr out
        lda #0
out     ldy col
        sta (PTR),y
        inc col
        rts
nextl   lda #0              ; next screen line (only used after flags)
        sta col
        clc
        lda PTR
        adc #40
        sta PTR
        bcc nl1
        inc PTR+1
nl1     rts

; ---- bouncing ball: y and vy are signed 8.8 fixed point ----
bounce  lda #0
        sta x
        sta y
        sta y+1
        sta vy
        sta vy+1
step    clc                 ; vy = vy + gravity
        lda vy
        adc #$30
        sta vy
        lda vy+1
        adc #0
        sta vy+1
        clc                 ; y = y + vy  (an ordinary 16-bit add)
        lda y
        adc vy
        sta y
        lda y+1
        adc vy+1
        sta y+1
        cmp #13             ; hit the floor (row 13)?
        bcs floor
        jmp plot
floor   lda #13
        sta y+1
        lda #0
        sta y
        jsr negvy           ; vy = -vy
        lda vy+1            ; vy = vy - vy/4 (keep 3/4 of the speed)
        sta t+1
        lda vy
        sta t
        jsr asr16
        jsr asr16
        sec
        lda vy
        sbc t
        sta vy
        lda vy+1
        sbc t+1
        sta vy+1
plot    lda y+1             ; screen row 10 + y
        clc
        adc #10
        tax
        lda SAVMSC
        clc
        adc rowlo,x
        sta PTR
        lda SAVMSC+1
        adc rowhi,x
        sta PTR+1
        ldy x
        lda #$0A            ; '*' in screen code
        sta (PTR),y
        inc x
        lda x
        cmp #40
        beq done
        jmp step
done    rts

negvy   sec                 ; 16-bit negate: 0 - vy
        lda #0
        sbc vy
        sta vy
        lda #0
        sbc vy+1
        sta vy+1
        rts
asr16   lda t+1             ; t = t / 2, keeping the sign
        cmp #$80
        ror t+1
        ror t
        rts

col     dta 0
x       dta 0
y       dta a(0)
vy      dta a(0)
t       dta a(0)
hexd    dta d'0123456789ABCDEF'
rowlo   :24 dta <[#*40]
rowhi   :24 dta >[#*40]
        run start

The four result lines at the top of the screen read, in order: tests 1–3 (result, C, V on each line), then test 4 (FB), test 5 (01), test 6 (01), test 7 (FF FD), test 8 (FC) and test 9 (00 D0).

Troubleshooting

  • An object moving left jumps to the far right: an 8-bit signed value was added to a 16-bit one without sign extension.
  • Comparisons fail only for large values: you used CMP/BCC on signed numbers. It works only while both values have the same sign. Use SBC and N EOR V.
  • A negative speed halves to a large positive one: you used LSR instead of CMP #$80 / ROR.
  • Results off by one after negation: the CLC before ADC #1 is missing, or SEC before SBC.
  • Overflow detection never triggers: V is only meaningful after ADC or SBC on the top byte. CMP doesn't change it.

Further reading

  • Mark Andrews, Atari Roots (Datamost, 1984): chapter 10 on assembly language maths, and the description of the processor status flags.
  • Avery Lee, Altirra Hardware Reference Manual, chapter 3: the 6502's flags.
  • On this site: Integer Maths on the 6502: Multiply, Divide, BCD Scores and Printing Numbers, 6502 Addressing Modes, the Atari Way, Writing a Small Complete Game in Assembly: Catch!.

Credits