It depends on the operands being unsigned or signed. Using 4-bit addition as an example.
Unsigned: Make a carry bit, for 4-bit numbers make a 5th bit and that is the overflow bit.
e.g.:
7+8 = 15, okay
0111+1000 = 0_1111 (5-bit value)
8+8 = 16, overflow
1000+1000 = 1_0000 (5-bit value)
Signed:
Overflow can only occur when: A+B=C or -A+-B=-C (add two positives, or add two negatives)
If the sign of the result changes you have an overflow (A,B,C all have the same width)
-1+-8=-9, overflow for 4-bits
1111+1000=0111 (oops this is +7)
7+1=8, overflow for 4-bits
0111+0001=1000 (oops this is -8)
so overflow = ~(a[3] ^ b[3]) ^ c[3]
~(a[3] ^ b[3]) checks if sign bits are the same
.... ^ c[3] checks if the sign changed.
- - - Updated - - -
Oh, and if you use the rule mentioned earlier (where you used the wrong bits) then you sign extend the 4-bit A & B inputs to 5-bits and truncate the addition to 5-bits:
-1+-8=-9
11111+11000=110111 (XOR the bits in red =1 overflow)
7+1=8
00111+00001=001000 (XOR the bits in red =1 overflow)
-2+-4=-6 (should be okay...prove it)
11110+11100=111010 (XOR the bits in red =0 okay)