[bug #51255] Calculations in util/setbaud.h are incorrect #657
Description
Sat 17 Jun 2017 07:56:37 AM CEST
Calculations are wrong, the error becomes severe at high baud rates. Correct formula in attached file includes correct calculation and finesse of (ratio * 100 + 50)/100 to achieve correct rounding to nearest integer value, not just truncation.
This should also address bug #37758 as the issue is more general than just a problem for Xmega devices. I personally find the updated formula version easier to follow than the suggestion in bug #37758.
table of results from attached spreadsheet, looks weird with the limited width to display here, hope it comes out at your end. The corrected formula agrees exactly with the corresponding table in an Atmel datasheet.
Osc 16000000 Error % after
Rounding
Baud UBRR bad error Rounded Rounded Correct bad
rate UBRR % UBRR bad UBRR UBRR UBRR
------------------------------------------------------------
1200 832.3 833.9 0.19 832 834 0.04 -0.20
2400 415.7 417.2 0.36 416 417 -0.08 -0.32
4800 207.3 208.8 0.72 207 209 0.16 -0.80
9600 103.2 104.7 1.45 103 105 0.16 -1.78
14400 68.4 69.9 2.19 68 70 0.65 -2.27
19200 51.1 52.6 2.94 51 53 0.16 -3.75
28800 33.7 35.2 4.45 34 35 -0.82 -3.79
38400 25.0 26.5 5.99 25 27 0.17 -7.82
57600 16.4 17.9 9.17 16 18 2.21 -10.02
76800 12.0 13.5 12.48 12 14 0.17 -16.46
115200 7.7 9.2 19.53 8 9 -4.16 -17.18
230400 3.3 4.8 44.91 3 5 10.19 -49.69
250000 3.0 4.5 50.00 3 5 0.00 -66.67
500000 1.0 2.5 150.00 1 3 0.00 -200.00
file #40930: setbuad errors.xlsx
file #40929: setbaud corrected.h
This issue was migrated from https://savannah.nongnu.org/bugs/?51255