RayBench EmbeddedInteractive engineering labs
EMBEDDED C

Bits, Fields, Endianness & Encodings

Safe mask construction, field extraction and insertion, shifts, Gray code, endianness, packing, and representation boundaries.

Reviewed 2026-08-223,348 words

Bit Fields in C: Masks, Shifts, and What Promotion Does to Them

Manipulating bits in C looks like the most elementary thing in the language and is one of the easiest places to invoke undefined behaviour. A shift count at or above the promoted operand's width is undefined. A left shift that moves a one into or past the sign bit of a signed type is undefined. An eight-bit mask silently becomes a signed int before the operator sees it, so complementing it produces a value with thirty-two bits set rather than eight. None of these produce a diagnostic by default, and all of them produce plausible results on the compiler you tested with. This chapter is about those rules. The bit algorithms built on top of them - counting set bits, reversing a word, isolating the lowest set bit, streaming parity - are the first stage of Embedded DSA at /dsa/bits, and are not repeated here.

How it is built

  • Every operand narrower than int is promoted before a bitwise operator applies. A uint8_t mask becomes a signed int, so ~mask has all the upper bits set too, and assigning it back to a uint8_t discards them - which is correct by accident. Used in a wider expression, the same complement corrupts the upper bits.
  • The shift operators do not perform the usual arithmetic conversions across their operands: each is promoted independently and the result has the promoted left operand's type. A shift count greater than or equal to that type's width is undefined, so the safe count depends on the promoted width and not on the type you wrote.
  • Left-shifting a signed value so that a one reaches or passes the sign bit is undefined. 1 << 31 is undefined on a thirty-two-bit int because 1 is signed; 1u << 31 is fine. This is why register masks should be written with unsigned literals throughout.
  • Right-shifting a negative signed value is implementation-defined - almost always arithmetic, but not guaranteed. Shifting unsigned values is always a logical shift, which is one more reason hardware masks belong in unsigned types.
  • A field update is a three-step operation: build a mask of width ones, shift it to the field's position, clear those bits in the register, then OR in the shifted value masked to width. Skipping the mask on the value lets an over-range input corrupt neighbouring fields, which is the most common register-access defect.
  • A width equal to the full type is the case that breaks the naive mask expression, because (1u << width) - 1 requires a shift by the type width. It needs an explicit branch or a different formulation, and it is exactly the case a test with typical widths never reaches.

Design procedure

  1. Write every mask and shift with unsigned literals and unsigned types, so no shift can reach a sign bit and no right shift is implementation-defined.
  2. Validate the shift count against the promoted operand's width before shifting, especially when the count comes from data or a parameter.
  3. Mask the value as well as the field when writing a register field, so an over-range input cannot reach a neighbouring field.
  4. Handle the full-width case explicitly rather than relying on (1u << width) - 1, which is undefined when width equals the type width.
  5. Use exact-width types for register variables so the promoted width is known rather than inferred from the target's int size.
  6. For the algorithms built on these rules - popcount, bit reversal, parity, isolating the lowest set bit - work through /dsa/bits, which has the implementations and step traces.

Key terms

Promotion in bitwise ops
Narrow operands become int first, so ~ and << operate on more bits than you wrote.
Shift count limit
Must be strictly below the PROMOTED left operand's width, or the behaviour is undefined.
Signed left shift
Undefined when a one reaches or passes the sign bit. 1 << 31 is a bug; 1u << 31 is not.
Signed right shift
Implementation-defined for negative values. Usually arithmetic, never guaranteed.
Mask-and-merge
Clear the field, then OR in the masked and shifted value. The only safe field update.
Full-width case
width equal to the type width, where the usual mask expression is undefined.

Worked example

#include <stdint.h>

/* Promotion makes this wrong in a wider expression: */
uint8_t mask = 0x0F;
uint32_t v = ~mask;         /* 0xFFFFFFF0, not 0xF0 */
uint32_t w = (uint8_t)~mask;/* 0x000000F0 - say what you mean */

/* Undefined, and it compiles: */
int32_t bad = 1 << 31;      /* signed, reaches the sign bit */
uint32_t ok = 1u << 31;     /* unsigned: fine               */

/* The full-width case the naive mask cannot express: */
static uint32_t width_mask(uint8_t width) {
    return (width >= 32u) ? UINT32_MAX : ((1u << width) - 1u);
}

/* Mask-and-merge, with the value masked too: */
static uint32_t field_write(uint32_t reg, uint8_t lsb,
                            uint8_t width, uint32_t value) {
    const uint32_t m = width_mask(width) << lsb;
    return (reg & ~m) | ((value << lsb) & m);
}

# The four rules, and what breaks each:
#
#   shift count >= promoted width      -> undefined
#   signed left shift into sign bit    -> undefined
#   signed right shift of a negative   -> implementation-defined
#   ~ on a narrow type in a wide expr  -> upper bits set

# The algorithms that use these rules are the next section:
#   /dsa/bits  - popcount, reverse, parity, lowest set bit,
#                next power of two, portable big-endian decode

Common pitfalls

Register fields: extract, insert, and the named constants

Almost all embedded bit work is one of two operations against a hardware register: pull a field out, or put one in without disturbing its neighbours. Both are three-line idioms, and both go wrong in the same two ways - a mask that is not the field's width, and a shift applied in the wrong order. Writing them once behind named constants is what keeps a driver readable and correct.

How it is built

  • A field is described by a position and a width; the mask is (1u << width) - 1, applied after shifting down.
  • Extraction is shift then mask: (reg >> POS) & MASK. Masking first requires the mask to be pre-shifted, which is easy to get wrong.
  • Insertion is clear then set: (reg & ~(MASK << POS)) | ((value & MASK) << POS).
  • Masking the incoming value matters: a value wider than the field silently corrupts the neighbouring one.
  • Vendor headers supply _Pos and _Msk constants, and the _Msk is already shifted - mixing conventions is a common defect.

Design procedure

  1. Define POS and MSK per field and use them consistently rather than writing literal shifts at each site.
  2. Mask the value on insertion even when you believe it is in range, since the cost is one instruction and the failure is silent.
  3. Read the register once into a local, modify, and write once, rather than reading it several times.
  4. Remember the read-modify-write hazard on registers with write-1-to-clear bits, covered in the MMIO topic.
  5. Add a static assertion that each mask fits its width, so a mistyped constant fails the build.

Key terms

Field position
The bit index of the field's least significant bit.
Field mask
Width ones. Unshifted for the extract idiom, pre-shifted in most vendor headers.
Insertion
Clearing the field then ORing the shifted value in.
_Pos / _Msk
CMSIS naming. _Msk is already shifted into place, which is the convention to check.
Field overflow
A value wider than its field, corrupting the neighbouring one with no diagnostic.

Worked example

The two idioms, and the convention mismatch that breaks them:

  #define BAUD_POS  8u
  #define BAUD_MSK  0xFFu              // UNSHIFTED width mask

  uint32_t baud = (reg >> BAUD_POS) & BAUD_MSK;
  reg = (reg & ~(BAUD_MSK << BAUD_POS))
      | ((value & BAUD_MSK) << BAUD_POS);

  CMSIS headers use the other convention:

  #define USART_BRR_DIV_Pos  4u
  #define USART_BRR_DIV_Msk  (0xFFFUL << USART_BRR_DIV_Pos)   // SHIFTED

  uint32_t div = (reg & USART_BRR_DIV_Msk) >> USART_BRR_DIV_Pos;

  Both are correct. Mixing them - shifting an already-shifted mask -
  produces a mask of zero on most fields, so the write does nothing
  and the read returns zero.

And the unmasked value, which corrupts a neighbour:

  reg |= (value << BAUD_POS);       // value = 0x1FF, field is 8 bits
                                    // bit 16 lands in the NEXT field

Common pitfalls

Gray code, and where one-bit-at-a-time matters

Gray code is any binary encoding in which consecutive values differ in exactly one bit. That single property makes it the right encoding whenever a value is sampled by something that is not synchronised to the thing producing it: a sample caught mid-transition returns either the old value or the new one, never a third value assembled from parts of both.

How it is built

  • Conversion to Gray is value XOR (value >> 1), which is two instructions and no table.
  • Conversion back is a cumulative XOR of the shifted values, so it is a short loop or an unrolled sequence.
  • The property is preserved across the wrap: the highest value and zero also differ in one bit.
  • Rotary encoders output Gray naturally, because their tracks are physically arranged so only one changes per step.
  • Asynchronous FIFO pointers use it so the read and write sides can sample each other's index safely.

Design procedure

  1. Convert a counter to Gray before it crosses a clock domain, and back after it is synchronised.
  2. Synchronise the Gray value through two flip-flops as usual; the encoding removes the multi-bit problem, not the metastability one.
  3. For a rotary encoder, decode the transition rather than the absolute value, since only the direction of change carries information.
  4. Remember the encoding does not preserve arithmetic: Gray values cannot be compared or added without converting back.
  5. Use a power-of-two range, since the one-bit property across the wrap depends on it.

Key terms

Gray code
An encoding where consecutive values differ in one bit.
Binary to Gray
g = b ^ (b >> 1). Two instructions.
Gray to binary
A cumulative XOR of successively shifted values.
Cyclic property
The wrap from maximum to zero also differs in one bit, given a power-of-two range.
Asynchronous FIFO
A queue between clock domains, using Gray-coded pointers on both sides.

Worked example

The problem it solves, in one comparison:

  binary 3 -> 4     011 -> 100     ALL THREE bits change
    a sample mid-transition can return 000, 001, 010, 111 - any of
    them, none of which the counter ever held

  gray   3 -> 4     010 -> 110     ONE bit changes
    a sample returns 010 or 110. Both real, both at most one count
    stale.

The conversion, which is cheap enough to be unconditional:

  uint32_t to_gray(uint32_t b)   { return b ^ (b >> 1); }

  uint32_t from_gray(uint32_t g) {
      g ^= g >> 16; g ^= g >> 8;
      g ^= g >>  4; g ^= g >> 2; g ^= g >> 1;
      return g;
  }

And the sequence, showing the wrap:

  0 000    4 110
  1 001    5 111
  2 011    6 101
  3 010    7 100  -> back to 000: one bit again

Common pitfalls

Fixed point: representing fractions without an FPU

Fixed-point arithmetic represents fractional values in an integer by agreeing where the binary point sits. The agreement exists only in your head and in the code's naming - the hardware sees ordinary integers - which is what makes it fast and what makes it error-prone. Every operation has to account for where the point is, and the compiler will not check any of it.

How it is built

  • Q notation names the format: Q15 is one sign bit and fifteen fractional bits in sixteen; Q16.16 is sixteen of each in thirty-two.
  • Addition and subtraction require the same Q on both operands and produce that Q, so they are free.
  • Multiplication adds the Q values: Q15 times Q15 gives Q30, which must be shifted right by 15 to return to Q15.
  • That intermediate needs double width: two 16-bit Q15 values multiply into a 32-bit Q30 before the shift.
  • Division subtracts them, and usually needs the numerator pre-shifted to keep precision.

Design procedure

  1. Write the Q format into every variable name or type, since the compiler tracks nothing.
  2. Promote to double width before multiplying, then shift back; multiplying at the operand width overflows silently.
  3. Round rather than truncate on the shift back by adding half an LSB first, which halves the accumulated error.
  4. Saturate rather than wrap on overflow for anything driving a physical output, since a wrap inverts the sign.
  5. Choose the Q from the required range and resolution: bits above the point set the range, bits below set the step.

Key terms

Q notation
Qn means n fractional bits. Qm.n names both integer and fractional widths.
Implicit scaling
The binary point exists only by convention; the hardware sees an integer.
Intermediate width
The double-width product a multiply needs before it is shifted back.
Rounding
Adding half an LSB before the shift, rather than truncating toward zero.
Saturation
Clamping at the representable limit instead of wrapping through the sign.

Worked example

Q15, and the multiply that must not be done at operand width:

  Q15: 1 sign bit, 15 fractional  ->  range [-1, 1)
       1.0 is not representable; 0x7FFF is 32767/32768

  int16_t a = 0x4000;          // 0.5
  int16_t b = 0x4000;          // 0.5

  int16_t bad = a * b;         // WRONG - overflows int16 en route

  int32_t p = (int32_t)a * b;  // Q30, 0x10000000
  int16_t r = (int16_t)(p >> 15);   // back to Q15 = 0x2000 = 0.25

With rounding, which halves the error for one add:

  int16_t r = (int16_t)((p + (1 << 14)) >> 15);

And choosing the format from requirements:

  need +/-100.0 with 0.01 resolution
    range needs 8 integer bits (128 > 100)
    resolution needs 7 fractional bits (1/128 < 0.01)
    -> Q8.7 minimum; Q16.16 in a 32-bit int is the comfortable choice

Common pitfalls

Bit manipulation without undefined behaviour

The bit idioms are short enough that the language rules around them get skipped, and three of those rules bite regularly: shifts of signed values, shifts by too much, and integer promotion turning a narrow operation into a wide one. Each produces code that compiles cleanly, looks correct, and is either wrong or undefined.

How it is built

  • Left-shifting a signed value into or past the sign bit is undefined behaviour; 1 << 31 is undefined and 1u << 31 is not.
  • Shifting by an amount greater than or equal to the type's width is undefined, and on x86 the count is taken modulo the width, so it silently does something plausible.
  • Every operand narrower than int is promoted to int, so an operation on uint8_t is performed on 32 bits.
  • That makes ~ on a narrow value produce set high bits, so a comparison against a narrow constant is never true.
  • Right-shifting a negative signed value is implementation-defined in older standards and arithmetic in practice, propagating the sign rather than shifting in zeros.

Design procedure

  1. Use unsigned types and unsigned literals for all bit work: 1u, not 1.
  2. Mask after any operation that can set bits above the intended width, particularly complement.
  3. Bound a computed shift count against the type width before shifting.
  4. Build with -Wconversion and -Wsign-conversion, which exist to surface exactly these.
  5. Prefer a named inline helper over a repeated idiom, so the guard is written once.

Key terms

Integer promotion
Narrow operands becoming int before an operation.
Undefined shift
A shift count at or above the type width, or a signed left shift into the sign bit.
Arithmetic right shift
Sign-propagating shift, what signed values get in practice.
-Wconversion
The warning reporting implicit narrowing. Noisy at first and finds real defects.
Masking back
Restoring the intended width after promotion widened the operation.

Worked example

Three lines that all compile and none of which do what they read as:

  uint32_t top = 1 << 31;
     1 is a SIGNED int; the result is not representable.
     Undefined behaviour. Fix: 1u << 31

  uint8_t flags = 0x0F;
  if (~flags == 0xF0) { }
     flags promotes to int 0x0000000F
     ~ gives 0xFFFFFFF0, never equal to 0xF0. ALWAYS FALSE.
     Fix: ((uint8_t)~flags == 0xF0)

  uint32_t m = (1u << width) - 1;
     if width is 32, the shift is UNDEFINED - not zero.
     On x86 the count is taken mod 32, giving 1u << 0 = 1,
     so m becomes 0 instead of 0xFFFFFFFF.
     Fix: width >= 32 ? 0xFFFFFFFFu : (1u << width) - 1

The third is the worst of the three, because the wrong answer is a plausible
mask rather than an obvious failure.

Common pitfalls

Packing, alignment, and reading a wire format

A protocol or a stored record has a byte layout the specification defines, and a C struct has a byte layout the compiler defines. They agree only by accident. Treating one as the other - casting a received buffer to a struct pointer - is the most common way an embedded parser is written and the most common reason it breaks on a different compiler, a different architecture, or an unaligned buffer.

How it is built

  • The compiler inserts padding to keep each member aligned, so a struct is usually larger than the sum of its members.
  • Padding placement and byte order are both properties of the target, not of the protocol.
  • A packed attribute removes the padding and makes every member potentially unaligned, which on some cores means the access faults and on others it is merely slow.
  • Casting a byte buffer to a struct pointer is undefined if the buffer is not suitably aligned, regardless of packing.
  • Explicit serialisation - reading each field with shifts from known offsets - has no alignment or padding dependency at all.

Design procedure

  1. Parse field by field from byte offsets, with explicit shifts for multi-byte values.
  2. Write the same way on the way out, so the layout lives in your code rather than in the compiler's choices.
  3. Reserve packed structs for register overlays where the vendor guarantees the layout, not for wire formats.
  4. If you must overlay a struct, copy the bytes into a properly aligned object rather than casting the buffer.
  5. Pin any layout you depend on with a static assertion on sizeof and offsetof, so a change fails the build.

Key terms

Padding
Bytes inserted for alignment. Contents unspecified; not part of any protocol.
Packed
An attribute removing padding, at the cost of possibly unaligned members.
Unaligned access
Reading a word at an address that is not a multiple of its size. Faults on some cores.
Explicit serialisation
Reading and writing each byte at a known offset. No layout dependency.
Static assertion
A compile-time check on a layout you rely on.

Worked example

The cast that works on your bench and nowhere else:

  struct Frame { uint8_t id; uint32_t value; uint16_t crc; };

  x86 / Arm layout:  id, 3 pad, value, crc, 2 pad  = 12 bytes
  the wire format:   id, value, crc                =  7 bytes

  struct Frame *f = (struct Frame *)rx;    // wrong size, wrong
                                           // offsets, and possibly
                                           // an unaligned load

The explicit parse, which has no dependency on any of that:

  uint8_t  id  = rx[0];
  uint32_t val = ((uint32_t)rx[1] << 24) | ((uint32_t)rx[2] << 16)
               | ((uint32_t)rx[3] <<  8) |  (uint32_t)rx[4];
  uint16_t crc = ((uint16_t)rx[5] << 8) | rx[6];

  Seven bytes, big-endian, on every compiler and every target. The
  shifts operate on VALUES, so neither padding nor byte order can
  reach it.

And pinning a register overlay you do depend on:

  _Static_assert(sizeof(USART_TypeDef) == 0x20, "layout changed");
  _Static_assert(offsetof(USART_TypeDef, CR1) == 0x0C, "moved");

Common pitfalls

Checksums and CRCs: what each actually detects

A checksum answers one question - has this data changed - and different algorithms answer it with very different confidence. A sum detects nothing about ordering, an XOR detects nothing about a pair of flipped bits, and a CRC gives a stated guarantee about burst errors up to its width. Choosing one is choosing an error class to be able to miss.

How it is built

  • A simple sum detects any single-byte change but not a reordering, and two changes can cancel exactly.
  • An XOR checksum detects an odd number of flipped bits in any column and misses an even number entirely.
  • A CRC treats the message as a polynomial and takes the remainder modulo a generator, which is what gives it its guarantees.
  • A CRC-n detects every burst error up to n bits, every odd number of bit errors if the polynomial has the right factor, and all single and double errors.
  • The parameters are the polynomial, the initial value, whether input and output are reflected, and the final XOR - and two implementations agreeing on the polynomial alone will not agree on the result.

Design procedure

  1. Use a CRC for anything crossing a wire or stored in flash, and a sum only where the check is a formality.
  2. State all five CRC parameters in the code, not just the polynomial, since that is what makes two implementations agree.
  3. Verify against a published test vector - the CRC of the ASCII string "123456789" is the standard one - before trusting an implementation.
  4. Use a table-driven implementation where throughput matters, and the bitwise one where flash is scarce.
  5. Use the hardware CRC unit if the part has one, and check its parameters, since they are often fixed and may not match your protocol.

Key terms

Generator polynomial
The divisor defining a CRC's error-detection properties.
Reflection
Whether bits are processed least significant first. A parameter, not an implementation detail.
Burst error
Consecutive corrupted bits. What CRCs are specifically designed to catch.
Check value
The CRC of "123456789", published for every standard variant as a test vector.
Residue
The constant a correct message plus its CRC produces, which allows checking without recomputing.

Worked example

What each misses, on the same corruption:

  data       01 02 03 04
  reordered  04 03 02 01

    sum   0x0A -> 0x0A    IDENTICAL. Not detected.
    XOR   0x04 -> 0x04    IDENTICAL. Not detected.
    CRC   differs                    Detected.

  two bits flipped in the same column, different bytes:

    XOR   unchanged        Not detected.
    CRC   differs          Detected.

And why the polynomial alone is not enough:

  CRC-16/CCITT-FALSE   poly 0x1021, init 0xFFFF, no reflect, xorout 0
  CRC-16/KERMIT        poly 0x1021, init 0x0000, reflected,  xorout 0

  Same polynomial. Different results for every input. Two devices
  that agreed on "CRC-16 with 0x1021" will not interoperate, and
  the failure looks like line noise.

  check("123456789") = 0x29B1 for CCITT-FALSE, 0x2189 for KERMIT.
  Running that one vector settles it in a minute.

Common pitfalls

More in Embedded C