Arrays, Strings & Buffers

Arrays, Strings, and Bounded Byte Processing

An array is one object containing a fixed number of adjacent elements. In most expressions its name converts to a pointer to the first element, which is why a called function cannot recover the array's element count from the pointer alone. A C string is a byte array whose first zero byte terminates the logical text; its allocated capacity and its current string length are different numbers. Embedded failures happen when an API forgets one of those facts, trusts unbounded input, performs unchecked size arithmetic, or publishes a partially written message after discovering the destination is too small.

How it is built

• Array extent exists where the array object is declared. sizeof array returns the full object size only in that scope; after array-to-pointer conversion, sizeof parameter returns the pointer size. APIs therefore pass pointer plus element count or wrap both in a view structure.
• Strings use a sentinel rather than stored metadata. Functions such as strlen must scan until a zero byte and are safe only when the input is already proven terminated within accessible storage. Binary packets are not strings and may legitimately contain zero bytes.
• A bounded reader stores source pointer, total length, and current offset. Every read first proves requested length is no greater than remaining length using subtraction that cannot wrap. State advances only after validation, so a rejected read is transactional.
• A bounded writer stores destination pointer, capacity, and current length. It checks the entire requested write before storing any byte. Returning status plus bytes written makes truncation policy visible; silently truncating often creates a syntactically valid but semantically corrupt protocol frame.
• Multidimensional arrays are contiguous arrays of arrays. A function parameter must preserve the inner dimension so pointer arithmetic knows each row stride. A flat pointer plus explicit rows, columns, and stride is more flexible for frame buffers and DMA surfaces.
• Library calls have overlap and termination contracts. memcpy requires non-overlapping regions, memmove handles overlap, memcmp compares bytes rather than numeric values, and snprintf reports the length that would have been written. Each contract must be checked rather than inferred from a familiar name.

Design procedure

1. Write the data contract first: element type, maximum capacity, current length, whether zero termination is required, whether overlap is allowed, and what happens on insufficient space.
2. Accept size_t for object sizes and indexes, then validate conversions at hardware or protocol boundaries. Check addition and multiplication before calculating byte counts: count <= SIZE_MAX / element_size.
3. Use half-open ranges [begin, end) or pointer plus count. Derive remaining = capacity - used only after proving used <= capacity; then compare need <= remaining. This order avoids the overflow created by used + need <= capacity.
4. Decode wire data byte by byte with explicit endian helpers. Do not cast a byte pointer to a wider integer pointer: alignment, effective type, aliasing, and byte order can all be wrong.
5. Make writes transactional. Validate complete frame size and field ranges first, or write into a temporary buffer and publish length only after success. Tests should confirm failure leaves output and offset unchanged.
6. Test empty, one-element, exact-fit, one-byte-short, maximum, embedded-zero, overlap, malformed length, and arithmetic-overflow cases. Fuzz parsers with arbitrary chunk boundaries and truncation at every byte.

Key terms

capacity

The number of elements allocated and safe to address.

length

The number of elements currently carrying valid logical content.

array decay

The usual conversion of an array expression to a pointer to its first element, losing extent information.

sentinel

A value such as the terminating zero byte that marks the logical end instead of storing a separate length.

stride

The byte or element distance between corresponding positions in adjacent rows or records.

transactional write

An operation that either commits the complete valid output or leaves the destination state unchanged.

Worked example

typedef struct {
    uint8_t *data;
    size_t capacity;
    size_t length;
} byte_writer_t;

bool writer_put_u16_be(byte_writer_t *w, uint16_t value)
{
    if ((w == NULL) || (w->data == NULL) ||
        (w->length > w->capacity) ||
        (2U > w->capacity - w->length)) {
        return false;                 /* no partial write */
    }

    w->data[w->length]     = (uint8_t)(value >> 8);
    w->data[w->length + 1] = (uint8_t)value;
    w->length += 2U;                  /* publish after both stores */
    return true;
}

Common pitfalls

Serialization & Byte-Level Parsing

Serialization is the deliberate conversion of values into a specified sequence of bytes. Parsing is the reverse operation under hostile conditions: input can be truncated at any byte, lengths can lie, versions can be unknown, integrity checks can fail and fields can begin at addresses unsuitable for native loads. A portable protocol therefore defines widths, signed interpretation, byte order, field order, framing, limits, version behavior and integrity coverage independently of any C object layout. A correct parser proves bounds before each read, validates semantic values and commits the destination only after the whole message succeeds.

How it is built

• The wire format begins with fixed-width integer types and an explicit byte order. A 32-bit numeric value is not four bytes in an unspecified arrangement: the protocol states whether the most-significant or least-significant byte appears first. Writers use shifts and masks or a defined conversion routine; readers combine uint8_t values into the destination width. The result is independent of host endianness.
• A frame gives a byte stream boundaries. A sync word can locate a candidate start, a fixed header carries version, type and declared payload length, the payload contains schema fields, and an integrity byte or checksum detects accidental corruption. Sync and length help resynchronization but are untrusted input. A CRC detects transmission errors but does not authenticate a sender or prevent deliberate modification.
• The parser owns a span described by a pointer, total length and cursor. Before a width-N read it first proves cursor is within the span, then checks N <= length - cursor. The subtraction form avoids overflow that can occur in cursor + N. Only after that proof may it inspect bytes, advance the cursor and publish the decoded value.
• Alignment-safe decoding never treats an arbitrary byte address as a pointer to uint16_t or uint32_t. Such a cast can violate natural alignment, effective-type rules and object lifetime, and it still reads in native byte order. Shift-and-combine works directly from uint8_t. memcpy into an aligned object can solve alignment and aliasing, but byte-order conversion is still required.
• A normal C struct may contain internal and tail padding selected by the ABI. A packed struct removes some padding with a compiler extension but can place multibyte members at misaligned addresses, does not standardize bit-field layout and does not convert native byte order. A structure definition is useful for the decoded destination, not as the protocol itself.
• Length fields create nested trust boundaries. Validate the protocol minimum and configured maximum, prove the declared payload plus trailer fits the remaining frame, then create a bounded sub-reader for exactly that payload. Parsing inside that view prevents one message from consuming bytes belonging to the next frame. Unknown permitted extension bytes can be skipped inside the declared payload, never outside it.
• Versioning needs an explicit compatibility policy. The header identifies the schema version, each version has a minimum payload size, optional appended fields have defaults when absent, and unsupported versions return a distinct status. A v1 decoder must not silently reinterpret v2 bytes. If forward-compatible extensions are allowed, their placement and skip rule are part of the protocol contract.
• Encoding should be transactional too. Compute or bound the encoded size, check capacity before each complete field, and either write into a temporary buffer or preserve a checkpoint so failure cannot expose a partial message. Parsing mirrors this: decode into a temporary Message, validate ranges, enums, flags, length and integrity, then copy it to the caller's output exactly once.
• Transport logic and message logic are separate state machines. A UART, TCP or DMA stream may deliver half a header, several frames together or noise before sync. The transport accumulator preserves partial bytes and finds candidate boundaries; the message parser accepts one complete bounded span. Keeping these layers separate makes truncation, resynchronization and timeout behavior testable.
• Proof combines exact vectors and generative failure cases. Golden vectors verify that independent implementations agree on every byte. Round-trip properties check decode(encode(x)) for valid values. A truncation loop tries every prefix length. Mutation and fuzz tests corrupt length, type, version, reserved bits, payload and checksum under sanitizers, followed by alignment and timing checks on the target.

Design procedure

1. Write a byte-level protocol table before writing a struct: offset or order, width, signedness, byte order, units, valid range, version introduced, default and integrity coverage for each field.
2. Define maximum frame and payload sizes from the transport and product budget. Treat every received length as untrusted and validate minimum, maximum and remaining capacity before forming a payload view.
3. Implement small checked readers and writers for u8, u16, u32, signed fixed-width values and byte spans. Make failure leave the cursor and output value unchanged whenever practical.
4. Build the frame parser as ordered gates: sync, fixed header, supported version, bounded payload, version-specific fields, semantic validation, permitted extensions, exact trailer and integrity check.
5. Decode into a temporary message and commit only on success. Return distinct statuses for truncation, invalid length, unsupported version, invalid field and integrity failure so the caller can apply policy without guessing.
6. Keep stream accumulation outside the message parser. Test partial headers, partial payloads, multiple frames in one delivery, noise before sync, timeout, buffer exhaustion and resynchronization after a rejected candidate.
7. Generate golden vectors for boundary values in every supported version and compare them with an independent implementation or carefully reviewed specification table.
8. Test every truncated prefix, lengths near zero and SIZE_MAX, invalid enums and reserved bits, checksum mutations and unchanged-output behavior. Fuzz the host parser with sanitizers, then confirm target-specific alignment and execution time.

Key terms

endianness

The order in which bytes of a multibyte value appear in memory or on a wire; it must be explicit at an interface.

framing

The rules that identify a complete message in a byte stream, commonly using sync, header, length, payload and trailer.

bounded reader

A pointer-length-cursor abstraction that proves remaining capacity before every read.

transactional parse

Decode and validate into temporary state, then commit the caller-visible output once after total success.

packed structure

A compiler-specific reduced-alignment object layout; not a portable serialization format.

golden vector

A named input and its exact expected byte sequence, shared across implementations as executable protocol evidence.

resynchronization

The stream policy for finding the next credible frame boundary after noise, truncation or a rejected candidate.

Worked example

The live lab constructs a versioned sensor frame byte by byte, displays every field at its wire offset, injects truncation and CRC faults, contrasts endian encodings, moves a 32-bit field across aligned and misaligned offsets, and provides complete bounded-reader, transactional-writer and frame-parser C patterns.

Common pitfalls

Arrays decay, and the length has to travel separately

An array in C is a block of objects with a size the compiler knows - until it is passed to a function, at which point it becomes a pointer to its first element and the size is gone. That conversion is called decay, it is silent, and it is why every buffer API in C carries a length parameter. A function cannot recover the size of what it was given.

How it is built

• An array expression converts to a pointer to its first element in nearly every context, including every function call.
• sizeof is one of the few exceptions, which is why it works on a local array and gives a pointer's size on a parameter.
• A parameter written T a[10] is adjusted to T *a; the 10 is documentation the compiler discards.
• Pointer arithmetic is defined only within one array object and one past its end, so a pointer derived outside that range is undefined even if never dereferenced.
• A pointer one past the end may be compared and not dereferenced, which is what makes the standard loop idiom legal.

Design procedure

1. Pass a pointer and a length together, always, and prefer size_t for the length.
2. Compute sizeof on the array where it is still an array, and pass the result rather than recomputing it later.
3. Validate an index against the length before forming the pointer, not after.
4. Prefer a small span struct holding both when a codebase passes buffers frequently, so they cannot be separated.
5. Never infer a length from a sentinel unless the format guarantees one; binary data contains zero bytes.

Key terms

Decay

The conversion of an array to a pointer to its first element.

Span

A pointer and a length together, describing a view of memory.

One past the end

A pointer that may be formed and compared but not dereferenced.

size_t

The unsigned type wide enough to index any object. The correct type for a length.

Sentinel

A terminating value marking the end. Works for text and not for binary.

Worked example

The size that disappears at the call:

  void clear(uint8_t buf[64]) {
      memset(buf, 0, sizeof buf);   // sizeof is 4 - buf is a POINTER
  }                                 // clears four bytes

  uint8_t data[64];
  sizeof data;                      // 64 here, where it is an array
  clear(data);                      // 64 is lost at the call

  void clear(uint8_t *buf, size_t n) { memset(buf, 0, n); }
  clear(data, sizeof data);         // the size travels

The span, when buffers are passed often:

  typedef struct { uint8_t *p; size_t n; } Span;
  #define SPAN_OF(a) ((Span){ (a), sizeof (a) })

  Span s = SPAN_OF(data);           // captured where it is an array
  process(s);                       // and cannot be separated

And the comparison that must use <, not !=:

  for (uint8_t *p = buf; p != buf + n; p++)   // fine
  for (uint8_t *p = buf; p < buf + n; p++)    // also fine, and
                                              // survives n == 0

Common pitfalls

Strings are a convention, not a type

A C string is an array of characters with a zero byte somewhere inside it. That is the whole definition - there is no length, no capacity, and no type distinguishing a string from any other byte array. Every string function trusts that the terminator is present and inside the allocation, and every classic buffer overflow is that trust being misplaced.

How it is built

• strlen walks until it finds a zero, so its cost is linear and it reads past the end if there is none.
• strcpy writes until it copies a zero, with no knowledge of the destination's size.
• strncpy does not guarantee termination: if the source is at least n bytes it copies n and stops, leaving no zero.
• snprintf always terminates within the buffer and returns the length it WOULD have written, which is how truncation is detected.
• Binary data is not a string; a sensor or network payload can contain zero bytes anywhere.

Design procedure

1. Use snprintf for formatting and check the return against the buffer size to detect truncation.
2. Prefer explicit length handling to the str functions wherever the data could be binary.
3. If you use strncpy, terminate manually afterwards; it is not a safe strcpy despite the name.
4. Track the length alongside the buffer for anything built incrementally, rather than calling strlen repeatedly.
5. Never build a path or a command by concatenating unvalidated input, regardless of the function used.

Key terms

NUL terminator

The zero byte marking the end. Not counted in strlen and required by every str function.

Truncation

Output that did not fit. snprintf reports it through the return value.

strncpy

A padding function, not a safe copy. Does not terminate when the source fills the buffer.

Off-by-one

Forgetting the terminator's byte. The most common sizing error in C.

Binary safety

Handling data that may contain zero bytes, which excludes every str function.

Worked example

The three copies, and what each actually guarantees:

  char dst[8];

  strcpy(dst, src);            // no bound at all. Overflows if
                               // src is longer than 7 + NUL.

  strncpy(dst, src, 8);        // bounded, and NOT terminated if
                               // strlen(src) >= 8. dst is now not
                               // a string, and the next strlen
                               // walks off the end.

  int n = snprintf(dst, sizeof dst, "%s", src);
  if (n < 0 || (size_t)n >= sizeof dst) { /* truncated */ }
                               // always terminated, truncation
                               // reported through the return

And why the return value is the length that WOULD have been written:

  char buf[4];
  int n = snprintf(buf, 4, "hello");    // buf = "hel", n = 5

  n is 5, not 3. Comparing n against the buffer size is what
  detects the truncation; comparing it against strlen(buf) does not.

Common pitfalls

Framing: finding message boundaries in a byte stream

A UART delivers bytes, not messages. Something has to decide where one message ends and the next begins, and that is framing. It matters more than it appears because the receiver may start listening mid-message, may lose bytes, and must resynchronise without human intervention - so a framing scheme is judged by how it recovers, not by how it works when nothing goes wrong.

How it is built

• Fixed length is the simplest and cannot resynchronise: one lost byte misaligns every message that follows.
• A delimiter byte marks the boundary and requires that the delimiter cannot appear in the payload, which needs escaping or an encoding.
• A length prefix is compact and fragile: a corrupted length makes the receiver consume the wrong number of bytes and lose sync.
• COBS encodes arbitrary data so that zero never appears, making zero an unambiguous frame delimiter at a cost of one byte per 254.
• Any scheme needs a checksum, because framing tells you where a message is and not whether it is intact.

Design procedure

1. Choose a delimiter-based scheme where resynchronisation matters, which on a real link it always does.
2. Use COBS or byte stuffing so the delimiter cannot occur in the payload, rather than hoping it will not.
3. Validate the length against a maximum before allocating or reading, so a corrupted length cannot hang the parser.
4. Add a CRC over the framed content and discard frames that fail it, rather than trying to repair them.
5. Write the parser as an explicit state machine that can always return to hunting for a delimiter.

Key terms

Framing

Determining message boundaries within a byte stream.

Byte stuffing

Escaping delimiter bytes in the payload so they cannot be mistaken for boundaries.

COBS

Consistent Overhead Byte Stuffing: removes all zero bytes with bounded overhead.

Resynchronisation

Recovering message alignment after corruption, without restarting the link.

Hunt state

The parser state that discards bytes until a delimiter appears.

Worked example

How each scheme behaves after one lost byte:

  FIXED 8 BYTES
    every subsequent message is misaligned, permanently.
    No recovery without an out-of-band reset.

  LENGTH PREFIX
    the length byte is now a data byte. The parser consumes a wrong
    count and stays wrong for as long as the numbers happen to work.

  ZERO DELIMITER + COBS
    the corrupted frame fails its CRC and is discarded. The next
    zero byte re-establishes the boundary. Recovery is automatic
    and bounded by one frame.

The parser, which can always get back to hunting:

  switch (state) {
  case HUNT:                              // discard until a zero
      if (b == 0) { len = 0; state = BODY; }
      break;
  case BODY:
      if (b == 0) {                       // frame complete
          if (cobs_decode(buf, len) && crc_ok()) deliver();
          len = 0;                        // either way, stay framed
      } else if (len < MAX) {
          buf[len++] = b;
      } else {
          state = HUNT;                   // oversized: resynchronise
      }
      break;
  }

Common pitfalls

Serialisation: getting a structure onto a wire

Serialisation is turning in-memory values into a defined byte sequence and back. The whole discipline is refusing to let the compiler's layout decisions leak into the format: byte order, padding and type sizes are all properties of the machine, and a wire format that depends on any of them works only between identical builds.

How it is built

• Write each field with explicit shifts and masks, so the bytes are determined by the value rather than by its storage.
• Fix the byte order in the format specification and convert at the boundary, in one place.
• Use fixed-width types for every field, since int is at least 16 bits and nothing more.
• Signed values need a defined encoding; two's complement is universal in practice and should still be stated.
• Floating point should be avoided on the wire, or sent as a scaled integer, because its representation is another machine property.

Design procedure

1. Write a pair of functions per message - encode and decode - and keep them adjacent so they cannot drift.
2. Return the number of bytes written and consumed, so the caller can bound and advance correctly.
3. Validate every decoded field against its legal range before using it, since the bytes came from outside.
4. Round-trip test with a fixed byte vector rather than only encode-then-decode, which passes even when both sides are wrong in the same way.
5. Version the format from the first release, because adding a version field later requires the change you are trying to avoid.

Key terms

Wire format

The byte layout defined by the protocol, independent of any machine.

Network byte order

Big-endian, as used by the internet protocols.

Round trip

Encoding then decoding. Necessary and not sufficient - both sides can be wrong together.

Golden vector

A fixed byte sequence with its known decoded value. What actually pins a format.

Format version

A field identifying the layout, needed from the first release rather than the second.

Worked example

Explicit encode and decode, with the length reported:

  size_t encode(uint8_t *out, size_t cap, const Reading *r) {
      if (cap < 7) return 0;
      out[0] = r->id;
      out[1] = (uint8_t)(r->value >> 24);
      out[2] = (uint8_t)(r->value >> 16);
      out[3] = (uint8_t)(r->value >>  8);
      out[4] = (uint8_t)(r->value      );
      out[5] = (uint8_t)(r->flags >> 8);
      out[6] = (uint8_t)(r->flags     );
      return 7;
  }

  bool decode(const uint8_t *in, size_t n, Reading *r) {
      if (n < 7) return false;
      r->id    = in[0];
      r->value = ((uint32_t)in[1] << 24) | ((uint32_t)in[2] << 16)
               | ((uint32_t)in[3] <<  8) |  (uint32_t)in[4];
      r->flags = (uint16_t)((in[5] << 8) | in[6]);
      return r->id <= MAX_ID;              // validate what came in
  }

And the test that actually pins the format:

  const uint8_t golden[7] = { 0x01, 0x00,0x00,0x01,0x00, 0x00,0x03 };
  decode(golden, 7, &r);
  assert(r.id == 1 && r.value == 256 && r.flags == 3);

  A round-trip test passes even if encode and decode share a bug.
  A golden vector does not.

Common pitfalls

Parsing untrusted input without a memory-safety bug

Every byte that arrives from a wire, a file or a user is untrusted, including the lengths and offsets inside it. A parser is the boundary between that data and the rest of the program, and essentially every remote memory-safety defect in embedded code is a parser trusting a field it read from the very buffer it is parsing.

How it is built

• A length or offset in the input is data, not a fact; it can be any value the type permits.
• Every read must be bounded by what remains in the buffer, not by what the header claims is present.
• Arithmetic on offsets can overflow, so `offset + length > total` is not a safe check when either can be large.
• A parser that returns pointers into its input creates lifetimes tied to that buffer, which must outlive every use.
• Recursive formats need an explicit depth limit, or a crafted input drives the stack into whatever lies below it.

Design procedure

1. Track remaining bytes and check each read against it before performing the read.
2. Compare using subtraction against the remaining count rather than adding to an offset, so overflow cannot pass the check.
3. Bound every length field against a maximum the design supports, before any allocation or copy.
4. Fail closed: return an error and discard the message rather than attempting partial interpretation.
5. Fuzz the parser with random and mutated input; it is the one place in embedded code where fuzzing finds bugs immediately.

Key terms

Untrusted input

Any data from outside the program, including every length inside it.

Bounds check

Confirming a read fits in what remains, performed before the read.

Integer overflow in a check

offset + len wrapping, so a check on the sum passes for values that should fail.

Fail closed

Rejecting malformed input rather than interpreting part of it.

Depth limit

A cap on nesting for a recursive format, preventing stack exhaustion.

Worked example

The check that overflows, and the one that does not:

  // UNSAFE - offset + len can wrap
  if (offset + len > total) return false;
  memcpy(dst, in + offset, len);

     offset = 0xFFFFFFF0, len = 0x20
     offset + len wraps to 0x10, which is <= total
     the check passes and the copy reads far out of bounds

  // SAFE - subtraction, and no wrap possible
  if (offset > total) return false;
  if (len > total - offset) return false;
  memcpy(dst, in + offset, len);

The remaining-bytes discipline, which makes each read obviously bounded:

  const uint8_t *p = in, *end = in + n;

  if (end - p < 1) return false;
  uint8_t type = *p++;

  if (end - p < 2) return false;
  uint16_t len = (uint16_t)((p[0] << 8) | p[1]); p += 2;

  if (len > MAX_PAYLOAD) return false;      // bound the CLAIM
  if (end - p < len)      return false;     // and the reality
  memcpy(payload, p, len);

Every read is preceded by a check against what is actually left, and the
length from the wire is bounded before it is used for anything.

Common pitfalls

More in Embedded C

• Interrupts, Rings & ConcurrencyThe handler is a second thread you did not declare. What is and is not atomic on a single core, the lock-free ring buffer and the conditions it requires, DMA and cache coherency, low-power modes and their wake sources, priority inversion, and debugging a race you cannot reproduce.
• Volatile RegistersMaster volatile keyword usage for memory-mapped registers in embedded C. Interactive simulator shows compiler optimization effects on register reads.
• User-Defined TypesDeclare your own types in embedded C: struct, union, enum, typedef, bitfields, designated initialisers, flexible array members and opaque handles.
• Embedded C + DSA 0 → 100One self-sufficient course connecting beginner C, Embedded C, hardware-facing APIs, bounded data structures, Embedded DSA practice, compiled code, diagnostics and production capstones.
• Bits, Fields & Fixed PointRegister fields and the mask conventions that silently disagree, Gray code and where one-bit-at-a-time matters, fixed-point arithmetic and the intermediate width a multiply needs, the undefined-behaviour traps in ordinary bit idioms, wire-format packing, and what each checksum detects.
• Types / PromotionUnderstand integer promotion and type conversion in embedded C. Interactive lab demonstrates implicit and explicit casting with signed/unsigned types.
• Compiler Workbench & TestingCompile real C for an embedded target and inspect what the compiler produced, then the discipline around it: where to draw the host-testable boundary, reading the generated assembly, undefined behaviour and the sanitizers, the warnings worth enabling, and measuring size and stack.
• Functions & ContractsDesign production C functions and modules with explicit preconditions, postconditions, ownership, status codes, opaque types, reentrancy, HAL seams and failure-state behavior.
• Object Layout & StorageWhere an object lives and what it costs: storage duration and linkage, the sections a declaration lands in, struct layout and the padding that makes a struct larger than its members, allocation without a heap, and integrity checks over stored data.
• C Basics & the Translation UnitFrom source text to a linked image: declarations against definitions, the translation unit the compiler actually sees, the preprocessor and what a macro can and cannot do, and the four build stages with the error vocabulary each one produces.
• FSM / DispatchDesign finite state machines with dispatch tables in embedded C. Interactive lab demonstrates state transitions and event handling for firmware.
• Embedded CA complete source-to-silicon workbench: C semantics, arrays and APIs, compiler and startup, MMIO, interrupts, DMA and caches, bounded systems, testing, safety evidence and production defense.