LwIP协议栈（1）：简介与pbuf

概述
　　Lwip是瑞典计算机科学院(SICS)的Adam Dunkels 开发的一个小型开源的TCP/IP协议栈。
　　LwIP是Light Weight (轻型)IP协议，有无操作系统的支持都可以运行。LwIP实现的重点是在保持TCP协议主要功能的基础上减少对RAM 的占用，它只需十几KB的RAM和40K左右的ROM就可以运行，这使LwIP协议栈适合在低端的嵌入式系统中使用。[1]
　　LwIP协议栈主要关注的是怎么样减少内存的使用和代码的大小，这样就可以让lwIP适用于资源有限的小型平台例如嵌入式系统。为了简化处理过程和内存要求，lwIP对API进行了裁减，可以不需要复制一些数据。
　　LwIP 由几个模块组成，除 TCP/IP 协议的实现模块外（ IP， ICMP， UDP， TCP），还有包括许多相关支持模块。这些支持模块包括：操作系统模拟层、缓冲与内存管理子系统、网络接口函数及一组 Internet 校验和计算函数。

进程模型
　　LwIP 则采取将所有协议驻留在同一个进程的方式，以便独立于操作系统内核之外。应用程序既可以驻留在 LwIP 的进程中，也可以使用一个单独的进程。应用程序与 TCP/IP 协议栈通讯可以采用两种方法：一种是函数调用，这适用于应用程序与 LwIP 使用同一个进程的情况；另一种是使用更抽象的 API。
　　LwIP 在用户空间而不是操作系统内核实现，既有优点也有缺点。把 LwIP 作为一个进程的主要优点是可以轻易的移植到不同的操作系统中。

操作系统模拟层
　　为了方便 LwIP 移植，属于操作系统的函数调用及数据结构并没有在代码中直接使用，而是用操作系统模拟层来代替对这些函数的使用。操作系统模拟层使用统一的接口提供定时器、进程同步及消息传递机制等诸如此类的系统服务。原则上，移植 LwIP，只需针对目标操作系统修改模拟层实现即可。

缓冲与内存管理
　　通讯系统里的内存与缓冲管理模块首要考虑的是如何适应不同大小的内存需求，一个TCP 段可能有几百个字节，而一个 ICMP 回显数据却仅有几个字节。还有，为了避免复制，应该尽可能的让缓冲区中的数据内容驻留在不能被网络子系统管理的存储区中，比如应用程序存储区或者 ROM。
　　LWIP 的动态内存管理机制可以有三种：C运行时库自带的内存分配策略、动态内存堆(HEAP)分配策略和动态内存池(POOL)分配策略。默认情况下，我们选择使用 LWIP 自身的动态内存堆分配策略。

　　一个典型的 LWIP 应用系统包括这样的三个进程：首先启动的是上层应用程序进程，然后是 LWIP 协议栈进程，最后是底层硬件数据包接收发送进程。通常 LWIP协议栈进程是在应用程序中调用 LWIP 协议栈初始化函数来创建的。注意 LWIP 协议栈进程一般具有最高的优先级，以便实时正确的对数据进行响应。

数据包pbuf：
　　
　　LwIP采用数据结构 pbuf 来描述数据包，其结构如下：
　　
　　

struct pbuf {
  /** next pbuf in singly linked pbuf chain */
  struct pbuf *next;

  /** pointer to the actual data in the buffer */
  void *payload;

  /**
   * total length of this buffer and all next buffers in chain
   * belonging to the same packet.
   *
   * For non-queue packet chains this is the invariant:
   * p->tot_len == p->len + (p->next? p->next->tot_len: 0)
   */
  u16_t tot_len;

  /** length of this buffer */
  u16_t len;

  /** pbuf_type as u8_t instead of enum to save space */
  u8_t /*pbuf_type*/ type;

  /** misc flags */
  u8_t flags;

  /**
   * the reference count always equals the number of pointers
   * that refer to this pbuf. This can be pointers from an application,
   * the stack itself, or pbuf->next pointers from a chain.
   */
  u16_t ref;
};

　　各成员含义上面的注释已经说得很清楚了。
　　关于采用链表结构，是因为实际发送或接收的数据包可能很大，而每个 pbuf 能够管理的数据可能很少，所以，往往需要多个 pbuf 结构才能完全描述一个数据包。
　　另外，最后的 ref 字段表示该 pbuf 被引用的次数。这里又是一个纠结的地方啊。初始化一个 pbuf 的时候， ref 字段值被设置为 1，当有其他 pbuf 的 next 指针指向该 pbuf 时，该 pbuf 的 ref 字段值加一。所以，要删除一个 pbuf 时， ref 的值必须为 1 才能删除成功，否则删除失败。
　　上图中注意 payload 并没有指向 ref 字段之后，而是隔了一定的区域。这段区域就是offset 的大小，这段区域用来存储数据的包头，如 TCP 包头， IP 包头等。当然， offset 也可以是 0。

来看代码：

/**
 * Allocates a pbuf of the given type (possibly a chain for PBUF_POOL type).
 *
 * The actual memory allocated for the pbuf is determined by the
 * layer at which the pbuf is allocated and the requested size
 * (from the size parameter).
 *
 * @param layer flag to define header size
 * @param length size of the pbuf‘s payload
 * @param type this parameter decides how and where the pbuf
 * should be allocated as follows:
 *
 * - PBUF_RAM: buffer memory for pbuf is allocated as one large
 *             chunk. This includes protocol headers as well.
 * - PBUF_ROM: no buffer memory is allocated for the pbuf, even for
 *             protocol headers. Additional headers must be prepended
 *             by allocating another pbuf and chain in to the front of
 *             the ROM pbuf. It is assumed that the memory used is really
 *             similar to ROM in that it is immutable and will not be
 *             changed. Memory which is dynamic should generally not
 *             be attached to PBUF_ROM pbufs. Use PBUF_REF instead.
 * - PBUF_REF: no buffer memory is allocated for the pbuf, even for
 *             protocol headers. It is assumed that the pbuf is only
 *             being used in a single thread. If the pbuf gets queued,
 *             then pbuf_take should be called to copy the buffer.
 * - PBUF_POOL: the pbuf is allocated as a pbuf chain, with pbufs from
 *              the pbuf pool that is allocated during pbuf_init().
 *
 * @return the allocated pbuf. If multiple pbufs where allocated, this
 * is the first pbuf of a pbuf chain.
 */

struct pbuf *
pbuf_alloc(pbuf_layer layer, u16_t length, pbuf_type type)
{
  struct pbuf *p, *q, *r;
  u16_t offset;
  s32_t rem_len; /* remaining length */
  LWIP_DEBUGF(PBUF_DEBUG | LWIP_DBG_TRACE, ("pbuf_alloc(length=%"U16_F")\n", length));

  /* determine header offset */
  offset = 0;
  switch (layer) { //注意这里从协议栈上层开始，方便offset从上层往下叠加，因此也没加 break
  case PBUF_TRANSPORT:
    /* add room for transport (often TCP) layer header */
    offset += PBUF_TRANSPORT_HLEN;
    /* FALLTHROUGH */
  case PBUF_IP:
    /* add room for IP layer header */
    offset += PBUF_IP_HLEN;
    /* FALLTHROUGH */
  case PBUF_LINK:
    /* add room for link layer header */
    offset += PBUF_LINK_HLEN;
    break;
  case PBUF_RAW:
    break;
  default:
    LWIP_ASSERT("pbuf_alloc: bad pbuf layer", 0);
    return NULL;
  }

  switch (type) {
  case PBUF_POOL:
    /* allocate head of pbuf chain into p */
    p = (struct pbuf *)memp_malloc(MEMP_PBUF_POOL); //分配第一个pbuf

    if (p == NULL) {
      return NULL;
    }
    p->type = type;
    p->next = NULL;

    /* make the payload pointer point ‘offset‘ bytes into pbuf data memory */
    p->payload = LWIP_MEM_ALIGN((void *)((u8_t *)p + (SIZEOF_STRUCT_PBUF + offset)));

    /* the total length of the pbuf chain is the requested size */
    p->tot_len = length; //该pbuf及其以后pbuf的负载数据总长度
    /* set the length of the first pbuf in the chain */
    p->len = LWIP_MIN(length, PBUF_POOL_BUFSIZE_ALIGNED - LWIP_MEM_ALIGN_SIZE(offset)); //负载数据可能大于分配空间长度，也有可能小于，取当前pbuf实际的负载长度

    /* set reference count (needed here in case we fail) */
    p->ref = 1;

    /* now allocate the tail of the pbuf chain */
    //如果一个pbuf不够的话，接着分配
    /* remember first pbuf for linkage in next iteration */
    r = p;
    /* remaining length to be allocated */
    rem_len = length - p->len;

    /* any remaining pbufs to be allocated? */
    while (rem_len > 0) {
      q = (struct pbuf *)memp_malloc(MEMP_PBUF_POOL); //从第二个pbuf开始，不再需要TCP/IP之类的头，所以没有offset
      if (q == NULL) {
        /* free chain so far allocated */
        pbuf_free(p);     //注意这里，如果当前pbuf分配不成功，要把之前分配的所有pbuf都释放掉
        /* bail out unsuccesfully */
        return NULL;
      }
      q->type = type;
      q->flags = 0;
      q->next = NULL;
      /* make previous pbuf point to this pbuf */
      r->next = q;
      /* set total length of this pbuf and next in chain */
      q->tot_len = (u16_t)rem_len;
      /* this pbuf length is pool size, unless smaller sized tail */
      q->len = LWIP_MIN((u16_t)rem_len, PBUF_POOL_BUFSIZE_ALIGNED);
      q->payload = (void *)((u8_t *)q + SIZEOF_STRUCT_PBUF);

      q->ref = 1;
      /* calculate remaining length to be allocated */
      rem_len -= q->len;
      /* remember this pbuf for linkage in next iteration */
      r = q;
    }
    /* end of chain */
    /*r->next = NULL;*/

    break;
  case PBUF_RAM:
    /* If pbuf is to be allocated in RAM, allocate memory for it. */
    p = (struct pbuf*)mem_malloc(LWIP_MEM_ALIGN_SIZE(SIZEOF_STRUCT_PBUF + offset) + LWIP_MEM_ALIGN_SIZE(length));
    if (p == NULL) {
      return NULL;
    }
    /* Set up internal structure of the pbuf. */
    p->payload = LWIP_MEM_ALIGN((void *)((u8_t *)p + SIZEOF_STRUCT_PBUF + offset));
    p->len = p->tot_len = length;
    p->next = NULL;
    p->type = type;
    break;
  /* pbuf references existing (non-volatile static constant) ROM payload? */
  case PBUF_ROM:
  /* pbuf references existing (externally allocated) RAM payload? */
  case PBUF_REF:
    /* only allocate memory for the pbuf structure */
    p = (struct pbuf *)memp_malloc(MEMP_PBUF);
    if (p == NULL) {
      return NULL;
    }
    /* caller must set this field properly, afterwards */
    p->payload = NULL;
    p->len = p->tot_len = length;
    p->next = NULL;
    p->type = type;
    break;
  default:
    return NULL;
  }
  /* set reference count */
  p->ref = 1;
  /* set flags */
  p->flags = 0;
  return p;
}



/**
 * Dereference a pbuf chain or queue and deallocate any no-longer-used
 * pbufs at the head of this chain or queue.
 *
 * Decrements the pbuf reference count. If it reaches zero, the pbuf is
 * deallocated.
 *
 * For a pbuf chain, this is repeated for each pbuf in the chain,
 * up to the first pbuf which has a non-zero reference count after
 * decrementing. So, when all reference counts are one, the whole
 * chain is free‘d.
 *
 * @param p The pbuf (chain) to be dereferenced.
 *
 * @return the number of pbufs that were de-allocated
 * from the head of the chain.
 *
 * @note MUST NOT be called on a packet queue (Not verified to work yet).
 * @note the reference counter of a pbuf equals the number of pointers
 * that refer to the pbuf (or into the pbuf).
 *
 * @internal examples:
 *
 * Assuming existing chains a->b->c with the following reference
 * counts, calling pbuf_free(a) results in:
 * 
 * 1->2->3 becomes ...1->3
 * 3->3->3 becomes 2->3->3
 * 1->1->2 becomes ......1
 * 2->1->1 becomes 1->1->1
 * 1->1->1 becomes .......
 *
 */
u8_t
pbuf_free(struct pbuf *p)
{
  u16_t type;
  struct pbuf *q;
  u8_t count;

  if (p == NULL) {
    return 0;
  }

  count = 0;
  /* de-allocate all consecutive pbufs from the head of the chain that
   * obtain a zero reference count after decrementing*/
  while (p != NULL) {
    u16_t ref;
    SYS_ARCH_DECL_PROTECT(old_level); //申请临界变量保护
    /* Since decrementing ref cannot be guaranteed to be a single machine operation
     * we must protect it. We put the new ref into a local variable to prevent
     * further protection. */
    SYS_ARCH_PROTECT(old_level);   //进入临界区
    /* all pbufs in a chain are referenced at least once */
    LWIP_ASSERT("pbuf_free: p->ref > 0", p->ref > 0);
    /* decrease reference count (number of pointers to pbuf) */
    ref = --(p->ref);
    SYS_ARCH_UNPROTECT(old_level); //退出临界区

    /* this pbuf is no longer referenced to? */
    if (ref == 0) {
      /* remember next pbuf in chain for next iteration */
      q = p->next;

      type = p->type;

      /* is this a pbuf from the pool? */
      if (type == PBUF_POOL) {
        memp_free(MEMP_PBUF_POOL, p);
      /* is this a ROM or RAM referencing pbuf? */
      } else if (type == PBUF_ROM || type == PBUF_REF) {
        memp_free(MEMP_PBUF, p);
      /* type == PBUF_RAM */
      } else {
        mem_free(p);
      }

      count++;
      /* proceed to next pbuf */
      p = q;
    /* p->ref > 0, this pbuf is still referenced to */
    /* (and so the remaining pbufs in chain as well) */
    } else {
      /* stop walking through the chain */
      p = NULL;
    }
  }
  /* return number of de-allocated pbufs */
  return count;
}


/**
 *
 * Create PBUF_RAM copies of pbufs.
 *
 * Used to queue packets on behalf of the lwIP stack, such as
 * ARP based queueing.
 *
 * @note You MUST explicitly use p = pbuf_take(p);
 *
 * @note Only one packet is copied, no packet queue!
 *
 * @param p_to pbuf destination of the copy
 * @param p_from pbuf source of the copy
 *
 * @return ERR_OK if pbuf was copied
 *         ERR_ARG if one of the pbufs is NULL or p_to is not big
 *                 enough to hold p_from
 */
err_t
pbuf_copy(struct pbuf *p_to, struct pbuf *p_from)
{
  u16_t offset_to=0, offset_from=0, len;

  /* is the target big enough to hold the source? */
  LWIP_ERROR("pbuf_copy: target not big enough to hold source", ((p_to != NULL) &&
             (p_from != NULL) && (p_to->tot_len >= p_from->tot_len)), return ERR_ARG;);

  /* iterate through pbuf chain */
  do
  {
    LWIP_ASSERT("p_to != NULL", p_to != NULL);

    /* copy one part of the original chain */
    if ((p_to->len - offset_to) >= (p_from->len - offset_from)) { //每次拷贝的长度是源端和目标端当前pbuf所剩空间的较小值，offset为当前pbuf拷贝数据的偏移量
      /* complete current p_from fits into current p_to */
      len = p_from->len - offset_from;
    } else {
      /* current p_from does not fit into current p_to */
      len = p_to->len - offset_to;
    }

    MEMCPY((u8_t*)p_to->payload + offset_to, (u8_t*)p_from->payload + offset_from, len);
    offset_to += len;
    offset_from += len;

    if (offset_to == p_to->len) { //目标端当前pbuf空间已满，转向下一个pbuf,记得offset清零
      /* on to next p_to (if any) */
      offset_to = 0;
      p_to = p_to->next;
    }

    if (offset_from >= p_from->len) { //源端当前pbuf数据已拷贝完，转向下一个pbuf,记得offset清零
      /* on to next p_from (if any) */
      offset_from = 0;
      p_from = p_from->next;
    }

    if((p_from != NULL) && (p_from->len == p_from->tot_len)) {
      /* don‘t copy more than one packet! */
      LWIP_ERROR("pbuf_copy() does not allow packet queues!\n",
                 (p_from->next == NULL), return ERR_VAL;);
    }
    if((p_to != NULL) && (p_to->len == p_to->tot_len)) {
      /* don‘t copy more than one packet! */
      LWIP_ERROR("pbuf_copy() does not allow packet queues!\n",
                  (p_to->next == NULL), return ERR_VAL;);
    }
  } while (p_from);

  LWIP_DEBUGF(PBUF_DEBUG | LWIP_DBG_TRACE, ("pbuf_copy: end of chain reached.\n"));
  return ERR_OK;
}

　　可以看到，回收 pbuf 使用pbuf_free()函数，该函数首先要减少 pbuf 索引计数（reference count）。如果引用计数已经减为 0，这个 pbuf 被回收。对于一个pbuf链来说，只有前一个pbuf被回收，才会考虑回收后面的pbuf，如果前面pbuf计数还不为0，则直接返回。