码迷,mamicode.com
首页 > 其他好文 > 详细

网口up不起来问题排查

时间:2017-07-29 11:29:09      阅读:1631      评论:0      收藏:0      [点我收藏+]

标签:kernel   cti   lag   base   sys   ace   cache   reserve   调用   

最近处理一个问题,发现有的网口up不起来。
   
  ethtool eth6
  Settings for eth6:
  Supported ports: [ FIBRE ]
  Supported link modes: 10000baseT/Full
  Supports auto-negotiation: No
  Advertised link modes: 10000baseT/Full
  Advertised pause frame use: No
  Advertised auto-negotiation: No
  Speed: 10000Mb/s
  Duplex: Full
  Port: FIBRE
  PHYAD: 0
  Transceiver: external
  Auto-negotiation: off
  Supports Wake-on: g
  Wake-on: g
  Current message level: 0x0000000f (15)
  drv probe link timer
  Link detected: no
   
  ifconfig eth6 up 没有反应。
   
  strace这个过程,打印如下:
  ioctl(4, SIOCGIFFLAGS, {ifr_name="eth6", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_SLAVE|IFF_MULTICAST}) = 0
  ioctl(4, SIOCSIFFLAGS, 0x7ffc98d26c90) = 0
  exit_group(0) = ?
   
  内核代码流程是:
  case SIOCSIFFLAGS: /* Set interface flags */
  return dev_change_flags(dev, ifr->ifr_flags);
  返回0按道理是正常的,为什么起不来。
   
  于是干脆再down一遍,奇迹出现了,down完之后再up,再进行strace,发现有如下打印:
   
  ioctl(4, SIOCSIFFLAGS, 0x7fff37670450) = -1 ENOMEM (Cannot allocate memory)
  dup(2) = 6
  fcntl(6, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE)
  fstat(6, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 28), ...}) = 0
  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9972e0e000
  lseek(6, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
  write(6, "SIOCSIFFLAGS: Cannot allocate me"..., 37SIOCSIFFLAGS: Cannot allocate memory
  ) = 37
  close(6) = 0
  munmap(0x7f9972e0e000, 4096) = 0
  exit_group(-1) = ?
   
  分配不到内存,于是查看dmesg,有新的发现:
   
  调用堆栈如下:
   
  JAedge6:~ # dmesg
  [9162409.155614] ifconfig: page allocation failure: order:5, mode:0x80d0
  [9162409.155619] Pid: 781, comm: ifconfig Tainted: G ENX 3.0.101-0.47.52-default #1
  [9162409.155620] Call Trace:
  [9162409.155634] [<ffffffff81004b95>] dump_trace+0x75/0x300
  [9162409.155641] [<ffffffff81461d43>] dump_stack+0x69/0x6f
  [9162409.155648] [<ffffffff811024e6>] warn_alloc_failed+0xc6/0x170
  [9162409.155652] [<ffffffff811040f1>] __alloc_pages_slowpath+0x561/0x7f0--------进入了slowpath,
  [9162409.155655] [<ffffffff81104569>] __alloc_pages_nodemask+0x1e9/0x200
  [9162409.155660] [<ffffffff81007746>] dma_generic_alloc_coherent+0xa6/0x160
  [9162409.155666] [<ffffffff81030f18>] x86_swiotlb_alloc_coherent+0x28/0x80
  [9162409.155684] [<ffffffffa05a5aa1>] i40e_setup_tx_descriptors+0xf1/0x140 [i40e]
  [9162409.155713] [<ffffffffa05869cd>] i40e_vsi_setup_tx_resources+0x2d/0x60 [i40e]
  [9162409.155722] [<ffffffffa058b347>] i40e_vsi_open+0x27/0x1c0 [i40e]
  [9162409.155731] [<ffffffffa058b53b>] i40e_open+0x5b/0x90 [i40e]
  [9162409.155741] [<ffffffff813a5da7>] __dev_open+0xa7/0x110
  [9162409.155745] [<ffffffff813a48fb>] __dev_change_flags+0x9b/0x180--------------熟悉的函数
  [9162409.155749] [<ffffffff813a5cb0>] dev_change_flags+0x20/0x70
  [9162409.155753] [<ffffffff8140a955>] devinet_ioctl+0x5b5/0x620
  [9162409.155758] [<ffffffff8138f509>] sock_do_ioctl+0x29/0x60
  [9162409.155761] [<ffffffff8138f86b>] sock_ioctl+0x7b/0x280
  [9162409.155765] [<ffffffff8116ec3b>] do_vfs_ioctl+0x8b/0x3b0
  [9162409.155768] [<ffffffff8116f001>] sys_ioctl+0xa1/0xb0
  [9162409.155772] [<ffffffff8146cda7>] tracesys+0xd9/0xde
  [9162409.155787] [<00007fb7de2a6ce7>] 0x7fb7de2a6ce6
   
  根据分配模式mode:0x80d0,可以知道宏值为:
  #define ___GFP_HIGH 0x20u
  #define ___GFP_IO 0x40u
  #define ___GFP_FS 0x80u
  #define ___GFP_ZERO 0x8000u
   
  要分配的内存大小为32*4k=128k
  既然进入了slowpath,那么说明没有合适的块。
   
  [9162409.162691] Node 0 DMA free:15904kB min:120kB low:148kB high:180kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15680kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
  [9162409.162696] lowmem_reserve[]: 0 1599 128859 128859
  [9162409.162698] Node 0 DMA32 free:521760kB min:12684kB low:15852kB high:19024kB active_anon:1372kB inactive_anon:17864kB active_file:396kB inactive_file:800932kB unevictable:6464kB isolated(anon):0kB isolated(file):0kB present:1637408kB mlocked:6464kB dirty:0kB writeback:0kB mapped:324kB shmem:17892kB slab_reclaimable:9804kB slab_unreclaimable:117188kB kernel_stack:16kB pagetables:204kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
  [9162409.162704] lowmem_reserve[]: 0 0 127260 127260
  [9162409.162706] Node 0 Normal free:1516580kB min:1009708kB low:1262132kB high:1514560kB active_anon:8746684kB inactive_anon:2298136kB active_file:1093096kB inactive_file:96852196kB unevictable:15591060kB isolated(anon):0kB isolated(file):0kB present:130314240kB mlocked:15591096kB dirty:608kB writeback:0kB mapped:128548kB shmem:2307616kB slab_reclaimable:367732kB slab_unreclaimable:2127656kB kernel_stack:31920kB pagetables:223776kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
  [9162409.162712] lowmem_reserve[]: 0 0 0 0
  [9162409.162714] Node 1 Normal free:1621196kB min:1025480kB low:1281848kB high:1538220kB active_anon:11279020kB inactive_anon:1941384kB active_file:1080860kB inactive_file:88023016kB unevictable:24280980kB isolated(anon):0kB isolated(file):0kB present:132349388kB mlocked:24280984kB dirty:812kB writeback:0kB mapped:5795476kB shmem:7620708kB slab_reclaimable:358276kB slab_unreclaimable:1995312kB kernel_stack:24040kB pagetables:271884kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
   
  [9162449.769662] ifconfig: page allocation failure: order:6, mode:0xd0-------------也有这种模式的
  [9162449.769669] Pid: 17429, comm: ifconfig Tainted: G ENX 3.0.101-0.47.52-default #1
  [9162449.769672] Call Trace:
  [9162449.769689] [<ffffffff81004b95>] dump_trace+0x75/0x300
  [9162449.769700] [<ffffffff81461d43>] dump_stack+0x69/0x6f
  [9162449.769712] [<ffffffff811024e6>] warn_alloc_failed+0xc6/0x170
  [9162449.769721] [<ffffffff811040f1>] __alloc_pages_slowpath+0x561/0x7f0
  [9162449.769731] [<ffffffff81104569>] __alloc_pages_nodemask+0x1e9/0x200
  [9162449.769740] [<ffffffff81145b73>] kmem_getpages+0x53/0x180
  [9162449.769746] [<ffffffff81146976>] fallback_alloc+0x196/0x270
  [9162449.769753] [<ffffffff8114701c>] __kmalloc+0x28c/0x330
  [9162449.769786] [<ffffffffa05a59ec>] i40e_setup_tx_descriptors+0x3c/0x140 [i40e]
  [9162449.769856] [<ffffffffa05869cd>] i40e_vsi_setup_tx_resources+0x2d/0x60 [i40e]
  [9162449.769876] [<ffffffffa058b347>] i40e_vsi_open+0x27/0x1c0 [i40e]
  [9162449.769907] [<ffffffffa058b53b>] i40e_open+0x5b/0x90 [i40e]
  [9162449.769930] [<ffffffff813a5da7>] __dev_open+0xa7/0x110
  [9162449.769939] [<ffffffff813a48fb>] __dev_change_flags+0x9b/0x180
  [9162449.769949] [<ffffffff813a5cb0>] dev_change_flags+0x20/0x70
  [9162449.769957] [<ffffffff8140a955>] devinet_ioctl+0x5b5/0x620
  [9162449.769964] [<ffffffff8138f509>] sock_do_ioctl+0x29/0x60
  [9162449.769970] [<ffffffff8138f86b>] sock_ioctl+0x7b/0x280
  [9162449.770006] [<ffffffff8116ec3b>] do_vfs_ioctl+0x8b/0x3b0
  [9162449.770012] [<ffffffff8116f001>] sys_ioctl+0xa1/0xb0
  [9162449.770019] [<ffffffff8146cda7>] tracesys+0xd9/0xde
  [9162449.770050] [<00007fdde024ace7>] 0x7fdde024ace6
   
  在合并之前,
  JAedge6:~ # cat /proc/buddyinfo
  Node 0, zone DMA 0 0 0 1 2 1 1 0 1 1 3
  Node 0, zone DMA32 419 643 403 451 313 225 235 251 148 48 3
  Node 0, zone Normal 20962 68138 35642 2430 161 2 0 0 0 0 1
  Node 1, zone Normal 68686 157150 906 8 0 0 0 0 0 0 1
  明明还有最大的一块内存可以拆分,为啥没拆?
   
  ok,手动合并块。
  echo 1 > /proc/sys/vm/drop_caches
  JAedge6:~ # cat /proc/buddyinfo
  Node 0, zone DMA 0 0 0 1 2 1 1 0 1 1 3
  Node 0, zone DMA32 8557 8460 7429 5915 3586 1499 380 260 148 48 3
  Node 0, zone Normal 2834074 2826437 1777328 672633 109003 4046 16 0 0 0 1
  Node 1, zone Normal 2943155 2702189 1645931 586739 83425 2255 2 0 0 0 1
   
  echo 2 > /proc/sys/vm/drop_caches
  echo 3 > /proc/sys/vm/drop_caches
   
  JAedge6:~ # cat /proc/buddyinfo
  Node 0, zone DMA 0 0 0 1 2 1 1 0 1 1 3
  Node 0, zone DMA32 6201 5597 4910 4359 3225 1898 624 332 153 48 3
  Node 0, zone Normal 1453979 2550503 1714926 796684 187727 13800 151 0 0 0 1
  Node 1, zone Normal 1619014 2484161 1619823 701543 144586 7165 7 0 0 0 1
  对比合并之后:发现再往大块的合并并没有明显增加,但是至少256的块是有了。
   
  再次操作先网口down,再up,都起来了。
   
  但是为什么要先down再up,这个一直没搞清楚,看strace的话,怀疑是__dev_close的时候会释放一些内存,然后立刻up的话,这些内存能给我用。

网口up不起来问题排查

标签:kernel   cti   lag   base   sys   ace   cache   reserve   调用   

原文地址:http://www.cnblogs.com/10087622blog/p/7253603.html

(0)
(1)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!