rabbitmq进程崩溃导致服务器宕机

标签：running   size   intern   https   system   操作系统   cat   file   shell   

背景

操作系统是 centos 6，RabbitMQ 3.8.1.

服务器突然无法连接，重启后查看/var/log/messages，报错日志如下：

epmd: erlang mapper daemon cant fork 2‘nd time Cannot allocate memory
epmd: erlang mapper daemon cant fork 2‘nd time Cannot allocate memory
epmd: erlang mapper daemon cant fork 2‘nd time Cannot allocate memory
epmd: erlang mapper daemon cant fork 2‘nd time Cannot allocate memory
sshd[1121]: error: fork: Cannot allocate memory
sshd[1121]: error: fork: Cannot allocate memory
sshd[1121]: error: fork: Cannot allocate memory
sshd[1121]: error: fork: Cannot allocate memory
sshd[1121]: error: fork: Cannot allocate memory
sshd[1121]: error: fork: Cannot allocate memory
sshd[1121]: error: fork: Cannot allocate memory
sshd[1121]: error: fork: Cannot allocate memory

使用locate erl_crash.dump找到崩溃日志 erl_crash.dump，使用 https://github.com/ferd/recon/edit/master/script/erl_crashdump_analyzer.sh 分析结果如下：

analyzing erl_crash.dump, generated on:  Wed Apr 22 16:39:10 2020 

Slogan: init terminating in do_boot ({error,{cannot_read_enabled_plugins_file,/etc/rabbitmq/enabled_plugins,eacces}})

Memory:
===
  processes: 17 Mb
  processes_used: 17 Mb
  system: 16 Mb
  atom: 0 Mb
  atom_used: 0 Mb
  binary: 0 Mb
  code: 6 Mb
  ets: 1 Mb
  ---
  total: 34 Mb

Different message queue lengths (5 largest different):
===
     89 0

Error logger queue length:
===
0

File descriptors open:
===
  UDP:  0
  TCP:  2
  Files:  0
  ---
  Total:  2

Number of processes:
===
89

Processes Heap+Stack memory sizes (words) used in the VM (5 largest different):
===
      1 46422
      1 10958
      2 6772
      2 4185
      2 2586

Processes OldHeap memory sizes (words) used in the VM (5 largest different):
===
      1 318187
      1 75113
      1 46422
      1 10958
      2 2586

Process States when crashing (sum): 
===
      1 CONNECTED
      2 CONNECTED|BINARY_IO
      2 CONNECTED|BINARY_IO|PORT_LOCK
      1 Current Process Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | ACTIVE | RUNNING | ACTIVE_SYS
      1 Current Process Running
      1 Internal ACT_PRIO_HIGH | USR_PRIO_HIGH | PRQ_PRIO_HIGH | OFF_HEAP_MSGQ
      3 Internal ACT_PRIO_MAX | USR_PRIO_MAX | PRQ_PRIO_MAX
      1 Internal ACT_PRIO_MAX | USR_PRIO_MAX | PRQ_PRIO_MAX | OFF_HEAP_MSGQ
      1 Internal ACT_PRIO_MAX | USR_PRIO_MAX | PRQ_PRIO_NORMAL
     76 Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL
      1 Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | ACTIVE | RUNNING | ACTIVE_SYS
      6 Internal ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | OFF_HEAP_MSGQ
      1 Running
     88 Waiting

没排查出问题来，这台机器是8c32g的机器，按道理应该不该出现内存不足的问题。

后面开发说RabbitMQ不用了，服务停掉。。。问题消失了。

rabbitmq进程崩溃导致服务器宕机

标签：running   size   intern   https   system   操作系统   cat   file   shell   

原文地址：https://www.cnblogs.com/fsckzy/p/14178630.html