Gangmax Blog

A Chef Server Installation Error

Install Chef server should be simple:

  1. Download the Chef server binary file from here. In my case, it’s “chef-server-11.0.8-1.el6.x86_64.rpm”.

  2. Run “rpm -Uvh chef-server-11.0.8-1.el6.x86_64.rpm” command with the root permission to install Chef server.

  3. Run “sudo chef-server-ctl reconfigure” command to do the configuration.

  4. Run “chef-server-ctl test” to verify.

  5. Done.

But I got stucked at step 3 by the following error:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
  * execute[/opt/chef-server/embedded/bin/chpst -u chef_server -U chef_server /opt/chef-server/embedded/bin/rabbitmqctl wait /var/opt/chef-server/rabbitmq/db/rabbit@localhost.pid] action run
================================================================================
Error executing action `run` on resource 'execute[/opt/chef-server/embedded/bin/chpst -u chef_server -U chef_server /opt/chef-server/embedded/bin/rabbitmqctl wait /var/opt/chef-server/rabbitmq/db/rabbit@localhost.pid]'
================================================================================


Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '2'
---- Begin output of /opt/chef-server/embedded/bin/chpst -u chef_server -U chef_server /opt/chef-server/embedded/bin/rabbitmqctl wait /var/opt/chef-server/rabbitmq/db/rabbit@localhost.pid ----
STDOUT: Waiting for rabbit@localhost ...
pid is 13744 ...
STDERR: Error: process_not_running
---- End output of /opt/chef-server/embedded/bin/chpst -u chef_server -U chef_server /opt/chef-server/embedded/bin/rabbitmqctl wait /var/opt/chef-server/rabbitmq/db/rabbit@localhost.pid ----
Ran /opt/chef-server/embedded/bin/chpst -u chef_server -U chef_server /opt/chef-server/embedded/bin/rabbitmqctl wait /var/opt/chef-server/rabbitmq/db/rabbit@localhost.pid returned 2


Resource Declaration:
---------------------
# In /opt/chef-server/embedded/cookbooks/chef-server/recipes/rabbitmq.rb

 80:   execute "/opt/chef-server/embedded/bin/chpst -u #{node["chef_server"]["user"]["username"]} -U #{node["chef_server"]["user"]["username"]} /opt/chef-server/embedded/bin/rabbitmqctl wait /var/opt/chef-server/rabbitmq/db/rabbit@localhost.pid" do
 81:     retries 10
 82:   end
 83: 



Compiled Resource:
------------------
# Declared in /opt/chef-server/embedded/cookbooks/chef-server/recipes/rabbitmq.rb:80:in `from_file'

execute("/opt/chef-server/embedded/bin/chpst -u chef_server -U chef_server /opt/chef-server/embedded/bin/rabbitmqctl wait /var/opt/chef-server/rabbitmq/db/rabbit@localhost.pid") do
  action "run"
  retries 0
  retry_delay 2
  command "/opt/chef-server/embedded/bin/chpst -u chef_server -U chef_server /opt/chef-server/embedded/bin/rabbitmqctl wait /var/opt/chef-server/rabbitmq/db/rabbit@localhost.pid"
  backup 5
  returns 0
  cookbook_name :"chef-server"
  recipe_name "rabbitmq"
end



[2013-06-06T01:48:09+00:00] ERROR: Running exception handlers
[2013-06-06T01:48:09+00:00] ERROR: Exception handlers complete
Chef Client failed. 2 resources updated
[2013-06-06T01:48:09+00:00] FATAL: Stacktrace dumped to /opt/chef-server/embedded/cookbooks/cache/chef-stacktrace.out
[2013-06-06T01:48:09+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: execute[/opt/chef-server/embedded/bin/chpst -u chef_server -U chef_server /opt/chef-server/embedded/bin/rabbitmqctl wait /var/opt/chef-server/rabbitmq/db/rabbit@localhost.pid] (chef-server::rabbitmq line 80) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '2'
---- Begin output of /opt/chef-server/embedded/bin/chpst -u chef_server -U chef_server /opt/chef-server/embedded/bin/rabbitmqctl wait /var/opt/chef-server/rabbitmq/db/rabbit@localhost.pid ----
STDOUT: Waiting for rabbit@localhost ...
pid is 13744 ...
STDERR: Error: process_not_running
---- End output of /opt/chef-server/embedded/bin/chpst -u chef_server -U chef_server /opt/chef-server/embedded/bin/rabbitmqctl wait /var/opt/chef-server/rabbitmq/db/rabbit@localhost.pid ----
Ran /opt/chef-server/embedded/bin/chpst -u chef_server -U chef_server /opt/chef-server/embedded/bin/rabbitmqctl wait /var/opt/chef-server/rabbitmq/db/rabbit@localhost.pid returned 2

If I run “sudo chef-server-ctl tail” command to view the Chef server log, I can see the following error messages again and again:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
2013-06-06_02:04:16.26950 Activating RabbitMQ plugins ...
2013-06-06_02:04:17.02079 0 plugins activated:
2013-06-06_02:04:17.02088 
2013-06-06_02:04:18.01948 
2013-06-06_02:04:18.01950 +---+   +---+
2013-06-06_02:04:18.01951 |   |   |   |
2013-06-06_02:04:18.01951 |   |   |   |
2013-06-06_02:04:18.01951 |   |   |   |
2013-06-06_02:04:18.01952 |   +---+   +-------+
2013-06-06_02:04:18.01952 |                   |
2013-06-06_02:04:18.01952 | RabbitMQ  +---+   |
2013-06-06_02:04:18.01952 |           |   |   |
2013-06-06_02:04:18.01953 |   v2.7.1  +---+   |
2013-06-06_02:04:18.01953 |                   |
2013-06-06_02:04:18.01953 +-------------------+
2013-06-06_02:04:18.01953 AMQP 0-9-1 / 0-9 / 0-8
2013-06-06_02:04:18.01953 Copyright (C) 2007-2011 VMware, Inc.
2013-06-06_02:04:18.01954 Licensed under the MPL.  See http://www.rabbitmq.com/
2013-06-06_02:04:18.01954 
2013-06-06_02:04:18.02649 node           : rabbit@localhost
2013-06-06_02:04:18.02669 app descriptor : /opt/chef-server/embedded/service/rabbitmq/sbin/../ebin/rabbit.app
2013-06-06_02:04:18.02687 home dir       : /var/opt/chef-server/rabbitmq
2013-06-06_02:04:18.02707 config file(s) : (none)
2013-06-06_02:04:18.02744 cookie hash    : k9l8J1nXB29ryHgMrqpENA==
2013-06-06_02:04:18.02769 log            : /var/opt/chef-server/rabbitmq/log/rabbit@localhost.log
2013-06-06_02:04:18.02835 sasl log       : /var/opt/chef-server/rabbitmq/log/rabbit@localhost-sasl.log
2013-06-06_02:04:18.02936 database dir   : /var/opt/chef-server/rabbitmq/db/rabbit@localhost
2013-06-06_02:04:18.02960 erlang version : 5.9.2
2013-06-06_02:04:18.02970 
2013-06-06_02:04:18.03472 -- rabbit boot start
2013-06-06_02:04:18.03489 starting file handle cache server                                     ...done
2013-06-06_02:04:18.03590 starting worker pool                                                  ...done
2013-06-06_02:04:18.03649 starting database                                                     ...done
2013-06-06_02:04:18.04631 starting codec correctness check                                      ...done
2013-06-06_02:04:18.04646 -- external infrastructure ready
2013-06-06_02:04:18.04656 starting statistics event manager                                     ...done
2013-06-06_02:04:18.04692 starting logging server                                               ...done
2013-06-06_02:04:18.04740 starting plugin registry                                              ...done
2013-06-06_02:04:18.04767 starting auth mechanism amqplain                                      ...done
2013-06-06_02:04:18.04785 starting auth mechanism cr-demo                                       ...done
2013-06-06_02:04:18.04803 starting auth mechanism plain                                         ...done
2013-06-06_02:04:18.04831 starting exchange type direct                                         ...done
2013-06-06_02:04:18.04848 starting exchange type fanout                                         ...done
2013-06-06_02:04:18.04866 starting exchange type headers                                        ...done
2013-06-06_02:04:18.04883 starting exchange type topic                                          ...done
2013-06-06_02:04:18.04897 -- kernel ready
2013-06-06_02:04:18.04907 starting node monitor                                                 ...done
2013-06-06_02:04:18.04954 starting cluster delegate                                             ...done
2013-06-06_02:04:18.05145 starting guid generator                                               ...done
2013-06-06_02:04:18.06185 starting alarm handler                                                ...done
2013-06-06_02:04:18.06360 starting memory monitor                                               ...done
2013-06-06_02:04:18.06413 -- core initialized
2013-06-06_02:04:18.06429 starting empty DB check                                               ...done
2013-06-06_02:04:18.06448 starting exchange, queue and binding recovery                         ...done
2013-06-06_02:04:18.07291 starting mirror queue slave sup                                       ...done
2013-06-06_02:04:18.07327 starting adding mirrors to queues                                     ...done
2013-06-06_02:04:18.07356 -- message delivery logic ready
2013-06-06_02:04:18.07367 starting error log relay                                              ...done
2013-06-06_02:04:18.07401 starting networking                                                   ...BOOT ERROR: FAILED
2013-06-06_02:04:18.07765 Reason: {badmatch,
2013-06-06_02:04:18.07765             {error,
2013-06-06_02:04:18.07766                 {shutdown,
2013-06-06_02:04:18.07766                     {child,undefined,
2013-06-06_02:04:18.07766                         'rabbit_tcp_listener_sup_127.0.0.1:5672',
2013-06-06_02:04:18.07766                         {tcp_listener_sup,start_link,
2013-06-06_02:04:18.07767                             [{127,0,0,1},
2013-06-06_02:04:18.07767                              5672,
2013-06-06_02:04:18.07767                              [inet,binary,
2013-06-06_02:04:18.07767                               {packet,raw},
2013-06-06_02:04:18.07768                               {reuseaddr,true},
2013-06-06_02:04:18.07768                               {backlog,128},
2013-06-06_02:04:18.07768                               {nodelay,true},
2013-06-06_02:04:18.07768                               {exit_on_close,false}],
2013-06-06_02:04:18.07769                              {rabbit_networking,tcp_listener_started,[amqp]},
2013-06-06_02:04:18.07769                              {rabbit_networking,tcp_listener_stopped,[amqp]},
2013-06-06_02:04:18.07770                              {rabbit_networking,start_client,[]},
2013-06-06_02:04:18.07771                              "TCP Listener"]},
2013-06-06_02:04:18.07771                         transient,infinity,supervisor,
2013-06-06_02:04:18.07771                         [tcp_listener_sup]}}}}
2013-06-06_02:04:18.07771 Stacktrace: [{rabbit_networking,start_listener0,4,[]},
2013-06-06_02:04:18.07772              {rabbit_networking,'-start_listener/4-lc$^0/1-0-',4,[]},
2013-06-06_02:04:18.07772              {rabbit_networking,start_listener,4,[]},
2013-06-06_02:04:18.07773              {rabbit_networking,'-boot_tcp/0-lc$^0/1-0-',1,[]},
2013-06-06_02:04:18.07773              {rabbit_networking,boot_tcp,0,[]},
2013-06-06_02:04:18.07773              {rabbit_networking,boot,0,[]},
2013-06-06_02:04:18.07773              {rabbit,'-run_boot_step/1-lc$^1/1-1-',1,[]},
2013-06-06_02:04:18.07774              {rabbit,run_boot_step,1,[]}]
2013-06-06_02:04:19.08754 [os_mon] cpu supervisor port (cpu_sup): Erlang has closed
2013-06-06_02:04:20.87345 Activating RabbitMQ plugins ...
2013-06-06_02:04:21.61271 0 plugins activated:

So it looks the error is caused by some network issue.

First, I stop the “iptables”:

1
2
3
sudo /etc/init.d/iptables save # Save
sudo /etc/init.d/iptables stop # Stop
sudo chkconfig iptables off    # Turn it off even restarting

But the error is still there.

Then I did a lot of google search and then I found this article which helps me. The root cause of this error is that when RabbitMQ service starts it need to listen on the “5672” port, but this port is used by some other service which starts when the server starts. I need to find out what it is and stop the service to release the port “5672”.

1
2
3
4
5
6
7
# Find it's qpidd which is using the port "5672".
sudo netstat -nltp | grep 5672
tcp        0      0 0.0.0.0:5672                0.0.0.0:*                   LISTEN      2979/qpidd          
tcp        0      0 :::5672                     :::*                        LISTEN      2979/qpidd  
# Stop it.
sudo service qpidd stop
sudo service iptables stop

Then run the “sudo chef-server-ctl reconfigure” command again and everything works as expected.

Comments