Playbook execution hangs on client with defunct process

[msaffar]

Hello,
The playbook execution on AIX '7.1 TL3 SP5' client hangs occasionally with a defunct porcess:

ansiuser 4980858 8585442 0 17:01:23 - 0:00 sshd: ansiuser@pts/1
root 8585442 4718616 0 17:01:22 - 0:00 sshd: ansiuser [priv]
ansiuser 11862210 4980858 8 0:00

Ansible Server : RedHat el6
Ansible: 1.9.1

[jimi-c]

@msaffar could you test the devel branch or 2.0.x against that system and see if it suffers the same problem? We've changed the way we execute remotely over ssh and this may have resolved the issue there.

[msaffar]

@jimi-c , even with ansible 2.0.0.1, we still have the same problem.

[msaffar]

@jimi-c, i found the solution in the following link:
https://groups.google.com/forum/#!msg/ansible-project/IQoTNwDBIiA/qiHUTgg31lkJ
Is-it possible to implement it and update the ssh.py in a next release ?
Thank you

[bcoca]

If you add -T to your ssh arguments it will override the -tt setting.

we cannot really remove -tt as this breaks interactive sudo prompts, we are looking at making it conditional though.

[jctanner]

!needs_info

[sousmangoosta]

Hello,

I have defunct process on Linux CentOS 7.1.1503 :

root 48615 0.0 0.0 168352 2224 ? S 16:22 0:00 /usr/sbin/CROND -n
newtprod 48616 0.0 0.0 108320 640 ? Ss 16:22 0:00 _ flock -nE 0 /tmp/ansible-playbook.lock -c (time /usr/bin/ansible-playbook -i /USR/newtprod/ansible/dev /USR/newtprod/ansible/deploy.yml
newtprod 48617 0.0 0.0 113116 1200 ? S 16:22 0:00 _ /bin/sh -c (time /usr/bin/ansible-playbook -i /USR/newtprod/ansible/dev /USR/newtprod/ansible/deploy.yml) >> /VAR/newtprod/ansible/
newtprod 48618 0.0 0.0 113116 636 ? S 16:22 0:00 _ /bin/sh -c (time /usr/bin/ansible-playbook -i /USR/newtprod/ansible/dev /USR/newtprod/ansible/deploy.yml) >> /VAR/newtprod/ansi
newtprod 48619 7.0 0.7 264496 29124 ? R 16:22 5:02 _ /usr/bin/python /usr/bin/ansible-playbook -i /USR/newtprod/ansible/dev /USR/newtprod/ansible/deploy.yml
newtprod 51196 6.9 0.6 338228 26080 ? Sl 16:24 4:51 _ /usr/bin/python /usr/bin/ansible-playbook -i /USR/newtprod/ansible/dev /USR/newtprod/ansible/deploy.yml
newtprod 51197 0.0 0.0 0 0 ? Z 16:24 0:00 _ [ansible-playboo]

gdb BT on process 51196 :

#0 0x00007f9f407218f3 in select () from /lib64/libc.so.6
#1 0x00007f9f39f0907b in time_sleep () from /usr/lib64/python2.7/lib-dynload/timemodule.so
#2 0x00007f9f413fab94 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#3 0x00007f9f413fc1ad in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#4 0x00007f9f413fa85f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#5 0x00007f9f413fa950 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#6 0x00007f9f413fc1ad in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#7 0x00007f9f41389098 in function_call () from /lib64/libpython2.7.so.1.0
#8 0x00007f9f41364073 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#9 0x00007f9f41373085 in instancemethod_call () from /lib64/libpython2.7.so.1.0
#10 0x00007f9f41364073 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#11 0x00007f9f413bb167 in slot_tp_init () from /lib64/libpython2.7.so.1.0
#12 0x00007f9f413b9e7f in type_call () from /lib64/libpython2.7.so.1.0
#13 0x00007f9f41364073 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#14 0x00007f9f413f834c in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#15 0x00007f9f413fa950 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#16 0x00007f9f413fc1ad in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#17 0x00007f9f413fa85f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#18 0x00007f9f413fc1ad in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#19 0x00007f9f413fa85f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#20 0x00007f9f413fc1ad in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#21 0x00007f9f413fa85f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#22 0x00007f9f413fc1ad in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#23 0x00007f9f413fa85f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#24 0x00007f9f413fc1ad in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#25 0x00007f9f413fc2b2 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
#26 0x00007f9f414156ef in run_mod () from /lib64/libpython2.7.so.1.0
#27 0x00007f9f414168ae in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0
#28 0x00007f9f41417b39 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0
#29 0x00007f9f41428b3f in Py_Main () from /lib64/libpython2.7.so.1.0
#30 0x00007f9f40655af5 in __libc_start_main () from /lib64/libc.so.6
#31 0x0000000000400721 in _start ()

gdb BT on process 48619 :

#0 0x00007f9f407218d3 in __select_nocancel () from /lib64/libc.so.6
#1 0x00007f9f39f0907b in time_sleep () from /usr/lib64/python2.7/lib-dynload/timemodule.so
#2 0x00007f9f413fab94 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#3 0x00007f9f413fc1ad in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#4 0x00007f9f413fa85f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#5 0x00007f9f413fc1ad in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#6 0x00007f9f413fa85f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#7 0x00007f9f413fc1ad in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#8 0x00007f9f413fa85f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#9 0x00007f9f413fc1ad in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#10 0x00007f9f413fa85f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#11 0x00007f9f413fc1ad in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#12 0x00007f9f413fa85f in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#13 0x00007f9f413fc1ad in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#14 0x00007f9f413fc2b2 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
#15 0x00007f9f414156ef in run_mod () from /lib64/libpython2.7.so.1.0
#16 0x00007f9f414168ae in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0
#17 0x00007f9f41417b39 in PyRun_SimpleFileExFlags () from /lib64/libpython2.7.so.1.0
#18 0x00007f9f41428b3f in Py_Main () from /lib64/libpython2.7.so.1.0
#19 0x00007f9f40655af5 in __libc_start_main () from /lib64/libc.so.6
#20 0x0000000000400721 in _start ()

Ansible version : ansible-2.0.1.0-2.el7.noarch

What can I do to give you more debug infos ?

[casusbelli]

Hi!
I'm also seeing this issue in CentOS 7.2 continuously in our CI systems. I'd estimate one in 50 runs hits this.
CentOS Linux release 7.2.1511 (Core), Ansible 2.1.0.0

[ansibot]

@msaffar Greetings! Thanks for taking the time to open this issue. In order for the community to handle your issue effectively, we need a bit more information.

Here are the items we could not find in your description:

• issue type
• ansible version
• component name

Please set the description of this issue with this template:
https://raw.githubusercontent.com/ansible/ansible/devel/.github/ISSUE_TEMPLATE.md

click here for bot help

[tinawenqiao]

Hi, all, I have the same issue. But the following config returns "....Bad configuration option -t". How to set?
[ssh_connection]
ssh_args= -o -T

[ansibotdev]

@msaffar You have not responded to information requests in this issue so we will assume it no longer affects you. If you are still interested in this, please create a new issue with the requested information.

click here for bot help