Child process sometimes hang, peg cpu @ 100%

Here is pstack trace for one of the hung processes pegging cpu @ 100% - looks like some kind of deadlock
#0  __lll_unlock_wake () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:371
#1  0x00007f2d659f0f9e in _L_unlock_738 () from /lib64/libpthread.so.0
#2  0x00007f2d659f0f10 in __pthread_mutex_unlock_usercnt (decr=1, mutex=0x7f2c7c5248f8) at pthread_mutex_unlock.c:55
#3  __GI___pthread_mutex_unlock (mutex=0x7f2c7c5248f8) at pthread_mutex_unlock.c:330
#4  0x00007f2d57cb140d in h2_beam_receive () from /etc/httpd/modules/mod_http2.so
#5  0x00007f2d57cc8ee3 in buffer_output_receive () from /etc/httpd/modules/mod_http2.so
#6  0x00007f2d57ccb1ec in stream_data_cb () from /etc/httpd/modules/mod_http2.so
#7  0x00007f2d66cba171 in nghttp2_session_pack_data () from /lib64/libnghttp2.so.14
#8  0x00007f2d66cbaedd in nghttp2_session_mem_send_internal () from /lib64/libnghttp2.so.14
#9  0x00007f2d66cbbae9 in nghttp2_session_send () from /lib64/libnghttp2.so.14
#10 0x00007f2d57cc7544 in h2_session_send () from /etc/httpd/modules/mod_http2.so
#11 0x00007f2d57cc777a in h2_session_process () from /etc/httpd/modules/mod_http2.so
#12 0x00007f2d57cb2149 in h2_c1_run () from /etc/httpd/modules/mod_http2.so
#13 0x00007f2d57cb2569 in h2_c1_hook_process_connection () from /etc/httpd/modules/mod_http2.so
#14 0x00005571f66c33c0 in ap_run_process_connection (c=c@entry=0x7f2d4006d5e0) at connection.c:42
#15 0x00007f2d5a8ab40a in process_socket (thd=thd@entry=0x5571f72c4510, p=, sock=, cs=0x7f2d4006d530, my_child_num=my_child_num@entry=11, my_thread_num=my_thread_num@entry=12) at event.c:1086
#16 0x00007f2d5a8ae6ae in worker_thread (thd=0x5571f72c4510, dummy=) at event.c:2179
#17 0x00007f2d659edea5 in start_thread (arg=0x7f2d3a7f4700) at pthread_create.c:307
#18 0x00007f2d65512b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Hmm, how could it deadlock when unlocking a mutex?
Which version of httpd and mod_http2 are you running?

Good point, just trying to sound competent, failing.
Server version: Apache/2.4.54 (codeit)
Name        : mod_http2
Arch        : x86_64
Version     : 2.0.3
Release     : 2.codeit.el7
Size        : 398 k
Repo        : installed
From repo   : CodeIT
Summary     : module implementing HTTP/2 for Apache 2
URL         : https://icing.github.io/mod_h2/
License     : ASL 2.0
Description : The mod_h2 Apache httpd module implements the HTTP2 protocol
: (h2+h2c) on top of libnghttp2 for httpd 2.4 servers.

Same problem here on several different servers:
OS CentOS 7
httpd-2.4.54-1.codeit.el7.x86_64
mod_http2-2.0.4-1.codeit.el7.x86_64
The problem appears when using mpm_worker or mpm_even
It doesn't happen using mpm_prefork
So we had to "downgrade" from mpm_event to prefork to keep hrrp2 on even if we lost a lot of mpm performance
Version 2.0.4 is not fixing the behaviour
httpd processes keep CPU at 100% and at memory while progressively every CPU Processor get filled and RAM also
strace ing the stuck httpd processes says they are all locked in read
Hope this may help to track down the problem

@alexskynet you do not have to switch mpm to disable http/2. http/2 is only used if you configure
Protocols h2 http/1.1

in your server. If you comment that out or just configure Protocols http/1.1, no h2 will be used.
That allows you to determine if the problem is related to http2 or if there is another problem in your server.

Thank you very much @icing
I realized that few minutes after my post so I'm running with event now
I also realized the reason it worked with fork is that mod_http2 doesn't work with this worker
Have you idea how we may workaround those hangs?
Thank you again and best regards

I want to add a little piece to the puzzle that may help
We run several servers and noticed that some of those work fin with http2 with no hangs
Digging a little bit we noticed that the servers that hang the apache process have the root on NFS filesystem and the ones that don't hang have local filesystem.
Running the NFS rooted servers without the mod_http2 loaded leads to fully functional wuth no hangs of sort.
It wold sound interesting know id @Adam7288 is NFS rooted or not

Another info
Previous message isn't completely perfect ...
We discovered a server with nfs based root that works ok with mod_http2
We see non differences in configuration of mount point:
<IP_ADDRESS>:/apachesites on /var/www/virtual type nfs4 (rw,noatime,nodiratime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=<IP_ADDRESS>,local_lock=none,addr=<IP_ADDRESS>)
The difference is that the hanging servers all use Joomla both versions 3 and 4 but have no errors in logs of the sites

Yeah, hard to see the pattern from here.

I think you'd need to vary one thing at a time in your setup in order to find out which combination is the critical one. For example, place your /var/www/virtual/ on an ext4 and see if the problem persists.

Tried to downgrade the mod_http2 from 2.0.4 to 2.0.3 on one server
Let's wait

Downgrading to 2.0.3 seems to fix the problem
We've run form more than two hours with no hangs and normal load
Now we are testing the downgrade on a second server

I was experiencing this issue on 2.0.3 and continue to experience it on 2.0.4. NFS is something we use too so that is an interesting avenue. @icing is there any way we can get better data to troubleshoot this?

Thank you for your answer @Adam7288 this sounds interesting
Wich version of NFS protocol are you using?
3, 4.0 or 4.1?
We are using protocol 4.0 with noatime and nodiratrime options on the servers where we're experimenting the anomaly
I suspect something with NFS on the release of locks maybe???

I looked at the changes between 2.0.3 and 2.0.4 and would have been surprised by differences. BUT I can make you a 2.0.5 with the changes I did in Apache trunk. There are difference in connection handling. Hard to tell if they improve your situation, but I'd like to know how those changes fare for you.
The alternative in your case would be to try the mod_http2 that is release in Apache 2.5.54.

Regarding NFS usage: I have no personal experience with running a server that way. As I understand your setups, you use NFS to share the htdocs between nodes.
Question is: are files on the NFS share being modified and may that have impact? Have you EnableMmap on on this for example?

Only time nfs is invoked on our end is within a php script to access them in some way - for instance a document download. We do not serve up static or dynamic content directly from the nfs share.

Thank you @icing
I appreciate your help very much
I understand prfectly thet sometimes software devel leads to unexpected results ... it happenetd to me also so many times :-)
If I can help in testing I'll be more than happy to do it
Yes you understood perfectely
We share the root among some nodes
The only files that are actually modified during access to the webserver are cache files, that may be regenerated if expired, and session files.
Yes EnableMmap is on

Ok, I'll make a v2.0.5 for testing. In the meantime, you might want to configure H2CopyFiles on to see if that makes any difference for you.

https://github.com/icing/mod_h2/releases/tag/v2.0.5

thank you @icing
I'm compiling right now
Stay tuned

Sorry @icing
Module installed and restarted httpd
Got 2 stuck processes in read in a couple of minutes

"stuck in read" as from the status handler or some backtrace?

strace -l
shows read
htop show cpu 100% for the process

Status shows

6781
0/0/0
W
0.00
22
0
0
0.0
0.00
0.00
<IP_ADDRESS>
h2

rolled back to 2.0.3: no hangs till now

Hmm, I assume your site is too busy to raise the http2 log level much?

I may reinstall the 2.0.5 and try to increase the log for few minutes if it may help
Which directive do I need to insert?

If you can take it LogLevel http2:trace1, but may start with LogLevel http2:debug.

Ok give me some minutes

Stuck in seconds with 2.0.5
This is trace log but it looks ok to me
[Wed Sep 21 16:09:53.574077 2022] [http2:debug] [pid 23712:tid<PHONE_NUMBER>49376] h2_workers.c(318): h2_workers: cleanup 25 workers idling
[Wed Sep 21 16:09:53.574635 2022] [http2:debug] [pid 23712:tid<PHONE_NUMBER>49376] h2_workers.c(346): h2_workers: cleanup all workers terminated
[Wed Sep 21 16:09:53.575301 2022] [http2:debug] [pid 23712:tid<PHONE_NUMBER>49376] h2_workers.c(349): h2_workers: cleanup zombie workers joined
[Wed Sep 21 16:12:20.930961 2022] [mpm_worker:notice] [pid 23636:tid<PHONE_NUMBER>49376] AH00297: SIGUSR1 received.  Doing graceful restart
[Wed Sep 21 16:12:21.009126 2022] [http2:debug] [pid 25117:tid<PHONE_NUMBER>49376] h2_workers.c(370): h2_workers: cleanup 25 workers (25 idle)
[Wed Sep 21 16:12:21.009464 2022] [http2:debug] [pid 25117:tid<PHONE_NUMBER>49376] h2_workers.c(406): h2_workers: cleanup all workers terminated
[Wed Sep 21 16:12:21.010195 2022] [http2:debug] [pid 25117:tid<PHONE_NUMBER>49376] h2_workers.c(411): h2_workers: cleanup zombie workers joined
[Wed Sep 21 16:12:22.385166 2022] [lbmethod_heartbeat:notice] [pid 23636:tid<PHONE_NUMBER>49376] AH02282: No slotmem from mod_heartmonitor
[Wed Sep 21 16:12:22.388610 2022] [http2:info] [pid 23636:tid<PHONE_NUMBER>49376] AH03090: mod_http2 (v2.0.5-git, feats=CHPRIO+SHA256+INVHD+DWINS, nghttp2 1.49.0), initializing...
[Wed Sep 21 16:12:22.388690 2022] [http2:trace1] [pid 23636:tid<PHONE_NUMBER>49376] h2_protocol.c(426): h2_h2, child_init
[Wed Sep 21 16:12:22.388773 2022] [http2:trace1] [pid 23636:tid<PHONE_NUMBER>49376] h2_switch.c(46): h2_switch init
[Wed Sep 21 16:12:22.526948 2022] [mpm_worker:notice] [pid 23636:tid<PHONE_NUMBER>49376] AH00292: Apache/2.4.54 (codeit) OpenSSL/1.1.1q+quic mod_fcgid/2.3.9 configured -- resuming normal operations
[Wed Sep 21 16:12:22.587929 2022] [core:notice] [pid 23636:tid<PHONE_NUMBER>49376] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Wed Sep 21 16:12:22.592326 2022] [http2:info] [pid 26351:tid<PHONE_NUMBER>49376] h2_workers: created with min=25 max=37 idle_ms=600000
[Wed Sep 21 16:12:23.448280 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>99168] h2_c1.c(218): [client <IP_ADDRESS>:39865] h2_h2, process_conn
[Wed Sep 21 16:12:23.448344 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>99168] h2_c1.c(226): [client <IP_ADDRESS>:39865] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:12:23.448371 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>99168] h2_c1.c(282): [client <IP_ADDRESS>:39865] h2_h2, declined
[Wed Sep 21 16:12:23.594873 2022] [http2:info] [pid 26438:tid<PHONE_NUMBER>49376] h2_workers: created with min=25 max=37 idle_ms=600000
[Wed Sep 21 16:12:25.171200 2022] [http2:debug] [pid 25034:tid<PHONE_NUMBER>49376] h2_workers.c(370): h2_workers: cleanup 25 workers (25 idle)
[Wed Sep 21 16:12:25.171708 2022] [http2:debug] [pid 25034:tid<PHONE_NUMBER>49376] h2_workers.c(406): h2_workers: cleanup all workers terminated
[Wed Sep 21 16:12:25.172210 2022] [http2:debug] [pid 25034:tid<PHONE_NUMBER>49376] h2_workers.c(411): h2_workers: cleanup zombie workers joined
[Wed Sep 21 16:12:33.463932 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>25760] h2_c1.c(218): [client <IP_ADDRESS>:12296] h2_h2, process_conn
[Wed Sep 21 16:12:33.463996 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>25760] h2_c1.c(226): [client <IP_ADDRESS>:12296] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:12:33.464018 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>25760] h2_c1.c(282): [client <IP_ADDRESS>:12296] h2_h2, declined
[Wed Sep 21 16:12:36.796691 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>91872] h2_c1.c(218): [client <IP_ADDRESS>:55726] h2_h2, process_conn
[Wed Sep 21 16:12:36.796740 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>91872] h2_c1.c(226): [client <IP_ADDRESS>:55726] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:12:36.796759 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>91872] h2_c1.c(282): [client <IP_ADDRESS>:55726] h2_h2, declined
[Wed Sep 21 16:12:43.466684 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>18464] h2_c1.c(218): [client <IP_ADDRESS>:16043] h2_h2, process_conn
[Wed Sep 21 16:12:43.466738 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>18464] h2_c1.c(226): [client <IP_ADDRESS>:16043] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:12:43.467413 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>18464] h2_c1.c(282): [client <IP_ADDRESS>:16043] h2_h2, declined
[Wed Sep 21 16:12:53.482230 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>50240] h2_c1.c(218): [client <IP_ADDRESS>:39973] h2_h2, process_conn
[Wed Sep 21 16:12:53.482277 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>50240] h2_c1.c(226): [client <IP_ADDRESS>:39973] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:12:53.482290 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>50240] h2_c1.c(282): [client <IP_ADDRESS>:39973] h2_h2, declined
[Wed Sep 21 16:13:00.838746 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>62240] h2_c1.c(218): [client <IP_ADDRESS>:56704] h2_h2, process_conn
[Wed Sep 21 16:13:00.838798 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>62240] h2_c1.c(226): [client <IP_ADDRESS>:56704] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:13:00.838821 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>62240] h2_c1.c(282): [client <IP_ADDRESS>:56704] h2_h2, declined
[Wed Sep 21 16:13:03.492551 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>91872] h2_c1.c(218): [client <IP_ADDRESS>:31642] h2_h2, process_conn
[Wed Sep 21 16:13:03.492615 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>91872] h2_c1.c(226): [client <IP_ADDRESS>:31642] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:13:03.492640 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>91872] h2_c1.c(282): [client <IP_ADDRESS>:31642] h2_h2, declined
[Wed Sep 21 16:13:13.506889 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>23200] h2_c1.c(218): [client <IP_ADDRESS>:24285] h2_h2, process_conn
[Wed Sep 21 16:13:13.506939 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>23200] h2_c1.c(226): [client <IP_ADDRESS>:24285] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:13:13.506957 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>23200] h2_c1.c(282): [client <IP_ADDRESS>:24285] h2_h2, declined
[Wed Sep 21 16:13:23.517390 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>59680] h2_c1.c(218): [client <IP_ADDRESS>:31236] h2_h2, process_conn
[Wed Sep 21 16:13:23.517450 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>59680] h2_c1.c(226): [client <IP_ADDRESS>:31236] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:13:23.517469 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>59680] h2_c1.c(282): [client <IP_ADDRESS>:31236] h2_h2, declined
[Wed Sep 21 16:13:33.515813 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>99168] h2_c1.c(218): [client <IP_ADDRESS>:18757] h2_h2, process_conn

[Wed Sep 21 16:14:03.559682 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>86720] h2_c1.c(282): [client <IP_ADDRESS>:6495] h2_h2, declined
[Wed Sep 21 16:14:13.577250 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>54944] h2_c1.c(218): [client <IP_ADDRESS>:4612] h2_h2, process_conn
[Wed Sep 21 16:14:13.577292 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>54944] h2_c1.c(226): [client <IP_ADDRESS>:4612] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:14:13.577310 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>54944] h2_c1.c(282): [client <IP_ADDRESS>:4612] h2_h2, declined
[Wed Sep 21 16:14:20.904512 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>76832] h2_c1.c(218): [client <IP_ADDRESS>:50797] h2_h2, process_conn
[Wed Sep 21 16:14:20.904551 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>76832] h2_c1.c(226): [client <IP_ADDRESS>:50797] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:14:20.904567 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>76832] h2_c1.c(282): [client <IP_ADDRESS>:50797] h2_h2, declined
[Wed Sep 21 16:14:23.578158 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>91424] h2_c1.c(218): [client <IP_ADDRESS>:47543] h2_h2, process_conn
[Wed Sep 21 16:14:23.578213 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>91424] h2_c1.c(226): [client <IP_ADDRESS>:47543] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:14:23.578243 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>91424] h2_c1.c(282): [client <IP_ADDRESS>:47543] h2_h2, declined
[Wed Sep 21 16:14:31.399700 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>27904] h2_c1.c(218): [client <IP_ADDRESS>:9498] h2_h2, process_conn
[Wed Sep 21 16:14:31.399763 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>27904] h2_c1.c(226): [client <IP_ADDRESS>:9498] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:14:31.399791 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>27904] h2_c1.c(282): [client <IP_ADDRESS>:9498] h2_h2, declined
[Wed Sep 21 16:14:31.834425 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>06464] h2_c1.c(218): [client <IP_ADDRESS>:19271] h2_h2, process_conn
[Wed Sep 21 16:14:31.834472 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>06464] h2_c1.c(226): [client <IP_ADDRESS>:19271] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:14:31.834488 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>06464] h2_c1.c(282): [client <IP_ADDRESS>:19271] h2_h2, declined
[Wed Sep 21 16:14:33.592091 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>35200] h2_c1.c(218): [client <IP_ADDRESS>:12914] h2_h2, process_conn
[Wed Sep 21 16:14:33.592147 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>35200] h2_c1.c(226): [client <IP_ADDRESS>:12914] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:14:33.592167 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>35200] h2_c1.c(282): [client <IP_ADDRESS>:12914] h2_h2, declined
[Wed Sep 21 16:14:36.961079 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>13760] h2_c1.c(218): [client <IP_ADDRESS>:42636] h2_h2, process_conn
[Wed Sep 21 16:14:36.961140 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>13760] h2_c1.c(226): [client <IP_ADDRESS>:42636] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:14:36.961168 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>13760] h2_c1.c(282): [client <IP_ADDRESS>:42636] h2_h2, declined
[Wed Sep 21 16:14:41.879892 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>42944] h2_c1.c(218): [client <IP_ADDRESS>:38879] h2_h2, process_conn
[Wed Sep 21 16:14:41.879944 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>42944] h2_c1.c(226): [client <IP_ADDRESS>:38879] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:14:41.879961 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>42944] h2_c1.c(282): [client <IP_ADDRESS>:38879] h2_h2, declined
[Wed Sep 21 16:14:43.588767 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>64384] h2_c1.c(218): [client <IP_ADDRESS>:20986] h2_h2, process_conn
[Wed Sep 21 16:14:43.588815 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>64384] h2_c1.c(226): [client <IP_ADDRESS>:20986] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:14:43.588831 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>64384] h2_c1.c(282): [client <IP_ADDRESS>:20986] h2_h2, declined
[Wed Sep 21 16:14:52.050764 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>78976] h2_c1.c(218): [client <IP_ADDRESS>:50798] h2_h2, process_conn
[Wed Sep 21 16:14:52.050810 2022] [http2:trace1] [pid 26438:tid<PHONE_NUMBER>78976] h2_c1.c(226): [client <IP_ADDRESS>:50798] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:14:53.602079 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>57536] h2_c1.c(218): [client <IP_ADDRESS>:25713] h2_h2, process_conn
[Wed Sep 21 16:14:53.602169 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>57536] h2_c1.c(226): [client <IP_ADDRESS>:25713] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:14:53.602209 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>57536] h2_c1.c(282): [client <IP_ADDRESS>:25713] h2_h2, declined
[Wed Sep 21 16:15:03.672347 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>30496] h2_c1.c(218): [client <IP_ADDRESS>:24106] h2_h2, process_conn
[Wed Sep 21 16:15:03.672415 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>30496] h2_c1.c(226): [client <IP_ADDRESS>:24106] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1
[Wed Sep 21 16:15:03.672437 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>30496] h2_c1.c(282): [client <IP_ADDRESS>:24106] h2_h2, declined
[Wed Sep 21 16:15:07.065872 2022] [http2:debug] [pid 26438:tid<PHONE_NUMBER>78976] h2_c1.c(246): (70007)The timeout specified has expired: [client <IP_ADDRESS>:50798] AH03054: h2_h2, error reading 24 bytes speculative
[Wed Sep 21 16:15:12.537993 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>15904] h2_c1.c(218): [client <IP_ADDRESS>:5330] h2_h2, process_conn
[Wed Sep 21 16:15:12.538054 2022] [http2:trace1] [pid 26351:tid<PHONE_NUMBER>15904] h2_c1.c(226): [client <IP_ADDRESS>:5330] h2_h2, process_conn, new connection using protocol 'http/1.1', direct=1, tls acceptable=1

Are the
h2_h2, declined
Ok?

yes give me a minute

Ok you shoud have it

Yes, thank you. Looking at it now.

What I see in the log:

there is no HTTP/2 connection made. mod_http2 inspects the incoming connection if it carries the magic bytes for a direct connection. This is not the case and declined is logged. No http2 session, negotiated via ALPN, is created.
On one occasion, this 24 bytes detected read times out. That could be a client opening the connection but not sending a request. Or it could be some interaction with the mod_security you have loaded (that one messes in several places with things).

One thing I recommend to try is configure H2Direct off to disable this sniffing by mod_http2 in case it interferes with mod_security.
If that has no effect, I'd recommend to make at least a try without mod_security loaded. Since mod_security is not from the Apache httpd project, I do not know how closely they followed our development. That they no longer sell it and have End-Of-Life announced makes me wonder. At least to see if it is in any way involved in the problems you see, I'd give this a try.
Any "real" http2 connections will appear in your logs as
h2_session.c(962): [client xxx] AH03200: h2_session(2-0,INIT,0): created, max_streams=100, stream_mem=32768, workers_limit=6, workers_max=37, push_diary(type=1,N=256)
...

If no such line is in your logs, no HTTP/2 is in play. With H2Direct off mod_http2 should then not interfere with any connctions (the  h2_h2, process_connand h2_h2, declined will still appear, but they do not really do anything).

Thank you @icing
I've started a new conf with better logging and the changes you suggested
I'll let you know

running 2.0.5 in debug mode now

Here I am with some info.
As I start httpd with version 2.0.5 the CPU load rises to 100%, look at this stats from mod_status
Server Version: Apache/2.4.54 (codeit) OpenSSL/1.1.1q+quic mod_fcgid/2.3.9
Server MPM: worker
Server Built: Jun 8 2022 09:34:58

Current Time: Thursday, 22-Sep-2022 09:15:24 CEST
Restart Time: Thursday, 22-Sep-2022 09:08:55 CEST
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 6 minutes 28 seconds
Server load: 5.16 6.63 6.25
Total accesses: 370 - Total Traffic: 3.6 MB - Total Duration: 945830
CPU Usage: u233.55 s675.72 cu0 cs0 - 234% CPU load
.954 requests/sec - 9.4 kB/second - 9.8 kB/request - 2556.3 ms/request
7 requests currently being processed, 43 idle workers

______LL__W____K____________________L________K___W..............
................................................................
......................

as you see 234% CPU load
Here is the same traffic with 2.0.3 loaded:
Server Version: Apache/2.4.54 (codeit) OpenSSL/1.1.1q+quic mod_fcgid/2.3.9
Server MPM: worker
Server Built: Jun 8 2022 09:34:58

Current Time: Thursday, 22-Sep-2022 09:25:37 CEST
Restart Time: Thursday, 22-Sep-2022 09:19:55 CEST
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 5 minutes 42 seconds
Server load: 1.29 2.39 4.36
Total accesses: 456 - Total Traffic: 8.1 MB - Total Duration: 224842
CPU Usage: u7.17 s1.15 cu0 cs0 - 2.43% CPU load
1.33 requests/sec - 24.3 kB/second - 18.2 kB/request - 493.075 ms/request
9 requests currently being processed, 41 idle workers

W__WL_____L_________K__L_____L_________K___R______..............
................................................................
......................

Here we have 2.43% load
I'm digging into the 10GB logs to extract something relevant
Stay tuned

sent you some logs by email

Hi !
FYI, same issue on Windows with mod_md 2.0.5 with nghttp2 1.48 (as nghttp2 1.49 and still 1.50 failed with curl when upgrading to h2)

@nono303  now we only need to find out what the issue is!🙈
But thanks for chiming in. Are you able to reproduce the issue at a http2:trace2 log level? Will be quite verbose...

Updated my log analyzer to v2.0.5 output. See h2wtf. Running that on the test.log that @alexskynet sent me, I see two connections where the request times out, for example:
id               created                          scheduled           started            response            ended              reset             cleanup         error
303-25-0         2022-09-22 09:17:01.488022           --                 --                      --            --                 --              +19.080493 70007 +19.076375
303-25-15        2022-09-22 09:17:01.511985           --                 --           200 +0.350654          +0.352962            --                 --           --
303-25-17        2022-09-22 09:17:02.012644           --                 --           304 +0.003347          +0.003376            --                 --           --
303-25-19        2022-09-22 09:17:02.130160           --                 --           304 +0.006242          +0.006319            --                 --           --
303-25-21        2022-09-22 09:17:05.028766           --                 --           302 +0.266084          +0.266340            --                 --           --
303-25-23        2022-09-22 09:17:05.420298           --                 --           302 +0.025555          +0.025604            --                 --           --
303-25-25        2022-09-22 09:17:05.541844           --                 --                      --            --                 --                 --           --

303-25-0 is the overall connection (always -0) and the others are the opened requests on that connection. 6 request are received quite rapidly and all but the last produce a response.
The requested URLs would only be visible on a higher log level (or I have to make a version that logs them on debug also). That would be interesting. I suspect that this request ends up via mod_fcgid in the php application.

@icing you're right
mod_fcgid is used for all our applications
At the moment I'm running a new test:
I recompiled 2.0.5 against  libnghttp2-devel-1.50.0-1.codeit.el7.x86_64 which has been updated this morning

Scanning my code and the log some more: What is the configuration for Timeout that you have configured? If may be that the new mod_http2 applies that in situations where it was not done before, e.g. waiting for a response should not really apply this.

That would match with the logged connection timeout. I'll do a v2.0.6 with a fix this evening.
You could try increasing that value in your config in the meantime to see if it makes a difference.

Ok

set Timeout to 30

pstack of httpd process at 100% running 2.0.5 timout30
pstack 3560
Thread 27 (Thread 0x7fe8649ee700 (LWP 3576)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec56ead90, wctx=0x558ec56e8f98) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fe8641ed700 (LWP 3577)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec56ef090, wctx=0x558ec56e8ff0) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fe8639ec700 (LWP 3578)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec56f3390, wctx=0x558ec56e9048) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fe8631eb700 (LWP 3581)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec56f7690, wctx=0x558ec56e90a0) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fe8629ea700 (LWP 3582)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec56fb990, wctx=0x558ec56e90f8) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fe8621e9700 (LWP 3585)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec56ffc90, wctx=0x558ec56e9150) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fe8619e8700 (LWP 3588)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec5703f90, wctx=0x558ec56e91a8) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fe8611e7700 (LWP 3589)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec5708290, wctx=0x558ec56e9200) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fe8609e6700 (LWP 3590)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec570c590, wctx=0x558ec56e9258) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fe8601e5700 (LWP 3591)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec5710890, wctx=0x558ec56e92b0) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fe85f9e4700 (LWP 3592)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec5714b90, wctx=0x558ec56e9308) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fe85f1e3700 (LWP 3595)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec5718e90, wctx=0x558ec56e9360) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fe85e9e2700 (LWP 3596)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec571d190, wctx=0x558ec56e93b8) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fe85e1e1700 (LWP 3599)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec5721490, wctx=0x558ec56e9410) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fe85d9e0700 (LWP 3600)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec5725790, wctx=0x558ec56e9468) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fe85d1df700 (LWP 3603)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec5729a90, wctx=0x558ec56e94c0) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fe85c9de700 (LWP 3604)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec572dd90, wctx=0x558ec56e9518) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fe85c1dd700 (LWP 3608)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec5732090, wctx=0x558ec56e9570) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fe85b9dc700 (LWP 3611)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec5736390, wctx=0x558ec56e95c8) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fe85b1db700 (LWP 3612)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec573a690, wctx=0x558ec56e9620) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fe85a9da700 (LWP 3615)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec573e990, wctx=0x558ec56e9678) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fe85a1d9700 (LWP 3618)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec5742c90, wctx=0x558ec56e96d0) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fe8599d8700 (LWP 3619)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec5746f90, wctx=0x558ec56e9728) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fe8591d7700 (LWP 3620)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec574b290, wctx=0x558ec56e9780) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fe8589d6700 (LWP 3621)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in slot_run (thread=0x558ec574f590, wctx=0x558ec56e97d8) at h2_workers.c:329
#2  0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fe84cff9700 (LWP 3668)):
#0  0x00007fe86ff431d4 in h2_brigade_mem_size (bb=0x7fe840002db8) at h2_util.c:1448
#1  0x00007fe86ff3b3a4 in buffer_output_receive (stream=stream@entry=0x7fe840002970) at h2_stream.c:925
#2  0x00007fe86ff3d96c in stream_data_cb (ng2s=, stream_id=1, buf=, length=16375, data_flags=0x7fe84cff89e4, source=, puser=0x7fe83802fd40) at h2_stream.c:1416
#3  0x00007fe87fd6a1a1 in nghttp2_session_pack_data () from /lib64/libnghttp2.so.14
#4  0x00007fe87fd6af0d in nghttp2_session_mem_send_internal () from /lib64/libnghttp2.so.14
#5  0x00007fe87fd6bb19 in nghttp2_session_send () from /lib64/libnghttp2.so.14
#6  0x00007fe86ff39afc in h2_session_send (session=session@entry=0x7fe83802fd40) at h2_session.c:1270
#7  0x00007fe86ff39d6c in h2_session_process (session=0x7fe83802fd40, async=) at h2_session.c:1815
#8  0x00007fe86ff24bc9 in h2_c1_run (c=c@entry=0x7fe850005798) at h2_c1.c:130
#9  0x00007fe86ff24fdd in h2_c1_hook_process_connection (c=0x7fe850005798) at h2_c1.c:278
#10 0x0000558ec27273c0 in ap_run_process_connection ()
#11 0x00007fe872f29d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so
#12 0x00007fe87ea85ea5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007fe87e5aab0d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fe87fe328c0 (LWP 3560)):
#0  0x00007fe87ea87017 in pthread_join () from /lib64/libpthread.so.0
#1  0x00007fe87ecc804b in apr_thread_join () from /lib64/libapr-1.so.0
#2  0x00007fe872f286f3 in join_workers () from /etc/httpd/modules/mod_mpm_worker.so
#3  0x00007fe872f2b061 in child_main () from /etc/httpd/modules/mod_mpm_worker.so
#4  0x00007fe872f2b48e in make_child () from /etc/httpd/modules/mod_mpm_worker.so
#5  0x00007fe872f2b534 in startup_children () from /etc/httpd/modules/mod_mpm_worker.so
#6  0x00007fe872f2c0a1 in worker_run () from /etc/httpd/modules/mod_mpm_worker.so
#7  0x0000558ec272a1ce in ap_run_mpm ()
#8  0x0000558ec271388d in main ()

Another 100%
hread 1 (process 3668):
#0  0x00007fe86ff431b4 in h2_brigade_concat_length () from /etc/httpd/modules/mod_http2.so
#1  0x00007fe8380304a0 in ?? ()
#2  0x00007fe87f342ef9 in heap_bucket_destroy () from /lib64/libaprutil-1.so.0
#3  0x00007fe86ff26a62 in h2_c1_io_init () from /etc/httpd/modules/mod_http2.so
#4  0x0000000000000001 in ?? ()
#5  0x0000000000000000 in ?? ()

And another one:
pstack 3560
Thread 27 (Thread 0x7fe8649ee700 (LWP 3576)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 26 (Thread 0x7fe8641ed700 (LWP 3577)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 25 (Thread 0x7fe8639ec700 (LWP 3578)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 24 (Thread 0x7fe8631eb700 (LWP 3581)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 23 (Thread 0x7fe8629ea700 (LWP 3582)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 22 (Thread 0x7fe8621e9700 (LWP 3585)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 21 (Thread 0x7fe8619e8700 (LWP 3588)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 20 (Thread 0x7fe8611e7700 (LWP 3589)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 19 (Thread 0x7fe8609e6700 (LWP 3590)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 18 (Thread 0x7fe8601e5700 (LWP 3591)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 17 (Thread 0x7fe85f9e4700 (LWP 3592)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 16 (Thread 0x7fe85f1e3700 (LWP 3595)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 15 (Thread 0x7fe85e9e2700 (LWP 3596)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 14 (Thread 0x7fe85e1e1700 (LWP 3599)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 13 (Thread 0x7fe85d9e0700 (LWP 3600)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 12 (Thread 0x7fe85d1df700 (LWP 3603)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 11 (Thread 0x7fe85c9de700 (LWP 3604)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 10 (Thread 0x7fe85c1dd700 (LWP 3608)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 9 (Thread 0x7fe85b9dc700 (LWP 3611)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 8 (Thread 0x7fe85b1db700 (LWP 3612)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 7 (Thread 0x7fe85a9da700 (LWP 3615)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7fe85a1d9700 (LWP 3618)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7fe8599d8700 (LWP 3619)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7fe8591d7700 (LWP 3620)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7fe8589d6700 (LWP 3621)):
#0  0x00007fe87ea89a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe86ff445c2 in activate_slot () from /etc/httpd/modules/mod_http2.so
#2  0x0000558ec56e8df0 in ?? ()
#3  0x0000558ec56e8dc0 in ?? ()
#4  0x0000558ec56e8dd0 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7fe84cff9700 (LWP 3668)):
#0  0x00007fe86ff431bb in h2_brigade_concat_length () from /etc/httpd/modules/mod_http2.so
#1  0x00007fe8380304a0 in ?? ()
#2  0x00007fe87f342ef9 in heap_bucket_destroy () from /lib64/libaprutil-1.so.0
#3  0x00007fe86ff26a62 in h2_c1_io_init () from /etc/httpd/modules/mod_http2.so
#4  0x0000000000000001 in ?? ()
#5  0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7fe87fe328c0 (LWP 3560)):
#0  0x00007fe87ea87017 in pthread_join () from /lib64/libpthread.so.0
#1  0x00007fe87ecc804b in apr_thread_join () from /lib64/libapr-1.so.0
#2  0x00007fe872f286f3 in join_workers () from /etc/httpd/modules/mod_mpm_worker.so
#3  0x00007fe872f2b061 in child_main () from /etc/httpd/modules/mod_mpm_worker.so
#4  0x00007fe872f2b48e in make_child () from /etc/httpd/modules/mod_mpm_worker.so
#5  0x00007fe872f2b534 in startup_children () from /etc/httpd/modules/mod_mpm_worker.so
#6  0x00007fe872f2c0a1 in worker_run () from /etc/httpd/modules/mod_mpm_worker.so
#7  0x0000558ec272a1ce in ap_run_mpm ()
#8  0x0000558ec271388d in main ()

Thanks for the traces. The first one really helped. The other seem to have the symbols/source mixed up.
Anyways, here is v2.0.6 with three fixes. Hope these work for you as well.

Thank you Stefan
Just compiled and installed on one server
I'll let you know soon
Best regards
Il 22 settembre 2022 20:23:49 CEST, Stefan Eissing @.***> ha scritto:

Thanks for the traces. The first one really helped. The other seem to have the symbols/source mixed up.
Anyways, here is v2.0.6 with three fixes. Hope these work for you as well.
--
Reply to this email directly or view it on GitHub:
https://github.com/icing/mod_h2/issues/234#issuecomment-1255392015
You are receiving this because you were mentioned.
Message ID: @.***>
SKNT Group SRLU
Via Maggiate 67/a
28021 Borgomanero (NO)
tel. +39 0322-836487/834765
fax +39 0322-836608
http://sknt.it

Tested 2.0.6 for about 20 minutes
Load on server greatly increases
It seems httpd processes still hang, but I will investigate better tomorrow morning since I'm operating from a tablet now.
I saw your Apache commit proposal: so it comes from Apache if I uderstand.
Thank you for you time
Best regards
Alessandro
Il 22 settembre 2022 20:23:49 CEST, Stefan Eissing @.***> ha scritto:

Thanks for the traces. The first one really helped. The other seem to have the symbols/source mixed up.
Anyways, here is v2.0.6 with three fixes. Hope these work for you as well.
--
Reply to this email directly or view it on GitHub:
https://github.com/icing/mod_h2/issues/234#issuecomment-1255392015
You are receiving this because you were mentioned.
Message ID: @.***>
SKNT Group SRLU
Via Maggiate 67/a
28021 Borgomanero (NO)
tel. +39 0322-836487/834765
fax +39 0322-836608
http://sknt.it

Here I am
Unfortunately as soon as I start httpd with 2.0.6 all httpd processes go to 100% CPU.
Here are a couple of pstack(s)
pstack 3491 Thread 75 (Thread 0x7ff9fb98f700 (LWP 3496)): #0  0x00007ffa14d41b43 in select () from /lib64/libc.so.6 #1  0x00007ffa15468725 in apr_sleep () from /lib64/libapr-1.so.0 #2  0x00007ffa0a351066 in wd_worker () from /etc/httpd/modules/mod_watchdog.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 74 (Thread 0x7ff9fb18e700 (LWP 3498)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 73 (Thread 0x7ff9fa98d700 (LWP 3499)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 72 (Thread 0x7ff9fa18c700 (LWP 3502)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 71 (Thread 0x7ff9f998b700 (LWP 3503)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 70 (Thread 0x7ff9f918a700 (LWP 3505)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 69 (Thread 0x7ff9f8989700 (LWP 3508)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 68 (Thread 0x7ff9f3fff700 (LWP 3511)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 67 (Thread 0x7ff9f37fe700 (LWP 3513)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 66 (Thread 0x7ff9f2ffd700 (LWP 3517)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 65 (Thread 0x7ff9f27fc700 (LWP 3516)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068858e0, wctx=0x55ce06883ae8) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 64 (Thread 0x7ff9f17fa700 (LWP 3520)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 63 (Thread 0x7ff9f1ffb700 (LWP 3521)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce06889be0, wctx=0x55ce06883b40) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 62 (Thread 0x7ff9f0ff9700 (LWP 3524)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 61 (Thread 0x7ff9f07f8700 (LWP 3526)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce0688dee0, wctx=0x55ce06883b98) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 60 (Thread 0x7ff9efff7700 (LWP 3527)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 59 (Thread 0x7ff9ef7f6700 (LWP 3531)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068921e0, wctx=0x55ce06883bf0) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 58 (Thread 0x7ff9eeff5700 (LWP 3530)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 57 (Thread 0x7ff9ee7f4700 (LWP 3533)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 56 (Thread 0x7ff9edff3700 (LWP 3535)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068964e0, wctx=0x55ce06883c48) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 55 (Thread 0x7ff9ed7f2700 (LWP 3536)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 54 (Thread 0x7ff9ecff1700 (LWP 3539)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa15af3f43 in thread_pool_func () from /lib64/libaprutil-1.so.0 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 53 (Thread 0x7ff9ec7f0700 (LWP 3538)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce0689a7e0, wctx=0x55ce06883ca0) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 52 (Thread 0x7ff9ebfef700 (LWP 3542)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce0689eae0, wctx=0x55ce06883cf8) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 51 (Thread 0x7ff9eb7ee700 (LWP 3544)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068a2de0, wctx=0x55ce06883d50) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 50 (Thread 0x7ff9eafed700 (LWP 3546)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068a70e0, wctx=0x55ce06883da8) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 49 (Thread 0x7ff9ea7ec700 (LWP 3548)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068ab3e0, wctx=0x55ce06883e00) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 48 (Thread 0x7ff9e9feb700 (LWP 3549)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068af6e0, wctx=0x55ce06883e58) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 47 (Thread 0x7ff9e97ea700 (LWP 3550)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068b39e0, wctx=0x55ce06883eb0) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 46 (Thread 0x7ff9e8fe9700 (LWP 3551)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068b7ce0, wctx=0x55ce06883f08) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 45 (Thread 0x7ff9e87e8700 (LWP 3552)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068bbfe0, wctx=0x55ce06883f60) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 44 (Thread 0x7ff9e7fe7700 (LWP 3553)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068c02e0, wctx=0x55ce06883fb8) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7ff9e77e6700 (LWP 3554)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068c45e0, wctx=0x55ce06884010) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7ff9e6fe5700 (LWP 3555)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068c88e0, wctx=0x55ce06884068) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7ff9e67e4700 (LWP 3556)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068ccbe0, wctx=0x55ce068840c0) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7ff9e5fe3700 (LWP 3557)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068d0ee0, wctx=0x55ce06884118) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7ff9e57e2700 (LWP 3558)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068d51e0, wctx=0x55ce06884170) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7ff9e4fe1700 (LWP 3559)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068d94e0, wctx=0x55ce068841c8) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7ff9e47e0700 (LWP 3560)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068dd7e0, wctx=0x55ce06884220) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7ff9e3fdf700 (LWP 3561)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068e1ae0, wctx=0x55ce06884278) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7ff9e37de700 (LWP 3562)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068e73a0, wctx=0x55ce068842d0) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7ff9e2fdd700 (LWP 3563)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068eb3c0, wctx=0x55ce06884328) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7ff9e1fdb700 (LWP 3571)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7ff9e17da700 (LWP 3573)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7ff9e0fd9700 (LWP 3575)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7ff9dbfff700 (LWP 3577)): #0  h2_session_process (session=0x7ff9a4016ba0, async=<optimized out>) at h2_session.c:1814 #1  0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9dc00ba38) at h2_c1.c:130 #2  0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9dc00ba38) at h2_c1.c:278 #3  0x000055ce02ab13c0 in ap_run_process_connection () #4  0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #5  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #6  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7ff9db7fe700 (LWP 3579)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7ff9daffd700 (LWP 3581)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7ff9da7fc700 (LWP 3583)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7ff9d9ffb700 (LWP 3585)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7ff9d97fa700 (LWP 3587)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7ff9d8ff9700 (LWP 3589)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7ff9d87f8700 (LWP 3590)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7ff9d7ff7700 (LWP 3592)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7ff9d77f6700 (LWP 3595)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7ff9d6ff5700 (LWP 3597)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7ff9d67f4700 (LWP 3599)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7ff9d5ff3700 (LWP 3601)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7ff9d57f2700 (LWP 3603)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7ff9d4ff1700 (LWP 3605)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7ff9d47f0700 (LWP 3607)): #0  0x00007ffa1522c75d in read () from /lib64/libpthread.so.0 #1  0x00007ffa1545fd4b in apr_socket_recv () from /lib64/libapr-1.so.0 #2  0x00007ffa15ae3ae4 in socket_bucket_read () from /lib64/libaprutil-1.so.0 #3  0x000055ce02ad2e71 in ap_core_input_filter () #4  0x00007ffa0d3cd04e in logio_in_filter () from /etc/httpd/modules/mod_logio.so #5  0x00007ffa07210a9a in bio_filter_in_read () from /etc/httpd/modules/mod_ssl.so #6  0x00007ffa06fc95ea in bread_conv () from /lib64/libcrypto.so.81.1.1 #7  0x00007ffa06fc82c6 in BIO_read () from /lib64/libcrypto.so.81.1.1 #8  0x00007ffa1655ee21 in ssl3_read_n () from /lib64/libssl.so.81.1.1 #9  0x00007ffa16563622 in ssl3_get_record () from /lib64/libssl.so.81.1.1 #10 0x00007ffa16560e24 in ssl3_read_bytes () from /lib64/libssl.so.81.1.1 #11 0x00007ffa16568b60 in ssl3_read () from /lib64/libssl.so.81.1.1 #12 0x00007ffa16575665 in SSL_read () from /lib64/libssl.so.81.1.1 #13 0x00007ffa07211f73 in ssl_io_input_read () from /etc/httpd/modules/mod_ssl.so #14 0x00007ffa07214e2c in ssl_io_filter_input () from /etc/httpd/modules/mod_ssl.so #15 0x00007ffa066c594a in read_and_feed (session=session@entry=0x7ff9cc084ee0) at h2_c1_io.c:486 #16 0x00007ffa066c6ecf in h2_c1_read (session=session@entry=0x7ff9cc084ee0) at h2_c1_io.c:513 #17 0x00007ffa066da08b in h2_session_process (session=0x7ff9cc084ee0, async=<optimized out>) at h2_session.c:1865 #18 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9dc005798) at h2_c1.c:130 #19 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9dc005798) at h2_c1.c:278 #20 0x000055ce02ab13c0 in ap_run_process_connection () #21 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #22 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7ff9d3fef700 (LWP 3609)): #0  0x00007ffa14d3fddd in poll () from /lib64/libc.so.6 #1  0x00007ffa069061ad in proc_read_ipc () from /etc/httpd/modules/mod_fcgid.so #2  0x00007ffa069069b6 in fcgid_feed_data () from /etc/httpd/modules/mod_fcgid.so #3  0x00007ffa06906b15 in fcgid_header_bucket_read () from /etc/httpd/modules/mod_fcgid.so #4  0x00007ffa068fe743 in getsfunc_fcgid_BRIGADE () from /etc/httpd/modules/mod_fcgid.so #5  0x000055ce02aaf558 in ap_scan_script_header_err_core_ex () #6  0x00007ffa068ff208 in handle_request () from /etc/httpd/modules/mod_fcgid.so #7  0x00007ffa068ff527 in bridge_request () from /etc/httpd/modules/mod_fcgid.so #8  0x00007ffa068fdcd1 in fcgid_handler () from /etc/httpd/modules/mod_fcgid.so #9  0x000055ce02a9f550 in ap_run_handler () #10 0x000055ce02a9fa99 in ap_invoke_handler () #11 0x000055ce02add40c in ap_internal_redirect () #12 0x00007ffa0c3961ec in handler_redirect () from /etc/httpd/modules/mod_rewrite.so #13 0x000055ce02a9f550 in ap_run_handler () #14 0x000055ce02a9fa99 in ap_invoke_handler () #15 0x000055ce02ade0ea in ap_process_async_request () #16 0x000055ce02ade3ce in ap_process_request () #17 0x000055ce02ada304 in ap_process_http_connection () #18 0x000055ce02ab13c0 in ap_run_process_connection () #19 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #20 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7ff9d37ee700 (LWP 3610)): #0  0x000055ce02ab42d7 in ap_run_mpm_query () #1  0x000055ce02ab52be in ap_mpm_query () #2  0x00007ffa066c4c77 in h2_c1_run (c=c@entry=0x7ff9dc007878) at h2_c1.c:142 #3  0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9dc007878) at h2_c1.c:278 #4  0x000055ce02ab13c0 in ap_run_process_connection () #5  0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #6  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #7  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7ff9d2fed700 (LWP 3613)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7ff9d27ec700 (LWP 3614)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7ff9d1feb700 (LWP 3615)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7ff9d17ea700 (LWP 3616)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7ff9d0fe9700 (LWP 3617)): #0  0x00007ffa14d4b0e3 in epoll_wait () from /lib64/libc.so.6 #1  0x00007ffa15463329 in impl_pollset_poll () from /lib64/libapr-1.so.0 #2  0x00007ffa096ca708 in listener_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7ff9cbfff700 (LWP 3629)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #2  0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7ff9cb7fe700 (LWP 3643)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #2  0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7ff9caffd700 (LWP 3644)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #2  0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7ff9ca7fc700 (LWP 3645)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #2  0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7ff9c9ffb700 (LWP 3646)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #2  0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7ff9c97fa700 (LWP 3707)): #0  0x00007ffa15229de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa019d8253 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #2  0x00007ffa019d0337 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ffa165d28c0 (LWP 3491)): #0  0x00007ffa1522c75d in read () from /lib64/libpthread.so.0 #1  0x000055ce02ab68e7 in ap_mpm_podx_check () #2  0x00007ffa096cb004 in child_main () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa096cb48e in make_child () from /etc/httpd/modules/mod_mpm_worker.so #4  0x00007ffa096cb534 in startup_children () from /etc/httpd/modules/mod_mpm_worker.so #5  0x00007ffa096cc0a1 in worker_run () from /etc/httpd/modules/mod_mpm_worker.so #6  0x000055ce02ab41ce in ap_run_mpm () #7  0x000055ce02a9d88d in main () [root@argo ~]# pstack 3492 Thread 59 (Thread 0x7ff9fb98f700 (LWP 3497)): #0  0x00007ffa14d41b43 in select () from /lib64/libc.so.6 #1  0x00007ffa15468725 in apr_sleep () from /lib64/libapr-1.so.0 #2  0x00007ffa0a350f02 in wd_worker () from /etc/httpd/modules/mod_watchdog.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 58 (Thread 0x7ff9fb18e700 (LWP 3500)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068858e0, wctx=0x55ce06883ae8) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 57 (Thread 0x7ff9fa98d700 (LWP 3501)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce06889be0, wctx=0x55ce06883b40) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 56 (Thread 0x7ff9fa18c700 (LWP 3504)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce0688dee0, wctx=0x55ce06883b98) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 55 (Thread 0x7ff9f998b700 (LWP 3506)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068921e0, wctx=0x55ce06883bf0) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 54 (Thread 0x7ff9f918a700 (LWP 3507)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068964e0, wctx=0x55ce06883c48) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 53 (Thread 0x7ff9f8989700 (LWP 3509)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce0689a7e0, wctx=0x55ce06883ca0) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 52 (Thread 0x7ff9f8188700 (LWP 3510)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce0689eae0, wctx=0x55ce06883cf8) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 51 (Thread 0x7ff9f7987700 (LWP 3512)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068a2de0, wctx=0x55ce06883d50) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 50 (Thread 0x7ff9f7186700 (LWP 3514)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068a70e0, wctx=0x55ce06883da8) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 49 (Thread 0x7ff9f6985700 (LWP 3515)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068ab3e0, wctx=0x55ce06883e00) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 48 (Thread 0x7ff9f6184700 (LWP 3518)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068af6e0, wctx=0x55ce06883e58) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 47 (Thread 0x7ff9f5983700 (LWP 3519)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068b39e0, wctx=0x55ce06883eb0) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 46 (Thread 0x7ff9f5182700 (LWP 3522)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068b7ce0, wctx=0x55ce06883f08) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 45 (Thread 0x7ff9f4981700 (LWP 3523)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068bbfe0, wctx=0x55ce06883f60) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 44 (Thread 0x7ff9f4180700 (LWP 3525)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068c02e0, wctx=0x55ce06883fb8) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7ff9f397f700 (LWP 3528)): #0  0x00007ffa14d3fddd in poll () from /lib64/libc.so.6 #1  0x00007ffa069061ad in proc_read_ipc () from /etc/httpd/modules/mod_fcgid.so #2  0x00007ffa069069b6 in fcgid_feed_data () from /etc/httpd/modules/mod_fcgid.so #3  0x00007ffa06906b15 in fcgid_header_bucket_read () from /etc/httpd/modules/mod_fcgid.so #4  0x00007ffa068fe743 in getsfunc_fcgid_BRIGADE () from /etc/httpd/modules/mod_fcgid.so #5  0x000055ce02aaf558 in ap_scan_script_header_err_core_ex () #6  0x00007ffa068ff208 in handle_request () from /etc/httpd/modules/mod_fcgid.so #7  0x00007ffa068ff527 in bridge_request () from /etc/httpd/modules/mod_fcgid.so #8  0x00007ffa068fdcd1 in fcgid_handler () from /etc/httpd/modules/mod_fcgid.so #9  0x000055ce02a9f550 in ap_run_handler () #10 0x000055ce02a9fa99 in ap_invoke_handler () #11 0x000055ce02ade0ea in ap_process_async_request () #12 0x000055ce02ade3ce in ap_process_request () #13 0x00007ffa066c8744 in c2_process (c=0x7ff91c0ce2a0, conn_ctx=0x7ff91c0ce840) at h2_c2.c:603 #14 h2_c2_hook_process (c=0x7ff91c0ce2a0) at h2_c2.c:633 #15 0x000055ce02ab13c0 in ap_run_process_connection () #16 0x00007ffa066c957a in h2_c2_process (c2=c2@entry=0x7ff91c0ce2a0, thread=thread@entry=0x55ce068c45e0, worker_id=<optimized out>) at h2_c2.c:545 #17 0x00007ffa066e44ae in slot_run (thread=0x55ce068c45e0, wctx=0x55ce06884010) at h2_workers.c:289 #18 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #19 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7ff9f317e700 (LWP 3529)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068c88e0, wctx=0x55ce06884068) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7ff9f297d700 (LWP 3532)): #0  0x00007ffa14d3fddd in poll () from /lib64/libc.so.6 #1  0x00007ffa069061ad in proc_read_ipc () from /etc/httpd/modules/mod_fcgid.so #2  0x00007ffa069069b6 in fcgid_feed_data () from /etc/httpd/modules/mod_fcgid.so #3  0x00007ffa06906b15 in fcgid_header_bucket_read () from /etc/httpd/modules/mod_fcgid.so #4  0x00007ffa068fe743 in getsfunc_fcgid_BRIGADE () from /etc/httpd/modules/mod_fcgid.so #5  0x000055ce02aaf558 in ap_scan_script_header_err_core_ex () #6  0x00007ffa068ff208 in handle_request () from /etc/httpd/modules/mod_fcgid.so #7  0x00007ffa068ff527 in bridge_request () from /etc/httpd/modules/mod_fcgid.so #8  0x00007ffa068fdcd1 in fcgid_handler () from /etc/httpd/modules/mod_fcgid.so #9  0x000055ce02a9f550 in ap_run_handler () #10 0x000055ce02a9fa99 in ap_invoke_handler () #11 0x000055ce02ade0ea in ap_process_async_request () #12 0x000055ce02ade3ce in ap_process_request () #13 0x00007ffa066c8744 in c2_process (c=0x7ff92804c0f0, conn_ctx=0x7ff92804c690) at h2_c2.c:603 #14 h2_c2_hook_process (c=0x7ff92804c0f0) at h2_c2.c:633 #15 0x000055ce02ab13c0 in ap_run_process_connection () #16 0x00007ffa066c957a in h2_c2_process (c2=c2@entry=0x7ff92804c0f0, thread=thread@entry=0x55ce068ccbe0, worker_id=<optimized out>) at h2_c2.c:545 #17 0x00007ffa066e44ae in slot_run (thread=0x55ce068ccbe0, wctx=0x55ce068840c0) at h2_workers.c:289 #18 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #19 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7ff9f217c700 (LWP 3534)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068d0ee0, wctx=0x55ce06884118) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7ff9f197b700 (LWP 3537)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068d51e0, wctx=0x55ce06884170) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7ff9f117a700 (LWP 3540)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068d94e0, wctx=0x55ce068841c8) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7ff9f0979700 (LWP 3541)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068dd7e0, wctx=0x55ce06884220) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7ff9f0178700 (LWP 3543)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068e1ae0, wctx=0x55ce06884278) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7ff9ef977700 (LWP 3545)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068e73a0, wctx=0x55ce068842d0) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7ff9ef176700 (LWP 3547)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa066e4682 in slot_run (thread=0x55ce068eb3c0, wctx=0x55ce06884328) at h2_workers.c:329 #2  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #3  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7ff9ee174700 (LWP 3565)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7ff9ed973700 (LWP 3566)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7ff9ed172700 (LWP 3568)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7ff9ec971700 (LWP 3569)): #0  0x00007ffa1522c75d in read () from /lib64/libpthread.so.0 #1  0x00007ffa1545fd4b in apr_socket_recv () from /lib64/libapr-1.so.0 #2  0x00007ffa15ae3ae4 in socket_bucket_read () from /lib64/libaprutil-1.so.0 #3  0x000055ce02ad2e71 in ap_core_input_filter () #4  0x00007ffa0d3cd04e in logio_in_filter () from /etc/httpd/modules/mod_logio.so #5  0x00007ffa07210a9a in bio_filter_in_read () from /etc/httpd/modules/mod_ssl.so #6  0x00007ffa06fc95ea in bread_conv () from /lib64/libcrypto.so.81.1.1 #7  0x00007ffa06fc82c6 in BIO_read () from /lib64/libcrypto.so.81.1.1 #8  0x00007ffa1655ee21 in ssl3_read_n () from /lib64/libssl.so.81.1.1 #9  0x00007ffa16563622 in ssl3_get_record () from /lib64/libssl.so.81.1.1 #10 0x00007ffa16560e24 in ssl3_read_bytes () from /lib64/libssl.so.81.1.1 #11 0x00007ffa16568b60 in ssl3_read () from /lib64/libssl.so.81.1.1 #12 0x00007ffa16575665 in SSL_read () from /lib64/libssl.so.81.1.1 #13 0x00007ffa07211f73 in ssl_io_input_read () from /etc/httpd/modules/mod_ssl.so #14 0x00007ffa07214e2c in ssl_io_filter_input () from /etc/httpd/modules/mod_ssl.so #15 0x00007ffa066c594a in read_and_feed (session=session@entry=0x7ff99c07fde0) at h2_c1_io.c:486 #16 0x00007ffa066c6ecf in h2_c1_read (session=session@entry=0x7ff99c07fde0) at h2_c1_io.c:513 #17 0x00007ffa066da08b in h2_session_process (session=0x7ff99c07fde0, async=<optimized out>) at h2_session.c:1865 #18 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9e8009958) at h2_c1.c:130 #19 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9e8009958) at h2_c1.c:278 #20 0x000055ce02ab13c0 in ap_run_process_connection () #21 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #22 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7ff9e7fff700 (LWP 3570)): #0  0x00007ffa14d4b0e3 in epoll_wait () from /lib64/libc.so.6 #1  0x00007ffa15463329 in impl_pollset_poll () from /lib64/libapr-1.so.0 #2  0x00007ffa066d1661 in mplx_pollset_poll (on_ctx=0x7ff97401b400, on_stream_output=0x7ffa066d5b50 <on_stream_output>, on_stream_input=0x7ffa066d5ca0 <on_stream_input>, timeout=30000000, m=0x7ff97401bc20) at h2_mplx.c:1126 #3  h2_mplx_c1_poll (m=0x7ff97401bc20, timeout=30000000, on_stream_input=on_stream_input@entry=0x7ffa066d5ca0 <on_stream_input>, on_stream_output=on_stream_output@entry=0x7ffa066d5b50 <on_stream_output>, on_ctx=on_ctx@entry=0x7ff97401b400) at h2_mplx.c:606 #4  0x00007ffa066da200 in h2_session_process (session=0x7ff97401b400, async=<optimized out>) at h2_session.c:1903 #5  0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9e8015e98) at h2_c1.c:130 #6  0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9e8015e98) at h2_c1.c:278 #7  0x000055ce02ab13c0 in ap_run_process_connection () #8  0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #9  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7ff9e77fe700 (LWP 3572)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7ff9e6ffd700 (LWP 3574)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7ff9e67fc700 (LWP 3576)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7ff9e5ffb700 (LWP 3578)): #0  0x00007ffa14d3fddd in poll () from /lib64/libc.so.6 #1  0x00007ffa069061ad in proc_read_ipc () from /etc/httpd/modules/mod_fcgid.so #2  0x00007ffa069069b6 in fcgid_feed_data () from /etc/httpd/modules/mod_fcgid.so #3  0x00007ffa06906b15 in fcgid_header_bucket_read () from /etc/httpd/modules/mod_fcgid.so #4  0x00007ffa068fe743 in getsfunc_fcgid_BRIGADE () from /etc/httpd/modules/mod_fcgid.so #5  0x000055ce02aaf558 in ap_scan_script_header_err_core_ex () #6  0x00007ffa068ff208 in handle_request () from /etc/httpd/modules/mod_fcgid.so #7  0x00007ffa068ff527 in bridge_request () from /etc/httpd/modules/mod_fcgid.so #8  0x00007ffa068fdcd1 in fcgid_handler () from /etc/httpd/modules/mod_fcgid.so #9  0x000055ce02a9f550 in ap_run_handler () #10 0x000055ce02a9fa99 in ap_invoke_handler () #11 0x000055ce02add40c in ap_internal_redirect () #12 0x00007ffa0c3961ec in handler_redirect () from /etc/httpd/modules/mod_rewrite.so #13 0x000055ce02a9f550 in ap_run_handler () #14 0x000055ce02a9fa99 in ap_invoke_handler () #15 0x000055ce02ade0ea in ap_process_async_request () #16 0x000055ce02ade3ce in ap_process_request () #17 0x000055ce02ada304 in ap_process_http_connection () #18 0x000055ce02ab13c0 in ap_run_process_connection () #19 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #20 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7ff9e57fa700 (LWP 3580)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7ff9e4ff9700 (LWP 3582)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7ff9e47f8700 (LWP 3584)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7ff9e3ff7700 (LWP 3586)): #0  0x00007ffa14d3fddd in poll () from /lib64/libc.so.6 #1  0x00007ffa069061ad in proc_read_ipc () from /etc/httpd/modules/mod_fcgid.so #2  0x00007ffa069069b6 in fcgid_feed_data () from /etc/httpd/modules/mod_fcgid.so #3  0x00007ffa06906b15 in fcgid_header_bucket_read () from /etc/httpd/modules/mod_fcgid.so #4  0x00007ffa068fe743 in getsfunc_fcgid_BRIGADE () from /etc/httpd/modules/mod_fcgid.so #5  0x000055ce02aaf558 in ap_scan_script_header_err_core_ex () #6  0x00007ffa068ff208 in handle_request () from /etc/httpd/modules/mod_fcgid.so #7  0x00007ffa068ff527 in bridge_request () from /etc/httpd/modules/mod_fcgid.so #8  0x00007ffa068fdcd1 in fcgid_handler () from /etc/httpd/modules/mod_fcgid.so #9  0x000055ce02a9f550 in ap_run_handler () #10 0x000055ce02a9fa99 in ap_invoke_handler () #11 0x000055ce02ade0ea in ap_process_async_request () #12 0x000055ce02ade3ce in ap_process_request () #13 0x000055ce02ada304 in ap_process_http_connection () #14 0x000055ce02ab13c0 in ap_run_process_connection () #15 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #16 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7ff9e37f6700 (LWP 3588)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7ff9e2ff5700 (LWP 3591)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7ff9e27f4700 (LWP 3593)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7ff9e1ff3700 (LWP 3594)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7ff9e17f2700 (LWP 3596)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7ff9e0ff1700 (LWP 3598)): #0  0x00007ffa14d4b0e3 in epoll_wait () from /lib64/libc.so.6 #1  0x00007ffa15463329 in impl_pollset_poll () from /lib64/libapr-1.so.0 #2  0x00007ffa066d1661 in mplx_pollset_poll (on_ctx=0x7ff96c0140b0, on_stream_output=0x7ffa066d5b50 <on_stream_output>, on_stream_input=0x7ffa066d5ca0 <on_stream_input>, timeout=30000000, m=0x7ff96c0148d0) at h2_mplx.c:1126 #3  h2_mplx_c1_poll (m=0x7ff96c0148d0, timeout=30000000, on_stream_input=on_stream_input@entry=0x7ffa066d5ca0 <on_stream_input>, on_stream_output=on_stream_output@entry=0x7ffa066d5b50 <on_stream_output>, on_ctx=on_ctx@entry=0x7ff96c0140b0) at h2_mplx.c:606 #4  0x00007ffa066da200 in h2_session_process (session=0x7ff96c0140b0, async=<optimized out>) at h2_session.c:1903 #5  0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9e8017f68) at h2_c1.c:130 #6  0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9e8017f68) at h2_c1.c:278 #7  0x000055ce02ab13c0 in ap_run_process_connection () #8  0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #9  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7ff9e07f0700 (LWP 3600)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7ff9dffef700 (LWP 3602)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7ff9df7ee700 (LWP 3604)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7ff9defed700 (LWP 3606)): #0  0x00007ffa1522c75d in read () from /lib64/libpthread.so.0 #1  0x00007ffa1545fd4b in apr_socket_recv () from /lib64/libapr-1.so.0 #2  0x00007ffa15ae3ae4 in socket_bucket_read () from /lib64/libaprutil-1.so.0 #3  0x000055ce02ad2e71 in ap_core_input_filter () #4  0x00007ffa0d3cd04e in logio_in_filter () from /etc/httpd/modules/mod_logio.so #5  0x00007ffa07210a9a in bio_filter_in_read () from /etc/httpd/modules/mod_ssl.so #6  0x00007ffa06fc95ea in bread_conv () from /lib64/libcrypto.so.81.1.1 #7  0x00007ffa06fc82c6 in BIO_read () from /lib64/libcrypto.so.81.1.1 #8  0x00007ffa1655ee21 in ssl3_read_n () from /lib64/libssl.so.81.1.1 #9  0x00007ffa16563622 in ssl3_get_record () from /lib64/libssl.so.81.1.1 #10 0x00007ffa16560e24 in ssl3_read_bytes () from /lib64/libssl.so.81.1.1 #11 0x00007ffa16568b60 in ssl3_read () from /lib64/libssl.so.81.1.1 #12 0x00007ffa16575665 in SSL_read () from /lib64/libssl.so.81.1.1 #13 0x00007ffa07211f73 in ssl_io_input_read () from /etc/httpd/modules/mod_ssl.so #14 0x00007ffa07214e2c in ssl_io_filter_input () from /etc/httpd/modules/mod_ssl.so #15 0x00007ffa066c594a in read_and_feed (session=session@entry=0x7ff98c040fb0) at h2_c1_io.c:486 #16 0x00007ffa066c6ecf in h2_c1_read (session=session@entry=0x7ff98c040fb0) at h2_c1_io.c:513 #17 0x00007ffa066da08b in h2_session_process (session=0x7ff98c040fb0, async=<optimized out>) at h2_session.c:1865 #18 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9e8007878) at h2_c1.c:130 #19 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9e8007878) at h2_c1.c:278 #20 0x000055ce02ab13c0 in ap_run_process_connection () #21 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #22 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7ff9de7ec700 (LWP 3608)): #0  0x00007ffa1522c75d in read () from /lib64/libpthread.so.0 #1  0x00007ffa1545fd4b in apr_socket_recv () from /lib64/libapr-1.so.0 #2  0x00007ffa15ae3ae4 in socket_bucket_read () from /lib64/libaprutil-1.so.0 #3  0x000055ce02ad2e71 in ap_core_input_filter () #4  0x00007ffa0d3cd04e in logio_in_filter () from /etc/httpd/modules/mod_logio.so #5  0x00007ffa07210a9a in bio_filter_in_read () from /etc/httpd/modules/mod_ssl.so #6  0x00007ffa06fc95ea in bread_conv () from /lib64/libcrypto.so.81.1.1 #7  0x00007ffa06fc82c6 in BIO_read () from /lib64/libcrypto.so.81.1.1 #8  0x00007ffa1655ee21 in ssl3_read_n () from /lib64/libssl.so.81.1.1 #9  0x00007ffa16563622 in ssl3_get_record () from /lib64/libssl.so.81.1.1 #10 0x00007ffa16560e24 in ssl3_read_bytes () from /lib64/libssl.so.81.1.1 #11 0x00007ffa16568b60 in ssl3_read () from /lib64/libssl.so.81.1.1 #12 0x00007ffa16575665 in SSL_read () from /lib64/libssl.so.81.1.1 #13 0x00007ffa07211f73 in ssl_io_input_read () from /etc/httpd/modules/mod_ssl.so #14 0x00007ffa07214e2c in ssl_io_filter_input () from /etc/httpd/modules/mod_ssl.so #15 0x00007ffa066c594a in read_and_feed (session=session@entry=0x7ff9d4082110) at h2_c1_io.c:486 #16 0x00007ffa066c6ecf in h2_c1_read (session=session@entry=0x7ff9d4082110) at h2_c1_io.c:513 #17 0x00007ffa066da08b in h2_session_process (session=0x7ff9d4082110, async=<optimized out>) at h2_session.c:1865 #18 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9e8005798) at h2_c1.c:130 #19 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9e8005798) at h2_c1.c:278 #20 0x000055ce02ab13c0 in ap_run_process_connection () #21 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #22 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #23 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7ff9ddfeb700 (LWP 3611)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x000055ce02ab789d in ap_queue_pop_something () #2  0x00007ffa096c9c3c in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7ff9dd7ea700 (LWP 3612)): #0  0x00007ffa15223bb8 in __pthread_mutex_lock_full () from /lib64/libpthread.so.0 #1  0x00007ffa15459b09 in proc_mutex_pthread_acquire_ex () from /lib64/libapr-1.so.0 #2  0x00007ffa096ca6c8 in listener_thread () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7ff9dcfe9700 (LWP 3659)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #2  0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7ff9cffff700 (LWP 3665)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #2  0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7ff9cf7fe700 (LWP 3666)): #0  0x00007ffa14d4b0e3 in epoll_wait () from /lib64/libc.so.6 #1  0x00007ffa15463329 in impl_pollset_poll () from /lib64/libapr-1.so.0 #2  0x00007ffa019da3de in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #3  0x00007ffa019cfb77 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #4  0x00007ffa019d02e5 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #5  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #6  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7ff9ceffd700 (LWP 3667)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #2  0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7ff9ce7fc700 (LWP 3668)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #2  0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7ff9cdffb700 (LWP 3669)): #0  0x00007ffa15229a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1  0x00007ffa018436aa in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #2  0x00007ffa019d7dd2 in ?? () from /usr/lib64/httpd/modules/mod_pagespeed_ap24.so #3  0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0 #4  0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ffa165d28c0 (LWP 3492)): #0  0x00007ffa1522c75d in read () from /lib64/libpthread.so.0 #1  0x000055ce02ab68e7 in ap_mpm_podx_check () #2  0x00007ffa096cb004 in child_main () from /etc/httpd/modules/mod_mpm_worker.so #3  0x00007ffa096cb48e in make_child () from /etc/httpd/modules/mod_mpm_worker.so #4  0x00007ffa096cb534 in startup_children () from /etc/httpd/modules/mod_mpm_worker.so #5  0x00007ffa096cc0a1 in worker_run () from /etc/httpd/modules/mod_mpm_worker.so #6  0x000055ce02ab41ce in ap_run_mpm () #7  0x000055ce02a9d88d in main ()
Reverting to 2.0.3 fixes CPU load but occasionally some httpd processes hang

Thanks @alexskynet for putting v2.0.6 into your grinder. Sorry, that it did nothing to improve.😢
Analyzing the backtraces now.

There was one change from v2.0.3 to newer ones that involved thread creation for the h2 workers. A quick check to see if that is causing problems would be to configure your server with a fixed number of h2 workers, like in:
H2MinWorkers 25
H2MaxWorkers 25

so all workers are created at startup and no dynamic creation/desctruction of thread does happen.
Could you give this a shot?

unfortunately no change
Still 100% CPU with your settings and 2.0.6

Thread 17 (Thread 0x7ff9e1ff3700 (LWP 8225)):
#0  0x00007ffa1522c75d in read () from /lib64/libpthread.so.0
#1  0x00007ffa1545fd4b in apr_socket_recv () from /lib64/libapr-1.so.0
#2  0x00007ffa15ae3ae4 in socket_bucket_read () from /lib64/libaprutil-1.so.0
#3  0x000055ce02ad2e71 in ap_core_input_filter ()
#4  0x00007ffa0d3cd04e in logio_in_filter () from /etc/httpd/modules/mod_logio.so
#5  0x00007ffa07210a9a in bio_filter_in_read () from /etc/httpd/modules/mod_ssl.so
#6  0x00007ffa06fc95ea in bread_conv () from /lib64/libcrypto.so.81.1.1
#7  0x00007ffa06fc82c6 in BIO_read () from /lib64/libcrypto.so.81.1.1
#8  0x00007ffa1655ee21 in ssl3_read_n () from /lib64/libssl.so.81.1.1
#9  0x00007ffa16563622 in ssl3_get_record () from /lib64/libssl.so.81.1.1
#10 0x00007ffa16560e24 in ssl3_read_bytes () from /lib64/libssl.so.81.1.1
#11 0x00007ffa16568b60 in ssl3_read () from /lib64/libssl.so.81.1.1
#12 0x00007ffa16575665 in SSL_read () from /lib64/libssl.so.81.1.1
#13 0x00007ffa07211f73 in ssl_io_input_read () from /etc/httpd/modules/mod_ssl.so
#14 0x00007ffa07214e2c in ssl_io_filter_input () from /etc/httpd/modules/mod_ssl.so
#15 0x00007ffa066c594a in read_and_feed (session=session@entry=0x7ff9d4004c60) at h2_c1_io.c:486
#16 0x00007ffa066c6ecf in h2_c1_read (session=session@entry=0x7ff9d4004c60) at h2_c1_io.c:513
#17 0x00007ffa066da08b in h2_session_process (session=0x7ff9d4004c60, async=) at h2_session.c:1865
#18 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9e8007878) at h2_c1.c:130
#19 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9e8007878) at h2_c1.c:278

Thread 25 (Thread 0x7ff9e5ffb700 (LWP 8217)):
#0  0x00007ffa1522c75d in read () from /lib64/libpthread.so.0
#1  0x00007ffa1545fd4b in apr_socket_recv () from /lib64/libapr-1.so.0
#2  0x00007ffa15ae3ae4 in socket_bucket_read () from /lib64/libaprutil-1.so.0
#3  0x000055ce02ad2e71 in ap_core_input_filter ()
#4  0x00007ffa0d3cd04e in logio_in_filter () from /etc/httpd/modules/mod_logio.so
#5  0x00007ffa07210a9a in bio_filter_in_read () from /etc/httpd/modules/mod_ssl.so
#6  0x00007ffa06fc95ea in bread_conv () from /lib64/libcrypto.so.81.1.1
#7  0x00007ffa06fc82c6 in BIO_read () from /lib64/libcrypto.so.81.1.1
#8  0x00007ffa1655ee21 in ssl3_read_n () from /lib64/libssl.so.81.1.1
#9  0x00007ffa16563622 in ssl3_get_record () from /lib64/libssl.so.81.1.1
#10 0x00007ffa16560e24 in ssl3_read_bytes () from /lib64/libssl.so.81.1.1
#11 0x00007ffa16568b60 in ssl3_read () from /lib64/libssl.so.81.1.1
#12 0x00007ffa16575665 in SSL_read () from /lib64/libssl.so.81.1.1
#13 0x00007ffa07211f73 in ssl_io_input_read () from /etc/httpd/modules/mod_ssl.so
#14 0x00007ffa07214e2c in ssl_io_filter_input () from /etc/httpd/modules/mod_ssl.so
#15 0x00007ffa066c594a in read_and_feed (session=session@entry=0x7ff9d002cbd0) at h2_c1_io.c:486
#16 0x00007ffa066c6ecf in h2_c1_read (session=session@entry=0x7ff9d002cbd0) at h2_c1_io.c:513
#17 0x00007ffa066da08b in h2_session_process (session=0x7ff9d002cbd0, async=) at h2_session.c:1865
#18 0x00007ffa066c4cc9 in h2_c1_run (c=c@entry=0x7ff9e8005798) at h2_c1.c:130
#19 0x00007ffa066c50dd in h2_c1_hook_process_connection (c=0x7ff9e8005798) at h2_c1.c:278
#20 0x000055ce02ab13c0 in ap_run_process_connection ()
#21 0x00007ffa096c9d9b in worker_thread () from /etc/httpd/modules/mod_mpm_worker.so
#22 0x00007ffa15225ea5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007ffa14d4ab0d in clone () from /lib64/libc.so.6

v2.0.7 released with a fix for mpm_worker setups that could (does?) result in busy loops.
Background: the v2.0.x line had improvements to return to mpm_event connection monitoring asap for efficiency. Unfortunately, the mpm_worker situation was not properly accounted for.

It is running with no apparent problems
The only thing I noticed is that httpd CPU usage raised from 6-7% to something around 10% but this is really not a problem

Thanks for your patience. Happy to hear that. Let's see what the day brings.
As to performance, I made some conservative changes to bring stability. When this version proves to be stable, I can dare to tighten the screws again somewhat.

You're doing a great job @icing
I'm happy to have been useful to help a little bit: that's called opensource!

running till now ok
Starting a second test server
Thank you @icing great job!

I just want to confirm that 2.0.7 works like a charm
CodeIt has already released the mod_http2-2.0.7 rpm with the fix so all world should be happy now :-)
It has been running for several hours on two server with absolutely no troubles
Problem solved: well done @icing ! :1st_place_medal:

Thanks again @alexskynet for the help!
Closing this as fixed in v2.0.7.

Sorry to reopen this bt today we had hangs on two different servers both running 2.0.7
Reverting to 2.0.3 seems to fix it
We runned 2.0.7 ok for nearly 4 days before the hang
The interesting part of pstack seems this at a first glance:
Thread 7 (Thread 0x7fdcfb7fe700 (LWP 20589)):
#0  0x00007fdd26670cd4 in h2_fifo_remove () from /etc/httpd/modules/mod_http2.so
#1  0x00007fdcdc03e478 in ?? ()
#2  0x00007fdd35a7167a in apr_bucket_shared_split () from /lib64/libaprutil-1.so.0
#3  0x00007fdcfc0057d8 in ?? ()
#4  0x0000000000003727 in ?? ()
#5  0x00007fdd26882f40 in http2_module () from /etc/httpd/modules/mod_http2.so
#6  0x0000000000003ff7 in ?? ()
#7  0x00007fdd2666b2cc in h2_stream_recv_DATA () from /etc/httpd/modules/mod_http2.so
#8  0x5613754aae48f400 in ?? ()
#9  0x00007fdcdc018c80 in ?? ()
#10 0x00007fdce80028d0 in ?? ()
#11 0x00007fdce800b0e0 in ?? ()
#12 0x00007fdce8003b60 in ?? ()
#13 0x00007fdce800b080 in ?? ()
#14 0x00007fdce800e0c8 in ?? ()
#15 0x00007fdce800e0c0 in ?? ()
#16 0x00007fdd364981a1 in nghttp2_session_pack_data () from /lib64/libnghttp2.so.14
#17 0x00007fdd36498f0d in nghttp2_session_mem_send_internal () from /lib64/libnghttp2.so.14
#18 0x00007fdd36499b19 in nghttp2_session_send () from /lib64/libnghttp2.so.14
#19 0x00007fdd266672ec in on_stream_state_enter () from /etc/httpd/modules/mod_http2.so
#20 0x00007fdd26667574 in on_stream_state_enter () from /etc/httpd/modules/mod_http2.so
#21 0x00007fdcdc018c08 in ?? ()
#22 0x00007fdcfb7fdb18 in ?? ()
#23 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7fdcf97fa700 (LWP 20593)):
#0  0x00007fdd26668946 in h2_session_process () from /etc/httpd/modules/mod_http2.so
#1  0x00007fdc8c072970 in ?? ()
#2  0x0000000000000142 in ?? ()
#3  0x00007fdcf97f9a50 in ?? ()
#4  0x00007fdcd0015b20 in ?? ()
#5  0xffffffffffffffff in ?? ()
#6  0x0000000000000000 in ?? ()

@alexskynet this strack trace does not look right. I assume there is another version of mod_http2 loaded than the v2.0.7 one. v2.0.7 never invokes h2_fifo_remove(). Assuming the APR lib symbols are correct, the apr_bucket_split is also never called in v2.0.7 (but was in versions v2.0.4 and earlier).
Could you double check?

v2.0.8 released. The fixes are unrelated to this, but I added a assertion before the point where the 100% cpu loop seems to happen in reports by @alexskynet. It would be interesting to know if this triggers and if so, what is logged (at level critical) in such a case.

The epic battle continues with v2.0.9:

Fixed a bug where errors during reponse body handling did not lead to
a proper RST_STREAM. Instead processing went into an infinite loop.
Extended test cases to catch this condition.

Hi @icing and thank you for your hard work
Testing 2.0.9 with worker
Till now what I see in mod_status looks OK: it has been running for 20 minutes now.
I cross my fingers and wait ...
I'll let you know

You tricked me before! I remain sceptical...😉

12 hours still running fine ...
It looks promising

Hi @icing did some testing for three days now and it works fine
Five servers are running 2.0.9 and one is running master with no anomalies
The battle is over and you won
Many compliments for the very good work

Hi @alexskynet! This is excellent news. We won. Thank you very much for helping on this.
In the meantime, I have added more edge test cases and made some more improvements on reliability. Will release that in some days as a v2.0.10.

Thanks you!
I'll start a test as soon as possible
Il 30 settembre 2022 16:59:17 CEST, Stefan Eissing @.***> ha scritto:

The epic battle continues with v2.0.9:

Fixed a bug where errors during reponse body handling did not lead to
a proper RST_STREAM. Instead processing went into an infinite loop.
Extended test cases to catch this condition.

--
Reply to this email directly or view it on GitHub:
https://github.com/icing/mod_h2/issues/234#issuecomment-1263686057
You are receiving this because you were mentioned.
Message ID: @.***>
SKNT Group SRLU
Via Maggiate 67/a
28021 Borgomanero (NO)
tel. +39 0322-836487/834765
fax +39 0322-836608
http://sknt.it

With v2.0.10 just being released and Alessandro's extensive testing, I think we have solve the issues.
Many, many thanks to everyone.