"kernel: traps: memcached[28432] general protection ip" with memcached SEGV

I run several memcached servers with identical setup and hardware,
with one having two memcached sudden exit in 1 week.
This server has about 8,600 average requests per second with peak 16,000 requests per second.
I have managed to find the syslog of the most recent exit.
Does this seem like a memcached bug or a problem with my server?
Here is the /var/log/messages log:
Nov 27 23:14:38 {hostnameREDACTED} kernel: traps: memcached[28432] general protection ip:417a2c sp:7f6c237fdd88 error:0 in memcached[400000+35000]
Nov 27 23:14:38 {hostnameREDACTED} systemd: memcached.service: main process exited, code=killed, status=11/SEGV
Nov 27 23:14:38 {hostnameREDACTED} systemd: Unit memcached.service entered failed state.
Nov 27 23:14:38 {hostnameREDACTED} systemd: memcached.service failed.
Here is some server information:
# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
# cat /etc/sysconfig/memcached
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="8192"
OPTIONS="-t 8 -l {thisisprivateip_REDACTED}"
# rpm -qa|grep memcached
memcached-1.5.20-1.el7.remi.x86_64
libmemcached-opt-libs-1.0.18-2.el7.remi.x86_64
php73-php-pecl-memcached-3.1.4-1.el7.remi.x86_64
# free -h
total        used        free      shared  buff/cache   available
Mem:            15G        1.8G        5.3G        218M        8.6G         13G
Swap:          399M         43M        356M
# uname -a
Linux {thisishostname_REDACTED} 3.10.0-1062.1.2.el7.x86_64 #1 SMP Mon Sep 30 14:19:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Hardware:
dual Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz
1GB ethernet

Hey,
Was there anything in dmesg/kernel messages as well? Something like an OOM killer or memory error.
Can you get an addr2line -e /path/to/your/servers/memcached 417a2c as well? (though that looks short?)
No core dumps?

Also, if you were on a previous version that was fine, would be nice to know what that was :)

Was there anything in dmesg/kernel messages as well? Something like an OOM killer or memory error.
Can you get an addr2line -e /path/to/your/servers/memcached 417a2c as well? (though that looks short?)
No core dumps?

Currently I rebooted the server and restarted everything.
No other error messages in /var/log/messages at all, so I don't think core dump happened.
No OOM killer or other error messages in /var/log/messages
addr2line -e /usr/bin/memcached 417a2c
??:0

Hey,
Was there anything in dmesg/kernel messages as well? Something like an OOM killer or memory error.
Can you get an addr2line -e /path/to/your/servers/memcached 417a2c as well? (though that looks short?)
No core dumps?

Was the line with the 417a2c clipped by chance?
I'll need the line from the segfault at very least. Ideally a backtrace from a coredump. Otherwise there's nothing for me to go on :(

Was the line with the 417a2c clipped by chance?
I'll need the line from the segfault at very least. Ideally a backtrace from a coredump. Otherwise there's nothing for me to go on :(

addr2line -e /usr/bin/memcached 417a2c
output is simply just
??:0
This is all my logs(I found the previous error too) with the word memcached in /var/log/messages:
cat /var/log/messages|grep memca
Nov 25 15:23:32 hostnameREDACTED kernel: traps: memcached[15687] general protection ip:417a2c sp:7fa1b2ffcd88 error:0 in memcached[400000+35000]
Nov 25 15:23:32 hostnameREDACTED systemd: memcached.service: main process exited, code=killed, status=11/SEGV
Nov 25 15:23:32 hostnameREDACTED systemd: Unit memcached.service entered failed state.
Nov 25 15:23:32 hostnameREDACTED systemd: memcached.service failed.
Nov 25 21:02:54 hostnameREDACTED rz[22214]: [root] mymemcached/ZMODEM: 269 Bytes, 53277 BPS
Nov 27 23:14:38 hostnameREDACTED kernel: traps: memcached[28432] general protection ip:417a2c sp:7f6c237fdd88 error:0 in memcached[400000+35000]
Nov 27 23:14:38 hostnameREDACTED systemd: memcached.service: main process exited, code=killed, status=11/SEGV
Nov 27 23:14:38 hostnameREDACTED systemd: Unit memcached.service entered failed state.
Nov 27 23:14:38 hostnameREDACTED systemd: memcached.service failed.

When I build https://rpms.remirepo.net/SRPMS/memcached-1.5.20-1.remi.src.rpm on a RHEL7 host, before symbols are stripped:
> addr2line -e memcached 417a2c
.../memcached-1.5.20/items.c:418 (discriminator 1)

406 static void do_item_link_q(item *it) { /* item is the new head */
407    item **head, **tail;
408    assert((it->it_flags & ITEM_SLABBED) == 0);
409
410    head = &heads[it->slabs_clsid];
411    tail = &tails[it->slabs_clsid];
412    assert(it != *head);
413    assert((*head && *tail) || (*head == 0 && *tail == 0));
414    it->prev = 0;
415    it->next = *head;
416    if (it->next) it->next->prev = it;
417    *head = it;
418    if (*tail == 0) *tail = it;            <--- HERE
419    sizes[it->slabs_clsid]++;

Hmm. That seems unlikely. If 418 would segfault 417 would've. In fact 410/411 would've as well?

Of note, on the problem server here is my memcached yum history for installs and updates.
I also have memcached running with similar load on a few other identical servers without problems.
Only this one server memcached SEGV twice in two days(I have currently rebooted the server).
Jun 16 21:51:59 Installed: memcached-1.4.15-10.el7_3.1.x86_64
Aug 18 11:23:51 Updated: memcached-1.5.16-1.el7.remi.x86_64
Oct 08 01:55:28 Updated: memcached-1.5.19-1.el7.remi.x86_64
Nov 16 21:21:15 Updated: memcached-1.5.20-1.el7.remi.x86_64

Can you try downgrading to 1.5.19 for a while? I'd love to see a full backtrace from 1.5.20 but it sounds like that might be hard for you to get. There's one potential commit in 1.5.20 that could cause a segfault in roughly that area of code but it's been tested for a while now.
How long were the instances running on 1.5.20 before they crashed?

How long were the instances running on 1.5.20 before they crashed?

My previous reply with yum update logs is very exact.
So that would be from
Nov 16 21:21:15 Updated: memcached-1.5.20-1.el7.remi.x86_64
to the first SEGV at
Nov 25 15:23:32 hostnameREDACTED kernel: traps: memcached[15687] general protection ip:417a2c sp:7fa1b2ffcd88 error:0 in memcached[400000+35000]
and the second SEGV at
Nov 27 23:14:38 hostnameREDACTED kernel: traps: memcached[28432] general protection ip:417a2c sp:7f6c237fdd88 error:0 in memcached[400000+35000]
with a memcached load of about 8,600 average requests per second with peak 16,000 requests per second.

Any chance you could share the output of stats, stats settings, stats items, and stats slabs from this server? Would give me a hint as to what features were being exercised.
I still need a better guide for how to get cores/backtraces on segfaults :\ Assuming systemd made this harder like it's made everything else hard.

Any chance you could share the output of stats, stats settings, stats items, and stats slabs from this server? Would give me a hint as to what features were being exercised.
I still need a better guide for how to get cores/backtraces on segfaults :\ Assuming systemd made this harder like it's made everything else hard.

stats
STAT pid 1740
STAT uptime 233293
STAT time<PHONE_NUMBER>
STAT version 1.5.20
STAT libevent 2.0.21-stable
STAT pointer_size 64
STAT rusage_user 20459.027184
STAT rusage_system 55569.973847
STAT max_connections 1024
STAT curr_connections 67
STAT total_connections 164491031
STAT rejected_connections 0
STAT connection_structures 471
STAT reserved_fds 40
STAT cmd_get<PHONE_NUMBER>
STAT cmd_set 74785057
STAT cmd_flush 0
STAT cmd_touch 0
STAT cmd_meta 0
STAT get_hits<PHONE_NUMBER>
STAT get_misses 68822357
STAT get_expired 1884606
STAT get_flushed 0
STAT delete_misses 30802924
STAT delete_hits 12162004
STAT incr_misses 18
STAT incr_hits 958584
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT touch_hits 0
STAT touch_misses 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read<PHONE_NUMBER>36
STAT bytes_written<PHONE_NUMBER>784
STAT limit_maxbytes<PHONE_NUMBER>
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT time_in_listen_disabled_us 0
STAT threads 8
STAT conn_yields 0
STAT hash_power_level 21
STAT hash_bytes 16777216
STAT hash_is_expanding 0
STAT slab_reassign_rescues 2366402
STAT slab_reassign_chunk_rescues 3
STAT slab_reassign_evictions_nomem 0
STAT slab_reassign_inline_reclaim 21284
STAT slab_reassign_busy_items 28
STAT slab_reassign_busy_deletes 0
STAT slab_reassign_running 0
STAT slabs_moved 6740
STAT lru_crawler_running 0
STAT lru_crawler_starts 1549484
STAT lru_maintainer_juggles 78049072
STAT malloc_fails 0
STAT log_worker_dropped 0
STAT log_worker_written 0
STAT log_watcher_skipped 0
STAT log_watcher_sent 0
STAT bytes 812653995
STAT curr_items 1091540
STAT total_items 77158251
STAT slab_global_page_pool 668
STAT expired_unfetched 15102969
STAT evicted_unfetched 0
STAT evicted_active 0
STAT evictions 0
STAT reclaimed 142866
STAT crawler_reclaimed 32249403
STAT crawler_items_checked<PHONE_NUMBER>
STAT lrutail_reflocked 411
STAT moves_to_cold 16363736
STAT moves_to_warm 7519571
STAT moves_within_lru 244478
STAT direct_reclaims 0
STAT lru_bumps_dropped 0
END
stats settings
STAT maxbytes<PHONE_NUMBER>
STAT maxconns 1024
STAT tcpport 11211
STAT udpport 0
STAT inter {IP_REDACTED}
STAT verbosity 0
STAT oldest 0
STAT evictions on
STAT domain_socket NULL
STAT umask 700
STAT growth_factor 1.25
STAT chunk_size 48
STAT num_threads 8
STAT num_threads_per_udp 8
STAT stat_key_prefix :
STAT detail_enabled no
STAT reqs_per_event 20
STAT cas_enabled yes
STAT tcp_backlog 1024
STAT binding_protocol auto-negotiate
STAT auth_enabled_sasl no
STAT auth_enabled_ascii no
STAT item_size_max 1048576
STAT maxconns_fast yes
STAT hashpower_init 0
STAT slab_reassign yes
STAT slab_automove 1
STAT slab_automove_ratio 0.80
STAT slab_automove_window 30
STAT slab_chunk_max 524288
STAT lru_crawler yes
STAT lru_crawler_sleep 100
STAT lru_crawler_tocrawl 0
STAT tail_repair_time 0
STAT flush_enabled yes
STAT dump_enabled yes
STAT hash_algorithm murmur3
STAT lru_maintainer_thread yes
STAT lru_segmented yes
STAT hot_lru_pct 20
STAT warm_lru_pct 40
STAT hot_max_factor 0.20
STAT warm_max_factor 2.00
STAT temp_lru no
STAT temporary_ttl 61
STAT idle_timeout 0
STAT watcher_logbuf_size 262144
STAT worker_logbuf_size 65536
STAT track_sizes no
STAT inline_ascii_response no
STAT ext_item_size 512
STAT ext_item_age<PHONE_NUMBER>
STAT ext_low_ttl 0
STAT ext_recache_rate 2000
STAT ext_wbuf_size 4194304
STAT ext_compact_under 0
STAT ext_drop_under 0
STAT ext_max_frag 0.80
STAT slab_automove_freeratio 0.010
STAT ext_drop_unread no
END
stats items
STAT items:1:number 2654
STAT items:1:number_hot 535
STAT items:1:number_warm 1052
STAT items:1:number_cold 1067
STAT items:1:age_hot 545
STAT items:1:age_warm 1330
STAT items:1:age 216806
STAT items:1:mem_requested 232196
STAT items:1:evicted 0
STAT items:1:evicted_nonzero 0
STAT items:1:evicted_time 0
STAT items:1:outofmemory 0
STAT items:1:tailrepairs 0
STAT items:1:reclaimed 2472
STAT items:1:expired_unfetched 26011
STAT items:1:evicted_unfetched 0
STAT items:1:evicted_active 0
STAT items:1:crawler_reclaimed 139086
STAT items:1:crawler_items_checked 48210514
STAT items:1:lrutail_reflocked 1
STAT items:1:moves_to_cold 145746
STAT items:1:moves_to_warm 147222
STAT items:1:moves_within_lru 152255
STAT items:1:direct_reclaims 0
STAT items:1:hits_to_hot 2858214
STAT items:1:hits_to_warm 4430761
STAT items:1:hits_to_cold 65135
STAT items:1:hits_to_temp 0
STAT items:2:number 307289
STAT items:2:number_hot 61510
STAT items:2:number_warm 80701
STAT items:2:number_cold 165078
STAT items:2:age_hot 8662
STAT items:2:age_warm 214096
STAT items:2:age 217790
STAT items:2:mem_requested 34547181
STAT items:2:evicted 0
STAT items:2:evicted_nonzero 0
STAT items:2:evicted_time 0
STAT items:2:outofmemory 0
STAT items:2:tailrepairs 0
STAT items:2:reclaimed 693
STAT items:2:expired_unfetched 5499597
STAT items:2:evicted_unfetched 0
STAT items:2:evicted_active 0
STAT items:2:crawler_reclaimed 7024789
STAT items:2:crawler_items_checked<PHONE_NUMBER>
STAT items:2:lrutail_reflocked 1
STAT items:2:moves_to_cold 2060116
STAT items:2:moves_to_warm 768963
STAT items:2:moves_within_lru 5335
STAT items:2:direct_reclaims 0
STAT items:2:hits_to_hot 482583535
STAT items:2:hits_to_warm 112719480
STAT items:2:hits_to_cold 961235
STAT items:2:hits_to_temp 0
STAT items:3:number 239470
STAT items:3:number_hot 47221
STAT items:3:number_warm 39856
STAT items:3:number_cold 152393
STAT items:3:age_hot 16161
STAT items:3:age_warm 142265
STAT items:3:age 172127
STAT items:3:mem_requested 29182670
STAT items:3:evicted 0
STAT items:3:evicted_nonzero 0
STAT items:3:evicted_time 0
STAT items:3:outofmemory 0
STAT items:3:tailrepairs 0
STAT items:3:reclaimed 37679
STAT items:3:expired_unfetched 806310
STAT items:3:evicted_unfetched 0
STAT items:3:evicted_active 0
STAT items:3:crawler_reclaimed 1215566
STAT items:3:crawler_items_checked 138531987
STAT items:3:lrutail_reflocked 0
STAT items:3:moves_to_cold 874941
STAT items:3:moves_to_warm 304369
STAT items:3:moves_within_lru 11416
STAT items:3:direct_reclaims 0
STAT items:3:hits_to_hot 108798771
STAT items:3:hits_to_warm 6150194
STAT items:3:hits_to_cold 488118
STAT items:3:hits_to_temp 0
STAT items:4:number 2096
STAT items:4:number_hot 415
STAT items:4:number_warm 828
STAT items:4:number_cold 853
STAT items:4:age_hot 2295
STAT items:4:age_warm 13734
STAT items:4:age 35683
STAT items:4:mem_requested 367616
STAT items:4:evicted 0
STAT items:4:evicted_nonzero 0
STAT items:4:evicted_time 0
STAT items:4:outofmemory 0
STAT items:4:tailrepairs 0
STAT items:4:reclaimed 3397
STAT items:4:expired_unfetched 26351
STAT items:4:evicted_unfetched 0
STAT items:4:evicted_active 0
STAT items:4:crawler_reclaimed 110220
STAT items:4:crawler_items_checked 32392825
STAT items:4:lrutail_reflocked 0
STAT items:4:moves_to_cold 67235
STAT items:4:moves_to_warm 31594
STAT items:4:moves_within_lru 2114
STAT items:4:direct_reclaims 0
STAT items:4:hits_to_hot 27529911
STAT items:4:hits_to_warm 92508872
STAT items:4:hits_to_cold 7549
STAT items:4:hits_to_temp 0
STAT items:5:number 2231
STAT items:5:number_hot 442
STAT items:5:number_warm 554
STAT items:5:number_cold 1235
STAT items:5:age_hot 816
STAT items:5:age_warm 30115
STAT items:5:age 28160
STAT items:5:mem_requested 485564
STAT items:5:evicted 0
STAT items:5:evicted_nonzero 0
STAT items:5:evicted_time 0
STAT items:5:outofmemory 0
STAT items:5:tailrepairs 0
STAT items:5:reclaimed 1984
STAT items:5:expired_unfetched 22402
STAT items:5:evicted_unfetched 0
STAT items:5:evicted_active 0
STAT items:5:crawler_reclaimed 275870
STAT items:5:crawler_items_checked 105859045
STAT items:5:lrutail_reflocked 0
STAT items:5:moves_to_cold 222446
STAT items:5:moves_to_warm 73750
STAT items:5:moves_within_lru 292
STAT items:5:direct_reclaims 0
STAT items:5:hits_to_hot 1229013
STAT items:5:hits_to_warm 1784990
STAT items:5:hits_to_cold 59494
STAT items:5:hits_to_temp 0
STAT items:6:number 3268
STAT items:6:number_hot 657
STAT items:6:number_warm 713
STAT items:6:number_cold 1898
STAT items:6:age_hot 791
STAT items:6:age_warm 30115
STAT items:6:age 34708
STAT items:6:mem_requested 889195
STAT items:6:evicted 0
STAT items:6:evicted_nonzero 0
STAT items:6:evicted_time 0
STAT items:6:outofmemory 0
STAT items:6:tailrepairs 0
STAT items:6:reclaimed 9739
STAT items:6:expired_unfetched 14555
STAT items:6:evicted_unfetched 0
STAT items:6:evicted_active 0
STAT items:6:crawler_reclaimed 404286
STAT items:6:crawler_items_checked 159245446
STAT items:6:lrutail_reflocked 0
STAT items:6:moves_to_cold 342060
STAT items:6:moves_to_warm 108149
STAT items:6:moves_within_lru 368
STAT items:6:direct_reclaims 0
STAT items:6:hits_to_hot 16394774
STAT items:6:hits_to_warm 18455690
STAT items:6:hits_to_cold 87408
STAT items:6:hits_to_temp 0
STAT items:7:number 4643
STAT items:7:number_hot 926
STAT items:7:number_warm 1267
STAT items:7:number_cold 2450
STAT items:7:age_hot 1077
STAT items:7:age_warm 173327
STAT items:7:age 172421
STAT items:7:mem_requested 1587199
STAT items:7:evicted 0
STAT items:7:evicted_nonzero 0
STAT items:7:evicted_time 0
STAT items:7:outofmemory 0
STAT items:7:tailrepairs 0
STAT items:7:reclaimed 3343
STAT items:7:expired_unfetched 28755
STAT items:7:evicted_unfetched 0
STAT items:7:evicted_active 0
STAT items:7:crawler_reclaimed 411473
STAT items:7:crawler_items_checked 181713247
STAT items:7:lrutail_reflocked 0
STAT items:7:moves_to_cold 347581
STAT items:7:moves_to_warm 112628
STAT items:7:moves_within_lru 188
STAT items:7:direct_reclaims 0
STAT items:7:hits_to_hot 14143939
STAT items:7:hits_to_warm 25532678
STAT items:7:hits_to_cold 85937
STAT items:7:hits_to_temp 0
STAT items:8:number 8413
STAT items:8:number_hot 1688
STAT items:8:number_warm 1515
STAT items:8:number_cold 5210
STAT items:8:age_hot 1644
STAT items:8:age_warm 173576
STAT items:8:age 173797
STAT items:8:mem_requested 3630980
STAT items:8:evicted 0
STAT items:8:evicted_nonzero 0
STAT items:8:evicted_time 0
STAT items:8:outofmemory 0
STAT items:8:tailrepairs 0
STAT items:8:reclaimed 6027
STAT items:8:expired_unfetched 159269
STAT items:8:evicted_unfetched 0
STAT items:8:evicted_active 0
STAT items:8:crawler_reclaimed 484880
STAT items:8:crawler_items_checked 226545981
STAT items:8:lrutail_reflocked 0
STAT items:8:moves_to_cold 423415
STAT items:8:moves_to_warm 116798
STAT items:8:moves_within_lru 118
STAT items:8:direct_reclaims 0
STAT items:8:hits_to_hot 2742204
STAT items:8:hits_to_warm 2403741
STAT items:8:hits_to_cold 117286
STAT items:8:hits_to_temp 0
STAT items:9:number 16898
STAT items:9:number_hot 3404
STAT items:9:number_warm 6790
STAT items:9:number_cold 6704
STAT items:9:age_hot 5235
STAT items:9:age_warm 139644
STAT items:9:age 174117
STAT items:9:mem_requested 9121149
STAT items:9:evicted 0
STAT items:9:evicted_nonzero 0
STAT items:9:evicted_time 0
STAT items:9:outofmemory 0
STAT items:9:tailrepairs 0
STAT items:9:reclaimed 2696
STAT items:9:expired_unfetched 101047
STAT items:9:evicted_unfetched 0
STAT items:9:evicted_active 0
STAT items:9:crawler_reclaimed 328777
STAT items:9:crawler_items_checked 85658674
STAT items:9:lrutail_reflocked 0
STAT items:9:moves_to_cold 233824
STAT items:9:moves_to_warm 82769
STAT items:9:moves_within_lru 305
STAT items:9:direct_reclaims 0
STAT items:9:hits_to_hot 57770095
STAT items:9:hits_to_warm 10529874
STAT items:9:hits_to_cold 43032
STAT items:9:hits_to_temp 0
STAT items:10:number 21642
STAT items:10:number_hot 4148
STAT items:10:number_warm 8607
STAT items:10:number_cold 8887
STAT items:10:age_hot 1956
STAT items:10:age_warm 145545
STAT items:10:age 174037
STAT items:10:mem_requested 15057924
STAT items:10:evicted 0
STAT items:10:evicted_nonzero 0
STAT items:10:evicted_time 0
STAT items:10:outofmemory 0
STAT items:10:tailrepairs 0
STAT items:10:reclaimed 8206
STAT items:10:expired_unfetched 451952
STAT items:10:evicted_unfetched 0
STAT items:10:evicted_active 0
STAT items:10:crawler_reclaimed 1096166
STAT items:10:crawler_items_checked 393180270
STAT items:10:lrutail_reflocked 0
STAT items:10:moves_to_cold 704611
STAT items:10:moves_to_warm 302237
STAT items:10:moves_within_lru 65
STAT items:10:direct_reclaims 0
STAT items:10:hits_to_hot 21728089
STAT items:10:hits_to_warm 11237634
STAT items:10:hits_to_cold 177411
STAT items:10:hits_to_temp 0
STAT items:11:number 37729
STAT items:11:number_hot 7874
STAT items:11:number_warm 13222
STAT items:11:number_cold 16633
STAT items:11:age_hot 573
STAT items:11:age_warm 24734
STAT items:11:age 24763
STAT items:11:mem_requested 30690339
STAT items:11:evicted 0
STAT items:11:evicted_nonzero 0
STAT items:11:evicted_time 0
STAT items:11:outofmemory 0
STAT items:11:tailrepairs 0
STAT items:11:reclaimed 8427
STAT items:11:expired_unfetched 1898525
STAT items:11:evicted_unfetched 0
STAT items:11:evicted_active 0
STAT items:11:crawler_reclaimed 6466791
STAT items:11:crawler_items_checked<PHONE_NUMBER>
STAT items:11:lrutail_reflocked 1
STAT items:11:moves_to_cold 4619438
STAT items:11:moves_to_warm 2784591
STAT items:11:moves_within_lru 159
STAT items:11:direct_reclaims 0
STAT items:11:hits_to_hot 49241721
STAT items:11:hits_to_warm 45294462
STAT items:11:hits_to_cold 1764090
STAT items:11:hits_to_temp 0
STAT items:12:number 170671
STAT items:12:number_hot 33947
STAT items:12:number_warm 58226
STAT items:12:number_cold 78498
STAT items:12:age_hot 7635
STAT items:12:age_warm 114975
STAT items:12:age 165677
STAT items:12:mem_requested 180800177
STAT items:12:evicted 0
STAT items:12:evicted_nonzero 0
STAT items:12:evicted_time 0
STAT items:12:outofmemory 0
STAT items:12:tailrepairs 0
STAT items:12:reclaimed 1568
STAT items:12:expired_unfetched 970552
STAT items:12:evicted_unfetched 0
STAT items:12:evicted_active 0
STAT items:12:crawler_reclaimed 2722551
STAT items:12:crawler_items_checked 520072246
STAT items:12:lrutail_reflocked 1
STAT items:12:moves_to_cold 1195546
STAT items:12:moves_to_warm 651065
STAT items:12:moves_within_lru 3340
STAT items:12:direct_reclaims 0
STAT items:12:hits_to_hot 85477883
STAT items:12:hits_to_warm 13849947
STAT items:12:hits_to_cold 534558
STAT items:12:hits_to_temp 0
STAT items:13:number 111316
STAT items:13:number_hot 22201
STAT items:13:number_warm 43467
STAT items:13:number_cold 45648
STAT items:13:age_hot 7211
STAT items:13:age_warm 103942
STAT items:13:age 173484
STAT items:13:mem_requested 144493581
STAT items:13:evicted 0
STAT items:13:evicted_nonzero 0
STAT items:13:evicted_time 0
STAT items:13:outofmemory 0
STAT items:13:tailrepairs 0
STAT items:13:reclaimed 654
STAT items:13:expired_unfetched 568235
STAT items:13:evicted_unfetched 0
STAT items:13:evicted_active 0
STAT items:13:crawler_reclaimed 1989146
STAT items:13:crawler_items_checked 723708499
STAT items:13:lrutail_reflocked 0
STAT items:13:moves_to_cold 638494
STAT items:13:moves_to_warm 452463
STAT items:13:moves_within_lru 2978
STAT items:13:direct_reclaims 0
STAT items:13:hits_to_hot 93078213
STAT items:13:hits_to_warm 26369978
STAT items:13:hits_to_cold 332939
STAT items:13:hits_to_temp 0
STAT items:14:number 98447
STAT items:14:number_hot 19671
STAT items:14:number_warm 39390
STAT items:14:number_cold 39386
STAT items:14:age_hot 10098
STAT items:14:age_warm 99924
STAT items:14:age 171999
STAT items:14:mem_requested 160959458
STAT items:14:evicted 0
STAT items:14:evicted_nonzero 0
STAT items:14:evicted_time 0
STAT items:14:outofmemory 0
STAT items:14:tailrepairs 0
STAT items:14:reclaimed 904
STAT items:14:expired_unfetched 503860
STAT items:14:evicted_unfetched 0
STAT items:14:evicted_active 0
STAT items:14:crawler_reclaimed 1968373
STAT items:14:crawler_items_checked 672926164
STAT items:14:lrutail_reflocked 1
STAT items:14:moves_to_cold 408798
STAT items:14:moves_to_warm 302440
STAT items:14:moves_within_lru 9414
STAT items:14:direct_reclaims 0
STAT items:14:hits_to_hot 114006575
STAT items:14:hits_to_warm 13056172
STAT items:14:hits_to_cold 200028
STAT items:14:hits_to_temp 0
STAT items:15:number 44824
STAT items:15:number_hot 8856
STAT items:15:number_warm 17979
STAT items:15:number_cold 17989
STAT items:15:age_hot 9967
STAT items:15:age_warm 97642
STAT items:15:age 170179
STAT items:15:mem_requested 88943103
STAT items:15:evicted 0
STAT items:15:evicted_nonzero 0
STAT items:15:evicted_time 0
STAT items:15:outofmemory 0
STAT items:15:tailrepairs 0
STAT items:15:reclaimed 705
STAT items:15:expired_unfetched 282602
STAT items:15:evicted_unfetched 0
STAT items:15:evicted_active 0
STAT items:15:crawler_reclaimed 1171787
STAT items:15:crawler_items_checked 325057373
STAT items:15:lrutail_reflocked 2
STAT items:15:moves_to_cold 206387
STAT items:15:moves_to_warm 162707
STAT items:15:moves_within_lru 9723
STAT items:15:direct_reclaims 0
STAT items:15:hits_to_hot 55087983
STAT items:15:hits_to_warm 6774932
STAT items:15:hits_to_cold 108013
STAT items:15:hits_to_temp 0
STAT items:16:number 6065
STAT items:16:number_hot 1205
STAT items:16:number_warm 2428
STAT items:16:number_cold 2432
STAT items:16:age_hot 3196
STAT items:16:age_warm 35248
STAT items:16:age 75853
STAT items:16:mem_requested 15576008
STAT items:16:evicted 0
STAT items:16:evicted_nonzero 0
STAT items:16:evicted_time 0
STAT items:16:outofmemory 0
STAT items:16:tailrepairs 0
STAT items:16:reclaimed 1495
STAT items:16:expired_unfetched 78397
STAT items:16:evicted_unfetched 0
STAT items:16:evicted_active 0
STAT items:16:crawler_reclaimed 366620
STAT items:16:crawler_items_checked 66604378
STAT items:16:lrutail_reflocked 0
STAT items:16:moves_to_cold 72751
STAT items:16:moves_to_warm 56573
STAT items:16:moves_within_lru 15681
STAT items:16:direct_reclaims 0
STAT items:16:hits_to_hot 12312794
STAT items:16:hits_to_warm 4319851
STAT items:16:hits_to_cold 41827
STAT items:16:hits_to_temp 0
STAT items:17:number 3412
STAT items:17:number_hot 675
STAT items:17:number_warm 1355
STAT items:17:number_cold 1382
STAT items:17:age_hot 2511
STAT items:17:age_warm 30832
STAT items:17:age 86613
STAT items:17:mem_requested 10998431
STAT items:17:evicted 0
STAT items:17:evicted_nonzero 0
STAT items:17:evicted_time 0
STAT items:17:outofmemory 0
STAT items:17:tailrepairs 0
STAT items:17:reclaimed 3346
STAT items:17:expired_unfetched 55920
STAT items:17:evicted_unfetched 0
STAT items:17:evicted_active 0
STAT items:17:crawler_reclaimed 232978
STAT items:17:crawler_items_checked 41304321
STAT items:17:lrutail_reflocked 0
STAT items:17:moves_to_cold 54066
STAT items:17:moves_to_warm 43446
STAT items:17:moves_within_lru 8426
STAT items:17:direct_reclaims 0
STAT items:17:hits_to_hot 26698819
STAT items:17:hits_to_warm 2360093
STAT items:17:hits_to_cold 24385
STAT items:17:hits_to_temp 0
STAT items:18:number 2290
STAT items:18:number_hot 459
STAT items:18:number_warm 890
STAT items:18:number_cold 941
STAT items:18:age_hot 1157
STAT items:18:age_warm 33996
STAT items:18:age 36713
STAT items:18:mem_requested 9260809
STAT items:18:evicted 0
STAT items:18:evicted_nonzero 0
STAT items:18:evicted_time 0
STAT items:18:outofmemory 0
STAT items:18:tailrepairs 0
STAT items:18:reclaimed 3866
STAT items:18:expired_unfetched 110787
STAT items:18:evicted_unfetched 0
STAT items:18:evicted_active 0
STAT items:18:crawler_reclaimed 275091
STAT items:18:crawler_items_checked 60697574
STAT items:18:lrutail_reflocked 1
STAT items:18:moves_to_cold 108525
STAT items:18:moves_to_warm 96394
STAT items:18:moves_within_lru 4310
STAT items:18:direct_reclaims 0
STAT items:18:hits_to_hot 9287161
STAT items:18:hits_to_warm 11441718
STAT items:18:hits_to_cold 25287
STAT items:18:hits_to_temp 0
STAT items:19:number 1690
STAT items:19:number_hot 325
STAT items:19:number_warm 651
STAT items:19:number_cold 714
STAT items:19:age_hot 420
STAT items:19:age_warm 15880
STAT items:19:age 35073
STAT items:19:mem_requested 8573765
STAT items:19:evicted 0
STAT items:19:evicted_nonzero 0
STAT items:19:evicted_time 0
STAT items:19:outofmemory 0
STAT items:19:tailrepairs 0
STAT items:19:reclaimed 907
STAT items:19:expired_unfetched 341322
STAT items:19:evicted_unfetched 0
STAT items:19:evicted_active 0
STAT items:19:crawler_reclaimed 573931
STAT items:19:crawler_items_checked 79217410
STAT items:19:lrutail_reflocked 0
STAT items:19:moves_to_cold 323616
STAT items:19:moves_to_warm 122864
STAT items:19:moves_within_lru 1531
STAT items:19:direct_reclaims 0
STAT items:19:hits_to_hot 3199308
STAT items:19:hits_to_warm 1886187
STAT items:19:hits_to_cold 29931
STAT items:19:hits_to_temp 0
STAT items:20:number 1574
STAT items:20:number_hot 297
STAT items:20:number_warm 626
STAT items:20:number_cold 651
STAT items:20:age_hot 221
STAT items:20:age_warm 20395
STAT items:20:age 20365
STAT items:20:mem_requested 10020965
STAT items:20:evicted 0
STAT items:20:evicted_nonzero 0
STAT items:20:evicted_time 0
STAT items:20:outofmemory 0
STAT items:20:tailrepairs 0
STAT items:20:reclaimed 422
STAT items:20:expired_unfetched 647583
STAT items:20:evicted_unfetched 0
STAT items:20:evicted_active 0
STAT items:20:crawler_reclaimed 1002179
STAT items:20:crawler_items_checked 100607464
STAT items:20:lrutail_reflocked 3
STAT items:20:moves_to_cold 610142
STAT items:20:moves_to_warm 162565
STAT items:20:moves_within_lru 1136
STAT items:20:direct_reclaims 0
STAT items:20:hits_to_hot 5495294
STAT items:20:hits_to_warm 6285705
STAT items:20:hits_to_cold 39002
STAT items:20:hits_to_temp 0
STAT items:21:number 1492
STAT items:21:number_hot 294
STAT items:21:number_warm 522
STAT items:21:number_cold 676
STAT items:21:age_hot 160
STAT items:21:age_warm 15059
STAT items:21:age 16495
STAT items:21:mem_requested 11857524
STAT items:21:evicted 0
STAT items:21:evicted_nonzero 0
STAT items:21:evicted_time 0
STAT items:21:outofmemory 0
STAT items:21:tailrepairs 0
STAT items:21:reclaimed 851
STAT items:21:expired_unfetched 733750
STAT items:21:evicted_unfetched 0
STAT items:21:evicted_active 0
STAT items:21:crawler_reclaimed 1145050
STAT items:21:crawler_items_checked 102233026
STAT items:21:lrutail_reflocked 1
STAT items:21:moves_to_cold 749531
STAT items:21:moves_to_warm 190183
STAT items:21:moves_within_lru 1064
STAT items:21:direct_reclaims 0
STAT items:21:hits_to_hot 6330063
STAT items:21:hits_to_warm 1254864
STAT items:21:hits_to_cold 47241
STAT items:21:hits_to_temp 0
STAT items:22:number 1067
STAT items:22:number_hot 211
STAT items:22:number_warm 314
STAT items:22:number_cold 542
STAT items:22:age_hot 154
STAT items:22:age_warm 11745
STAT items:22:age 12951
STAT items:22:mem_requested 10530706
STAT items:22:evicted 0
STAT items:22:evicted_nonzero 0
STAT items:22:evicted_time 0
STAT items:22:outofmemory 0
STAT items:22:tailrepairs 0
STAT items:22:reclaimed 638
STAT items:22:expired_unfetched 672629
STAT items:22:evicted_unfetched 0
STAT items:22:evicted_active 0
STAT items:22:crawler_reclaimed 1027892
STAT items:22:crawler_items_checked 87064938
STAT items:22:lrutail_reflocked 1
STAT items:22:moves_to_cold 736594
STAT items:22:moves_to_warm 130737
STAT items:22:moves_within_lru 1121
STAT items:22:direct_reclaims 0
STAT items:22:hits_to_hot 6162490
STAT items:22:hits_to_warm 23694192
STAT items:22:hits_to_cold 41457
STAT items:22:hits_to_temp 0
STAT items:23:number 804
STAT items:23:number_hot 160
STAT items:23:number_warm 217
STAT items:23:number_cold 427
STAT items:23:age_hot 130
STAT items:23:age_warm 13418
STAT items:23:age 13974
STAT items:23:mem_requested 9950127
STAT items:23:evicted 0
STAT items:23:evicted_nonzero 0
STAT items:23:evicted_time 0
STAT items:23:outofmemory 0
STAT items:23:tailrepairs 0
STAT items:23:reclaimed 567
STAT items:23:expired_unfetched 491873
STAT items:23:evicted_unfetched 0
STAT items:23:evicted_active 0
STAT items:23:crawler_reclaimed 797595
STAT items:23:crawler_items_checked 66243506
STAT items:23:lrutail_reflocked 0
STAT items:23:moves_to_cold 560940
STAT items:23:moves_to_warm 120254
STAT items:23:moves_within_lru 1137
STAT items:23:direct_reclaims 0
STAT items:23:hits_to_hot 3185934
STAT items:23:hits_to_warm 245381
STAT items:23:hits_to_cold 44228
STAT items:23:hits_to_temp 0
STAT items:24:number 480
STAT items:24:number_hot 94
STAT items:24:number_warm 152
STAT items:24:number_cold 234
STAT items:24:age_hot 161
STAT items:24:age_warm 11300
STAT items:24:age 13575
STAT items:24:mem_requested 7423936
STAT items:24:evicted 0
STAT items:24:evicted_nonzero 0
STAT items:24:evicted_time 0
STAT items:24:outofmemory 0
STAT items:24:tailrepairs 0
STAT items:24:reclaimed 236
STAT items:24:expired_unfetched 278984
STAT items:24:evicted_unfetched 0
STAT items:24:evicted_active 0
STAT items:24:crawler_reclaimed 515501
STAT items:24:crawler_items_checked 41942212
STAT items:24:lrutail_reflocked 0
STAT items:24:moves_to_cold 320772
STAT items:24:moves_to_warm 114885
STAT items:24:moves_within_lru 950
STAT items:24:direct_reclaims 0
STAT items:24:hits_to_hot 3785209
STAT items:24:hits_to_warm 311829
STAT items:24:hits_to_cold 42560
STAT items:24:hits_to_temp 0
STAT items:25:number 264
STAT items:25:number_hot 52
STAT items:25:number_warm 73
STAT items:25:number_cold 139
STAT items:25:age_hot 132
STAT items:25:age_warm 13992
STAT items:25:age 19012
STAT items:25:mem_requested 5040362
STAT items:25:evicted 0
STAT items:25:evicted_nonzero 0
STAT items:25:evicted_time 0
STAT items:25:outofmemory 0
STAT items:25:tailrepairs 0
STAT items:25:reclaimed 143
STAT items:25:expired_unfetched 138430
STAT items:25:evicted_unfetched 0
STAT items:25:evicted_active 0
STAT items:25:crawler_reclaimed 245723
STAT items:25:crawler_items_checked 18852736
STAT items:25:lrutail_reflocked 0
STAT items:25:moves_to_cold 146919
STAT items:25:moves_to_warm 42684
STAT items:25:moves_within_lru 468
STAT items:25:direct_reclaims 0
STAT items:25:hits_to_hot 2979579
STAT items:25:hits_to_warm 89470
STAT items:25:hits_to_cold 30386
STAT items:25:hits_to_temp 0
STAT items:26:number 138
STAT items:26:number_hot 18
STAT items:26:number_warm 41
STAT items:26:number_cold 79
STAT items:26:age_hot 123
STAT items:26:age_warm 13624
STAT items:26:age 16214
STAT items:26:mem_requested 3346427
STAT items:26:evicted 0
STAT items:26:evicted_nonzero 0
STAT items:26:evicted_time 0
STAT items:26:outofmemory 0
STAT items:26:tailrepairs 0
STAT items:26:reclaimed 675
STAT items:26:expired_unfetched 85159
STAT items:26:evicted_unfetched 0
STAT items:26:evicted_active 0
STAT items:26:crawler_reclaimed 137424
STAT items:26:crawler_items_checked 9499777
STAT items:26:lrutail_reflocked 0
STAT items:26:moves_to_cold 83677
STAT items:26:moves_to_warm 16731
STAT items:26:moves_within_lru 267
STAT items:26:direct_reclaims 0
STAT items:26:hits_to_hot 1123452
STAT items:26:hits_to_warm 56916
STAT items:26:hits_to_cold 20777
STAT items:26:hits_to_temp 0
STAT items:27:number 74
STAT items:27:number_hot 12
STAT items:27:number_warm 18
STAT items:27:number_cold 44
STAT items:27:age_hot 63
STAT items:27:age_warm 15248
STAT items:27:age 21042
STAT items:27:mem_requested 2206106
STAT items:27:evicted 0
STAT items:27:evicted_nonzero 0
STAT items:27:evicted_time 0
STAT items:27:outofmemory 0
STAT items:27:tailrepairs 0
STAT items:27:reclaimed 5817
STAT items:27:expired_unfetched 48225
STAT items:27:evicted_unfetched 0
STAT items:27:evicted_active 0
STAT items:27:crawler_reclaimed 68656
STAT items:27:crawler_items_checked 4511468
STAT items:27:lrutail_reflocked 0
STAT items:27:moves_to_cold 49945
STAT items:27:moves_to_warm 8589
STAT items:27:moves_within_lru 305
STAT items:27:direct_reclaims 0
STAT items:27:hits_to_hot 327186
STAT items:27:hits_to_warm 46406
STAT items:27:hits_to_cold 14143
STAT items:27:hits_to_temp 0
STAT items:28:number 56
STAT items:28:number_hot 10
STAT items:28:number_warm 11
STAT items:28:number_cold 35
STAT items:28:age_hot 80
STAT items:28:age_warm 17499
STAT items:28:age 26523
STAT items:28:mem_requested 2103497
STAT items:28:evicted 0
STAT items:28:evicted_nonzero 0
STAT items:28:evicted_time 0
STAT items:28:outofmemory 0
STAT items:28:tailrepairs 0
STAT items:28:reclaimed 9415
STAT items:28:expired_unfetched 25004
STAT items:28:evicted_unfetched 0
STAT items:28:evicted_active 0
STAT items:28:crawler_reclaimed 27547
STAT items:28:crawler_items_checked 1751064
STAT items:28:lrutail_reflocked 0
STAT items:28:moves_to_cold 23134
STAT items:28:moves_to_warm 4036
STAT items:28:moves_within_lru 236
STAT items:28:direct_reclaims 0
STAT items:28:hits_to_hot 213412
STAT items:28:hits_to_warm 23993
STAT items:28:hits_to_cold 7125
STAT items:28:hits_to_temp 0
STAT items:29:number 33
STAT items:29:number_hot 6
STAT items:29:number_warm 6
STAT items:29:number_cold 21
STAT items:29:age_hot 29
STAT items:29:age_warm 3934
STAT items:29:age 35647
STAT items:29:mem_requested 1577678
STAT items:29:evicted 0
STAT items:29:evicted_nonzero 0
STAT items:29:evicted_time 0
STAT items:29:outofmemory 0
STAT items:29:tailrepairs 0
STAT items:29:reclaimed 7829
STAT items:29:expired_unfetched 13636
STAT items:29:evicted_unfetched 0
STAT items:29:evicted_active 0
STAT items:29:crawler_reclaimed 11182
STAT items:29:crawler_items_checked 471573
STAT items:29:lrutail_reflocked 25
STAT items:29:moves_to_cold 9752
STAT items:29:moves_to_warm 1948
STAT items:29:moves_within_lru 7878
STAT items:29:direct_reclaims 0
STAT items:29:hits_to_hot 32483
STAT items:29:hits_to_warm 1480528
STAT items:29:hits_to_cold 2992
STAT items:29:hits_to_temp 0
STAT items:30:number 29
STAT items:30:number_hot 4
STAT items:30:number_warm 3
STAT items:30:number_cold 22
STAT items:30:age_hot 63
STAT items:30:age_warm 7730
STAT items:30:age 27486
STAT items:30:mem_requested 1650302
STAT items:30:evicted 0
STAT items:30:evicted_nonzero 0
STAT items:30:evicted_time 0
STAT items:30:outofmemory 0
STAT items:30:tailrepairs 0
STAT items:30:reclaimed 3614
STAT items:30:expired_unfetched 8230
STAT items:30:evicted_unfetched 0
STAT items:30:evicted_active 0
STAT items:30:crawler_reclaimed 7544
STAT items:30:crawler_items_checked 241248
STAT items:30:lrutail_reflocked 0
STAT items:30:moves_to_cold 5143
STAT items:30:moves_to_warm 975
STAT items:30:moves_within_lru 61
STAT items:30:direct_reclaims 0
STAT items:30:hits_to_hot 64157
STAT items:30:hits_to_warm 228718
STAT items:30:hits_to_cold 970
STAT items:30:hits_to_temp 0
STAT items:31:number 13
STAT items:31:number_hot 2
STAT items:31:number_warm 1
STAT items:31:number_cold 10
STAT items:31:age_hot 15
STAT items:31:age_warm 314
STAT items:31:age 10518
STAT items:31:mem_requested 958734
STAT items:31:evicted 0
STAT items:31:evicted_nonzero 0
STAT items:31:evicted_time 0
STAT items:31:outofmemory 0
STAT items:31:tailrepairs 0
STAT items:31:reclaimed 3071
STAT items:31:expired_unfetched 4328
STAT items:31:evicted_unfetched 0
STAT items:31:evicted_active 0
STAT items:31:crawler_reclaimed 2872
STAT items:31:crawler_items_checked 39534
STAT items:31:lrutail_reflocked 0
STAT items:31:moves_to_cold 3085
STAT items:31:moves_to_warm 718
STAT items:31:moves_within_lru 300
STAT items:31:direct_reclaims 0
STAT items:31:hits_to_hot 2774
STAT items:31:hits_to_warm 795
STAT items:31:hits_to_cold 723
STAT items:31:hits_to_temp 0
STAT items:32:number 4
STAT items:32:number_hot 0
STAT items:32:number_warm 0
STAT items:32:number_cold 4
STAT items:32:age_hot 0
STAT items:32:age_warm 0
STAT items:32:age 12455
STAT items:32:mem_requested 367955
STAT items:32:evicted 0
STAT items:32:evicted_nonzero 0
STAT items:32:evicted_time 0
STAT items:32:outofmemory 0
STAT items:32:tailrepairs 0
STAT items:32:reclaimed 4776
STAT items:32:expired_unfetched 4432
STAT items:32:evicted_unfetched 0
STAT items:32:evicted_active 0
STAT items:32:crawler_reclaimed 1816
STAT items:32:crawler_items_checked 8858
STAT items:32:lrutail_reflocked 1
STAT items:32:moves_to_cold 3693
STAT items:32:moves_to_warm 1245
STAT items:32:moves_within_lru 594
STAT items:32:direct_reclaims 0
STAT items:32:hits_to_hot 3908
STAT items:32:hits_to_warm 769
STAT items:32:hits_to_cold 1163
STAT items:32:hits_to_temp 0
STAT items:33:number 2
STAT items:33:number_hot 0
STAT items:33:number_warm 0
STAT items:33:number_cold 2
STAT items:33:age_hot 0
STAT items:33:age_warm 0
STAT items:33:age 105
STAT items:33:mem_requested 215194
STAT items:33:evicted 0
STAT items:33:evicted_nonzero 0
STAT items:33:evicted_time 0
STAT items:33:outofmemory 0
STAT items:33:tailrepairs 0
STAT items:33:reclaimed 1716
STAT items:33:expired_unfetched 1881
STAT items:33:evicted_unfetched 0
STAT items:33:evicted_active 0
STAT items:33:crawler_reclaimed 945
STAT items:33:crawler_items_checked 2594
STAT items:33:lrutail_reflocked 2
STAT items:33:moves_to_cold 2417
STAT items:33:moves_to_warm 456
STAT items:33:moves_within_lru 296
STAT items:33:direct_reclaims 0
STAT items:33:hits_to_hot 609
STAT items:33:hits_to_warm 320
STAT items:33:hits_to_cold 824
STAT items:33:hits_to_temp 0
END
stats slabs
STAT 1:chunk_size 96
STAT 1:chunks_per_page 10922
STAT 1:total_pages 1
STAT 1:total_chunks 10922
STAT 1:used_chunks 2646
STAT 1:free_chunks 8276
STAT 1:free_chunks_end 0
STAT 1:get_hits 7354283
STAT 1:cmd_set 11367085
STAT 1:delete_hits 11216788
STAT 1:incr_hits 958534
STAT 1:decr_hits 0
STAT 1:cas_hits 0
STAT 1:cas_badval 0
STAT 1:touch_hits 0
STAT 2:chunk_size 120
STAT 2:chunks_per_page 8738
STAT 2:total_pages 37
STAT 2:total_chunks 323306
STAT 2:used_chunks 307266
STAT 2:free_chunks 16040
STAT 2:free_chunks_end 0
STAT 2:get_hits 596274827
STAT 2:cmd_set 13197787
STAT 2:delete_hits 2700
STAT 2:incr_hits 123
STAT 2:decr_hits 0
STAT 2:cas_hits 0
STAT 2:cas_badval 0
STAT 2:touch_hits 0
STAT 3:chunk_size 152
STAT 3:chunks_per_page 6898
STAT 3:total_pages 36
STAT 3:total_chunks 248328
STAT 3:used_chunks 239492
STAT 3:free_chunks 8836
STAT 3:free_chunks_end 0
STAT 3:get_hits 115444350
STAT 3:cmd_set 1952813
STAT 3:delete_hits 7489
STAT 3:incr_hits 0
STAT 3:decr_hits 0
STAT 3:cas_hits 0
STAT 3:cas_badval 0
STAT 3:touch_hits 0
STAT 4:chunk_size 192
STAT 4:chunks_per_page 5461
STAT 4:total_pages 2
STAT 4:total_chunks 10922
STAT 4:used_chunks 2094
STAT 4:free_chunks 8828
STAT 4:free_chunks_end 0
STAT 4:get_hits 120049773
STAT 4:cmd_set 331584
STAT 4:delete_hits 5235
STAT 4:incr_hits 0
STAT 4:decr_hits 0
STAT 4:cas_hits 0
STAT 4:cas_badval 0
STAT 4:touch_hits 0
STAT 5:chunk_size 240
STAT 5:chunks_per_page 4369
STAT 5:total_pages 2
STAT 5:total_chunks 8738
STAT 5:used_chunks 2234
STAT 5:free_chunks 6504
STAT 5:free_chunks_end 0
STAT 5:get_hits 3073559
STAT 5:cmd_set 305572
STAT 5:delete_hits 8617
STAT 5:incr_hits 0
STAT 5:decr_hits 0
STAT 5:cas_hits 0
STAT 5:cas_badval 0
STAT 5:touch_hits 0
STAT 6:chunk_size 304
STAT 6:chunks_per_page 3449
STAT 6:total_pages 3
STAT 6:total_chunks 10347
STAT 6:used_chunks 3266
STAT 6:free_chunks 7081
STAT 6:free_chunks_end 0
STAT 6:get_hits 34937911
STAT 6:cmd_set 430010
STAT 6:delete_hits 2551
STAT 6:incr_hits 0
STAT 6:decr_hits 0
STAT 6:cas_hits 0
STAT 6:cas_badval 0
STAT 6:touch_hits 0
STAT 7:chunk_size 384
STAT 7:chunks_per_page 2730
STAT 7:total_pages 4
STAT 7:total_chunks 10920
STAT 7:used_chunks 4645
STAT 7:free_chunks 6275
STAT 7:free_chunks_end 0
STAT 7:get_hits 39764998
STAT 7:cmd_set 427271
STAT 7:delete_hits 1875
STAT 7:incr_hits 0
STAT 7:decr_hits 0
STAT 7:cas_hits 0
STAT 7:cas_badval 0
STAT 7:touch_hits 0
STAT 8:chunk_size 480
STAT 8:chunks_per_page 2184
STAT 8:total_pages 6
STAT 8:total_chunks 13104
STAT 8:used_chunks 8420
STAT 8:free_chunks 4684
STAT 8:free_chunks_end 0
STAT 8:get_hits 5263317
STAT 8:cmd_set 508599
STAT 8:delete_hits 1786
STAT 8:incr_hits 0
STAT 8:decr_hits 0
STAT 8:cas_hits 0
STAT 8:cas_badval 0
STAT 8:touch_hits 0
STAT 9:chunk_size 600
STAT 9:chunks_per_page 1747
STAT 9:total_pages 12
STAT 9:total_chunks 20964
STAT 9:used_chunks 16901
STAT 9:free_chunks 4063
STAT 9:free_chunks_end 0
STAT 9:get_hits 68345190
STAT 9:cmd_set 360638
STAT 9:delete_hits 2374
STAT 9:incr_hits 0
STAT 9:decr_hits 0
STAT 9:cas_hits 0
STAT 9:cas_badval 0
STAT 9:touch_hits 0
STAT 10:chunk_size 752
STAT 10:chunks_per_page 1394
STAT 10:total_pages 17
STAT 10:total_chunks 23698
STAT 10:used_chunks 21655
STAT 10:free_chunks 2043
STAT 10:free_chunks_end 0
STAT 10:get_hits 33144248
STAT 10:cmd_set 1137402
STAT 10:delete_hits 2230
STAT 10:incr_hits 0
STAT 10:decr_hits 0
STAT 10:cas_hits 0
STAT 10:cas_badval 0
STAT 10:touch_hits 0
STAT 11:chunk_size 944
STAT 11:chunks_per_page 1110
STAT 11:total_pages 36
STAT 11:total_chunks 39960
STAT 11:used_chunks 37797
STAT 11:free_chunks 2163
STAT 11:free_chunks_end 0
STAT 11:get_hits 96302943
STAT 11:cmd_set 6586856
STAT 11:delete_hits 22279
STAT 11:incr_hits 0
STAT 11:decr_hits 0
STAT 11:cas_hits 0
STAT 11:cas_badval 0
STAT 11:touch_hits 0
STAT 12:chunk_size 1184
STAT 12:chunks_per_page 885
STAT 12:total_pages 195
STAT 12:total_chunks 172575
STAT 12:used_chunks 170700
STAT 12:free_chunks 1875
STAT 12:free_chunks_end 0
STAT 12:get_hits 99864187
STAT 12:cmd_set 3215169
STAT 12:delete_hits 219006
STAT 12:incr_hits 0
STAT 12:decr_hits 0
STAT 12:cas_hits 0
STAT 12:cas_badval 0
STAT 12:touch_hits 0
STAT 13:chunk_size 1480
STAT 13:chunks_per_page 708
STAT 13:total_pages 159
STAT 13:total_chunks 112572
STAT 13:used_chunks 111339
STAT 13:free_chunks 1233
STAT 13:free_chunks_end 0
STAT 13:get_hits 119783673
STAT 13:cmd_set 2373142
STAT 13:delete_hits 186969
STAT 13:incr_hits 0
STAT 13:decr_hits 0
STAT 13:cas_hits 0
STAT 13:cas_badval 0
STAT 13:touch_hits 0
STAT 14:chunk_size 1856
STAT 14:chunks_per_page 564
STAT 14:total_pages 176
STAT 14:total_chunks 99264
STAT 14:used_chunks 98442
STAT 14:free_chunks 822
STAT 14:free_chunks_end 0
STAT 14:get_hits 127265213
STAT 14:cmd_set 2259356
STAT 14:delete_hits 94899
STAT 14:incr_hits 0
STAT 14:decr_hits 0
STAT 14:cas_hits 0
STAT 14:cas_badval 0
STAT 14:touch_hits 0
STAT 15:chunk_size 2320
STAT 15:chunks_per_page 451
STAT 15:total_pages 101
STAT 15:total_chunks 45551
STAT 15:used_chunks 44837
STAT 15:free_chunks 714
STAT 15:free_chunks_end 0
STAT 15:get_hits 61971296
STAT 15:cmd_set 1309035
STAT 15:delete_hits 51322
STAT 15:incr_hits 0
STAT 15:decr_hits 0
STAT 15:cas_hits 0
STAT 15:cas_badval 0
STAT 15:touch_hits 0
STAT 16:chunk_size 2904
STAT 16:chunks_per_page 361
STAT 16:total_pages 19
STAT 16:total_chunks 6859
STAT 16:used_chunks 6068
STAT 16:free_chunks 791
STAT 16:free_chunks_end 0
STAT 16:get_hits 16675010
STAT 16:cmd_set 419872
STAT 16:delete_hits 27128
STAT 16:incr_hits 0
STAT 16:decr_hits 0
STAT 16:cas_hits 0
STAT 16:cas_badval 0
STAT 16:touch_hits 0
STAT 17:chunk_size 3632
STAT 17:chunks_per_page 288
STAT 17:total_pages 14
STAT 17:total_chunks 4032
STAT 17:used_chunks 3420
STAT 17:free_chunks 612
STAT 17:free_chunks_end 0
STAT 17:get_hits 29083384
STAT 17:cmd_set 21047075
STAT 17:delete_hits 22492
STAT 17:incr_hits 0
STAT 17:decr_hits 0
STAT 17:cas_hits 0
STAT 17:cas_badval 0
STAT 17:touch_hits 0
STAT 18:chunk_size 4544
STAT 18:chunks_per_page 230
STAT 18:total_pages 12
STAT 18:total_chunks 2760
STAT 18:used_chunks 2290
STAT 18:free_chunks 470
STAT 18:free_chunks_end 0
STAT 18:get_hits 20754389
STAT 18:cmd_set 367731
STAT 18:delete_hits 25641
STAT 18:incr_hits 0
STAT 18:decr_hits 0
STAT 18:cas_hits 0
STAT 18:cas_badval 0
STAT 18:touch_hits 0
STAT 19:chunk_size 5680
STAT 19:chunks_per_page 184
STAT 19:total_pages 11
STAT 19:total_chunks 2024
STAT 19:used_chunks 1689
STAT 19:free_chunks 335
STAT 19:free_chunks_end 0
STAT 19:get_hits 5115520
STAT 19:cmd_set 724755
STAT 19:delete_hits 29793
STAT 19:incr_hits 0
STAT 19:decr_hits 0
STAT 19:cas_hits 0
STAT 19:cas_badval 0
STAT 19:touch_hits 0
STAT 20:chunk_size 7104
STAT 20:chunks_per_page 147
STAT 20:total_pages 12
STAT 20:total_chunks 1764
STAT 20:used_chunks 1587
STAT 20:free_chunks 177
STAT 20:free_chunks_end 0
STAT 20:get_hits 11820041
STAT 20:cmd_set 1249628
STAT 20:delete_hits 50322
STAT 20:incr_hits 0
STAT 20:decr_hits 0
STAT 20:cas_hits 0
STAT 20:cas_badval 0
STAT 20:touch_hits 0
STAT 21:chunk_size 8880
STAT 21:chunks_per_page 118
STAT 21:total_pages 14
STAT 21:total_chunks 1652
STAT 21:used_chunks 1503
STAT 21:free_chunks 149
STAT 21:free_chunks_end 0
STAT 21:get_hits 7632248
STAT 21:cmd_set 1415413
STAT 21:delete_hits 95532
STAT 21:incr_hits 0
STAT 21:decr_hits 0
STAT 21:cas_hits 0
STAT 21:cas_badval 0
STAT 21:touch_hits 0
STAT 22:chunk_size 11104
STAT 22:chunks_per_page 94
STAT 22:total_pages 13
STAT 22:total_chunks 1222
STAT 22:used_chunks 1073
STAT 22:free_chunks 149
STAT 22:free_chunks_end 0
STAT 22:get_hits 29898252
STAT 22:cmd_set 1191405
STAT 22:delete_hits 50009
STAT 22:incr_hits 0
STAT 22:decr_hits 0
STAT 22:cas_hits 0
STAT 22:cas_badval 0
STAT 22:touch_hits 0
STAT 23:chunk_size 13880
STAT 23:chunks_per_page 75
STAT 23:total_pages 12
STAT 23:total_chunks 900
STAT 23:used_chunks 814
STAT 23:free_chunks 86
STAT 23:free_chunks_end 0
STAT 23:get_hits 3475766
STAT 23:cmd_set 950313
STAT 23:delete_hits 20459
STAT 23:incr_hits 0
STAT 23:decr_hits 0
STAT 23:cas_hits 0
STAT 23:cas_badval 0
STAT 23:touch_hits 0
STAT 24:chunk_size 17352
STAT 24:chunks_per_page 60
STAT 24:total_pages 10
STAT 24:total_chunks 600
STAT 24:used_chunks 483
STAT 24:free_chunks 117
STAT 24:free_chunks_end 0
STAT 24:get_hits 4139606
STAT 24:cmd_set 620495
STAT 24:delete_hits 12609
STAT 24:incr_hits 0
STAT 24:decr_hits 0
STAT 24:cas_hits 0
STAT 24:cas_badval 0
STAT 24:touch_hits 0
STAT 25:chunk_size 21696
STAT 25:chunks_per_page 48
STAT 25:total_pages 7
STAT 25:total_chunks 336
STAT 25:used_chunks 265
STAT 25:free_chunks 71
STAT 25:free_chunks_end 0
STAT 25:get_hits 3099435
STAT 25:cmd_set 328219
STAT 25:delete_hits 2405
STAT 25:incr_hits 0
STAT 25:decr_hits 0
STAT 25:cas_hits 0
STAT 25:cas_badval 0
STAT 25:touch_hits 0
STAT 26:chunk_size 27120
STAT 26:chunks_per_page 38
STAT 26:total_pages 6
STAT 26:total_chunks 228
STAT 26:used_chunks 137
STAT 26:free_chunks 91
STAT 26:free_chunks_end 0
STAT 26:get_hits 1201145
STAT 26:cmd_set 430944
STAT 26:delete_hits 156
STAT 26:incr_hits 0
STAT 26:decr_hits 0
STAT 26:cas_hits 0
STAT 26:cas_badval 0
STAT 26:touch_hits 0
STAT 27:chunk_size 33904
STAT 27:chunks_per_page 30
STAT 27:total_pages 4
STAT 27:total_chunks 120
STAT 27:used_chunks 74
STAT 27:free_chunks 46
STAT 27:free_chunks_end 0
STAT 27:get_hits 387740
STAT 27:cmd_set 118922
STAT 27:delete_hits 21
STAT 27:incr_hits 0
STAT 27:decr_hits 0
STAT 27:cas_hits 0
STAT 27:cas_badval 0
STAT 27:touch_hits 0
STAT 28:chunk_size 42384
STAT 28:chunks_per_page 24
STAT 28:total_pages 3
STAT 28:total_chunks 72
STAT 28:used_chunks 57
STAT 28:free_chunks 15
STAT 28:free_chunks_end 0
STAT 28:get_hits 244530
STAT 28:cmd_set 91636
STAT 28:delete_hits 14
STAT 28:incr_hits 0
STAT 28:decr_hits 0
STAT 28:cas_hits 0
STAT 28:cas_badval 0
STAT 28:touch_hits 0
STAT 29:chunk_size 52984
STAT 29:chunks_per_page 19
STAT 29:total_pages 3
STAT 29:total_chunks 57
STAT 29:used_chunks 33
STAT 29:free_chunks 24
STAT 29:free_chunks_end 0
STAT 29:get_hits 1516024
STAT 29:cmd_set 26257
STAT 29:delete_hits 10
STAT 29:incr_hits 0
STAT 29:decr_hits 0
STAT 29:cas_hits 0
STAT 29:cas_badval 0
STAT 29:touch_hits 0
STAT 30:chunk_size 66232
STAT 30:chunks_per_page 15
STAT 30:total_pages 3
STAT 30:total_chunks 45
STAT 30:used_chunks 29
STAT 30:free_chunks 16
STAT 30:free_chunks_end 0
STAT 30:get_hits 293845
STAT 30:cmd_set 15476
STAT 30:delete_hits 1
STAT 30:incr_hits 0
STAT 30:decr_hits 0
STAT 30:cas_hits 0
STAT 30:cas_badval 0
STAT 30:touch_hits 0
STAT 31:chunk_size 82792
STAT 31:chunks_per_page 12
STAT 31:total_pages 2
STAT 31:total_chunks 24
STAT 31:used_chunks 13
STAT 31:free_chunks 11
STAT 31:free_chunks_end 0
STAT 31:get_hits 4292
STAT 31:cmd_set 6858
STAT 31:delete_hits 2
STAT 31:incr_hits 0
STAT 31:decr_hits 0
STAT 31:cas_hits 0
STAT 31:cas_badval 0
STAT 31:touch_hits 0
STAT 32:chunk_size 103496
STAT 32:chunks_per_page 10
STAT 32:total_pages 2
STAT 32:total_chunks 20
STAT 32:used_chunks 4
STAT 32:free_chunks 16
STAT 32:free_chunks_end 0
STAT 32:get_hits 5840
STAT 32:cmd_set 9824
STAT 32:delete_hits 0
STAT 32:incr_hits 0
STAT 32:decr_hits 0
STAT 32:cas_hits 0
STAT 32:cas_badval 0
STAT 32:touch_hits 0
STAT 33:chunk_size 129376
STAT 33:chunks_per_page 8
STAT 33:total_pages 2
STAT 33:total_chunks 16
STAT 33:used_chunks 2
STAT 33:free_chunks 14
STAT 33:free_chunks_end 0
STAT 33:get_hits 1753
STAT 33:cmd_set 3308
STAT 33:delete_hits 0
STAT 33:incr_hits 0
STAT 33:decr_hits 0
STAT 33:cas_hits 0
STAT 33:cas_badval 0
STAT 33:touch_hits 0
STAT 34:chunk_size 161720
STAT 34:chunks_per_page 6
STAT 34:total_pages 2
STAT 34:total_chunks 12
STAT 34:used_chunks 0
STAT 34:free_chunks 12
STAT 34:free_chunks_end 0
STAT 34:get_hits 837
STAT 34:cmd_set 1584
STAT 34:delete_hits 0
STAT 34:incr_hits 0
STAT 34:decr_hits 0
STAT 34:cas_hits 0
STAT 34:cas_badval 0
STAT 34:touch_hits 0
STAT 35:chunk_size 202152
STAT 35:chunks_per_page 5
STAT 35:total_pages 2
STAT 35:total_chunks 10
STAT 35:used_chunks 0
STAT 35:free_chunks 10
STAT 35:free_chunks_end 0
STAT 35:get_hits 686
STAT 35:cmd_set 1019
STAT 35:delete_hits 0
STAT 35:incr_hits 0
STAT 35:decr_hits 0
STAT 35:cas_hits 0
STAT 35:cas_badval 0
STAT 35:touch_hits 0
STAT 36:chunk_size 252696
STAT 36:chunks_per_page 4
STAT 36:total_pages 2
STAT 36:total_chunks 8
STAT 36:used_chunks 0
STAT 36:free_chunks 8
STAT 36:free_chunks_end 0
STAT 36:get_hits 293
STAT 36:cmd_set 414
STAT 36:delete_hits 0
STAT 36:incr_hits 0
STAT 36:decr_hits 0
STAT 36:cas_hits 0
STAT 36:cas_badval 0
STAT 36:touch_hits 0
STAT 37:chunk_size 315872
STAT 37:chunks_per_page 3
STAT 37:total_pages 1
STAT 37:total_chunks 3
STAT 37:used_chunks 0
STAT 37:free_chunks 3
STAT 37:free_chunks_end 0
STAT 37:get_hits 118
STAT 37:cmd_set 148
STAT 37:delete_hits 0
STAT 37:incr_hits 0
STAT 37:decr_hits 0
STAT 37:cas_hits 0
STAT 37:cas_badval 0
STAT 37:touch_hits 0
STAT 38:chunk_size 394840
STAT 38:chunks_per_page 2
STAT 38:total_pages 2
STAT 38:total_chunks 4
STAT 38:used_chunks 0
STAT 38:free_chunks 4
STAT 38:free_chunks_end 0
STAT 38:get_hits 183
STAT 38:cmd_set 208
STAT 38:delete_hits 0
STAT 38:incr_hits 0
STAT 38:decr_hits 0
STAT 38:cas_hits 0
STAT 38:cas_badval 0
STAT 38:touch_hits 0
STAT 39:chunk_size 524288
STAT 39:chunks_per_page 2
STAT 39:total_pages 2
STAT 39:total_chunks 4
STAT 39:used_chunks 0
STAT 39:free_chunks 4
STAT 39:free_chunks_end 0
STAT 39:get_hits 3854
STAT 39:cmd_set 3605
STAT 39:delete_hits 0
STAT 39:incr_hits 0
STAT 39:decr_hits 0
STAT 39:cas_hits 0
STAT 39:cas_badval 0
STAT 39:touch_hits 0
STAT active_slabs 39
STAT total_malloced<PHONE_NUMBER>
END

Thanks. Have you seen any segfaults since your original ones?

Thanks. Have you seen any segfaults since your original ones?

No.
After reboot currently no segfaults.

Same version right, .20?
Lets keep this open for a while longer; please update with as much detail as possible if you see the issue again.
It's possible that the server ran out of memory, but validating that is pretty hard and depends on a lot of settings (overcommit/etc). You also didn't see anything obvious in the kernel messages history.
The hash table in memcached is a huge allocation that doesn't get used all at once, so it's possible to allocate it, get a sparse allocation, then if it's still filling when the system runs out of memory it can give you a really odd looking segfault. All other allocations are small or quickly written to, so they tend to get backed by real memory.
I may have to just add a memset to the hashtable allocation to force this, as the calloc that's there now may not be doing it. HT maintenance is already in a side thread so it may be okay.

memcached just SEGV on a totally different server with completely different specs
so I think there is a software problem rather than hardware problem
this occurred a few hours after a huge event which caused nearly full cpu utilization on this server
Here is the /var/log/messages log
Dec  8 06:12:37 [HOSTNAME_REDACTED] kernel: traps: memcached[15525] general protection ip:416b76 sp:7f65427fbdc0 error:0 in memcached[400000+35000]
Dec  8 06:12:37 [HOSTNAME_REDACTED] systemd: memcached.service: main process exited, code=killed, status=11/SEGV
Dec  8 06:12:37 [HOSTNAME_REDACTED] systemd: Unit memcached.service entered failed state.
Dec  8 06:12:37 [HOSTNAME_REDACTED] systemd: memcached.service failed.
Dec  8 06:12:59 [HOSTNAME_REDACTED] monit: 'memcached' process is not running
Dec  8 06:12:59 [HOSTNAME_REDACTED] monit: 'memcached' trying to restart
Dec  8 06:12:59 [HOSTNAME_REDACTED] monit: 'memcached' start: '/usr/bin/systemctl start memcached'
Dec  8 06:12:59 [HOSTNAME_REDACTED] systemd: Started Memcached.
Dec  8 06:13:30 [HOSTNAME_REDACTED] monit: 'memcached' process is running with pid 16515
No other related logs at all
CPU:
Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
Memory:
64 GB
Memcached config:
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="12288"
OPTIONS="-l [IP_REDACTED]"
Last memcached restart:
Dec  6 21:58:38 [HOSTNAME_REDACTED] systemd: Stopped Memcached.
Dec  6 21:58:38 [HOSTNAME_REDACTED] systemd: Started Memcached.
Only about 2 days ago
Yum logs:
cat /var/log/yum.log|grep memcached
Nov 16 21:20:40 Updated: memcached-1.5.20-1.el7.remi.x86_64
Dec 05 14:28:01 Updated: php73-php-pecl-memcached-3.1.5-1.el7.remi.x86_64
Notes:
I have attached a whole bunch of memcached monitoring graphs(plotted every 5 minutes by munin with same timezone as above) and cpu and memory graphs
The server has PHP running at the same time, php-fpm version 7.3.12
The server had a very high CPU utilization load(due to php) a few hours before this happened

One question is for the above workload would you recommend default thread settings or increase with -t option?

Dying in a different spot, need an addr2line again, if you or someone would like to assist.
For your workload -t 4 is fine. 4 worker threads can do a million unpipelined ops per second. It wouldn't crash if it weren't fine, as well.
is the access pattern any different on this host/workload than from the original one?
need to think about this a bit. are these machines dedicated to memcached or do they run any other apps as well?

need to think about this a bit. are these machines dedicated to memcached or do they run any other apps as well?

Like I mentioned in the above long bug report, both are running php and memcached.
Both are the same app and php is load balanced with similar random traffic, memcached is used with ketama consistent hashing library so there should be similar traffic.
addr2line -e /usr/bin/memcached 416b76
output is (again)
??:0

Ah I missed those few Notes: lines in your graphs post.
could you provide just a 'stats' and 'stats settings' output for this latest workload to see a crash? I'd like to see if they differ much.
I was referring to a non-stripped binary for the addr2line, maybe @thatsafunnyname can save the day again :P I'm juggling too much stuff to get a centos7 instance for a few days.
I've rusted out on redhat-related, but it should be possible to install a debuginfo package matching the memcached 1.5.20 binary, and get the addr2line to work from there. Shouldn't need to build from the source rpm or anything. I don't know the exact steps offhand, if you can get some help you can get the info sooner. Else I'll get to it in a few days.
1.5.20's been surviving my testing so far.

could you provide just a 'stats' and 'stats settings' output for this latest workload to see a crash? I'd like to see if they differ much.

I didn't understand "to see a crash" but here is the output in the currently running server that SEGV recently for stats and stats setting
stats
STAT pid 16515
STAT uptime 120542
STAT time<PHONE_NUMBER>
STAT version 1.5.20
STAT libevent 2.0.21-stable
STAT pointer_size 64
STAT rusage_user 10539.798145
STAT rusage_system 19908.911663
STAT max_connections 1024
STAT curr_connections 127
STAT total_connections 127119219
STAT rejected_connections 0
STAT connection_structures 258
STAT reserved_fds 20
STAT cmd_get<PHONE_NUMBER>
STAT cmd_set 185752873
STAT cmd_flush 0
STAT cmd_touch 0
STAT cmd_meta 0
STAT get_hits<PHONE_NUMBER>
STAT get_misses 72691041
STAT get_expired 1685024
STAT get_flushed 0
STAT delete_misses 31955823
STAT delete_hits 8657707
STAT incr_misses 16
STAT incr_hits 568695
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT touch_hits 0
STAT touch_misses 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read<PHONE_NUMBER>08
STAT bytes_written<PHONE_NUMBER>308
STAT limit_maxbytes<PHONE_NUMBER>8
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT time_in_listen_disabled_us 0
STAT threads 4
STAT conn_yields 0
STAT hash_power_level 22
STAT hash_bytes 33554432
STAT hash_is_expanding 0
STAT slab_reassign_rescues 4634503
STAT slab_reassign_chunk_rescues 0
STAT slab_reassign_evictions_nomem 0
STAT slab_reassign_inline_reclaim 23146
STAT slab_reassign_busy_items 168
STAT slab_reassign_busy_deletes 0
STAT slab_reassign_running 0
STAT slabs_moved 17150
STAT lru_crawler_running 0
STAT lru_crawler_starts 950926
STAT lru_maintainer_juggles 87548035
STAT malloc_fails 0
STAT log_worker_dropped 0
STAT log_worker_written 0
STAT log_watcher_skipped 0
STAT log_watcher_sent 0
STAT bytes<PHONE_NUMBER>
STAT curr_items 3095776
STAT total_items 190393259
STAT slab_global_page_pool 111
STAT expired_unfetched 16276742
STAT evicted_unfetched 0
STAT evicted_active 0
STAT evictions 0
STAT reclaimed 306868
STAT crawler_reclaimed 39213685
STAT crawler_items_checked<PHONE_NUMBER>
STAT lrutail_reflocked 649
STAT moves_to_cold 22339836
STAT moves_to_warm 11708167
STAT moves_within_lru 338162
STAT direct_reclaims 0
STAT lru_bumps_dropped 0
END
stats settings
STAT maxbytes<PHONE_NUMBER>8
STAT maxconns 1024
STAT tcpport 11211
STAT udpport 0
STAT inter [IP_REDACTED]
STAT verbosity 0
STAT oldest 0
STAT evictions on
STAT domain_socket NULL
STAT umask 700
STAT growth_factor 1.25
STAT chunk_size 48
STAT num_threads 4
STAT num_threads_per_udp 4
STAT stat_key_prefix :
STAT detail_enabled no
STAT reqs_per_event 20
STAT cas_enabled yes
STAT tcp_backlog 1024
STAT binding_protocol auto-negotiate
STAT auth_enabled_sasl no
STAT auth_enabled_ascii no
STAT item_size_max 1048576
STAT maxconns_fast yes
STAT hashpower_init 0
STAT slab_reassign yes
STAT slab_automove 1
STAT slab_automove_ratio 0.80
STAT slab_automove_window 30
STAT slab_chunk_max 524288
STAT lru_crawler yes
STAT lru_crawler_sleep 100
STAT lru_crawler_tocrawl 0
STAT tail_repair_time 0
STAT flush_enabled yes
STAT dump_enabled yes
STAT hash_algorithm murmur3
STAT lru_maintainer_thread yes
STAT lru_segmented yes
STAT hot_lru_pct 20
STAT warm_lru_pct 40
STAT hot_max_factor 0.20
STAT warm_max_factor 2.00
STAT temp_lru no
STAT temporary_ttl 61
STAT idle_timeout 0
STAT watcher_logbuf_size 262144
STAT worker_logbuf_size 65536
STAT track_sizes no
STAT inline_ascii_response no
STAT ext_item_size 512
STAT ext_item_age<PHONE_NUMBER>
STAT ext_low_ttl 0
STAT ext_recache_rate 2000
STAT ext_wbuf_size 4194304
STAT ext_compact_under 0
STAT ext_drop_under 0
STAT ext_max_frag 0.80
STAT slab_automove_freeratio 0.010
STAT ext_drop_unread no
END

I was referring to a non-stripped binary for the addr2line, maybe @thatsafunnyname can save the day again :P I'm juggling too much stuff to get a centos7 instance for a few days.

I'm not familiar with addr2line. I'll wait to see if anyone can help.

> addr2line -e memcached-1.5.20/memcached 416b76
memcached-1.5.20/slabs.c:882

879        /* ITEM_FETCHED when ITEM_SLABBED is overloaded to mean we've cleared
880         * the chunk for move. Only these two flags should exist.
881         */
882        if ( (slab_rebal.completed[offset] == 0)
883              && (it->it_flags != (ITEM_SLABBED|ITEM_FETCHED)) ) {
884            /* ITEM_SLABBED can only be added/removed under the slabs_lock */
885            if (it->it_flags & ITEM_SLABBED) {

hm... slab_rebal.completed is an unchecked calloc (I just found/fixed that earlier today in a branch). This particular crash would've been an out of memory situation (the OOM killer doesn't always need to fire. you can see segfaults, depending).
I can't really reconcile the first crash though, the line doesn't make sense. :/ if it was going to segfault it would've done so earlier.
I'm leaning towards this being an out of memory situation, though slightly unusual. That (tiny!) calloc in the slab rebalancer was added in 1.5.20, so it might be possible that your otherwise stable system could trip on it sometimes if there's an out of memory issue. Regardless of checked/unchecked calloc/mallocs, the OS won't always give you all of the memory you request up front; especially for larger allocations like the hash table and slab pages. No matter how hard I try to protect the memory it'll still randomly page fault if you're out of RAM.
I'll prioritize adjusting that for the next release just in case... might pre-alloc the largest possible set, but otherwise having it fail-to-retry is easy enough.
So; TL;DR: Maybe RAM, but I'm not 100% convinced. A third crash in a different place would help, or a repro case but I'm still failing to do that on my end.

Thanks for the response and also for a great project.
Of note the last server does have plenty of RAM as you can see from the graphs.
For that server it has 64 GB, enough RAM that linux is not even using it for cache. Some RAM is completely unused.
I think reproducing the crash is going to be hard because I have 8 servers running memcached with the same setup 24 hours a day and only 3 crashes have been noted with the latest version.
I do think I had memcached die on me in the past on a previous version, but I wasn't monitoring like right now so I am not sure.

I noticed in the memcached memory chart:

the current bytes used max is 1.99G at the time of the crash, when the crashes have been seen previously was the "Current Bytes Used" also just below 2GB?
If the other crashes have been at just below 2GB, does the process or user running memcached have any ulimits set on it.  As the same user you should be able to cat /proc/<MEMCACHED_PID>/limits to check.

Here is the "cat /proc/<MEMCACHED_PID>/limit" output
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             256933               256933               processes
Max open files            1024                 1024                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       256933               256933               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Also here is a graph near the previous first logged crash in the different server(the first comment in this github issue).
Of note this should be 30 minute averages so it won't be that accurate to pinpoint values.

First: sorry this is taking so long to work through. I've been under the weather until the last few days, so haven't been able to focus on it. Usually the crashes we get are more straightforward as well; consistent back trace, other errors in dmesg, odd feature usage, etc. In your case you're not doing anything special, load is low, etc.
For your graphs, it's going to be hard to see sudden spikes in memory usage. A few seconds is enough to fill things up, and often lots of "free / uncached" memory on a server indicates that large processes could be starting and dying frequently, since linux really likes allocating all ram to various caches. Still, I'm not convinced it's a memory issue: more simply all evidence is pointing to it being a memory issue.
The 2GB guess was a good one, but I see the 'bytes' stat from the most recent server is sitting at 2.4G. You can quickly test with this script:
https://github.com/memcached/mc-crusher/blob/master/bench-warmer
edit my $exptime = 0 to be 300 or so. Then run with: ./bench-warmer <IP_ADDRESS>:11211 filltest 50000 50000 - use non-prod server or off-peak! If you change the final 50000 to 250000 it should fill about 12G of RAM. It shouldn't force out (much) existing data due to the way memory slabbing works. After 5 minutes the values will expire.
Are there any cgroups in use as well? containers? systemd loves to cgroup things.

Thanks @digirave .
@dormando , I am not familiar with the code and what the max difference between slab_start and slab_pos can be, but should offset at https://github.com/memcached/memcached/blob/master/slabs.c#L864 be size_t and not int ?
#include <iostream>
using namespace std;

int main() {
  unsigned int m_size = 1;
  void* slab_start = (void*)1073741824;
  void* slab_pos   = (void*)3221225472;

  int offset = ((char*)slab_pos-(char*)slab_start)/(m_size);
  cout << "offset    = " << offset << endl;

  unsigned int u_offset = ((char*)slab_pos-(char*)slab_start)/(m_size);
  cout << "u_offset  = " << u_offset << endl;

  size_t st_offset = ((char*)slab_pos-(char*)slab_start)/(m_size);
  cout << "st_offset = " << st_offset << endl;

  return 0;
}

// > g++ -Wall -Wextra -pedantic test_offset.cc && ./a.out
// offset    = -2147483648
// u_offset  =<PHONE_NUMBER>
// st_offset =<PHONE_NUMBER>

@thatsafunnyname the max range of offset should be 0 - 20,000 or so. it's a chunked offset into a 1MB page.

Are there any cgroups in use as well? containers? systemd loves to cgroup things.
Nope.
This is a yum install memcached -y with remi repository install.
With a very simple memcached configuration file.

Not sure how to compile https://github.com/memcached/mc-crusher/blob/master/bench-warmer is seems to have no ./compile unlike the install instructions.

You just chmod +x it and run it, so long as you have perl installed. Uses a hash bang in the top there, like a bash script.

You just chmod +x it and run it, so long as you have perl installed. Uses a hash bang in the top there, like a bash script.

Ah... I missed your initial instructions and tried to install the mc-crusher inself
Finished testing on the production server in live enviroment... no problems or crashes.
perl bench-warmer [IP_REDACTED]:11211 filltest 50000 50000
Run: 5000
Run: 10000
Run: 15000
Run: 20000
Run: 25000
Run: 30000
Run: 35000
Run: 40000
Run: 45000
Run: 50000
Result: No errors in /var/log/messages

You just chmod +x it and run it, so long as you have perl installed. Uses a hash bang in the top there, like a bash script.

Ah... I missed your initial instructions and tried to install the mc-crusher inself
Finished testing on the production server in live enviroment... no problems or crashes.
perl bench-warmer [IP_REDACTED]:11211 filltest 50000 50000
Run: 5000
Run: 10000
Run: 15000
Run: 20000
Run: 25000
Run: 30000
Run: 35000
Run: 40000
Run: 45000
Run: 50000
Result: No errors in /var/log/messages

Great, thanks! See any bumps on your graphs?

I see a temporary bump in evictions and memory usage, which seems like a normal result for the test.
See graphs below:

Alright, fired up a centos VM to figure this out (added remirepo stuff)
yum --enablerepo=remi install memcached
debuginfo-install memcached-1.5.20-1.el7.remi.x86_64
# addr2line -e /usr/bin/memcached 417a2c
/usr/src/debug/memcached-1.5.20/items.c:462 (discriminator 1)

addr2line should work if you run that debuginfo-install bit first (and maybe install gdb if it still doesn't work). The addr2line for the first error gives me a better line:
    if (it->next) it->next->prev = it->prev;

which is the first time it is dereferenced during do_item_unlink.
I also get a different line for the slabs check:
# addr2line -e /usr/bin/memcached 416b76
/usr/src/debug/memcached-1.5.20/slabs.c:817

which is:
static void slab_rebalance_cut_free(slabclass_t *s_cls, item *it) {
    /* Ensure this was on the freelist and nothing else. */
    assert(it->it_flags == ITEM_SLABBED);
    if (s_cls->slots == it) {
        s_cls->slots = it->next;
    }
    if (it->next) it->next->prev = it->prev;
    if (it->prev) it->prev->next = it->next;
    s_cls->sl_curr--;
}

So very likely that page mover patch is buggy somehow, despite surviving torture tests for quite a while. Finding a slightly different line makes a world of difference when investigating bugs :)
I'll keep working on a repro case. If you get another crash, try the debuginfo-install to resolve the addr2line, or simply give me the number since I have this VM while debugging this.

I have a crude load test running now with a roughly similar load as yours under 1.5.20, though not exactly the same binary. It causes consistent page moves, with sets/gets/deletes across a random space. encouraging extra page moves by varying which slab classes get sets every million writes.
I suspect if this crashes at all it'll take a few days. Will leave this one running as a baseline and try some other more aggressive tests + continue working on another repro system.

just a note for several internal reasons (including decreasing frequent updates).
I swapped the memcached with an older
memcached-1.4.15-10.el7_3.1.x86_64 default CentOS version
so I probably won't be able to produce any more SEGV information for the time being

Sorry that we weren't able to figure it out in time, though I agree with the outcome.. but sad you revert below 1.5.0 :( in general new versions of major software should be qualified and slow rolled, but it looks like you were autoupdating which isn't ideal.

@digirave is there anything else you can tell me about your workload that might not be obvious from the stats? IE; do you use TTL's? if so, are they all the same or a range?

@digirave is there anything else you can tell me about your workload that might not be obvious from the stats? IE; do you use TTL's? if so, are they all the same or a range?

expiration ranged from very short 2 seconds to a few minutes and some a few hours~2days.
due to a very fast changing workload, short a few second expiration were commonly used.
no sets without expiration were used.

I think #608 is the fix. My tests didn't catch it because they were never making chunked items. I'll see if I can get mctester to reproduce the crash for the affected versions as well.

Fix released in 1.5.22

@dormando Thanks for the fix, and the debugging work to get there.
I found this same issue on one host running 1.5.16, which crashed 4x on (slabs.c:817) and 3x on (items.c:462)
Unfortunately I don't know the details of the workload and what makes this different from our other memcache instances, reproducing this has been a head-scratcher for sure.

@phaas open up a new issue if you have a different bug?
Either your 1.5.16 is modified or those crashes aren't from 1.5.16; the line numbers don't make sense. Did you mean to say 1.5.19? and that it's fixed now? :)

@dormando I was misled by a docker container label, the actual source files and lines that I looked at match 1.5.20 and the exact lines highlighted in this issue.
I haven't verified the fix locally yet, will start that Monday. Thanks again :)


lake update does not always update

I was trying to move the LeanInk test suite to the new Lake format:
~/lean/LeanInk/test/dep $ cat lakefile.lean
import Lake
open Lake DSL

package dep

require mathlib from git "https://github.com/leanprover-community/mathlib4.git"

lean_lib dep

~/lean/LeanInk/test/dep $ lake update
error: ./lean_packages/mathlib/./lakefile.lean:9:18: error: unknown constant 'Lake.PackageFacet.oleans'
error: package configuration `./lean_packages/mathlib/./lakefile.lean` has errors

~/lean/LeanInk/test/dep $ git -C lean_packages/mathlib rev-parse @
84ca33e940678954dc146f732f447ddc8984cf39  # old!

~/lean/LeanInk/test/dep $ nix shell nixpkgs#elan -c lake --version
Lake version 3.2.0 (Lean version 4.0.0-nightly-2022-07-03)

~/lean/LeanInk/test/dep $ rm -rf lean_packages/

~/lean/LeanInk/test/dep $ lake update
info: mathlib: cloning https://github.com/leanprover-community/mathlib4.git to ./lean_packages/mathlib

~/lean/LeanInk/test/dep $ git -C lean_packages/mathlib rev-parse @
ecd37441047e490ff2ad339e16f45bb8b58591bd

I'm not completely sure what's happening, does it decide not to update the dep or does it try to parse the dep's lakefile before deciding whether to update it?

The old lean_packages, before deleting it, probably did not have a manifest.json yet. If that's the case, this issue can probably be ignored as the bug will fix itself with time.

@Kha, I'm confused -- is this or is this not a reproducible bug?

Yes, it should be reproducible by replaying LeanInk's history
$ git checkout 0a160d91458c1873937449a7c78d25b34b8686df
$ cd test/dep
$ lake build  # checks out old mathlib, does not create manifest.json
$ git checkout 4b5e606ea8cc54c2447ce48706f8ec1d133d19e9
$ git rm lean_packages/manifest.json  # restore state before successful `lake update`
$ lake update  # as above

Restoring the manifest.json leads to another unexpected error:
$ lake update
info: mathlib: updating ./lean_packages/mathlib
error: stdout:

stderr:
You are not currently on a branch.
Please specify which branch you want to rebase against.
See git-pull(1) for details.

    git pull <remote> <branch>
error: git exited with code 1

I don't think the branch state of the current checkout should matter. However, after switching to mathlib's master, everything works as expected, so it seems the issue really was the lack of manifest.json resulting from the Lake version transition.


Make Line2D be white by default

Closes https://github.com/godotengine/godot-proposals/issues/751

Thanks!


Invalid UpdateRolePositions body

The serialized body of UpdateRolePositions is wrongly an array of arrays instead of an array of objects. First reported in the following support thread https://discord.com/channels/745809834183753828/1105721932021760050.
The endpoint is also lacking an AuditLogReason implementation.

This issue was fixed by #2342. This issue was probably overlooked because of #2338.


Convenient Action Handler

Looks like if you are working with a list where It's possible to have multiple selections, you have to provide both handlers - Action and MultipleAction. I had to spend some time to comprehend it.
Why not just create an action handler, where if a single option is selected it will be a slice with one element only?

@NicolasSiver
I think there was a reason or a poor design plan on my part. Not sure which since this was one of my first GoLang projects. I have since done better with internal tools for work. I'll look at it when I get time.

@NicolasSiver Looking into this and it seems it was just poor design. It will take some time for me to fix. Things Todo:

[ ] Remove MultiFunc
[ ] Add Multi-answer boolean
[ ] Change defaultFunc to work with slices of options
[ ] Make sure nothing else broke
[ ] Release Version 3 as this is a breaking change

I'll try to keep this comment up to date with my progress

Ran through UTs, Report Card, and GoDocs and couldn't find anything I missed.
I:

Removed MultipleAction()
Edited Action() to accept a slice of Options
Added AllowMultiple() to still allow for that check
Fixed UTs
Checked Code Coverage
Checked Code Quality (Go Report Card)
Made a V3 branch to keep up with that
Master is now on V4 (tags)
Updated the Readme

Hope this helps! @NicolasSiver


Consider using dataloader

Dataloader is a project from Facebook and allows removing pressure from the database: https://github.com/facebook/dataloader .
As per the README, it does not replace shared application level caches. It's meant to be used per request, and prevent multiple DB calls when one will do. See https://github.com/facebook/dataloader/blob/master/README.md.
Premature optimisation for now, but will come in handy when we are in need for performance improvements.
See https://github.com/withspectrum/spectrum/tree/alpha/api/loaders for a JS implementation.

Closing for now.

Reopening this, it's not too complicated to get in and will really help offload the database.

Note to self: article with examples of dataloader and Mongo at http://www.petecorey.com/blog/2017/08/14/batching-graphql-queries-with-dataloader/

Like #89 this makes sense when we have have types that refer to each other.


remove status from downloader url

@jcarroll-mediafly Could you take a look at this PR. I have removed /status from all downloader urls. Having /status in any of the URLs didn't seem RESTful. Feel free to merge if it looks good and I can update the spreadsheet or let me know if you have any concerns.

I could go either way, adding @eugirdor

/status in addToDownloader() definitely doesn't make sense.
getDownloadStatus(id) probably should have been something like /items/{id}/download-status, but I don't think it makes much of a difference now.

True, the method name is contains status so having it in the url makes sense.
I would say the absence of the word status in the GET calls implies you're getting a download object, if we are being "restful".
Note any changes requires a new iOS build which im hesitant right now


Calling function from wrong derived class after loading from pluginlib

Programming is not really my strong suit and I really have no idea what went wrong here:
pkg_1::derived_1:
base_class: pkg_2::derived_2
init(a,b,c);
getAllIK(a,b,c);
pkg_2::derived_2:
base_class: pkg_3::base
init(a,b,c);
getAllIK(a,b,c);
... a couple more
pkg_3::base:
init(a,b,c);
getAllIK(a,b,c);
... a couple more
//other_code
static pluginlib::ClassLoader<pkg_name::base> r_model_loader("pkg_3", "pkg_3::base");
basePtr ptr = r_model_loader.createInstance("pkg1::derived_1");
When i call the function init(a,b,c) , the function called is the one in derived_2. When i call getAllIK, the function called is in derived_1 , but it doesn't receive the values from the call and instead gets junk. The calls are identical. Sometimes segfaults happen.
Base is defined in the header base.h , and all of its functions are actually in derived_2. In my code I am loading derived_2 with plugin lib as follows:
I have exported the derived_1 plugin with the base declared as "pkg_name::base" but that didn't help. Does anyone have any idea what might be the problem?

Please ask questions on http://answers.ros.org and when you do please provide a reproducible example so we know exactly what you're doing and what you're expecting. Guidelines for asking questions can be found at: http://wiki.ros.org/Support


Ignore leading and trailing white space inside certain or all attributes?

I am having issues since these elements are not considered equivalent?

I'd like to add that

and

should also be equivalent.


Rust-Analyzer's diagnostics not updating on Windows

nvim --version: NVIM v0.5.0-dev+1059-g55add1c1c
nvim-lsp version(commit hash): 0fb71129de54577254a3ae143850d11b6902a279
What language server (If the problem is related to a specific language server): Rust_Analyzer
Can you reproduce this behavior on other language server clients (vscode, languageclient-neovim, coc.nvim, etc.): I don't use others
Can you reproduce this behavior on other language servers offered in the nvim-lspconfig repo? (pyls -> pyright): I don't use it
Is the problem isolated to a particular language server: Yes
Operating system/version: Windows 10

How to reproduce the problem from neovim startup

nvim -u minimal.vim
!cargo new rust_project_name
e (...)/main.rs
do something similar to the demo video I attached
(PS: I prerecorded the video using my actual config, thus the colorscheme and lightline. But I assure that using my minimal.vim exactly the same happens)

Actual behaviour
Diagnostics aren't updating properly.
Expected behaviour
Diagnostics should update on save instead of having to exit and reload nvim.
Minimal init.vim or init.lua and code sample
E5113: Error while calling lua chunk: minimal_init.lua:10: Vim(packadd):E919: Directory not found in 'packpath': "pack/*/opt/packer.nvim"
Press ENTER or type command to continue

I'm on Windows and I have never used packer.nvim in my life so I have no idea why it's giving me a directory error, but I have reproduced the error using only a minimal config with plug.nvim.
call plug#begin('~/AppData/Local/nvim/autoload')
Plug 'neovim/nvim-lspconfig'
Plug 'nvim-lua/completion-nvim'
call plug#end()

" --- Define mapleader for keymaps --------------------------------------"
let mapleader = ' '

" RUST_ANALYZER
lua require'lspconfig'.rust_analyzer.setup{ cmd={"rust-analyzer-windows.exe"} }

" Keymaps
nnoremap <silent>gD          :lua vim.lsp.buf.declaration()<CR>
nnoremap <silent>gd          :lua vim.lsp.buf.definition()<CR>
nnoremap <silent><leader>gk  :lua vim.lsp.buf.hover()<CR>
nnoremap <silent><leader>gr  :lua vim.lsp.buf.rename()<CR>
nnoremap <silent><leader>gR  :lua vim.lsp.buf.references()<CR>
nnoremap <silent><leader>gp  :lua vim.lsp.diagnostic.goto_prev()<CR>
nnoremap <silent><leader>gn  :lua vim.lsp.diagnostic.goto_next()<CR>

autocmd BufEnter * lua require'completion'.on_attach()
let g:completion_confirm_key = '<CR>'
let g:completion_enable_auto_paren = 0
let g:completion_matching_strategy_list =
      \ ['exact', 'substring', 'fuzzy', 'all']

set termguicolors
set background=dark

" --- Disable recommended styles ----------------------------------------"
let g:python_recommended_style = 0
let g:rust_recommended_style = 0

" --- Set g:python3_host_prog -------------------------------------------"
let g:python3_host_prog=
      \'~/AppData/Local/Programs/Python/Python38/python.exe'

" --- Time format language ----------------------------------------------"
language time en_us

" --- Global settings ---------------------------------------------------"
set nowrap
set clipboard=unnamed
set noerrorbells
set nocursorline
set tabstop=2 softtabstop=2
set shiftwidth=2
set expandtab
set smartindent
set nu rnu
set nohls
set ignorecase
set smartcase
set noswapfile
set nobackup
set undofile
set textwidth=80
set colorcolumn=+1
set updatetime=100
set completeopt=menuone,noinsert,noselect
set shortmess+=c
set laststatus=2
set guicursor=a:block-Cursor

" --- Mouse options -----------------------------------------------------"
set mouse=a
behave xterm

nnoremap <leader>w :w<CR>

" --- Window handling ---------------------------------------------------"
nnoremap <silent><leader>h :wincmd h<CR>
nnoremap <silent><leader>j :wincmd j<CR>
nnoremap <silent><leader>k :wincmd k<CR>
nnoremap <silent><leader>l :wincmd l<CR>
nnoremap <silent><leader>x :wincmd c<CR>
nnoremap <silent><leader>r :wincmd r<CR>
nnoremap <silent><leader>T :wincmd T<CR>
nnoremap <silent><leader>= :wincmd =<CR>
nnoremap <silent><leader>o :wincmd o<CR>
nnoremap <silent><leader>sv :wincmd v<CR>
nnoremap <silent><leader>sh :wincmd s<CR>
" Split Resizing
nnoremap <silent> <leader>+ :vertical resize +5<CR>
nnoremap <silent> <leader>- :vertical resize -5<CR>
nnoremap <silent> <A-+> :resize +2<CR>
nnoremap <silent> <A--> :resize -2<CR>

" --- Scroll up/down with keys ------------------------------------------"
nnoremap <silent><C-j> <C-e>
nnoremap <silent><C-k> <C-y>
" 5x FASTER
nnoremap <silent><A-j> 5<C-e>
nnoremap <silent><A-k> 5<C-y>

" --- Scroll left/right with keys ---------------------------------------"
nnoremap <C-h> 3zh
nnoremap <C-l> 3zl

" --- Move lines up/down ------------------------------------------------"
nnoremap <silent><A-s> :m .+1<CR>==
nnoremap <silent><A-w> :m .-2<CR>==
inoremap <silent><A-s> <Esc>:m .+1<CR>==gi
inoremap <silent><A-w> <Esc>:m .-2<CR>==gi
vnoremap <silent><A-s> :m '>+1<CR>gv=gv
vnoremap <silent><A-w> :m '<-2<CR>gv=gv

" --- Tab handling ------------------------------------------------------"
nnoremap <silent><leader>tc :tabc<CR>
nnoremap <silent><leader>tn :tabn<CR>
nnoremap <silent><leader>tp :tabp<CR>

Health check

Checkhealth result

```
health#lspconfig#check
========================================================================
## Checking language server protocol configuration
  - INFO: rust_analyzer: configuration checked.
```

LSP log

Log file

It doesn't log anything.

I've never used rust on Windows before. Following the rust-analyzer instructions, I managed to get to set everything up:

This seems like a configuration issue on your end, and you should ask in our discussions forum for rust-analyzer.

Oh, and my diagnostics update fine

Of note, the reason we ask for:
Can you reproduce this behavior on other language server clients (vscode, languageclient-neovim, coc.nvim, etc.): I don't use others
Can you reproduce this behavior on other language servers offered in the nvim-lspconfig repo? (pyls -> pyright): I don't use it
Is the problem isolated to a particular language server: Yes

Is not to survey users, but to ensure the problem is isolated to neovim's language server. The first step I would ask you to do to debug your config would be to try an alternate client and see if the issue reproduces.

Oh, I see your problem. I think this is a rust-analyzer issue.

FYI when I try to use the vscode rust-analyzer extension, I get frequent rust-analyzer crahses:
The Rust Analyzer server crashed 5 times in the last 3 minutes. The server will not be restarted.

Sorry for not presenting the issue properly, the reason is actually that currently I have no access to wifi, so I'm trying to use the least amount of mobile data. So for now I'll present the log but if that information doesn't suffice I'll do the rest later (like trying other servers, etc.)
log: https://paste.sh/4J4qX2vk#w8u6A8YfX-b8ImnGQfMGoclG
(since it was gigantic I pasted it there)

Since I already had the necessary components I went and tried rls and it works fine. The problem seems to be rust_analyzer related I guess

I should open an issue in Rust_Analyzer.... @mjlbach I'll close this, if you still think it could be related to lspconfig reopen it please

I'll keep an eye, but you should create an issue in the rust-analyzer repo and link it here so others can follow. I don't use Windows regularly so won't be able to actively test.

Alright thank you :)

@mjlbach still nothing on rust-analyzer's repo. Should I do some other testing or something else?

What are you asking? I can't fix it here, and I don't have time to dig into the rust-analyzer codebase and fix it there.

Okay


Different exception behavior when creating clients between full framework and .net core

When creating a client, for example with
new AmazonSqsClient();

if now credentials or region are available the above code throw an AmazonClientException on the full framework but an HttpRequestException on .NET Core. The problem with the HttpRequestException is that it is in System.Net.Http and would require to add a reference to that package just to be able to handle it. Ideally, the client would throw AmazonClientException  on both frameworks.
Expected Behavior
Throw AmazonClientException for client configuration problems on both frameworks
Current Behavior
.NET Core throws HttpRequestException while full framework throws AmazonClientException
Possible Solution
Steps to Reproduce (for bugs)
Without any credentials available
new AmazonSqsClient();

run on full framework --> throws an AmazonClientException
run on .net core 2.1 --> throw an HttpRequestException
Context
At the moment we are forced to handle this behavior with a generic exception catch because we don't want to take a reference to System.NET.Http just to be able to guard against HttpRequestException
Your Environment

AWSSDK.Core version used: latest
Service assembly and version used:
Operating System and version:
Visual Studio version:
Targeted .NET platform: Fullframework 4.6.1 and Net core 2.1.x

.NET Core Info

.NET Core version used for development:
.NET Core version installed in the environment where application runs:
Output of dotnet --info:
Contents of project.json/project.csproj:

Hi @danielmarbach, thank you for the feedback on this.  I can address this with the team, but my guess is that anything that is already implemented will not be synced to avoid issues with breaking changes.

What were the exception messages?  I am curious as to where this branching is happening.

@sstevenkang It is a bit embarrassing but I can't repro it anymore. That is one of the downsides of transitive dependencies with the new csproj :( I guess I'm closing it for now until I have further intel


Is there any plan to release the training files?

Hi, this is great work, considering that I would like to train my own LUT or weights, are there any plans to release the training files? Thanks!

Hi, this is great work, considering that I would like to train my own LUT or weights, are there any plans to release the training files? Thanks!

Thanks for your interest in our work. Our code has been reimplemented in PyTorch and released along with the extended version. Find it here: https://github.com/ddlee-cn/LeRF-PyTorch .

您的邮件我已收到，感谢。祝好。


The maximum value of conversion rate is not getting altered if we increase the annual sales figure to a greater value

To Reproduce
Steps to reproduce the behavior:

Launch https://storybook-preview-290--upbeat-sinoussi-f675aa.netlify.app/?path=/story/chart--default
2- Click on the chart
2- select the default chart
3-Click on the chart tab
4-Press 'A' key
5- Click on the Controls tab
6- navigate to componentProps
7-Click on the Y axis
8- click on the tickValues
9-Change the value and observe

Expected behavior
The maximum value of conversion rate should get altered if we increase the annual sales figure to a greater value.
Screenshots
https://www.screenpresso.com/=EPGub
Desktop (please complete the following information):
Windows 10 : Chrome [ 95.0.4638.69] /Edge [ Version 95.0.1020.40]
Mac 14 : Safari [14.1.2]
Additional context
Add any other context about the problem here.

If you change the tick values, you must also change the domain prop.
Not an issue.


Fix a few typos in the comments

While reading through the code I spotted some typos, couldn't help myself and had to create a PR. 😄

Wow. Thanks for the once-over. It really needed it, :).


Cleanup for New Parser

Codecov Report

Merging #574 (f816f3c) into reducer-dev (949b57c) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@               Coverage Diff               @@
##           reducer-dev     #574      +/-   ##
===============================================
- Coverage        55.12%   55.11%   -0.02%
===============================================
  Files               45       45
  Lines             2886     2885       -1
===============================================
- Hits              1591     1590       -1
  Misses            1295     1295

Impacted Files
Coverage Δ

...ucerInterface/ReducerInterface_ExpressionValue.res
56.71% <ø> (ø)

.../Reducer/Reducer_Expression/Reducer_Expression.res
85.36% <100.00%> (-0.35%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 949b57c...f816f3c. Read the comment docs.

Merge those asap so that I donot create merge conflicts on further work...

Just wanted to get your thumbs up, happy to merge

I guess it would be good to later have a story to make sure we don't use Prettier on the generated file.


How to stretch the image

Now I can cropped the answer sheet from image. But at sometimes, the answer sheet is not birds-eye-view image. So I can not cropped precise. I need to stretch four points of contour into a rectangle and then cropped answer sheet from image.

Finished with one line of code on Oct 3rd.


Add direct token usage

Going to get to this asap so things are usable without callback and code workflow.

Done.


Edit - Somewhat misleading documentation

Attached is a screenshot of the "enhanced" result taken from the example image using the API provided here http://max-image-resolution-enhancer.codait-prod-41208c73af8fca213512856c7a09db52-0000.us-east.containers.appdomain.cloud/. While it may work for increasing resolution size it's clearly a misleading example, and doesn't do anything to reconstruct image details. Why falsely advertise?

Sorry to raise this issue, it's not a bug, but the documentation seems slightly misleading. Yes this does upscale low-resolution images, but "generating photo-realistic details" is  stretch. This model doesn't work to reconstruct meaningful details not captured in the original photo.

For what it's worth the following is the enhancement I got from running it locally within a Docker image. You can see the actual enhanced details in the upscaling. I'm not a contributor to the project, so have no understanding what the reasoning behind the difference might be, but it does appear to work for me.


[feature-request] hybrid PoW + PoS mining, defend against 51% attacks

For Dash: vs PoW it is better to build a hybrid PoW+PoS because we could do odd numbered blocks via PoW Mining and even numbered blocks via Masternodes (PoS), increasing defences vs 51% PoW attacks (attacker will need to have both 51% POW + 51% stake to attack Dash... )
i.e. we should teach Masternodes to “mine” blocks by themselves.
PoW is still needed to solve certain problems with PoS staking (like 'nothing-at-stake' problem, etc...), I find hybrid mining beneficial.
If we also lower bounty for PoW mining, this will make our world a greener place (since less electricity is gonna be wasted on Dash network defences, while protection itself would improve) - I propose to cut miners income in half to 22.5% and increase Masternode income. (and maybe increase Dash budget, too)
This is relevant for Dash because small altcoins are constantly getting 51% attacked and exchanges are suffering, including Bitcoin Gold (BTG), Litecoin Cash (LCC) and Horizen (ZEN)...
and now Dr. Craig Wright publicly threats to attack Bitcoin Cash network, a network bigger than Dash, and he might be able to pull this off.
https://www.reddit.com/r/btc/comments/9b9mf0/csw_threatens_to_double_spend_steal_and_attack/
Attacking Dash is not very expensive, according to this website:
https://www.crypto51.app/
https://en.m.wikipedia.org/wiki/2018_double-spend_attacks_on_Equihash-based_cryptocurrencies
What do you think ?

Resolved by the implementation of ChainLocks

Suggesting these kind of very big changes to the network is probably more appropriate for the forum first. Also, there's been no activity on this in a year+ with no responses.
Closing for the reasons above