segmentation fault 11 on running my unit tests

Last few lines of output on the server looks like:
[88313] 07-17 19:44:00,753 INFO [RocksDB]Options.compaction_options_fifo.max_table_files_size:<PHONE_NUMBER>
[88313] 07-17 19:44:00,753 INFO [RocksDB]                  Options.table_properties_collectors:
[88313] 07-17 19:44:00,753 INFO [RocksDB]                  Options.inplace_update_support: 0
[88313] 07-17 19:44:00,753 INFO [RocksDB]                Options.inplace_update_num_locks: 10000
[88313] 07-17 19:44:00,753 INFO [RocksDB]              Options.min_partial_merge_operands: 2
[88313] 07-17 19:44:00,753 INFO [RocksDB]              Options.memtable_prefix_bloom_bits: 0
[88313] 07-17 19:44:00,753 INFO [RocksDB]            Options.memtable_prefix_bloom_probes: 6
[88313] 07-17 19:44:00,753 INFO [RocksDB]  Options.memtable_prefix_bloom_huge_page_tlb_size: 0
[88313] 07-17 19:44:00,753 INFO [RocksDB]                          Options.bloom_locality: 0
[88313] 07-17 19:44:00,753 INFO [RocksDB]                   Options.max_successive_merges: 0
[88313] 07-17 19:44:00,753 INFO [RocksDB]               Options.optimize_fllters_for_hits: 0
[88313] 07-17 19:44:00,753 INFO [RocksDB]               Options.paranoid_file_checks: 0
[88313] 07-17 19:44:00,753 INFO [RocksDB]               Options.compaction_measure_io_stats: 0
[88313] 07-17 19:44:00,755 INFO [RocksDB]Created column family [0] (ID 43)
[88313] 07-17 19:44:00,755 INFO Create ColumnFamilyHandle with name:0 success.
Segmentation fault: 11

Anything else I should do to gather info? It is very easy to repro, but I haven't isolated it to a certain command yet.

can u use gdb with the core dump file to get the crash stack trace?
to get the core dump file ,u need to ulimit -c unlimited before start ardb-server in the same terminal shell.

Another variation (not sure if it is related or not) looks like this:
[88390] 07-17 19:48:51,157 INFO RocksDB drop column family:0.
[88390] 07-17 19:48:51,159 INFO [RocksDB]Dropped column family with id 67

ardb-server(88390,0x700007055000) malloc: *** error for object 0xffffffff: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

I'll look into gdb.

seems u are using flushdb or flushall

Yes, my unit test suite depends on them to clear state at each test run. Are they unsupported commands?

it's supported.

I was able to get the core dump file after a bit of research. I'm running this on my mac. Can try running it on ubuntu and see if it happens there too. The core dump file is 1GB and when I run it I get:
AL Decode second key prefix failed in comparator.
Abort trap: 6 (core dumped)
jloehrer@jl-mac~/Code/ardb/src$ gdb ./ardb-server /cores/core.88797
GNU gdb (GDB) 7.11
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin15.4.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./ardb-server...done.
"/cores/core.88797": no core file handler recognizes format
(gdb)

So, not very useful. I'll switch to debugging in an ubuntu vm.

just type bt to get the stack trace

and in my mac, i use lldb ./ardb-server -c <core file> to debug

Here ya go:
jloehrer@jl-mac~/Code/ardb$ lldb ./src/ardb-server -c /cores/core.97974
(lldb) target create "./src/ardb-server" --core "/cores/core.97974"
warning: (x86_64) /cores/core.97974 load command 148 LC_SEGMENT_64 has a fileoff + filesize (0x37ff8000) that extends beyond the end of the file (0x37ff7000), the segment will be truncated to match
warning: (x86_64) /cores/core.97974 load command 149 LC_SEGMENT_64 has a fileoff (0x37ff8000) that extends beyond the end of the file (0x37ff7000), ignoring this section
Core file '/cores/core.97974' (x86_64) was loaded.
(lldb) bt
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10
    frame #1: 0x0000000102d804aa ardb-server`aeProcessEvents [inlined] aeApiPoll(eventLoop=0x00007fe983f00078, tvp=0x00007fff5cecc248) + 87 at ae_select.cc:48
    frame #2: 0x0000000102d80453 ardb-server`aeProcessEvents(eventLoop=0x00007fe983f00078, flags=<unavailable>) + 307 at ae.c:413
    frame #3: 0x0000000102d8095b ardb-server`aeMain(eventLoop=0x00007fe983f00078) + 43 at ae.c:485
    frame #4: 0x0000000102d3d721 ardb-server`ardb::Server::Start(this=0x00007fff5cecc460) + 2417 at network.cpp:300
    frame #5: 0x0000000102d34858 ardb-server`main(argc=<unavailable>, argv=<unavailable>) + 472 at main.cpp:94
    frame #6: 0x00007fff9c9e55ad libdyld.dylib`start + 1
    frame #7: 0x00007fff9c9e55ad libdyld.dylib`start + 1

Hope it helps. I'll try to narrow it down to a repeatable set of commands to cause the crash.

can u use thread list to list all threads?

(lldb) thread list
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #2: tid = 0x0001, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #3: tid = 0x0002, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #4: tid = 0x0003, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #5: tid = 0x0004, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #6: tid = 0x0005, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #7: tid = 0x0006, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #8: tid = 0x0007, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #9: tid = 0x0008, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #10: tid = 0x0009, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #11: tid = 0x000a, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #12: tid = 0x000b, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #13: tid = 0x000c, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #14: tid = 0x000d, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #15: tid = 0x000e, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #16: tid = 0x000f, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #17: tid = 0x0010, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #18: tid = 0x0011, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #19: tid = 0x0012, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #20: tid = 0x0013, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #21: tid = 0x0014, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #22: tid = 0x0015, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #23: tid = 0x0016, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #24: tid = 0x0017, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #25: tid = 0x0018, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
  thread #26: tid = 0x0019, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #27: tid = 0x001a, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #28: tid = 0x001b, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP

it's crashed in thread #25 where there is __pthread_kill, u can use thread select 25 to switch current thread to 25, and then bt to get the thread 25's stack trace.

(lldb) thread select 25
* thread #25: tid = 0x0018, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
    frame #0: 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff87cc8f06 <+10>: jae    0x7fff87cc8f10            ; <+20>
    0x7fff87cc8f08 <+12>: movq   %rax, %rdi
    0x7fff87cc8f0b <+15>: jmp    0x7fff87cc37cd            ; cerror_nocancel
    0x7fff87cc8f10 <+20>: retq
(lldb) bt
warning: could not load any Objective-C class information. This will significantly reduce the quality of type information available.
* thread #25: tid = 0x0018, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff95bdf4ec libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff959466e7 libsystem_c.dylib`abort + 129
    frame #3: 0x0000000102e19dd7 ardb-server`ardb::RocksDBPrefixExtractor::Transform(this=<unavailable>, src=<unavailable>) const + 247 at rocksdb_engine.cpp:385
    frame #4: 0x0000000102eb6682 ardb-server`rocksdb::MemTable::Get(this=0x00007fe985010c00, key=0x00007000048456e0, value="", s=0x00007fe983e05a60, merge_context=0x0000700004845698, seq=0x00007000048456d0) + 210 at memtable.cc:521
    frame #5: 0x0000000102e6d727 ardb-server`rocksdb::DBImpl::MultiGet(rocksdb::ReadOptions const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> > const&, std::__1::vector<rocksdb::Slice, std::__1::allocator<rocksdb::Slice> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >*) [inlined] rocksdb::MemTable::Get(this=<unavailable>, key=<unavailable>, value=<unavailable>, s=<unavailable>, merge_context=<unavailable>) + 1911 at memtable.h:175
    frame #6: 0x0000000102e6d704 ardb-server`rocksdb::DBImpl::MultiGet(this=0x00007fe98402ac00, read_options=0x00007000048458a8, column_family=0x0000700004845920, keys=0x0000700004845900, values=0x00007fe983e03c18 size=23) + 1876 at db_impl.cc:3199
    frame #7: 0x0000000102e14652 ardb-server`ardb::RocksDBEngine::MultiGet(this=<unavailable>, ctx=<unavailable>, keys=<unavailable>, values=size=23, errs=size=23) + 1010 at rocksdb_engine.cpp:1018
    frame #8: 0x0000000102dbae1f ardb-server`ardb::Ardb::HMGet(this=0x00007fff5cecc500, ctx=0x00007fe983c547f0, cmd=<unavailable>) + 655 at t_hash.cpp:324
    frame #9: 0x0000000102de7f92 ardb-server`ardb::Ardb::DoCall(this=0x00007fff5cecc500, ctx=0x00007fe983c547f0, setting=0x00007fe98403e7d8, args=0x0000700004845c68) + 258 at db.cpp:1395
    frame #10: 0x0000000102de870b ardb-server`ardb::Ardb::Call(this=0x00007fff5cecc500, ctx=0x00007fe983c547f0, args=0x0000700004845c68) + 1483 at db.cpp:1574
    frame #11: 0x0000000102d3e487 ardb-server`ardb::RedisRequestHandler::MessageReceived(this=0x00007fe983c54790, ctx=<unavailable>, e=<unavailable>) + 231 at network.cpp:86
    frame #12: 0x0000000102d5537a ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] ardb::ChannelUpstreamHandler<ardb::codec::RedisCommandFrame>::HandleStreamEvent(e=<unavailable>) + 506 at channel_upstream_handler.hpp:133
    frame #13: 0x0000000102d55367 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::codec::RedisCommandFrame>(ardb::ChannelHandlerContext*, ardb::MessageEvent<ardb::codec::RedisCommandFrame>&) at all_includes.hpp:89
    frame #14: 0x0000000102d55367 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelHandlerContext::SendUpstream<ardb::MessageEvent<ardb::codec::RedisCommandFrame> >(this=<unavailable>) at all_includes.hpp:164
    frame #15: 0x0000000102d55367 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::fire_message_received<ardb::codec::RedisCommandFrame>(ctx=0x00007fe983c54690) at channel_helper.hpp:91
    frame #16: 0x0000000102d55367 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(this=0x00007fe983c54650, context=0x00007fe983c54690, channel=0x00007fe983c54530, cumulation=0x00007fe983c54598) + 487 at stack_frame_decoder.hpp:102
    frame #17: 0x0000000102d532e7 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::MessageReceived(this=0x00007fe983c54650, ctx=<unavailable>, e=0x0000700004845db0) + 327 at stack_frame_decoder.hpp:157
    frame #18: 0x0000000102d43605 ardb-server`ardb::Channel::OnRead() [inlined] ardb::ChannelUpstreamHandler<ardb::Buffer>::HandleStreamEvent(e=<unavailable>) + 389 at channel_upstream_handler.hpp:133
    frame #19: 0x0000000102d435f2 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::Buffer>(ardb::ChannelHandlerContext*, ardb::MessageEvent<ardb::Buffer>&) + 66 at all_includes.hpp:89
    frame #20: 0x0000000102d435b0 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::MessageEvent<ardb::Buffer> >(this=<unavailable>) + 16 at all_includes.hpp:128
    frame #21: 0x0000000102d435a0 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::fire_message_received<ardb::Buffer>(message=0x0000000103046f40) + 43 at channel_helper.hpp:83
    frame #22: 0x0000000102d43575 ardb-server`ardb::Channel::OnRead(this=<unavailable>) + 245 at channel.cpp:550
    frame #23: 0x0000000102d41a5e ardb-server`ardb::Channel::IOEventCallback(eventLoop=<unavailable>, fd=<unavailable>, clientData=0x00007fe983c54530, mask=1) + 30 at channel.cpp:53
    frame #24: 0x0000000102d80596 ardb-server`aeProcessEvents(eventLoop=0x00007fe983c53c78, flags=<unavailable>) + 630 at ae.c:429
    frame #25: 0x0000000102d8095b ardb-server`aeMain(eventLoop=0x00007fe983c53c78) + 43 at ae.c:485
    frame #26: 0x0000000102d6d295 ardb-server`ardb::Thread::ThreadFunc(data=0x00007fe983e01fc0) + 21 at thread.cpp:38
    frame #27: 0x00007fff95bdc99d libsystem_pthread.dylib`_pthread_body + 131
    frame #28: 0x00007fff95bdc91a libsystem_pthread.dylib`_pthread_start + 168
    frame #29: 0x00007fff95bda351 libsystem_pthread.dylib`thread_start + 13

it's an assert crash at https://github.com/yinqiwen/ardb/blob/0.9/src/db/rocksdb/rocksdb_engine.cpp#L385, seems the data is corrupted.

hmmm, so this is strange. I start fresh at the beginning of each test, and each test only lasts a second or so, and I run hundreds of them. The data types primarily used are sorted sets and hashes with very simple commands. All single key commands. But I'll see if I can narrow it down to a single command. So far when I restart the ardb-server and run a single test, it passes. but if I run multiple ones, it fails. Maybe something about doing flushall then right away writing again?

Would it help to have the database files? There's nothing confidential in it since it is just test data, so I can pass it along to you.

The crash is related the data in memtable, so the data file maybe not helpful.
Is the unit test complated? maybe i can use to reproduce this issue in my mac.

And in the stack trace, the crash is happened with hmget.

Let me see if I can come up with the simplest test to recreate the issue. hmget is probably one of the most common commands I run so that may not signify anything in particular.

I was able to repro the crash by running only two commands repeatedly from redis-cli over and over:
[MP]~/Code/happybits-devops/sites/marcopolo> redis-cli -p 16379
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> hgetall test1
1) "modified_by"
2) "test"
3) "resolved"
4) "False"
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> hgetall test1
(empty list or set)
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> hgetall test1
(empty list or set)
<IP_ADDRESS>:16379> "hmset" "test1" "resolved" "False" "modified_by" "test"
OK
<IP_ADDRESS>:16379> hgetall test1
1) "modified_by"
2) "test"
3) "resolved"
4) "False"
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> hgetall test1
(empty list or set)
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> hgetall test1
(empty list or set)
<IP_ADDRESS>:16379> flushall
OK
<IP_ADDRESS>:16379> hgetall test1
Could not connect to Redis at <IP_ADDRESS>:16379: Connection refused
not connected>

Here's the last few lines of ardb-server output:
[98921] 07-17 21:50:25,639 INFO [RocksDB]--------------- Options for column family [0]:

[98921] 07-17 21:50:25,639 INFO [RocksDB]              Options.comparator: rocksdb.InternalKeyComparator:ardb.comparator
[98921] 07-17 21:50:25,639 INFO [RocksDB]          Options.merge_operator: ardb.merger
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compaction_filter: None
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compaction_filter_factory: ardb.compact_filter_factory
[98921] 07-17 21:50:25,639 INFO [RocksDB]        Options.memtable_factory: SkipListFactory
[98921] 07-17 21:50:25,639 INFO [RocksDB]           Options.table_factory: BlockBasedTable
[98921] 07-17 21:50:25,639 INFO [RocksDB]           table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7fd3f2c4d320)
  cache_index_and_filter_blocks: 0
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x7fd3f2c4e698
  block_cache_size: 536870912
  block_cache_compressed: 0x0
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 1
  skip_table_builder_flush: 0
  format_version: 0

[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.write_buffer_size: 134217728
[98921] 07-17 21:50:25,639 INFO [RocksDB] Options.max_write_buffer_number: 6
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compression[0]: NoCompression
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compression[1]: NoCompression
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compression[2]: Snappy
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compression[3]: Snappy
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compression[4]: Snappy
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compression[5]: Snappy
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.compression[6]: Snappy
[98921] 07-17 21:50:25,639 INFO [RocksDB]      Options.prefix_extractor: ardb.prefix_extractor
[98921] 07-17 21:50:25,639 INFO [RocksDB]            Options.num_levels: 7
[98921] 07-17 21:50:25,639 INFO [RocksDB]       Options.min_write_buffer_number_to_merge: 2
[98921] 07-17 21:50:25,639 INFO [RocksDB]    Options.max_write_buffer_number_to_maintain: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]           Options.compression_opts.window_bits: -14
[98921] 07-17 21:50:25,639 INFO [RocksDB]                 Options.compression_opts.level: -1
[98921] 07-17 21:50:25,639 INFO [RocksDB]              Options.compression_opts.strategy: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]     Options.level0_file_num_compaction_trigger: 2
[98921] 07-17 21:50:25,639 INFO [RocksDB]         Options.level0_slowdown_writes_trigger: 20
[98921] 07-17 21:50:25,639 INFO [RocksDB]             Options.level0_stop_writes_trigger: 24
[98921] 07-17 21:50:25,639 INFO [RocksDB]                  Options.target_file_size_base: 67108864
[98921] 07-17 21:50:25,639 INFO [RocksDB]            Options.target_file_size_multiplier: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]               Options.max_bytes_for_level_base: 536870912
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.level_compaction_dynamic_level_bytes: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]         Options.max_bytes_for_level_multiplier: 10
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[0]: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[1]: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[2]: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[3]: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[4]: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[5]: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[6]: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]      Options.max_sequential_skip_in_iterations: 8
[98921] 07-17 21:50:25,639 INFO [RocksDB]             Options.expanded_compaction_factor: 25
[98921] 07-17 21:50:25,639 INFO [RocksDB]               Options.source_compaction_factor: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]         Options.max_grandparent_overlap_factor: 10
[98921] 07-17 21:50:25,639 INFO [RocksDB]                       Options.arena_block_size: 16777216
[98921] 07-17 21:50:25,639 INFO [RocksDB]                      Options.soft_rate_limit: 0.00
[98921] 07-17 21:50:25,639 INFO [RocksDB]  Options.hard_pending_compaction_bytes_limit: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]      Options.rate_limit_delay_max_milliseconds: 1000
[98921] 07-17 21:50:25,639 INFO [RocksDB]               Options.disable_auto_compactions: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]                          Options.filter_deletes: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]          Options.verify_checksums_in_compaction: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]                        Options.compaction_style: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]                          Options.compaction_pri: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB] Options.compaction_options_universal.size_ratio: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.compaction_options_universal.min_merge_width: 2
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.compaction_options_universal.max_merge_width:<PHONE_NUMBER>
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.compaction_options_universal.max_size_amplification_percent: 200
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.compaction_options_universal.compression_size_percent: -1
[98921] 07-17 21:50:25,639 INFO [RocksDB]Options.compaction_options_fifo.max_table_files_size:<PHONE_NUMBER>
[98921] 07-17 21:50:25,639 INFO [RocksDB]                  Options.table_properties_collectors:
[98921] 07-17 21:50:25,639 INFO [RocksDB]                  Options.inplace_update_support: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]                Options.inplace_update_num_locks: 10000
[98921] 07-17 21:50:25,639 INFO [RocksDB]              Options.min_partial_merge_operands: 2
[98921] 07-17 21:50:25,639 INFO [RocksDB]              Options.memtable_prefix_bloom_bits: 100000000
[98921] 07-17 21:50:25,639 INFO [RocksDB]            Options.memtable_prefix_bloom_probes: 6
[98921] 07-17 21:50:25,639 INFO [RocksDB]  Options.memtable_prefix_bloom_huge_page_tlb_size: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]                          Options.bloom_locality: 1
[98921] 07-17 21:50:25,639 INFO [RocksDB]                   Options.max_successive_merges: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]               Options.optimize_fllters_for_hits: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]               Options.paranoid_file_checks: 0
[98921] 07-17 21:50:25,639 INFO [RocksDB]               Options.compaction_measure_io_stats: 0
[98921] 07-17 21:50:25,641 INFO [RocksDB]Created column family [0] (ID 161)
[98921] 07-17 21:50:25,641 INFO Create ColumnFamilyHandle with name:0 success.
[98921] 07-17 21:50:29,268 INFO RocksDB drop column family:0.
[98921] 07-17 21:50:29,272 INFO [RocksDB]Dropped column family with id 161

Segmentation fault: 11 (core dumped)

It's fixed by latest commit.

With latest commit, I now get:
[99295] 07-18 06:21:51,732 INFO [RocksDB]--------------- Options for column family [0]:

[99295] 07-18 06:21:51,732 INFO [RocksDB]              Options.comparator: rocksdb.InternalKeyComparator:ardb.comparator
[99295] 07-18 06:21:51,732 INFO [RocksDB]          Options.merge_operator: ardb.merger
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compaction_filter: None
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compaction_filter_factory: ardb.compact_filter_factory
[99295] 07-18 06:21:51,732 INFO [RocksDB]        Options.memtable_factory: SkipListFactory
[99295] 07-18 06:21:51,732 INFO [RocksDB]           Options.table_factory: BlockBasedTable
[99295] 07-18 06:21:51,732 INFO [RocksDB]           table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7fea51c56d20)
  cache_index_and_filter_blocks: 0
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x7fea51c58098
  block_cache_size: 536870912
  block_cache_compressed: 0x0
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 1
  skip_table_builder_flush: 0
  format_version: 0

[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.write_buffer_size: 134217728
[99295] 07-18 06:21:51,732 INFO [RocksDB] Options.max_write_buffer_number: 6
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compression[0]: NoCompression
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compression[1]: NoCompression
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compression[2]: Snappy
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compression[3]: Snappy
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compression[4]: Snappy
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compression[5]: Snappy
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.compression[6]: Snappy
[99295] 07-18 06:21:51,732 INFO [RocksDB]      Options.prefix_extractor: ardb.prefix_extractor
[99295] 07-18 06:21:51,732 INFO [RocksDB]            Options.num_levels: 7
[99295] 07-18 06:21:51,732 INFO [RocksDB]       Options.min_write_buffer_number_to_merge: 2
[99295] 07-18 06:21:51,732 INFO [RocksDB]    Options.max_write_buffer_number_to_maintain: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]           Options.compression_opts.window_bits: -14
[99295] 07-18 06:21:51,732 INFO [RocksDB]                 Options.compression_opts.level: -1
[99295] 07-18 06:21:51,732 INFO [RocksDB]              Options.compression_opts.strategy: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]     Options.level0_file_num_compaction_trigger: 2
[99295] 07-18 06:21:51,732 INFO [RocksDB]         Options.level0_slowdown_writes_trigger: 20
[99295] 07-18 06:21:51,732 INFO [RocksDB]             Options.level0_stop_writes_trigger: 24
[99295] 07-18 06:21:51,732 INFO [RocksDB]                  Options.target_file_size_base: 67108864
[99295] 07-18 06:21:51,732 INFO [RocksDB]            Options.target_file_size_multiplier: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]               Options.max_bytes_for_level_base: 536870912
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.level_compaction_dynamic_level_bytes: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]         Options.max_bytes_for_level_multiplier: 10
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[0]: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[1]: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[2]: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[3]: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[4]: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[5]: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[6]: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]      Options.max_sequential_skip_in_iterations: 8
[99295] 07-18 06:21:51,732 INFO [RocksDB]             Options.expanded_compaction_factor: 25
[99295] 07-18 06:21:51,732 INFO [RocksDB]               Options.source_compaction_factor: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]         Options.max_grandparent_overlap_factor: 10
[99295] 07-18 06:21:51,732 INFO [RocksDB]                       Options.arena_block_size: 16777216
[99295] 07-18 06:21:51,732 INFO [RocksDB]                      Options.soft_rate_limit: 0.00
[99295] 07-18 06:21:51,732 INFO [RocksDB]  Options.hard_pending_compaction_bytes_limit: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]      Options.rate_limit_delay_max_milliseconds: 1000
[99295] 07-18 06:21:51,732 INFO [RocksDB]               Options.disable_auto_compactions: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]                          Options.filter_deletes: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]          Options.verify_checksums_in_compaction: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]                        Options.compaction_style: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]                          Options.compaction_pri: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB] Options.compaction_options_universal.size_ratio: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.compaction_options_universal.min_merge_width: 2
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.compaction_options_universal.max_merge_width:<PHONE_NUMBER>
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.compaction_options_universal.max_size_amplification_percent: 200
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.compaction_options_universal.compression_size_percent: -1
[99295] 07-18 06:21:51,732 INFO [RocksDB]Options.compaction_options_fifo.max_table_files_size:<PHONE_NUMBER>
[99295] 07-18 06:21:51,732 INFO [RocksDB]                  Options.table_properties_collectors:
[99295] 07-18 06:21:51,732 INFO [RocksDB]                  Options.inplace_update_support: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]                Options.inplace_update_num_locks: 10000
[99295] 07-18 06:21:51,732 INFO [RocksDB]              Options.min_partial_merge_operands: 2
[99295] 07-18 06:21:51,732 INFO [RocksDB]              Options.memtable_prefix_bloom_bits: 100000000
[99295] 07-18 06:21:51,732 INFO [RocksDB]            Options.memtable_prefix_bloom_probes: 6
[99295] 07-18 06:21:51,732 INFO [RocksDB]  Options.memtable_prefix_bloom_huge_page_tlb_size: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]                          Options.bloom_locality: 1
[99295] 07-18 06:21:51,732 INFO [RocksDB]                   Options.max_successive_merges: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]               Options.optimize_fllters_for_hits: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]               Options.paranoid_file_checks: 0
[99295] 07-18 06:21:51,732 INFO [RocksDB]               Options.compaction_measure_io_stats: 0
[99295] 07-18 06:21:51,734 INFO [RocksDB]Created column family [0] (ID 175)
[99295] 07-18 06:21:51,734 INFO Create ColumnFamilyHandle with name:0 success.
[99295] 07-18 06:21:51,753 FATAL Not a valid key slice in PrefixExtractor with len:0
Abort trap: 6 (core dumped)
jloehrer@jl-mac~/Code/ardb$ lldb ./src/ardb-server -c /cores/core.99295
(lldb) target create "./src/ardb-server" --core "/cores/core.99295"
warning: (x86_64) /cores/core.99295 load command 151 LC_SEGMENT_64 has a fileoff + filesize (0x390f7000) that extends beyond the end of the file (0x390f6000), the segment will be truncated to match
warning: (x86_64) /cores/core.99295 load command 152 LC_SEGMENT_64 has a fileoff (0x390f7000) that extends beyond the end of the file (0x390f6000), ignoring this section
Core file '/cores/core.99295' (x86_64) was loaded.
(lldb) bt
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10
    frame #1: 0x000000010a2f9c6a ardb-server`aeProcessEvents [inlined] aeApiPoll(eventLoop=0x00007fea53000118, tvp=0x00007fff55952248) + 87 at ae_select.cc:48
    frame #2: 0x000000010a2f9c13 ardb-server`aeProcessEvents(eventLoop=0x00007fea53000118, flags=<unavailable>) + 307 at ae.c:413
    frame #3: 0x000000010a2fa11b ardb-server`aeMain(eventLoop=0x00007fea53000118) + 43 at ae.c:485
    frame #4: 0x000000010a2b6ee1 ardb-server`ardb::Server::Start(this=0x00007fff55952460) + 2417 at network.cpp:300
    frame #5: 0x000000010a2ae018 ardb-server`main(argc=<unavailable>, argv=<unavailable>) + 472 at main.cpp:94
    frame #6: 0x00007fff9c9e55ad libdyld.dylib`start + 1
(lldb) thread list
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #2: tid = 0x0001, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #3: tid = 0x0002, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #4: tid = 0x0003, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #5: tid = 0x0004, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #6: tid = 0x0005, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #7: tid = 0x0006, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #8: tid = 0x0007, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #9: tid = 0x0008, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #10: tid = 0x0009, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #11: tid = 0x000a, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #12: tid = 0x000b, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #13: tid = 0x000c, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #14: tid = 0x000d, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #15: tid = 0x000e, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #16: tid = 0x000f, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #17: tid = 0x0010, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #18: tid = 0x0011, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #19: tid = 0x0012, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #20: tid = 0x0013, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #21: tid = 0x0014, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #22: tid = 0x0015, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #23: tid = 0x0016, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #24: tid = 0x0017, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #25: tid = 0x0018, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #26: tid = 0x0019, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
  thread #27: tid = 0x001a, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #28: tid = 0x001b, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
(lldb) thread select 26
* thread #26: tid = 0x0019, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
    frame #0: 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff87cc8f06 <+10>: jae    0x7fff87cc8f10            ; <+20>
    0x7fff87cc8f08 <+12>: movq   %rax, %rdi
    0x7fff87cc8f0b <+15>: jmp    0x7fff87cc37cd            ; cerror_nocancel
    0x7fff87cc8f10 <+20>: retq
(lldb) bt
warning: could not load any Objective-C class information. This will significantly reduce the quality of type information available.
* thread #26: tid = 0x0019, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff95bdf4ec libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff959466e7 libsystem_c.dylib`abort + 129
    frame #3: 0x000000010a392e92 ardb-server`ardb::RocksDBPrefixExtractor::Transform(this=<unavailable>, src=<unavailable>) const + 162 at rocksdb_engine.cpp:418
    frame #4: 0x000000010a42f782 ardb-server`rocksdb::MemTable::Get(this=0x00007fea5400aa00, key=0x00007000050486e0, value="", s=0x00007fea51e03d90, merge_context=0x0000700005048698, seq=0x00007000050486d0) + 210 at memtable.cc:521
    frame #5: 0x000000010a3e6827 ardb-server`rocksdb::DBImpl::MultiGet(rocksdb::ReadOptions const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> > const&, std::__1::vector<rocksdb::Slice, std::__1::allocator<rocksdb::Slice> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >*) [inlined] rocksdb::MemTable::Get(this=<unavailable>, key=<unavailable>, value=<unavailable>, s=<unavailable>, merge_context=<unavailable>) + 1911 at memtable.h:175
    frame #6: 0x000000010a3e6804 ardb-server`rocksdb::DBImpl::MultiGet(this=0x00007fea5202ac00, read_options=0x00007000050488a8, column_family=0x0000700005048920, keys=0x0000700005048900, values=0x00007fea51c5bc88 size=23) + 1876 at db_impl.cc:3199
    frame #7: 0x000000010a38ddb2 ardb-server`ardb::RocksDBEngine::MultiGet(this=<unavailable>, ctx=<unavailable>, keys=<unavailable>, values=size=23, errs=size=23) + 1010 at rocksdb_engine.cpp:1051
    frame #8: 0x000000010a3345df ardb-server`ardb::Ardb::HMGet(this=0x00007fff55952500, ctx=0x00007fea51c59b30, cmd=<unavailable>) + 655 at t_hash.cpp:325
    frame #9: 0x000000010a361752 ardb-server`ardb::Ardb::DoCall(this=0x00007fff55952500, ctx=0x00007fea51c59b30, setting=0x00007fea5203e7d8, args=0x0000700005048c68) + 258 at db.cpp:1395
    frame #10: 0x000000010a361ecb ardb-server`ardb::Ardb::Call(this=0x00007fff55952500, ctx=0x00007fea51c59b30, args=0x0000700005048c68) + 1483 at db.cpp:1574
    frame #11: 0x000000010a2b7c47 ardb-server`ardb::RedisRequestHandler::MessageReceived(this=0x00007fea51c59ad0, ctx=<unavailable>, e=<unavailable>) + 231 at network.cpp:86
    frame #12: 0x000000010a2ceb3a ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] ardb::ChannelUpstreamHandler<ardb::codec::RedisCommandFrame>::HandleStreamEvent(e=<unavailable>) + 506 at channel_upstream_handler.hpp:133
    frame #13: 0x000000010a2ceb27 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::codec::RedisCommandFrame>(ardb::ChannelHandlerContext*, ardb::MessageEvent<ardb::codec::RedisCommandFrame>&) at all_includes.hpp:89
    frame #14: 0x000000010a2ceb27 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelHandlerContext::SendUpstream<ardb::MessageEvent<ardb::codec::RedisCommandFrame> >(this=<unavailable>) at all_includes.hpp:164
    frame #15: 0x000000010a2ceb27 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::fire_message_received<ardb::codec::RedisCommandFrame>(ctx=0x00007fea51c5adf0) at channel_helper.hpp:91
    frame #16: 0x000000010a2ceb27 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(this=0x00007fea51c5ba60, context=0x00007fea51c5adf0, channel=0x00007fea51c5b940, cumulation=0x00007fea51c5b9a8) + 487 at stack_frame_decoder.hpp:102
    frame #17: 0x000000010a2ccaa7 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::MessageReceived(this=0x00007fea51c5ba60, ctx=<unavailable>, e=0x0000700005048db0) + 327 at stack_frame_decoder.hpp:157
    frame #18: 0x000000010a2bcdc5 ardb-server`ardb::Channel::OnRead() [inlined] ardb::ChannelUpstreamHandler<ardb::Buffer>::HandleStreamEvent(e=<unavailable>) + 389 at channel_upstream_handler.hpp:133
    frame #19: 0x000000010a2bcdb2 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::Buffer>(ardb::ChannelHandlerContext*, ardb::MessageEvent<ardb::Buffer>&) + 66 at all_includes.hpp:89
    frame #20: 0x000000010a2bcd70 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::MessageEvent<ardb::Buffer> >(this=<unavailable>) + 16 at all_includes.hpp:128
    frame #21: 0x000000010a2bcd60 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::fire_message_received<ardb::Buffer>(message=0x000000010a5bff40) + 43 at channel_helper.hpp:83
    frame #22: 0x000000010a2bcd35 ardb-server`ardb::Channel::OnRead(this=<unavailable>) + 245 at channel.cpp:550
    frame #23: 0x000000010a2bb21e ardb-server`ardb::Channel::IOEventCallback(eventLoop=<unavailable>, fd=<unavailable>, clientData=0x00007fea51c5b940, mask=1) + 30 at channel.cpp:53
    frame #24: 0x000000010a2f9d56 ardb-server`aeProcessEvents(eventLoop=0x00007fea51e02098, flags=<unavailable>) + 630 at ae.c:429
    frame #25: 0x000000010a2fa11b ardb-server`aeMain(eventLoop=0x00007fea51e02098) + 43 at ae.c:485
    frame #26: 0x000000010a2e6a55 ardb-server`ardb::Thread::ThreadFunc(data=0x00007fea51c53990) + 21 at thread.cpp:38
    frame #27: 0x00007fff95bdc99d libsystem_pthread.dylib`_pthread_body + 131
    frame #28: 0x00007fff95bdc91a libsystem_pthread.dylib`_pthread_start + 168
    frame #29: 0x00007fff95bda351 libsystem_pthread.dylib`thread_start + 13
(lldb)

seems there is a bug for empty key in rocksdb.    fixed in last commit.
is there any hsetnx/setnx in your unit test? setnx/hetnx may generate empty keys in rocksdb.

yes, it is used frequently.

okay that latest commit allowed one set of previously failing unit tests to pass. Here's the next core dump I get:
[201] 07-18 06:44:35,087 INFO [RocksDB]--------------- Options for column family [0]:

[201] 07-18 06:44:35,087 INFO [RocksDB]              Options.comparator: rocksdb.InternalKeyComparator:ardb.comparator
[201] 07-18 06:44:35,087 INFO [RocksDB]          Options.merge_operator: ardb.merger
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compaction_filter: None
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compaction_filter_factory: ardb.compact_filter_factory
[201] 07-18 06:44:35,087 INFO [RocksDB]        Options.memtable_factory: SkipListFactory
[201] 07-18 06:44:35,087 INFO [RocksDB]           Options.table_factory: BlockBasedTable
[201] 07-18 06:44:35,087 INFO [RocksDB]           table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7f91d2c56d20)
  cache_index_and_filter_blocks: 0
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x7f91d2c58098
  block_cache_size: 536870912
  block_cache_compressed: 0x0
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 1
  skip_table_builder_flush: 0
  format_version: 0

[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.write_buffer_size: 134217728
[201] 07-18 06:44:35,087 INFO [RocksDB] Options.max_write_buffer_number: 6
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compression[0]: NoCompression
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compression[1]: NoCompression
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compression[2]: Snappy
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compression[3]: Snappy
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compression[4]: Snappy
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compression[5]: Snappy
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.compression[6]: Snappy
[201] 07-18 06:44:35,087 INFO [RocksDB]      Options.prefix_extractor: ardb.prefix_extractor
[201] 07-18 06:44:35,087 INFO [RocksDB]            Options.num_levels: 7
[201] 07-18 06:44:35,087 INFO [RocksDB]       Options.min_write_buffer_number_to_merge: 2
[201] 07-18 06:44:35,087 INFO [RocksDB]    Options.max_write_buffer_number_to_maintain: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]           Options.compression_opts.window_bits: -14
[201] 07-18 06:44:35,087 INFO [RocksDB]                 Options.compression_opts.level: -1
[201] 07-18 06:44:35,087 INFO [RocksDB]              Options.compression_opts.strategy: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]     Options.level0_file_num_compaction_trigger: 2
[201] 07-18 06:44:35,087 INFO [RocksDB]         Options.level0_slowdown_writes_trigger: 20
[201] 07-18 06:44:35,087 INFO [RocksDB]             Options.level0_stop_writes_trigger: 24
[201] 07-18 06:44:35,087 INFO [RocksDB]                  Options.target_file_size_base: 67108864
[201] 07-18 06:44:35,087 INFO [RocksDB]            Options.target_file_size_multiplier: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]               Options.max_bytes_for_level_base: 536870912
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.level_compaction_dynamic_level_bytes: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]         Options.max_bytes_for_level_multiplier: 10
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[0]: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[1]: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[2]: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[3]: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[4]: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[5]: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[6]: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]      Options.max_sequential_skip_in_iterations: 8
[201] 07-18 06:44:35,087 INFO [RocksDB]             Options.expanded_compaction_factor: 25
[201] 07-18 06:44:35,087 INFO [RocksDB]               Options.source_compaction_factor: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]         Options.max_grandparent_overlap_factor: 10
[201] 07-18 06:44:35,087 INFO [RocksDB]                       Options.arena_block_size: 16777216
[201] 07-18 06:44:35,087 INFO [RocksDB]                      Options.soft_rate_limit: 0.00
[201] 07-18 06:44:35,087 INFO [RocksDB]  Options.hard_pending_compaction_bytes_limit: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]      Options.rate_limit_delay_max_milliseconds: 1000
[201] 07-18 06:44:35,087 INFO [RocksDB]               Options.disable_auto_compactions: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]                          Options.filter_deletes: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]          Options.verify_checksums_in_compaction: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]                        Options.compaction_style: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]                          Options.compaction_pri: 0
[201] 07-18 06:44:35,087 INFO [RocksDB] Options.compaction_options_universal.size_ratio: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.compaction_options_universal.min_merge_width: 2
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.compaction_options_universal.max_merge_width:<PHONE_NUMBER>
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.compaction_options_universal.max_size_amplification_percent: 200
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.compaction_options_universal.compression_size_percent: -1
[201] 07-18 06:44:35,087 INFO [RocksDB]Options.compaction_options_fifo.max_table_files_size:<PHONE_NUMBER>
[201] 07-18 06:44:35,087 INFO [RocksDB]                  Options.table_properties_collectors:
[201] 07-18 06:44:35,087 INFO [RocksDB]                  Options.inplace_update_support: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]                Options.inplace_update_num_locks: 10000
[201] 07-18 06:44:35,087 INFO [RocksDB]              Options.min_partial_merge_operands: 2
[201] 07-18 06:44:35,087 INFO [RocksDB]              Options.memtable_prefix_bloom_bits: 100000000
[201] 07-18 06:44:35,087 INFO [RocksDB]            Options.memtable_prefix_bloom_probes: 6
[201] 07-18 06:44:35,087 INFO [RocksDB]  Options.memtable_prefix_bloom_huge_page_tlb_size: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]                          Options.bloom_locality: 1
[201] 07-18 06:44:35,087 INFO [RocksDB]                   Options.max_successive_merges: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]               Options.optimize_fllters_for_hits: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]               Options.paranoid_file_checks: 0
[201] 07-18 06:44:35,087 INFO [RocksDB]               Options.compaction_measure_io_stats: 0
[201] 07-18 06:44:35,089 INFO [RocksDB]Created column family [0] (ID 245)
[201] 07-18 06:44:35,089 INFO Create ColumnFamilyHandle with name:0 success.
Segmentation fault: 11 (core dumped)
jloehrer@jl-mac~/Code/ardb$ lldb ./src/ardb-server -c /cores/core.201
(lldb) target create "./src/ardb-server" --core "/cores/core.201"
warning: (x86_64) /cores/core.201 load command 152 LC_SEGMENT_64 has a fileoff + filesize (0x398f7000) that extends beyond the end of the file (0x398f6000), the segment will be truncated to match
warning: (x86_64) /cores/core.201 load command 153 LC_SEGMENT_64 has a fileoff (0x398f7000) that extends beyond the end of the file (0x398f6000), ignoring this section
Core file '/cores/core.201' (x86_64) was loaded.
(lldb) bt
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10
    frame #1: 0x000000010132fc4a ardb-server`aeProcessEvents [inlined] aeApiPoll(eventLoop=0x00007f91d2c07ea8, tvp=0x00007fff5e91c248) + 87 at ae_select.cc:48
    frame #2: 0x000000010132fbf3 ardb-server`aeProcessEvents(eventLoop=0x00007f91d2c07ea8, flags=<unavailable>) + 307 at ae.c:413
    frame #3: 0x00000001013300fb ardb-server`aeMain(eventLoop=0x00007f91d2c07ea8) + 43 at ae.c:485
    frame #4: 0x00000001012ecec1 ardb-server`ardb::Server::Start(this=0x00007fff5e91c460) + 2417 at network.cpp:300
    frame #5: 0x00000001012e3ff8 ardb-server`main(argc=<unavailable>, argv=<unavailable>) + 472 at main.cpp:94
    frame #6: 0x00007fff9c9e55ad libdyld.dylib`start + 1
(lldb) thread list
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #2: tid = 0x0001, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #3: tid = 0x0002, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #4: tid = 0x0003, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #5: tid = 0x0004, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #6: tid = 0x0005, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #7: tid = 0x0006, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #8: tid = 0x0007, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #9: tid = 0x0008, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #10: tid = 0x0009, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #11: tid = 0x000a, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #12: tid = 0x000b, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #13: tid = 0x000c, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #14: tid = 0x000d, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #15: tid = 0x000e, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #16: tid = 0x000f, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #17: tid = 0x0010, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #18: tid = 0x0011, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #19: tid = 0x0012, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #20: tid = 0x0013, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #21: tid = 0x0014, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #22: tid = 0x0015, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #23: tid = 0x0016, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #24: tid = 0x0017, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #25: tid = 0x0018, 0x0000000101360024 ardb-server`ardb::monitor_write_callback(ardb::Channel*, void*) [inlined] ardb::Buffer::Readable() const at buffer.hpp:122, stop reason = signal SIGSTOP
  thread #26: tid = 0x0019, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #27: tid = 0x001a, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #28: tid = 0x001b, 0x00007fff9593b76e libsystem_c.dylib`_vsnprintf + 173, stop reason = signal SIGSTOP
(lldb) thread select 25
* thread #25: tid = 0x0018, 0x0000000101360024 ardb-server`ardb::monitor_write_callback(ardb::Channel*, void*) [inlined] ardb::Buffer::Readable() const at buffer.hpp:122, stop reason = signal SIGSTOP
    frame #0: 0x0000000101360024 ardb-server`ardb::monitor_write_callback(ardb::Channel*, void*) [inlined] ardb::Buffer::Readable() const at buffer.hpp:122
   119 				}
   120 				inline bool Readable() const
   121 				{
-> 122 					return m_write_idx > m_read_idx;
   123 				}
   124 				inline bool Writeable() const
   125 				{
(lldb) bt
warning: could not load any Objective-C class information. This will significantly reduce the quality of type information available.
* thread #25: tid = 0x0018, 0x0000000101360024 ardb-server`ardb::monitor_write_callback(ardb::Channel*, void*) [inlined] ardb::Buffer::Readable() const at buffer.hpp:122, stop reason = signal SIGSTOP
  * frame #0: 0x0000000101360024 ardb-server`ardb::monitor_write_callback(ardb::Channel*, void*) [inlined] ardb::Buffer::Readable() const at buffer.hpp:122
    frame #1: 0x0000000101360024 ardb-server`ardb::monitor_write_callback(ardb::Channel*, void*) [inlined] ardb::Buffer::ReadableBytes() const at buffer.hpp:130
    frame #2: 0x0000000101360024 ardb-server`ardb::monitor_write_callback(ch=0x00007f91d50002e0, data=0x7000000000000000) + 52 at server.cpp:1021
    frame #3: 0x00000001012f77f3 ardb-server`ardb::ChannelService::OnSoftSignal(this=0x00007f91d2e02c10, soft_signo=<unavailable>, appendinfo=<unavailable>) + 67 at channel_service.cpp:175
    frame #4: 0x000000010130ec33 ardb-server`ardb::SoftSignalChannel::FireSignalReceived(this=<unavailable>, signo=4, append_info=1) + 243 at soft_signal_channel.cpp:72
    frame #5: 0x000000010130f764 ardb-server`ardb::codec::StackFrameDecoder<unsigned long long>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] ardb::ChannelUpstreamHandler<unsigned long long>::HandleStreamEvent(e=<unavailable>) + 292 at channel_upstream_handler.hpp:133
    frame #6: 0x000000010130f751 ardb-server`ardb::codec::StackFrameDecoder<unsigned long long>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelPipeline::SendUpstream<unsigned long long>(ardb::ChannelHandlerContext*, ardb::MessageEvent<unsigned long long>&) at all_includes.hpp:89
    frame #7: 0x000000010130f751 ardb-server`ardb::codec::StackFrameDecoder<unsigned long long>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelHandlerContext::SendUpstream<ardb::MessageEvent<unsigned long long> >(this=<unavailable>) at all_includes.hpp:164
    frame #8: 0x000000010130f751 ardb-server`ardb::codec::StackFrameDecoder<unsigned long long>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::fire_message_received<unsigned long long>(ctx=0x00007f91d2e02b50) at channel_helper.hpp:91
    frame #9: 0x000000010130f751 ardb-server`ardb::codec::StackFrameDecoder<unsigned long long>::CallDecode(this=0x00007f91d2e03058, context=0x00007f91d2e02b50, channel=0x00007f91d2e02f40, cumulation=0x00007f91d2e02fa8) + 273 at stack_frame_decoder.hpp:102
    frame #10: 0x000000010130f2b7 ardb-server`ardb::codec::StackFrameDecoder<unsigned long long>::MessageReceived(this=0x00007f91d2e03058, ctx=<unavailable>, e=0x0000700004845db0) + 327 at stack_frame_decoder.hpp:157
    frame #11: 0x00000001012f2da5 ardb-server`ardb::Channel::OnRead() [inlined] ardb::ChannelUpstreamHandler<ardb::Buffer>::HandleStreamEvent(e=<unavailable>) + 389 at channel_upstream_handler.hpp:133
    frame #12: 0x00000001012f2d92 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::Buffer>(ardb::ChannelHandlerContext*, ardb::MessageEvent<ardb::Buffer>&) + 66 at all_includes.hpp:89
    frame #13: 0x00000001012f2d50 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::MessageEvent<ardb::Buffer> >(this=<unavailable>) + 16 at all_includes.hpp:128
    frame #14: 0x00000001012f2d40 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::fire_message_received<ardb::Buffer>(message=0x00000001015f5f40) + 43 at channel_helper.hpp:83
    frame #15: 0x00000001012f2d15 ardb-server`ardb::Channel::OnRead(this=<unavailable>) + 245 at channel.cpp:550
    frame #16: 0x00000001012f11fe ardb-server`ardb::Channel::IOEventCallback(eventLoop=<unavailable>, fd=<unavailable>, clientData=0x00007f91d2e02f40, mask=1) + 30 at channel.cpp:53
    frame #17: 0x000000010132fd36 ardb-server`aeProcessEvents(eventLoop=0x00007f91d2e02b08, flags=<unavailable>) + 630 at ae.c:429
    frame #18: 0x00000001013300fb ardb-server`aeMain(eventLoop=0x00007f91d2e02b08) + 43 at ae.c:485
    frame #19: 0x000000010131ca35 ardb-server`ardb::Thread::ThreadFunc(data=0x00007f91d2e02bd0) + 21 at thread.cpp:38
    frame #20: 0x00007fff95bdc99d libsystem_pthread.dylib`_pthread_body + 131
    frame #21: 0x00007fff95bdc91a libsystem_pthread.dylib`_pthread_start + 168
    frame #22: 0x00007fff95bda351 libsystem_pthread.dylib`thread_start + 13
(lldb)

is it crashed with monitor command?

I was running monitor in another window, but that doesn't occur in my unit tests anywhere. I'll re-run the tests with no monitor.

here you go:
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.write_buffer_size: 134217728
[324] 07-18 06:59:49,159 INFO [RocksDB] Options.max_write_buffer_number: 6
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.compression[0]: NoCompression
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.compression[1]: NoCompression
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.compression[2]: Snappy
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.compression[3]: Snappy
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.compression[4]: Snappy
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.compression[5]: Snappy
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.compression[6]: Snappy
[324] 07-18 06:59:49,159 INFO [RocksDB]      Options.prefix_extractor: ardb.prefix_extractor
[324] 07-18 06:59:49,159 INFO [RocksDB]            Options.num_levels: 7
[324] 07-18 06:59:49,159 INFO [RocksDB]       Options.min_write_buffer_number_to_merge: 2
[324] 07-18 06:59:49,159 INFO [RocksDB]    Options.max_write_buffer_number_to_maintain: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]           Options.compression_opts.window_bits: -14
[324] 07-18 06:59:49,159 INFO [RocksDB]                 Options.compression_opts.level: -1
[324] 07-18 06:59:49,159 INFO [RocksDB]              Options.compression_opts.strategy: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]     Options.level0_file_num_compaction_trigger: 2
[324] 07-18 06:59:49,159 INFO [RocksDB]         Options.level0_slowdown_writes_trigger: 20
[324] 07-18 06:59:49,159 INFO [RocksDB]             Options.level0_stop_writes_trigger: 24
[324] 07-18 06:59:49,159 INFO [RocksDB]                  Options.target_file_size_base: 67108864
[324] 07-18 06:59:49,159 INFO [RocksDB]            Options.target_file_size_multiplier: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]               Options.max_bytes_for_level_base: 536870912
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.level_compaction_dynamic_level_bytes: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]         Options.max_bytes_for_level_multiplier: 10
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[0]: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[1]: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[2]: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[3]: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[4]: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[5]: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.max_bytes_for_level_multiplier_addtl[6]: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]      Options.max_sequential_skip_in_iterations: 8
[324] 07-18 06:59:49,159 INFO [RocksDB]             Options.expanded_compaction_factor: 25
[324] 07-18 06:59:49,159 INFO [RocksDB]               Options.source_compaction_factor: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]         Options.max_grandparent_overlap_factor: 10
[324] 07-18 06:59:49,159 INFO [RocksDB]                       Options.arena_block_size: 16777216
[324] 07-18 06:59:49,159 INFO [RocksDB]                      Options.soft_rate_limit: 0.00
[324] 07-18 06:59:49,159 INFO [RocksDB]  Options.hard_pending_compaction_bytes_limit: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]      Options.rate_limit_delay_max_milliseconds: 1000
[324] 07-18 06:59:49,159 INFO [RocksDB]               Options.disable_auto_compactions: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]                          Options.filter_deletes: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]          Options.verify_checksums_in_compaction: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]                        Options.compaction_style: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]                          Options.compaction_pri: 0
[324] 07-18 06:59:49,159 INFO [RocksDB] Options.compaction_options_universal.size_ratio: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.compaction_options_universal.min_merge_width: 2
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.compaction_options_universal.max_merge_width:<PHONE_NUMBER>
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.compaction_options_universal.max_size_amplification_percent: 200
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.compaction_options_universal.compression_size_percent: -1
[324] 07-18 06:59:49,159 INFO [RocksDB]Options.compaction_options_fifo.max_table_files_size:<PHONE_NUMBER>
[324] 07-18 06:59:49,159 INFO [RocksDB]                  Options.table_properties_collectors:
[324] 07-18 06:59:49,159 INFO [RocksDB]                  Options.inplace_update_support: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]                Options.inplace_update_num_locks: 10000
[324] 07-18 06:59:49,159 INFO [RocksDB]              Options.min_partial_merge_operands: 2
[324] 07-18 06:59:49,159 INFO [RocksDB]              Options.memtable_prefix_bloom_bits: 100000000
[324] 07-18 06:59:49,159 INFO [RocksDB]            Options.memtable_prefix_bloom_probes: 6
[324] 07-18 06:59:49,159 INFO [RocksDB]  Options.memtable_prefix_bloom_huge_page_tlb_size: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]                          Options.bloom_locality: 1
[324] 07-18 06:59:49,159 INFO [RocksDB]                   Options.max_successive_merges: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]               Options.optimize_fllters_for_hits: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]               Options.paranoid_file_checks: 0
[324] 07-18 06:59:49,159 INFO [RocksDB]               Options.compaction_measure_io_stats: 0
[324] 07-18 06:59:49,162 INFO [RocksDB]Created column family [0] (ID 246)
[324] 07-18 06:59:49,162 INFO Create ColumnFamilyHandle with name:0 success.
Segmentation fault: 11 (core dumped)
jloehrer@jl-mac~/Code/ardb$ lldb src/ardb-server -c /cores/core.
core.201  core.324
jloehrer@jl-mac~/Code/ardb$ lldb src/ardb-server -c /cores/core.324
(lldb) target create "src/ardb-server" --core "/cores/core.324"
warning: (x86_64) /cores/core.324 load command 148 LC_SEGMENT_64 has a fileoff + filesize (0x37ff7000) that extends beyond the end of the file (0x37ff6000), the segment will be truncated to match
warning: (x86_64) /cores/core.324 load command 149 LC_SEGMENT_64 has a fileoff (0x37ff7000) that extends beyond the end of the file (0x37ff6000), ignoring this section
Core file '/cores/core.324' (x86_64) was loaded.
(lldb) bt
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10
    frame #1: 0x0000000103996c4a ardb-server`aeProcessEvents [inlined] aeApiPoll(eventLoop=0x00007fc601d016f8, tvp=0x00007fff5c2b5248) + 87 at ae_select.cc:48
    frame #2: 0x0000000103996bf3 ardb-server`aeProcessEvents(eventLoop=0x00007fc601d016f8, flags=<unavailable>) + 307 at ae.c:413
    frame #3: 0x00000001039970fb ardb-server`aeMain(eventLoop=0x00007fc601d016f8) + 43 at ae.c:485
    frame #4: 0x0000000103953ec1 ardb-server`ardb::Server::Start(this=0x00007fff5c2b5460) + 2417 at network.cpp:300
    frame #5: 0x000000010394aff8 ardb-server`main(argc=<unavailable>, argv=<unavailable>) + 472 at main.cpp:94
    frame #6: 0x00007fff9c9e55ad libdyld.dylib`start + 1
(lldb) thread list
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #2: tid = 0x0001, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #3: tid = 0x0002, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #4: tid = 0x0003, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #5: tid = 0x0004, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #6: tid = 0x0005, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #7: tid = 0x0006, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #8: tid = 0x0007, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #9: tid = 0x0008, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #10: tid = 0x0009, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #11: tid = 0x000a, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #12: tid = 0x000b, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #13: tid = 0x000c, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #14: tid = 0x000d, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #15: tid = 0x000e, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #16: tid = 0x000f, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #17: tid = 0x0010, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #18: tid = 0x0011, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #19: tid = 0x0012, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #20: tid = 0x0013, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #21: tid = 0x0014, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #22: tid = 0x0015, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #23: tid = 0x0016, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #24: tid = 0x0017, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #25: tid = 0x0018, 0x0000000103a837f0 ardb-server`rocksdb::DBImpl::MultiGet(rocksdb::ReadOptions const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> > const&, std::__1::vector<rocksdb::Slice, std::__1::allocator<rocksdb::Slice> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >*) + 20 at __hash_table:2020, stop reason = signal SIGSTOP
  thread #26: tid = 0x0019, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #27: tid = 0x001a, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #28: tid = 0x001b, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
(lldb) thread select 25
* thread #25: tid = 0x0018, 0x0000000103a837f0 ardb-server`rocksdb::DBImpl::MultiGet(rocksdb::ReadOptions const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> > const&, std::__1::vector<rocksdb::Slice, std::__1::allocator<rocksdb::Slice> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >*) + 20 at __hash_table:2020, stop reason = signal SIGSTOP
    frame #0: 0x0000000103a837f0 ardb-server`rocksdb::DBImpl::MultiGet(rocksdb::ReadOptions const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> > const&, std::__1::vector<rocksdb::Slice, std::__1::allocator<rocksdb::Slice> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >*) + 20 at __hash_table:2020
   2017	        __node_pointer __nd = __bucket_list_[__chash];
   2018	        if (__nd != nullptr)
   2019	        {
-> 2020	            for (__nd = __nd->__next_; __nd != nullptr &&
   2021	                                       __constrain_hash(__nd->__hash_, __bc) == __chash;
   2022	                                                           __nd = __nd->__next_)
   2023	            {
(lldb)

it's crashed in rocksdb's code now.
can u make clean and then rebuild to test again?

did make clean and then storage_engine=rocksdb make. I also removed the data dir to start fresh, then started it up and ran my tests:
[1161] 07-18 07:52:26,190 INFO [RocksDB]Options.compaction_options_fifo.max_table_files_size:<PHONE_NUMBER>
[1161] 07-18 07:52:26,190 INFO [RocksDB]                  Options.table_properties_collectors:
[1161] 07-18 07:52:26,190 INFO [RocksDB]                  Options.inplace_update_support: 0
[1161] 07-18 07:52:26,190 INFO [RocksDB]                Options.inplace_update_num_locks: 10000
[1161] 07-18 07:52:26,190 INFO [RocksDB]              Options.min_partial_merge_operands: 2
[1161] 07-18 07:52:26,190 INFO [RocksDB]              Options.memtable_prefix_bloom_bits: 100000000
[1161] 07-18 07:52:26,190 INFO [RocksDB]            Options.memtable_prefix_bloom_probes: 6
[1161] 07-18 07:52:26,190 INFO [RocksDB]  Options.memtable_prefix_bloom_huge_page_tlb_size: 0
[1161] 07-18 07:52:26,190 INFO [RocksDB]                          Options.bloom_locality: 1
[1161] 07-18 07:52:26,190 INFO [RocksDB]                   Options.max_successive_merges: 0
[1161] 07-18 07:52:26,190 INFO [RocksDB]               Options.optimize_fllters_for_hits: 0
[1161] 07-18 07:52:26,190 INFO [RocksDB]               Options.paranoid_file_checks: 0
[1161] 07-18 07:52:26,190 INFO [RocksDB]               Options.compaction_measure_io_stats: 0
[1161] 07-18 07:52:26,192 INFO [RocksDB]Created column family [0] (ID 37)
[1161] 07-18 07:52:26,192 INFO Create ColumnFamilyHandle with name:0 success.
[1161] 07-18 07:52:26,211 FATAL Not a valid key slice in PrefixExtractor with len:39
Abort trap: 6 (core dumped)
jloehrer@jl-mac~/Code/ardb$ lldb ./src/ardb-server -c /cores/core.11
core.1115  core.1126  core.1161
jloehrer@jl-mac~/Code/ardb$ lldb ./src/ardb-server -c /cores/core.1161
(lldb) target create "./src/ardb-server" --core "/cores/core.1161"
warning: (x86_64) /cores/core.1161 load command 152 LC_SEGMENT_64 has a fileoff + filesize (0x398f7000) that extends beyond the end of the file (0x398f6000), the segment will be truncated to match
warning: (x86_64) /cores/core.1161 load command 153 LC_SEGMENT_64 has a fileoff (0x398f7000) that extends beyond the end of the file (0x398f6000), ignoring this section
Core file '/cores/core.1161' (x86_64) was loaded.
(lldb) bt
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10
    frame #1: 0x0000000104822c4a ardb-server`aeProcessEvents [inlined] aeApiPoll(eventLoop=0x00007f8313000158, tvp=0x00007fff5b429248) + 87 at ae_select.cc:48
    frame #2: 0x0000000104822bf3 ardb-server`aeProcessEvents(eventLoop=0x00007f8313000158, flags=<unavailable>) + 307 at ae.c:413
    frame #3: 0x00000001048230fb ardb-server`aeMain(eventLoop=0x00007f8313000158) + 43 at ae.c:485
    frame #4: 0x00000001047dfec1 ardb-server`ardb::Server::Start(this=0x00007fff5b429460) + 2417 at network.cpp:300
    frame #5: 0x00000001047d6ff8 ardb-server`main(argc=<unavailable>, argv=<unavailable>) + 472 at main.cpp:94
    frame #6: 0x00007fff9c9e55ad libdyld.dylib`start + 1
(lldb) thread list
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #2: tid = 0x0001, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #3: tid = 0x0002, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #4: tid = 0x0003, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #5: tid = 0x0004, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #6: tid = 0x0005, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #7: tid = 0x0006, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #8: tid = 0x0007, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #9: tid = 0x0008, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #10: tid = 0x0009, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #11: tid = 0x000a, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #12: tid = 0x000b, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #13: tid = 0x000c, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #14: tid = 0x000d, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #15: tid = 0x000e, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #16: tid = 0x000f, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #17: tid = 0x0010, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #18: tid = 0x0011, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #19: tid = 0x0012, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #20: tid = 0x0013, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #21: tid = 0x0014, 0x00007fff87cc8db6 libsystem_kernel.dylib`__psynch_cvwait + 10, stop reason = signal SIGSTOP
  thread #22: tid = 0x0015, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #23: tid = 0x0016, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #24: tid = 0x0017, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #25: tid = 0x0018, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
  thread #26: tid = 0x0019, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #27: tid = 0x001a, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
  thread #28: tid = 0x001b, 0x00007fff87cc907a libsystem_kernel.dylib`__select + 10, stop reason = signal SIGSTOP
(lldb) thread select 25
* thread #25: tid = 0x0018, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
    frame #0: 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff87cc8f06 <+10>: jae    0x7fff87cc8f10            ; <+20>
    0x7fff87cc8f08 <+12>: movq   %rax, %rdi
    0x7fff87cc8f0b <+15>: jmp    0x7fff87cc37cd            ; cerror_nocancel
    0x7fff87cc8f10 <+20>: retq
(lldb) bt
warning: could not load any Objective-C class information. This will significantly reduce the quality of type information available.
* thread #25: tid = 0x0018, 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff87cc8f06 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff95bdf4ec libsystem_pthread.dylib`pthread_kill + 90
    frame #2: 0x00007fff959466e7 libsystem_c.dylib`abort + 129
    frame #3: 0x00000001048bbe7d ardb-server`ardb::RocksDBPrefixExtractor::Transform(this=<unavailable>, src=<unavailable>) const + 173 at rocksdb_engine.cpp:422
    frame #4: 0x0000000104958782 ardb-server`rocksdb::MemTable::Get(this=0x00007f8311034000, key=0x00007000048456e0, value="", s=0x00007f83130033a0, merge_context=0x0000700004845698, seq=0x00007000048456d0) + 210 at memtable.cc:521
    frame #5: 0x000000010490f827 ardb-server`rocksdb::DBImpl::MultiGet(rocksdb::ReadOptions const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> > const&, std::__1::vector<rocksdb::Slice, std::__1::allocator<rocksdb::Slice> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >*) [inlined] rocksdb::MemTable::Get(this=<unavailable>, key=<unavailable>, value=<unavailable>, s=<unavailable>, merge_context=<unavailable>) + 1911 at memtable.h:175
    frame #6: 0x000000010490f804 ardb-server`rocksdb::DBImpl::MultiGet(this=0x00007f831102b400, read_options=0x00007000048458a8, column_family=0x0000700004845920, keys=0x0000700004845900, values=0x00007f8310e02048 size=11) + 1876 at db_impl.cc:3199
    frame #7: 0x00000001048b6d92 ardb-server`ardb::RocksDBEngine::MultiGet(this=<unavailable>, ctx=<unavailable>, keys=<unavailable>, values=size=11, errs=size=11) + 1010 at rocksdb_engine.cpp:1055
    frame #8: 0x000000010485d5bf ardb-server`ardb::Ardb::HMGet(this=0x00007fff5b429500, ctx=0x00007f8310d02940, cmd=<unavailable>) + 655 at t_hash.cpp:325
    frame #9: 0x000000010488a732 ardb-server`ardb::Ardb::DoCall(this=0x00007fff5b429500, ctx=0x00007f8310d02940, setting=0x00007f831103e7d8, args=0x0000700004845c68) + 258 at db.cpp:1395
    frame #10: 0x000000010488aeab ardb-server`ardb::Ardb::Call(this=0x00007fff5b429500, ctx=0x00007f8310d02940, args=0x0000700004845c68) + 1483 at db.cpp:1574
    frame #11: 0x00000001047e0c27 ardb-server`ardb::RedisRequestHandler::MessageReceived(this=0x00007f8310d028e0, ctx=<unavailable>, e=<unavailable>) + 231 at network.cpp:86
    frame #12: 0x00000001047f7b1a ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] ardb::ChannelUpstreamHandler<ardb::codec::RedisCommandFrame>::HandleStreamEvent(e=<unavailable>) + 506 at channel_upstream_handler.hpp:133
    frame #13: 0x00000001047f7b07 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::codec::RedisCommandFrame>(ardb::ChannelHandlerContext*, ardb::MessageEvent<ardb::codec::RedisCommandFrame>&) at all_includes.hpp:89
    frame #14: 0x00000001047f7b07 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::ChannelHandlerContext::SendUpstream<ardb::MessageEvent<ardb::codec::RedisCommandFrame> >(this=<unavailable>) at all_includes.hpp:164
    frame #15: 0x00000001047f7b07 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(ardb::ChannelHandlerContext&, ardb::Channel*, ardb::Buffer&) [inlined] bool ardb::fire_message_received<ardb::codec::RedisCommandFrame>(ctx=0x00007f8310d027c0) at channel_helper.hpp:91
    frame #16: 0x00000001047f7b07 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::CallDecode(this=0x00007f8310d02780, context=0x00007f8310d027c0, channel=0x00007f8310d02660, cumulation=0x00007f8310d026c8) + 487 at stack_frame_decoder.hpp:102
    frame #17: 0x00000001047f5a87 ardb-server`ardb::codec::StackFrameDecoder<ardb::codec::RedisCommandFrame>::MessageReceived(this=0x00007f8310d02780, ctx=<unavailable>, e=0x0000700004845db0) + 327 at stack_frame_decoder.hpp:157
    frame #18: 0x00000001047e5da5 ardb-server`ardb::Channel::OnRead() [inlined] ardb::ChannelUpstreamHandler<ardb::Buffer>::HandleStreamEvent(e=<unavailable>) + 389 at channel_upstream_handler.hpp:133
    frame #19: 0x00000001047e5d92 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::Buffer>(ardb::ChannelHandlerContext*, ardb::MessageEvent<ardb::Buffer>&) + 66 at all_includes.hpp:89
    frame #20: 0x00000001047e5d50 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::ChannelPipeline::SendUpstream<ardb::MessageEvent<ardb::Buffer> >(this=<unavailable>) + 16 at all_includes.hpp:128
    frame #21: 0x00000001047e5d40 ardb-server`ardb::Channel::OnRead() [inlined] bool ardb::fire_message_received<ardb::Buffer>(message=0x0000000104ae8f40) + 43 at channel_helper.hpp:83
    frame #22: 0x00000001047e5d15 ardb-server`ardb::Channel::OnRead(this=<unavailable>) + 245 at channel.cpp:550
    frame #23: 0x00000001047e41fe ardb-server`ardb::Channel::IOEventCallback(eventLoop=<unavailable>, fd=<unavailable>, clientData=0x00007f8310d02660, mask=1) + 30 at channel.cpp:53
    frame #24: 0x0000000104822d36 ardb-server`aeProcessEvents(eventLoop=0x00007f8310c54648, flags=<unavailable>) + 630 at ae.c:429
    frame #25: 0x00000001048230fb ardb-server`aeMain(eventLoop=0x00007f8310c54648) + 43 at ae.c:485
    frame #26: 0x000000010480fa35 ardb-server`ardb::Thread::ThreadFunc(data=0x00007f8310c56f40) + 21 at thread.cpp:38
    frame #27: 0x00007fff95bdc99d libsystem_pthread.dylib`_pthread_body + 131
    frame #28: 0x00007fff95bdc91a libsystem_pthread.dylib`_pthread_start + 168
    frame #29: 0x00007fff95bda351 libsystem_pthread.dylib`thread_start + 13
(lldb)

what commands used before hmget test?
there is FATAL Not a valid key slice in PrefixExtractor with len:39 in log which indicate that the stored key is not valid.

I re-ran the test with monitor connected. Here's the output, but I'm not sure how much it will help.

That seems to fix the segfault issue, at least at first pass. will do a bit more thorough testing. Now that I am able to run my unit tests I am discovering other subtle differences between ardb and the redis reference implementation. I'll open those as separate issues.

made it through several full runs of my application's test suite without segfaults. Progress! Before I couldn't even make it through 1 run of a subset of my tests without a crash. Closing this issue. I'll re-open if I see additional issues in this area.


Change export zip name to distinguish between export types

Is your feature request related to a problem? Please describe.
Currently when labelling sometimes I want to export to VOC format while other times to Brush Labels. So for example switching between image segmentation and object detection. The export zip file name is now hardcodes to the datetime of the export moment.
Describe the solution you'd like
Being able to give in the export name combined with the datetime would be ideal
Describe alternatives you've considered
an other option i can think of is that the format chosen is added to the export name together with the datetime. I think this one will require less changes in the code.

@bram2506 Could you check this PR: https://github.com/heartexlabs/label-studio/pull/551, I've added this option there.

@bram2506 Could you check this PR: https://github.com/heartexlabs/label-studio/pull/551, I've added this option there.


Added isFirstIn utility function

Added the isFirstIn utility function. This function is useful to implement a scroll to first error behaviour. For example, one could write a hook using this utility function:
export const useOnFirstError = (name: string, scrollRef: React.RefObject<HTMLElement>) => {
  const { errors, isSubmitting, isValidating } = useFormikContext()

  useEffect(() => {
    if (isSubmitting && !isValidating && isFirstIn(errors, name)) {
      scrollRef.current?.scrollIntoView()
    }
  }, [errors, name, isSubmitting, isValidating, scrollRef])
}

The isFirstIn function works a bit similar to the getIn function. It accepts an object (like the Formik errors object) and a formik field name. Given these two values it determines if the given name is the first to occur in the given object.
The order of the keys in the errors object is determined by the order of the fields in the Yup validationSchema, so one can order these in the same order as the fields appear on the page.
Finally, one can use this hook in the following way:
import React, { useRef } from 'react'
import { useField, useFormikContext } from 'formik'

import { useOnFirstError } from './useOnFirstError'

type Props = {
  name: string
}

const InputField = ({ name }: Props) => {
  const [field, meta, helpers] = useField(name)
  const inputRef = useRef<HTMLInputElement>(null)

  useOnFirstError(name, inputRef)

  return <input {...field} ref={inputRef}  />
}

<Formik ...>
  <InputField name="firstName" />
  <InputField name="lastName" />
</Formik>

The isFirstIn function is a small function, but it would be beneficial to include this function in Formik. This will allow consumers to use it without requiring a direct dependency on Lodash. Secondary, it's behaviour is strongly coupled to the behaviour of setIn en getIn, changing one of these will like also mean changing the isFirstIn function, so it would make sense to keep these three functions closely together :)
A more complete version of the useOnFirstError hook can be found here:
https://github.com/dhlparcel/formik/pull/2
We can add this hook to this PR if desired.

I spoke with Dan Abramov about this exact thing use case yesterday. We agreed on the following:
The reality is that this functionality really can't be easily done in pure React with refs because we actually don't know if the order of the fields in the errors object actually matches the order in the UI. We also don't know if the field is still mounted just by looking at the errors object either (becuase of conditional rendering or suspense).
The way to do this is with the DOM. One solution we came up with is to add data attribute to mark a scrollable area and then use an effect to scroll to the dom node when you need to. This works well with conditional logic and is safe in Concurrent Mode.
<Form>

    ...

  {something &&

     ...

  }

    ...

</Form>

// Somewhere else...
useEffect(() => {
	// get the list of fields in the order they show in the DOM
	const orderedFieldsElements = document.querySelectorAll('[data-scrollable]')
    smoothScrollIntoViewIfNeeded(orderedFieledElenets.some(el => ... ))
}, [errors])