IVFPQ使用SIFT1M数据集测试召回率低（57%），和官方测试结果（99.99%）存在较大出入

Vearch 3.2.5版本
IVFPQ索引使用SIFT1M数据集自己进行的召回率测试仅有57，和官网召回率测试结果有较大出入，不知道问题出在哪里
下面是我的测试结果

ncen
nsub
nprobe
recall@100

256
32
1
33.3205

256
32
5
52.5332

256
32
10
55.7455

256
32
20
56.8641

256
32
30
57.0442

256
32
40
57.092403

256
32
50
57.1042

256
32
80
57.1102

256
32
100
57.1106

256
32
200
57.1106

建表空间如下
{
"name":"mark",
"partition_num":1,
"replica_num":1,
"engine":{
"name":"gamma",
"index_size":900000,
"id_type":"String",
"retrieval_type":"IVFPQ",
"retrieval_param":{
"metric_type":"InnerProduct",
"ncentroids":256,
"nsubvector":32
}
},
"properties":{
"vec":{
"type":"vector",
"dimension":128,
"format":"normalization",
"store_type":"RocksDB",
"store_param":{
"cache_size":2048,
"compress":{
"rate":16
}
}
}
}
}
如上数据达到900000开始建索引，原始向量数1000000，感觉应该不是这个造成的误差

搜索语句可以提供一下吗？看看gamma.log 索引构建的信息，类似于这种 Add time xxx ms, count xxx

INFO 2021-04-02 02:52:07,126 table.cc:48 Table created success! INFO 2021-04-02 02:52:07,126 gamma_engine.cc:285 GammaEngine setup successed! bitmap_bytes_size=268435456 INFO 2021-04-02 02:52:07,126 gamma_api.cc:52 Engine init successed! INFO 2021-04-02 02:52:07,127 vector_manager.cc:117 store params={"store_type":,"cache_size":2048,"segment_size":500000,"compress":{"rate":16}} INFO 2021-04-02 02:52:07,127 raw_vector_common.h:172 zfp compress rate=16, zfpsize=256 INFO 2021-04-02 02:52:07,141 rocksdb_raw_vector.cc:64 rocks raw vector init success! name=vec, block cache size=2147483648Bytes INFO 2021-04-02 02:52:07,142 raw_vector.cc:221 raw vector init success! name=vec, has source=0, multi_vids=0, vector_byte_size=256, dimension=128, compress={"rate":16} INFO 2021-04-02 02:52:07,142 vector_manager.cc:153 Create index model [IVFPQ] INFO 2021-04-02 02:52:07,142 gamma_index_ivfpq.cc:319 ncentroids =256, nsubvector =32, nbits_per_idx =8, metric_type =0, bucket_init_size =1000, bucket_max_size =1280000 INFO 2021-04-02 02:52:07,154 realtime_mem_data.cc:89 init success! total_mem_bytes=10243072, current max size=256000 INFO 2021-04-02 02:52:07,154 vector_manager.cc:177 create vectors and indexes success! models=[IVFPQ] INFO 2021-04-02 02:52:07,154 table.cc:124 Table compress [0] INFO 2021-04-02 02:52:07,154 table.cc:131 Add field name [_id], type [4], index [0] INFO 2021-04-02 02:52:07,154 table.cc:173 Create table 365 success! item length=5, field num=1 INFO 2021-04-02 02:52:07,154 gamma_engine.cc:636 create table [365] success! INFO 2021-04-02 02:52:07,154 async_flush.h:70 async flush executor is started! INFO 2021-04-02 02:52:07,255 async_flush.h:75 async flush executor exit successfully! INFO 2021-04-02 02:52:07,256 table.cc:109 Table load successed! doc num=0 INFO 2021-04-02 02:52:07,256 table_io.cc:22 Table load successed! doc num=0 INFO 2021-04-02 02:52:07,256 vector_manager.cc:692 vector [vec] load success! INFO 2021-04-02 02:52:07,257 gamma_engine.cc:1229 load engine success! max docid=0, load directory=, clean directorys(not done)=[] INFO 2021-04-02 02:52:07,257 async_flush.h:70 async flush executor is started! INFO 2021-04-02 02:53:12,834 table.cc:390 Add item _id [0], num [0] INFO 2021-04-02 02:53:12,835 table.cc:410 table cost [5.86499]ms INFO 2021-04-02 02:53:16,937 gamma_engine.cc:768 Add total cost [3.77686]ms INFO 2021-04-02 02:53:16,938 table.cc:390 Add item _id [10000], num [10000] INFO 2021-04-02 02:53:16,939 table.cc:410 table cost [1.55005]ms INFO 2021-04-02 02:53:18,703 gamma_engine.cc:768 Add total cost [4.10205]ms INFO 2021-04-02 02:53:18,708 table.cc:390 Add item _id [20200], num [20000] INFO 2021-04-02 02:53:18,709 table.cc:410 table cost [1.49512]ms INFO 2021-04-02 02:53:23,794 gamma_engine.cc:768 Add total cost [3.26904]ms INFO 2021-04-02 02:53:23,795 table.cc:390 Add item _id [30100], num [30000] INFO 2021-04-02 02:53:23,796 table.cc:410 table cost [1.70581]ms INFO 2021-04-02 02:53:25,741 gamma_engine.cc:768 Add total cost [4.35791]ms INFO 2021-04-02 02:53:25,745 table.cc:390 Add item _id [40000], num [40000] INFO 2021-04-02 02:53:25,746 table.cc:410 table cost [1.41821]ms INFO 2021-04-02 02:53:27,826 gamma_engine.cc:768 Add total cost [4.83398]ms INFO 2021-04-02 02:53:27,833 table.cc:390 Add item _id [50100], num [50000] INFO 2021-04-02 02:53:27,834 table.cc:410 table cost [3.08203]ms INFO 2021-04-02 02:53:35,675 gamma_engine.cc:768 Add total cost [3.93921]ms INFO 2021-04-02 02:53:35,682 table.cc:390 Add item _id [60000], num [60000] INFO 2021-04-02 02:53:35,683 table.cc:410 table cost [2.06616]ms INFO 2021-04-02 02:53:37,686 gamma_engine.cc:768 Add total cost [2.97607]ms INFO 2021-04-02 02:53:37,699 table.cc:390 Add item _id [69900], num [70000] INFO 2021-04-02 02:53:37,701 table.cc:410 table cost [1.63281]ms INFO 2021-04-02 02:53:44,749 gamma_engine.cc:768 Add total cost [3.36987]ms INFO 2021-04-02 02:53:44,761 table.cc:390 Add item _id [80200], num [80000] INFO 2021-04-02 02:53:44,762 table.cc:410 table cost [2.65601]ms INFO 2021-04-02 02:53:46,779 gamma_engine.cc:768 Add total cost [3.68604]ms INFO 2021-04-02 02:53:46,801 table.cc:390 Add item _id [90100], num [90000] INFO 2021-04-02 02:53:46,802 table.cc:410 table cost [2.01587]ms INFO 2021-04-02 02:53:48,443 gamma_engine.cc:768 Add total cost [4.50806]ms INFO 2021-04-02 02:53:48,444 table.cc:390 Add item _id [100100], num [100000] INFO 2021-04-02 02:53:48,445 table.cc:410 table cost [1.56201]ms INFO 2021-04-02 02:53:56,995 gamma_engine.cc:768 Add total cost [3.04199]ms INFO 2021-04-02 02:53:56,997 table.cc:390 Add item _id [110100], num [110000] INFO 2021-04-02 02:53:56,999 table.cc:410 table cost [1.71899]ms INFO 2021-04-02 02:53:58,520 gamma_engine.cc:768 Add total cost [3.90991]ms INFO 2021-04-02 02:53:58,522 table.cc:390 Add item _id [120000], num [120000] INFO 2021-04-02 02:53:58,523 table.cc:410 table cost [1.75513]ms INFO 2021-04-02 02:54:01,004 gamma_engine.cc:768 Add total cost [3.21802]ms INFO 2021-04-02 02:54:01,015 table.cc:390 Add item _id [130400], num [130000] INFO 2021-04-02 02:54:01,016 table.cc:410 table cost [1.75415]ms INFO 2021-04-02 02:54:02,811 gamma_engine.cc:768 Add total cost [3.47681]ms INFO 2021-04-02 02:54:02,840 table.cc:390 Add item _id [140000], num [140000] INFO 2021-04-02 02:54:02,841 table.cc:410 table cost [1.57202]ms INFO 2021-04-02 02:54:04,514 gamma_engine.cc:768 Add total cost [5.20801]ms INFO 2021-04-02 02:54:04,520 table.cc:390 Add item _id [150000], num [150000] INFO 2021-04-02 02:54:04,522 table.cc:410 table cost [1.69385]ms INFO 2021-04-02 02:54:06,200 gamma_engine.cc:768 Add total cost [3.24902]ms INFO 2021-04-02 02:54:06,220 table.cc:390 Add item _id [160000], num [160000] INFO 2021-04-02 02:54:06,222 table.cc:410 table cost [1.95312]ms INFO 2021-04-02 02:54:07,630 gamma_engine.cc:768 Add total cost [3.46387]ms INFO 2021-04-02 02:54:07,639 table.cc:390 Add item _id [169900], num [170000] INFO 2021-04-02 02:54:07,641 table.cc:410 table cost [1.70508]ms INFO 2021-04-02 02:54:10,467 gamma_engine.cc:768 Add total cost [3.38208]ms INFO 2021-04-02 02:54:10,468 table.cc:390 Add item _id [179600], num [180000] INFO 2021-04-02 02:54:10,469 table.cc:410 table cost [1.62402]ms INFO 2021-04-02 02:54:12,716 gamma_engine.cc:768 Add total cost [4.50513]ms INFO 2021-04-02 02:54:12,721 table.cc:390 Add item _id [190000], num [190000] INFO 2021-04-02 02:54:12,722 table.cc:410 table cost [1.64697]ms INFO 2021-04-02 02:54:14,950 gamma_engine.cc:768 Add total cost [3.47705]ms INFO 2021-04-02 02:54:14,951 table.cc:390 Add item _id [200000], num [200000] INFO 2021-04-02 02:54:14,952 table.cc:410 table cost [1.38306]ms INFO 2021-04-02 02:54:15,258 gamma_index_ivfpq.cc:1287 gamma index is not trained, skip dumping INFO 2021-04-02 02:54:15,258 vector_manager.cc:661 vector vec_IVFPQ dump gamma index success! INFO 2021-04-02 02:54:15,258 vector_manager.cc:675 vector vec dump success! INFO 2021-04-02 02:54:15,926 gamma_engine.cc:1078 Dumped to [/data/db/data/365/retrieval_model_index/2021-04-02-02:54:15], next dump docid [201600], last dump directory(removed)= INFO 2021-04-02 02:54:16,991 gamma_engine.cc:768 Add total cost [3.24194]ms INFO 2021-04-02 02:54:17,063 table.cc:390 Add item _id [210000], num [210000] INFO 2021-04-02 02:54:17,064 table.cc:410 table cost [1.56421]ms INFO 2021-04-02 02:54:19,142 gamma_engine.cc:768 Add total cost [4.2168]ms INFO 2021-04-02 02:54:19,159 table.cc:390 Add item _id [220000], num [220000] INFO 2021-04-02 02:54:19,160 table.cc:410 table cost [1.5752]ms INFO 2021-04-02 02:54:21,122 gamma_engine.cc:768 Add total cost [3.16797]ms INFO 2021-04-02 02:54:21,125 table.cc:390 Add item _id [230000], num [230000] INFO 2021-04-02 02:54:21,127 table.cc:410 table cost [2.38086]ms INFO 2021-04-02 02:54:23,091 gamma_engine.cc:768 Add total cost [3.52808]ms INFO 2021-04-02 02:54:23,092 table.cc:390 Add item _id [239900], num [240000] INFO 2021-04-02 02:54:23,094 table.cc:410 table cost [1.77197]ms INFO 2021-04-02 02:54:24,662 gamma_engine.cc:768 Add total cost [3.38013]ms INFO 2021-04-02 02:54:24,663 table.cc:390 Add item _id [250000], num [250000] INFO 2021-04-02 02:54:24,664 table.cc:410 table cost [1.81299]ms INFO 2021-04-02 02:54:27,028 gamma_engine.cc:768 Add total cost [3.35596]ms INFO 2021-04-02 02:54:27,060 table.cc:390 Add item _id [260100], num [260000] INFO 2021-04-02 02:54:27,061 table.cc:410 table cost [1.44092]ms INFO 2021-04-02 02:54:28,804 gamma_engine.cc:768 Add total cost [3.73315]ms INFO 2021-04-02 02:54:28,805 table.cc:390 Add item _id [270000], num [270000] INFO 2021-04-02 02:54:28,806 table.cc:410 table cost [1.35889]ms INFO 2021-04-02 02:54:31,164 gamma_engine.cc:768 Add total cost [3.51904]ms INFO 2021-04-02 02:54:31,169 table.cc:390 Add item _id [279100], num [280000] INFO 2021-04-02 02:54:31,170 table.cc:410 table cost [1.8252]ms INFO 2021-04-02 02:54:32,852 gamma_engine.cc:768 Add total cost [3.09204]ms INFO 2021-04-02 02:54:32,853 table.cc:390 Add item _id [290100], num [290000] INFO 2021-04-02 02:54:32,854 table.cc:410 table cost [1.48486]ms INFO 2021-04-02 02:54:35,006 gamma_engine.cc:768 Add total cost [3.6062]ms INFO 2021-04-02 02:54:35,007 table.cc:390 Add item _id [299900], num [300000] INFO 2021-04-02 02:54:35,008 table.cc:410 table cost [1.48315]ms INFO 2021-04-02 02:54:37,105 gamma_engine.cc:768 Add total cost [4.14014]ms INFO 2021-04-02 02:54:37,137 table.cc:390 Add item _id [310200], num [310000] INFO 2021-04-02 02:54:37,138 table.cc:410 table cost [1.57104]ms INFO 2021-04-02 02:54:39,214 gamma_engine.cc:768 Add total cost [4.0249]ms INFO 2021-04-02 02:54:39,215 table.cc:390 Add item _id [320000], num [320000] INFO 2021-04-02 02:54:39,217 table.cc:410 table cost [1.46509]ms INFO 2021-04-02 02:54:41,193 gamma_engine.cc:768 Add total cost [3.35498]ms INFO 2021-04-02 02:54:41,194 table.cc:390 Add item _id [329900], num [330000] INFO 2021-04-02 02:54:41,195 table.cc:410 table cost [1.38794]ms INFO 2021-04-02 02:54:44,129 gamma_engine.cc:768 Add total cost [3.43408]ms INFO 2021-04-02 02:54:44,143 table.cc:390 Add item _id [340000], num [340000] INFO 2021-04-02 02:54:44,144 table.cc:410 table cost [1.60791]ms INFO 2021-04-02 02:54:46,137 gamma_engine.cc:768 Add total cost [3.46484]ms INFO 2021-04-02 02:54:46,138 table.cc:390 Add item _id [350200], num [350000] INFO 2021-04-02 02:54:46,140 table.cc:410 table cost [1.73804]ms INFO 2021-04-02 02:54:48,301 gamma_engine.cc:768 Add total cost [2.99683]ms INFO 2021-04-02 02:54:48,302 table.cc:390 Add item _id [359400], num [360000] INFO 2021-04-02 02:54:48,303 table.cc:410 table cost [1.51587]ms INFO 2021-04-02 02:54:50,062 gamma_engine.cc:768 Add total cost [3.76196]ms INFO 2021-04-02 02:54:50,063 table.cc:390 Add item _id [369900], num [370000] INFO 2021-04-02 02:54:50,064 table.cc:410 table cost [1.48193]ms INFO 2021-04-02 02:54:51,420 gamma_engine.cc:768 Add total cost [3.31006]ms INFO 2021-04-02 02:54:51,420 table.cc:390 Add item _id [379800], num [380000] INFO 2021-04-02 02:54:51,422 table.cc:410 table cost [1.51099]ms INFO 2021-04-02 02:54:53,282 gamma_engine.cc:768 Add total cost [4.01099]ms INFO 2021-04-02 02:54:53,309 table.cc:390 Add item _id [390100], num [390000] INFO 2021-04-02 02:54:53,311 table.cc:410 table cost [1.50195]ms INFO 2021-04-02 02:54:55,365 gamma_engine.cc:768 Add total cost [3.24805]ms INFO 2021-04-02 02:54:55,407 table.cc:390 Add item _id [400000], num [400000] INFO 2021-04-02 02:54:55,408 table.cc:410 table cost [1.54199]ms INFO 2021-04-02 02:54:57,546 gamma_engine.cc:768 Add total cost [4.60107]ms INFO 2021-04-02 02:54:57,551 table.cc:390 Add item _id [409800], num [410000] INFO 2021-04-02 02:54:57,552 table.cc:410 table cost [1.88696]ms INFO 2021-04-02 02:54:59,432 gamma_engine.cc:768 Add total cost [3.2002]ms INFO 2021-04-02 02:54:59,443 table.cc:390 Add item _id [419900], num [420000] INFO 2021-04-02 02:54:59,444 table.cc:410 table cost [1.7168]ms INFO 2021-04-02 02:55:01,197 gamma_engine.cc:768 Add total cost [4.33301]ms INFO 2021-04-02 02:55:01,202 table.cc:390 Add item _id [430000], num [430000] INFO 2021-04-02 02:55:01,204 table.cc:410 table cost [1.73096]ms INFO 2021-04-02 02:55:02,570 gamma_engine.cc:768 Add total cost [5.27905]ms INFO 2021-04-02 02:55:02,584 table.cc:390 Add item _id [440100], num [440000] INFO 2021-04-02 02:55:02,586 table.cc:410 table cost [1.89917]ms INFO 2021-04-02 02:55:03,940 gamma_engine.cc:768 Add total cost [4.70288]ms INFO 2021-04-02 02:55:03,973 table.cc:390 Add item _id [450000], num [450000] INFO 2021-04-02 02:55:03,974 table.cc:410 table cost [2.85889]ms INFO 2021-04-02 02:55:05,207 gamma_engine.cc:768 Add total cost [3.58301]ms INFO 2021-04-02 02:55:05,215 table.cc:390 Add item _id [460000], num [460000] INFO 2021-04-02 02:55:05,216 table.cc:410 table cost [1.55811]ms INFO 2021-04-02 02:55:06,787 gamma_engine.cc:768 Add total cost [3.53687]ms INFO 2021-04-02 02:55:06,792 table.cc:390 Add item _id [470400], num [470000] INFO 2021-04-02 02:55:06,794 table.cc:410 table cost [1.67603]ms INFO 2021-04-02 02:55:08,064 gamma_engine.cc:768 Add total cost [4.65381]ms INFO 2021-04-02 02:55:08,071 table.cc:390 Add item _id [480000], num [480000] INFO 2021-04-02 02:55:08,072 table.cc:410 table cost [2.02808]ms INFO 2021-04-02 02:55:09,139 gamma_engine.cc:768 Add total cost [3.51196]ms INFO 2021-04-02 02:55:09,155 table.cc:390 Add item _id [490000], num [490000] INFO 2021-04-02 02:55:09,156 table.cc:410 table cost [1.51318]ms INFO 2021-04-02 02:55:10,289 gamma_engine.cc:768 Add total cost [5.61304]ms INFO 2021-04-02 02:55:10,290 table.cc:390 Add item _id [500200], num [500000] INFO 2021-04-02 02:55:10,292 table.cc:410 table cost [2.38794]ms INFO 2021-04-02 02:55:11,420 gamma_engine.cc:768 Add total cost [3.87891]ms INFO 2021-04-02 02:55:11,421 table.cc:390 Add item _id [510000], num [510000] INFO 2021-04-02 02:55:11,423 table.cc:410 table cost [1.43896]ms INFO 2021-04-02 02:55:12,659 gamma_engine.cc:768 Add total cost [6.00195]ms INFO 2021-04-02 02:55:12,662 table.cc:390 Add item _id [519300], num [520000] INFO 2021-04-02 02:55:12,663 table.cc:410 table cost [1.80688]ms INFO 2021-04-02 02:55:13,771 gamma_engine.cc:768 Add total cost [3.53003]ms INFO 2021-04-02 02:55:13,833 table.cc:390 Add item _id [530100], num [530000] INFO 2021-04-02 02:55:13,834 table.cc:410 table cost [1.63306]ms INFO 2021-04-02 02:55:15,141 gamma_engine.cc:768 Add total cost [3.45312]ms INFO 2021-04-02 02:55:15,159 table.cc:390 Add item _id [540100], num [540000] INFO 2021-04-02 02:55:15,161 table.cc:410 table cost [1.93408]ms INFO 2021-04-02 02:55:16,454 gamma_engine.cc:768 Add total cost [3.71606]ms INFO 2021-04-02 02:55:16,472 table.cc:390 Add item _id [550000], num [550000] INFO 2021-04-02 02:55:16,474 table.cc:410 table cost [1.67188]ms INFO 2021-04-02 02:55:17,744 gamma_engine.cc:768 Add total cost [3.21411]ms INFO 2021-04-02 02:55:17,753 table.cc:390 Add item _id [560000], num [560000] INFO 2021-04-02 02:55:17,755 table.cc:410 table cost [1.91602]ms INFO 2021-04-02 02:55:19,250 gamma_engine.cc:768 Add total cost [4.31787]ms INFO 2021-04-02 02:55:19,258 table.cc:390 Add item _id [570100], num [570000] INFO 2021-04-02 02:55:19,259 table.cc:410 table cost [1.85303]ms INFO 2021-04-02 02:55:20,553 gamma_engine.cc:768 Add total cost [7.7998]ms INFO 2021-04-02 02:55:20,554 table.cc:390 Add item _id [580000], num [580000] INFO 2021-04-02 02:55:20,556 table.cc:410 table cost [1.67114]ms INFO 2021-04-02 02:55:21,733 gamma_engine.cc:768 Add total cost [3.85107]ms INFO 2021-04-02 02:55:21,734 table.cc:390 Add item _id [590100], num [590000] INFO 2021-04-02 02:55:21,735 table.cc:410 table cost [1.51685]ms INFO 2021-04-02 02:55:22,900 gamma_engine.cc:768 Add total cost [3.89404]ms INFO 2021-04-02 02:55:22,901 table.cc:390 Add item _id [600000], num [600000] INFO 2021-04-02 02:55:22,903 table.cc:410 table cost [1.58496]ms INFO 2021-04-02 02:55:24,219 gamma_engine.cc:768 Add total cost [5.11011]ms INFO 2021-04-02 02:55:24,221 table.cc:390 Add item _id [610000], num [610000] INFO 2021-04-02 02:55:24,222 table.cc:410 table cost [1.88989]ms INFO 2021-04-02 02:55:25,693 gamma_engine.cc:768 Add total cost [4.97607]ms INFO 2021-04-02 02:55:25,699 table.cc:390 Add item _id [620100], num [620000] INFO 2021-04-02 02:55:25,701 table.cc:410 table cost [1.99609]ms INFO 2021-04-02 02:55:27,162 gamma_engine.cc:768 Add total cost [5.81396]ms INFO 2021-04-02 02:55:27,166 table.cc:390 Add item _id [630000], num [630000] INFO 2021-04-02 02:55:27,167 table.cc:410 table cost [1.49805]ms INFO 2021-04-02 02:55:28,499 gamma_engine.cc:768 Add total cost [3.80786]ms INFO 2021-04-02 02:55:28,502 table.cc:390 Add item _id [640000], num [640000] INFO 2021-04-02 02:55:28,503 table.cc:410 table cost [1.59888]ms INFO 2021-04-02 02:55:29,744 gamma_engine.cc:768 Add total cost [3.18018]ms INFO 2021-04-02 02:55:29,752 table.cc:390 Add item _id [650000], num [650000] INFO 2021-04-02 02:55:29,754 table.cc:410 table cost [1.87109]ms INFO 2021-04-02 02:55:31,128 gamma_engine.cc:768 Add total cost [3.93994]ms INFO 2021-04-02 02:55:31,144 table.cc:390 Add item _id [660100], num [660000] INFO 2021-04-02 02:55:31,146 table.cc:410 table cost [1.67383]ms INFO 2021-04-02 02:55:32,697 gamma_engine.cc:768 Add total cost [3.69995]ms INFO 2021-04-02 02:55:32,698 table.cc:390 Add item _id [670000], num [670000] INFO 2021-04-02 02:55:32,700 table.cc:410 table cost [1.67212]ms INFO 2021-04-02 02:55:34,242 gamma_engine.cc:768 Add total cost [4.31201]ms INFO 2021-04-02 02:55:34,243 table.cc:390 Add item _id [680200], num [680000] INFO 2021-04-02 02:55:34,245 table.cc:410 table cost [1.8291]ms INFO 2021-04-02 02:55:35,589 gamma_engine.cc:768 Add total cost [3.63696]ms INFO 2021-04-02 02:55:35,591 table.cc:390 Add item _id [689900], num [690000] INFO 2021-04-02 02:55:35,592 table.cc:410 table cost [1.47485]ms INFO 2021-04-02 02:55:37,014 gamma_engine.cc:768 Add total cost [3.84399]ms INFO 2021-04-02 02:55:37,022 table.cc:390 Add item _id [700100], num [700000] INFO 2021-04-02 02:55:37,024 table.cc:410 table cost [1.75293]ms INFO 2021-04-02 02:55:38,270 gamma_engine.cc:768 Add total cost [4.61597]ms INFO 2021-04-02 02:55:38,313 table.cc:390 Add item _id [710000], num [710000] INFO 2021-04-02 02:55:38,315 table.cc:410 table cost [1.73389]ms INFO 2021-04-02 02:55:39,521 gamma_engine.cc:768 Add total cost [3.90698]ms INFO 2021-04-02 02:55:39,531 table.cc:390 Add item _id [719900], num [720000] INFO 2021-04-02 02:55:39,532 table.cc:410 table cost [6.38306]ms INFO 2021-04-02 02:55:40,698 gamma_engine.cc:768 Add total cost [3.28589]ms INFO 2021-04-02 02:55:40,713 table.cc:390 Add item _id [730000], num [730000] INFO 2021-04-02 02:55:40,714 table.cc:410 table cost [1.41699]ms INFO 2021-04-02 02:55:41,909 gamma_engine.cc:768 Add total cost [3.34106]ms INFO 2021-04-02 02:55:41,909 table.cc:390 Add item _id [740100], num [740000] INFO 2021-04-02 02:55:41,911 table.cc:410 table cost [1.42822]ms INFO 2021-04-02 02:55:43,033 gamma_engine.cc:768 Add total cost [4.14697]ms INFO 2021-04-02 02:55:43,048 table.cc:390 Add item _id [750000], num [750000] INFO 2021-04-02 02:55:43,050 table.cc:410 table cost [1.87305]ms INFO 2021-04-02 02:55:44,625 gamma_engine.cc:768 Add total cost [5.58887]ms INFO 2021-04-02 02:55:44,643 table.cc:390 Add item _id [760000], num [760000] INFO 2021-04-02 02:55:44,644 table.cc:410 table cost [1.729]ms INFO 2021-04-02 02:55:46,058 gamma_engine.cc:768 Add total cost [3.06396]ms INFO 2021-04-02 02:55:46,060 table.cc:390 Add item _id [770100], num [770000] INFO 2021-04-02 02:55:46,062 table.cc:410 table cost [1.698]ms INFO 2021-04-02 02:55:47,240 gamma_engine.cc:768 Add total cost [3.58594]ms INFO 2021-04-02 02:55:47,241 table.cc:390 Add item _id [780100], num [780000] INFO 2021-04-02 02:55:47,242 table.cc:410 table cost [1.5271]ms INFO 2021-04-02 02:55:48,607 gamma_engine.cc:768 Add total cost [4.00708]ms INFO 2021-04-02 02:55:48,608 table.cc:390 Add item _id [790000], num [790000] INFO 2021-04-02 02:55:48,610 table.cc:410 table cost [1.48804]ms INFO 2021-04-02 02:55:49,827 gamma_engine.cc:768 Add total cost [4.14307]ms INFO 2021-04-02 02:55:49,828 table.cc:390 Add item _id [799900], num [800000] INFO 2021-04-02 02:55:49,829 table.cc:410 table cost [1.37695]ms INFO 2021-04-02 02:55:51,161 gamma_engine.cc:768 Add total cost [3.67896]ms INFO 2021-04-02 02:55:51,162 table.cc:390 Add item _id [809900], num [810000] INFO 2021-04-02 02:55:51,163 table.cc:410 table cost [1.75903]ms INFO 2021-04-02 02:55:52,344 gamma_engine.cc:768 Add total cost [3.04712]ms INFO 2021-04-02 02:55:52,352 table.cc:390 Add item _id [820000], num [820000] INFO 2021-04-02 02:55:52,353 table.cc:410 table cost [1.41211]ms INFO 2021-04-02 02:55:53,482 gamma_engine.cc:768 Add total cost [3.229]ms INFO 2021-04-02 02:55:53,483 table.cc:390 Add item _id [830000], num [830000] INFO 2021-04-02 02:55:53,484 table.cc:410 table cost [1.36108]ms INFO 2021-04-02 02:55:54,450 gamma_engine.cc:768 Add total cost [3.69702]ms INFO 2021-04-02 02:55:54,452 table.cc:390 Add item _id [840000], num [840000] INFO 2021-04-02 02:55:54,453 table.cc:410 table cost [1.45996]ms INFO 2021-04-02 02:55:55,407 gamma_engine.cc:768 Add total cost [3.21094]ms INFO 2021-04-02 02:55:55,409 table.cc:390 Add item _id [850000], num [850000] INFO 2021-04-02 02:55:55,410 table.cc:410 table cost [1.55811]ms INFO 2021-04-02 02:55:56,355 gamma_engine.cc:768 Add total cost [3.85205]ms INFO 2021-04-02 02:55:56,373 table.cc:390 Add item _id [860100], num [860000] INFO 2021-04-02 02:55:56,374 table.cc:410 table cost [1.50195]ms INFO 2021-04-02 02:55:57,347 gamma_engine.cc:768 Add total cost [4.35303]ms INFO 2021-04-02 02:55:57,348 table.cc:390 Add item _id [870000], num [870000] INFO 2021-04-02 02:55:57,349 table.cc:410 table cost [2.10889]ms INFO 2021-04-02 02:55:58,522 gamma_engine.cc:768 Add total cost [5.06909]ms INFO 2021-04-02 02:55:58,533 table.cc:390 Add item _id [880000], num [880000] INFO 2021-04-02 02:55:58,534 table.cc:410 table cost [1.55493]ms INFO 2021-04-02 02:55:59,966 gamma_engine.cc:768 Add total cost [3.83887]ms INFO 2021-04-02 02:55:59,967 table.cc:390 Add item _id [890200], num [890000] INFO 2021-04-02 02:55:59,968 table.cc:410 table cost [1.43188]ms INFO 2021-04-02 02:56:01,324 gamma_engine.cc:761 Begin indexing. INFO 2021-04-02 02:56:01,324 gamma_engine.cc:768 Add total cost [3.71899]ms INFO 2021-04-02 02:56:01,337 table.cc:390 Add item _id [900000], num [900000] INFO 2021-04-02 02:56:01,338 table.cc:410 table cost [3.66699]ms INFO 2021-04-02 02:56:02,686 gamma_engine.cc:768 Add total cost [4.01904]ms INFO 2021-04-02 02:56:02,704 table.cc:390 Add item _id [910100], num [910000] INFO 2021-04-02 02:56:02,706 table.cc:410 table cost [2.18506]ms INFO 2021-04-02 02:56:04,146 gamma_engine.cc:768 Add total cost [4.32007]ms INFO 2021-04-02 02:56:04,147 table.cc:390 Add item _id [920000], num [920000] INFO 2021-04-02 02:56:04,149 table.cc:410 table cost [1.76099]ms INFO 2021-04-02 02:56:05,581 gamma_engine.cc:768 Add total cost [4.1189]ms INFO 2021-04-02 02:56:05,587 table.cc:390 Add item _id [930100], num [930000] INFO 2021-04-02 02:56:05,589 table.cc:410 table cost [5.69312]ms INFO 2021-04-02 02:56:06,791 gamma_engine.cc:768 Add total cost [4.22095]ms INFO 2021-04-02 02:56:06,794 table.cc:390 Add item _id [940000], num [940000] INFO 2021-04-02 02:56:06,796 table.cc:410 table cost [1.78906]ms INFO 2021-04-02 02:56:07,955 gamma_engine.cc:768 Add total cost [3.81909]ms INFO 2021-04-02 02:56:07,965 table.cc:390 Add item _id [950100], num [950000] INFO 2021-04-02 02:56:07,967 table.cc:410 table cost [1.79224]ms INFO 2021-04-02 02:56:09,323 gamma_engine.cc:768 Add total cost [3.12305]ms INFO 2021-04-02 02:56:09,333 table.cc:390 Add item _id [960000], num [960000] INFO 2021-04-02 02:56:09,334 table.cc:410 table cost [1.66479]ms INFO 2021-04-02 02:56:10,912 gamma_engine.cc:768 Add total cost [6.04614]ms INFO 2021-04-02 02:56:10,948 table.cc:390 Add item _id [970000], num [970000] INFO 2021-04-02 02:56:10,949 table.cc:410 table cost [1.55713]ms INFO 2021-04-02 02:56:12,851 gamma_engine.cc:768 Add total cost [3.7439]ms INFO 2021-04-02 02:56:12,885 table.cc:390 Add item _id [980100], num [980000] INFO 2021-04-02 02:56:12,886 table.cc:410 table cost [1.67993]ms INFO 2021-04-02 02:56:14,758 gamma_engine.cc:768 Add total cost [3.82007]ms INFO 2021-04-02 02:56:14,761 table.cc:390 Add item _id [990000], num [990000] INFO 2021-04-02 02:56:14,762 table.cc:410 table cost [2.21021]ms INFO 2021-04-02 02:56:15,258 gamma_index_ivfpq.cc:1287 gamma index is not trained, skip dumping INFO 2021-04-02 02:56:15,258 vector_manager.cc:661 vector vec_IVFPQ dump gamma index success! INFO 2021-04-02 02:56:15,258 vector_manager.cc:675 vector vec dump success! INFO 2021-04-02 02:56:16,065 gamma_engine.cc:1078 Dumped to [/data/db/data/365/retrieval_model_index/2021-04-02-02:56:15], next dump docid [993200], last dump directory(removed)=/data/db/data/365/retrieval_model_index/2021-04-02-02:54:15 INFO 2021-04-02 02:56:16,317 gamma_engine.cc:768 Add total cost [3.83594]ms INFO 2021-04-02 02:56:41,629 gamma_index_ivfpq.cc:540 train successed! INFO 2021-04-02 02:56:41,639 gamma_engine.cc:939 vector manager indexing success! INFO 2021-04-02 02:56:41,879 gamma_index_ivfpq.cc:691 Add time [0.01149]ms, count 10000 INFO 2021-04-02 02:56:42,135 gamma_index_ivfpq.cc:691 Add time [0.0147]ms, count 20000 INFO 2021-04-02 02:56:42,382 gamma_index_ivfpq.cc:691 Add time [0.0137571]ms, count 30000 INFO 2021-04-02 02:56:42,670 gamma_index_ivfpq.cc:691 Add time [0.0182969]ms, count 40000 INFO 2021-04-02 02:56:42,936 gamma_index_ivfpq.cc:691 Add time [0.0151838]ms, count 50000 INFO 2021-04-02 02:56:43,199 gamma_index_ivfpq.cc:691 Add time [0.013188]ms, count 60000 INFO 2021-04-02 02:56:43,438 gamma_index_ivfpq.cc:691 Add time [0.0137349]ms, count 70000 INFO 2021-04-02 02:56:43,692 gamma_index_ivfpq.cc:691 Add time [0.0160601]ms, count 80000 INFO 2021-04-02 02:56:43,951 gamma_index_ivfpq.cc:691 Add time [0.0197681]ms, count 90000 INFO 2021-04-02 02:56:44,196 gamma_index_ivfpq.cc:691 Add time [0.0118979]ms, count 100000 INFO 2021-04-02 02:56:44,437 gamma_index_ivfpq.cc:691 Add time [0.012363]ms, count 110000 INFO 2021-04-02 02:56:44,768 gamma_index_ivfpq.cc:691 Add time [0.0159211]ms, count 120000 INFO 2021-04-02 02:56:45,073 gamma_index_ivfpq.cc:691 Add time [0.0251519]ms, count 130000 INFO 2021-04-02 02:56:45,343 gamma_index_ivfpq.cc:691 Add time [0.011531]ms, count 140000 INFO 2021-04-02 02:56:45,597 gamma_index_ivfpq.cc:691 Add time [0.0232671]ms, count 150000 INFO 2021-04-02 02:56:45,876 gamma_index_ivfpq.cc:691 Add time [0.019981]ms, count 160000 INFO 2021-04-02 02:56:46,150 gamma_index_ivfpq.cc:691 Add time [0.012915]ms, count 170000 INFO 2021-04-02 02:56:46,454 gamma_index_ivfpq.cc:691 Add time [0.018459]ms, count 180000 INFO 2021-04-02 02:56:46,765 gamma_index_ivfpq.cc:691 Add time [0.0165222]ms, count 190000 INFO 2021-04-02 02:56:47,037 gamma_index_ivfpq.cc:691 Add time [0.0210491]ms, count 200000 INFO 2021-04-02 02:56:47,314 gamma_index_ivfpq.cc:691 Add time [0.016551]ms, count 210000 INFO 2021-04-02 02:56:47,629 gamma_index_ivfpq.cc:691 Add time [0.026741]ms, count 220000 INFO 2021-04-02 02:56:47,917 gamma_index_ivfpq.cc:691 Add time [0.0158611]ms, count 230000 INFO 2021-04-02 02:56:48,184 gamma_index_ivfpq.cc:691 Add time [0.00901489]ms, count 240000 INFO 2021-04-02 02:56:48,439 gamma_index_ivfpq.cc:691 Add time [0.019447]ms, count 250000 INFO 2021-04-02 02:56:48,748 gamma_index_ivfpq.cc:691 Add time [0.0128372]ms, count 260000 INFO 2021-04-02 02:56:49,002 gamma_index_ivfpq.cc:691 Add time [0.0166411]ms, count 270000 INFO 2021-04-02 02:56:49,287 gamma_index_ivfpq.cc:691 Add time [0.015074]ms, count 280000 INFO 2021-04-02 02:56:49,563 gamma_index_ivfpq.cc:691 Add time [0.0169299]ms, count 290000 INFO 2021-04-02 02:56:49,824 gamma_index_ivfpq.cc:691 Add time [0.012345]ms, count 300000 INFO 2021-04-02 02:56:50,087 gamma_index_ivfpq.cc:691 Add time [0.0121641]ms, count 310000 INFO 2021-04-02 02:56:50,328 gamma_index_ivfpq.cc:691 Add time [0.0169419]ms, count 320000 INFO 2021-04-02 02:56:50,615 gamma_index_ivfpq.cc:691 Add time [0.01504]ms, count 330000 INFO 2021-04-02 02:56:50,871 gamma_index_ivfpq.cc:691 Add time [0.012304]ms, count 340000 INFO 2021-04-02 02:56:51,133 gamma_index_ivfpq.cc:691 Add time [0.014531]ms, count 350000 INFO 2021-04-02 02:56:51,388 gamma_index_ivfpq.cc:691 Add time [0.0147351]ms, count 360000 INFO 2021-04-02 02:56:51,690 gamma_index_ivfpq.cc:691 Add time [0.0171189]ms, count 370000 INFO 2021-04-02 02:56:51,944 gamma_index_ivfpq.cc:691 Add time [0.01676]ms, count 380000 INFO 2021-04-02 02:56:52,205 gamma_index_ivfpq.cc:691 Add time [0.01678]ms, count 390000 INFO 2021-04-02 02:56:52,580 gamma_index_ivfpq.cc:691 Add time [0.018749]ms, count 400000 INFO 2021-04-02 02:56:52,852 gamma_index_ivfpq.cc:691 Add time [0.0139302]ms, count 410000 INFO 2021-04-02 02:56:53,118 gamma_index_ivfpq.cc:691 Add time [0.018626]ms, count 420000 INFO 2021-04-02 02:56:53,368 gamma_index_ivfpq.cc:691 Add time [0.0120652]ms, count 430000 INFO 2021-04-02 02:56:53,644 gamma_index_ivfpq.cc:691 Add time [0.013395]ms, count 440000 INFO 2021-04-02 02:56:53,920 gamma_index_ivfpq.cc:691 Add time [0.012509]ms, count 450000 INFO 2021-04-02 02:56:54,186 gamma_index_ivfpq.cc:691 Add time [0.0138159]ms, count 460000 INFO 2021-04-02 02:56:54,444 gamma_index_ivfpq.cc:691 Add time [0.013134]ms, count 470000 INFO 2021-04-02 02:56:54,723 gamma_index_ivfpq.cc:691 Add time [0.0206338]ms, count 480000 INFO 2021-04-02 02:56:54,987 gamma_index_ivfpq.cc:691 Add time [0.0163369]ms, count 490000 INFO 2021-04-02 02:56:55,262 gamma_index_ivfpq.cc:691 Add time [0.014053]ms, count 500000 INFO 2021-04-02 02:56:55,519 gamma_index_ivfpq.cc:691 Add time [0.0170999]ms, count 510000 INFO 2021-04-02 02:56:55,828 gamma_index_ivfpq.cc:691 Add time [0.015988]ms, count 520000 INFO 2021-04-02 02:56:56,083 gamma_index_ivfpq.cc:691 Add time [0.0215549]ms, count 530000 INFO 2021-04-02 02:56:56,348 gamma_index_ivfpq.cc:691 Add time [0.0195439]ms, count 540000 INFO 2021-04-02 02:56:56,626 gamma_index_ivfpq.cc:691 Add time [0.0167009]ms, count 550000 INFO 2021-04-02 02:56:56,897 gamma_index_ivfpq.cc:691 Add time [0.00986499]ms, count 560000 INFO 2021-04-02 02:56:57,143 gamma_index_ivfpq.cc:691 Add time [0.011894]ms, count 570000 INFO 2021-04-02 02:56:57,386 gamma_index_ivfpq.cc:691 Add time [0.0171721]ms, count 580000 INFO 2021-04-02 02:56:57,685 gamma_index_ivfpq.cc:691 Add time [0.022731]ms, count 590000 INFO 2021-04-02 02:56:57,952 gamma_index_ivfpq.cc:691 Add time [0.0139041]ms, count 600000 INFO 2021-04-02 02:56:58,216 gamma_index_ivfpq.cc:691 Add time [0.0122529]ms, count 610000 INFO 2021-04-02 02:56:58,456 gamma_index_ivfpq.cc:691 Add time [0.012512]ms, count 620000 INFO 2021-04-02 02:56:58,749 gamma_index_ivfpq.cc:691 Add time [0.01571]ms, count 630000 INFO 2021-04-02 02:56:59,003 gamma_index_ivfpq.cc:691 Add time [0.0119521]ms, count 640000 INFO 2021-04-02 02:56:59,264 gamma_index_ivfpq.cc:691 Add time [0.0137979]ms, count 650000 INFO 2021-04-02 02:56:59,522 gamma_index_ivfpq.cc:691 Add time [0.0122461]ms, count 660000 INFO 2021-04-02 02:56:59,807 gamma_index_ivfpq.cc:691 Add time [0.0173501]ms, count 670000 INFO 2021-04-02 02:57:00,075 gamma_index_ivfpq.cc:691 Add time [0.0175698]ms, count 680000 INFO 2021-04-02 02:57:00,330 gamma_index_ivfpq.cc:691 Add time [0.01423]ms, count 690000 INFO 2021-04-02 02:57:00,640 gamma_index_ivfpq.cc:691 Add time [0.0157839]ms, count 700000 INFO 2021-04-02 02:57:00,916 gamma_index_ivfpq.cc:691 Add time [0.018365]ms, count 710000 INFO 2021-04-02 02:57:01,178 gamma_index_ivfpq.cc:691 Add time [0.0120979]ms, count 720000 INFO 2021-04-02 02:57:01,455 gamma_index_ivfpq.cc:691 Add time [0.012364]ms, count 730000 INFO 2021-04-02 02:57:01,758 gamma_index_ivfpq.cc:691 Add time [0.018082]ms, count 740000 INFO 2021-04-02 02:57:02,047 gamma_index_ivfpq.cc:691 Add time [0.022356]ms, count 750000 INFO 2021-04-02 02:57:02,327 gamma_index_ivfpq.cc:691 Add time [0.014561]ms, count 760000 INFO 2021-04-02 02:57:02,628 gamma_index_ivfpq.cc:691 Add time [0.024718]ms, count 770000 INFO 2021-04-02 02:57:02,936 gamma_index_ivfpq.cc:691 Add time [0.0211201]ms, count 780000 INFO 2021-04-02 02:57:03,232 gamma_index_ivfpq.cc:691 Add time [0.016179]ms, count 790000 INFO 2021-04-02 02:57:03,509 gamma_index_ivfpq.cc:691 Add time [0.0146211]ms, count 800000 INFO 2021-04-02 02:57:03,825 gamma_index_ivfpq.cc:691 Add time [0.016575]ms, count 810000 INFO 2021-04-02 02:57:04,129 gamma_index_ivfpq.cc:691 Add time [0.0158252]ms, count 820000 INFO 2021-04-02 02:57:04,417 gamma_index_ivfpq.cc:691 Add time [0.0142549]ms, count 830000 INFO 2021-04-02 02:57:04,726 gamma_index_ivfpq.cc:691 Add time [0.0223511]ms, count 840000 INFO 2021-04-02 02:57:05,011 gamma_index_ivfpq.cc:691 Add time [0.0162939]ms, count 850000 INFO 2021-04-02 02:57:05,274 gamma_index_ivfpq.cc:691 Add time [0.0153931]ms, count 860000 INFO 2021-04-02 02:57:05,582 gamma_index_ivfpq.cc:691 Add time [0.023887]ms, count 870000 INFO 2021-04-02 02:57:05,933 gamma_index_ivfpq.cc:691 Add time [0.019176]ms, count 880000 INFO 2021-04-02 02:57:06,194 gamma_index_ivfpq.cc:691 Add time [0.0140579]ms, count 890000 INFO 2021-04-02 02:57:06,439 gamma_index_ivfpq.cc:691 Add time [0.012262]ms, count 900000 INFO 2021-04-02 02:57:06,710 gamma_index_ivfpq.cc:691 Add time [0.01296]ms, count 910000 INFO 2021-04-02 02:57:06,960 gamma_index_ivfpq.cc:691 Add time [0.0121838]ms, count 920000 INFO 2021-04-02 02:57:07,203 gamma_index_ivfpq.cc:691 Add time [0.015032]ms, count 930000 INFO 2021-04-02 02:57:07,437 gamma_index_ivfpq.cc:691 Add time [0.0125981]ms, count 940000 INFO 2021-04-02 02:57:07,720 gamma_index_ivfpq.cc:691 Add time [0.016554]ms, count 950000 INFO 2021-04-02 02:57:07,968 gamma_index_ivfpq.cc:691 Add time [0.012801]ms, count 960000 INFO 2021-04-02 02:57:08,229 gamma_index_ivfpq.cc:691 Add time [0.0136951]ms, count 970000 INFO 2021-04-02 02:57:08,505 gamma_index_ivfpq.cc:691 Add time [0.0165691]ms, count 980000 INFO 2021-04-02 02:57:08,828 gamma_index_ivfpq.cc:691 Add time [0.016342]ms, count 990000 INFO 2021-04-02 02:57:09,156 gamma_index_ivfpq.cc:691 Add time [0.0210149]ms, count 1000000

INFO 2021-04-02 02:52:07,126 table.cc:48 Table created success!
INFO 2021-04-02 02:52:07,126 gamma_engine.cc:285 GammaEngine setup successed! bitmap_bytes_size=268435456
INFO 2021-04-02 02:52:07,126 gamma_api.cc:52 Engine init successed!
INFO 2021-04-02 02:52:07,127 vector_manager.cc:117 store params={"store_type":,"cache_size":2048,"segment_size":500000,"compress":{"rate":16}}
INFO 2021-04-02 02:52:07,127 raw_vector_common.h:172 zfp compress rate=16, zfpsize=256
INFO 2021-04-02 02:52:07,141 rocksdb_raw_vector.cc:64 rocks raw vector init success! name=vec, block cache size=2147483648Bytes
INFO 2021-04-02 02:52:07,142 raw_vector.cc:221 raw vector init success! name=vec, has source=0, multi_vids=0, vector_byte_size=256, dimension=128, compress={"rate":16}
INFO 2021-04-02 02:52:07,142 vector_manager.cc:153 Create index model [IVFPQ]
INFO 2021-04-02 02:52:07,142 gamma_index_ivfpq.cc:319 ncentroids =256, nsubvector =32, nbits_per_idx =8, metric_type =0, bucket_init_size =1000, bucket_max_size =1280000
INFO 2021-04-02 02:52:07,154 realtime_mem_data.cc:89 init success! total_mem_bytes=10243072, current max size=256000
INFO 2021-04-02 02:52:07,154 vector_manager.cc:177 create vectors and indexes success! models=[IVFPQ]
INFO 2021-04-02 02:52:07,154 table.cc:124 Table compress [0]
INFO 2021-04-02 02:52:07,154 table.cc:131 Add field name [_id], type [4], index [0]
INFO 2021-04-02 02:52:07,154 table.cc:173 Create table 365 success! item length=5, field num=1
INFO 2021-04-02 02:52:07,154 gamma_engine.cc:636 create table [365] success!
INFO 2021-04-02 02:52:07,154 async_flush.h:70 async flush executor is started!
INFO 2021-04-02 02:52:07,255 async_flush.h:75 async flush executor exit successfully!
INFO 2021-04-02 02:52:07,256 table.cc:109 Table load successed! doc num=0
INFO 2021-04-02 02:52:07,256 table_io.cc:22 Table load successed! doc num=0
INFO 2021-04-02 02:52:07,256 vector_manager.cc:692 vector [vec] load success!
INFO 2021-04-02 02:52:07,257 gamma_engine.cc:1229 load engine success! max docid=0, load directory=, clean directorys(not done)=[]
INFO 2021-04-02 02:52:07,257 async_flush.h:70 async flush executor is started!
INFO 2021-04-02 02:53:12,834 table.cc:390 Add item _id [0], num [0]
INFO 2021-04-02 02:53:12,835 table.cc:410 table cost [5.86499]ms
INFO 2021-04-02 02:53:16,937 gamma_engine.cc:768 Add total cost [3.77686]ms
INFO 2021-04-02 02:53:16,938 table.cc:390 Add item _id [10000], num [10000]
INFO 2021-04-02 02:53:16,939 table.cc:410 table cost [1.55005]ms
INFO 2021-04-02 02:53:18,703 gamma_engine.cc:768 Add total cost [4.10205]ms
INFO 2021-04-02 02:53:18,708 table.cc:390 Add item _id [20200], num [20000]
INFO 2021-04-02 02:53:18,709 table.cc:410 table cost [1.49512]ms
INFO 2021-04-02 02:53:23,794 gamma_engine.cc:768 Add total cost [3.26904]ms
INFO 2021-04-02 02:53:23,795 table.cc:390 Add item _id [30100], num [30000]
INFO 2021-04-02 02:53:23,796 table.cc:410 table cost [1.70581]ms
INFO 2021-04-02 02:53:25,741 gamma_engine.cc:768 Add total cost [4.35791]ms
INFO 2021-04-02 02:53:25,745 table.cc:390 Add item _id [40000], num [40000]
INFO 2021-04-02 02:53:25,746 table.cc:410 table cost [1.41821]ms
INFO 2021-04-02 02:53:27,826 gamma_engine.cc:768 Add total cost [4.83398]ms
INFO 2021-04-02 02:53:27,833 table.cc:390 Add item _id [50100], num [50000]
INFO 2021-04-02 02:53:27,834 table.cc:410 table cost [3.08203]ms
INFO 2021-04-02 02:53:35,675 gamma_engine.cc:768 Add total cost [3.93921]ms
INFO 2021-04-02 02:53:35,682 table.cc:390 Add item _id [60000], num [60000]
INFO 2021-04-02 02:53:35,683 table.cc:410 table cost [2.06616]ms
INFO 2021-04-02 02:53:37,686 gamma_engine.cc:768 Add total cost [2.97607]ms
INFO 2021-04-02 02:53:37,699 table.cc:390 Add item _id [69900], num [70000]
INFO 2021-04-02 02:53:37,701 table.cc:410 table cost [1.63281]ms
INFO 2021-04-02 02:53:44,749 gamma_engine.cc:768 Add total cost [3.36987]ms
INFO 2021-04-02 02:53:44,761 table.cc:390 Add item _id [80200], num [80000]
INFO 2021-04-02 02:53:44,762 table.cc:410 table cost [2.65601]ms
INFO 2021-04-02 02:53:46,779 gamma_engine.cc:768 Add total cost [3.68604]ms
INFO 2021-04-02 02:53:46,801 table.cc:390 Add item _id [90100], num [90000]
INFO 2021-04-02 02:53:46,802 table.cc:410 table cost [2.01587]ms
INFO 2021-04-02 02:53:48,443 gamma_engine.cc:768 Add total cost [4.50806]ms
INFO 2021-04-02 02:53:48,444 table.cc:390 Add item _id [100100], num [100000]
INFO 2021-04-02 02:53:48,445 table.cc:410 table cost [1.56201]ms
INFO 2021-04-02 02:53:56,995 gamma_engine.cc:768 Add total cost [3.04199]ms
INFO 2021-04-02 02:53:56,997 table.cc:390 Add item _id [110100], num [110000]
INFO 2021-04-02 02:53:56,999 table.cc:410 table cost [1.71899]ms
INFO 2021-04-02 02:53:58,520 gamma_engine.cc:768 Add total cost [3.90991]ms
INFO 2021-04-02 02:53:58,522 table.cc:390 Add item _id [120000], num [120000]
INFO 2021-04-02 02:53:58,523 table.cc:410 table cost [1.75513]ms
INFO 2021-04-02 02:54:01,004 gamma_engine.cc:768 Add total cost [3.21802]ms
INFO 2021-04-02 02:54:01,015 table.cc:390 Add item _id [130400], num [130000]
INFO 2021-04-02 02:54:01,016 table.cc:410 table cost [1.75415]ms
INFO 2021-04-02 02:54:02,811 gamma_engine.cc:768 Add total cost [3.47681]ms
INFO 2021-04-02 02:54:02,840 table.cc:390 Add item _id [140000], num [140000]
INFO 2021-04-02 02:54:02,841 table.cc:410 table cost [1.57202]ms
INFO 2021-04-02 02:54:04,514 gamma_engine.cc:768 Add total cost [5.20801]ms
INFO 2021-04-02 02:54:04,520 table.cc:390 Add item _id [150000], num [150000]
INFO 2021-04-02 02:54:04,522 table.cc:410 table cost [1.69385]ms
INFO 2021-04-02 02:54:06,200 gamma_engine.cc:768 Add total cost [3.24902]ms
INFO 2021-04-02 02:54:06,220 table.cc:390 Add item _id [160000], num [160000]
INFO 2021-04-02 02:54:06,222 table.cc:410 table cost [1.95312]ms
INFO 2021-04-02 02:54:07,630 gamma_engine.cc:768 Add total cost [3.46387]ms
INFO 2021-04-02 02:54:07,639 table.cc:390 Add item _id [169900], num [170000]
INFO 2021-04-02 02:54:07,641 table.cc:410 table cost [1.70508]ms
INFO 2021-04-02 02:54:10,467 gamma_engine.cc:768 Add total cost [3.38208]ms
INFO 2021-04-02 02:54:10,468 table.cc:390 Add item _id [179600], num [180000]
INFO 2021-04-02 02:54:10,469 table.cc:410 table cost [1.62402]ms
INFO 2021-04-02 02:54:12,716 gamma_engine.cc:768 Add total cost [4.50513]ms
INFO 2021-04-02 02:54:12,721 table.cc:390 Add item _id [190000], num [190000]
INFO 2021-04-02 02:54:12,722 table.cc:410 table cost [1.64697]ms
INFO 2021-04-02 02:54:14,950 gamma_engine.cc:768 Add total cost [3.47705]ms
INFO 2021-04-02 02:54:14,951 table.cc:390 Add item _id [200000], num [200000]
INFO 2021-04-02 02:54:14,952 table.cc:410 table cost [1.38306]ms
INFO 2021-04-02 02:54:15,258 gamma_index_ivfpq.cc:1287 gamma index is not trained, skip dumping
INFO 2021-04-02 02:54:15,258 vector_manager.cc:661 vector vec_IVFPQ dump gamma index success!
INFO 2021-04-02 02:54:15,258 vector_manager.cc:675 vector vec dump success!
INFO 2021-04-02 02:54:15,926 gamma_engine.cc:1078 Dumped to [/data/db/data/365/retrieval_model_index/2021-04-02-02:54:15], next dump docid [201600], last dump directory(removed)=
INFO 2021-04-02 02:54:16,991 gamma_engine.cc:768 Add total cost [3.24194]ms
INFO 2021-04-02 02:54:17,063 table.cc:390 Add item _id [210000], num [210000]
INFO 2021-04-02 02:54:17,064 table.cc:410 table cost [1.56421]ms
INFO 2021-04-02 02:54:19,142 gamma_engine.cc:768 Add total cost [4.2168]ms
INFO 2021-04-02 02:54:19,159 table.cc:390 Add item _id [220000], num [220000]
INFO 2021-04-02 02:54:19,160 table.cc:410 table cost [1.5752]ms
INFO 2021-04-02 02:54:21,122 gamma_engine.cc:768 Add total cost [3.16797]ms
INFO 2021-04-02 02:54:21,125 table.cc:390 Add item _id [230000], num [230000]
INFO 2021-04-02 02:54:21,127 table.cc:410 table cost [2.38086]ms
INFO 2021-04-02 02:54:23,091 gamma_engine.cc:768 Add total cost [3.52808]ms
INFO 2021-04-02 02:54:23,092 table.cc:390 Add item _id [239900], num [240000]
INFO 2021-04-02 02:54:23,094 table.cc:410 table cost [1.77197]ms
INFO 2021-04-02 02:54:24,662 gamma_engine.cc:768 Add total cost [3.38013]ms
INFO 2021-04-02 02:54:24,663 table.cc:390 Add item _id [250000], num [250000]
INFO 2021-04-02 02:54:24,664 table.cc:410 table cost [1.81299]ms
INFO 2021-04-02 02:54:27,028 gamma_engine.cc:768 Add total cost [3.35596]ms
INFO 2021-04-02 02:54:27,060 table.cc:390 Add item _id [260100], num [260000]
INFO 2021-04-02 02:54:27,061 table.cc:410 table cost [1.44092]ms
INFO 2021-04-02 02:54:28,804 gamma_engine.cc:768 Add total cost [3.73315]ms
INFO 2021-04-02 02:54:28,805 table.cc:390 Add item _id [270000], num [270000]
INFO 2021-04-02 02:54:28,806 table.cc:410 table cost [1.35889]ms
INFO 2021-04-02 02:54:31,164 gamma_engine.cc:768 Add total cost [3.51904]ms
INFO 2021-04-02 02:54:31,169 table.cc:390 Add item _id [279100], num [280000]
INFO 2021-04-02 02:54:31,170 table.cc:410 table cost [1.8252]ms
INFO 2021-04-02 02:54:32,852 gamma_engine.cc:768 Add total cost [3.09204]ms
INFO 2021-04-02 02:54:32,853 table.cc:390 Add item _id [290100], num [290000]
INFO 2021-04-02 02:54:32,854 table.cc:410 table cost [1.48486]ms
INFO 2021-04-02 02:54:35,006 gamma_engine.cc:768 Add total cost [3.6062]ms
INFO 2021-04-02 02:54:35,007 table.cc:390 Add item _id [299900], num [300000]
INFO 2021-04-02 02:54:35,008 table.cc:410 table cost [1.48315]ms
INFO 2021-04-02 02:54:37,105 gamma_engine.cc:768 Add total cost [4.14014]ms
INFO 2021-04-02 02:54:37,137 table.cc:390 Add item _id [310200], num [310000]
INFO 2021-04-02 02:54:37,138 table.cc:410 table cost [1.57104]ms
INFO 2021-04-02 02:54:39,214 gamma_engine.cc:768 Add total cost [4.0249]ms
INFO 2021-04-02 02:54:39,215 table.cc:390 Add item _id [320000], num [320000]
INFO 2021-04-02 02:54:39,217 table.cc:410 table cost [1.46509]ms
INFO 2021-04-02 02:54:41,193 gamma_engine.cc:768 Add total cost [3.35498]ms
INFO 2021-04-02 02:54:41,194 table.cc:390 Add item _id [329900], num [330000]
INFO 2021-04-02 02:54:41,195 table.cc:410 table cost [1.38794]ms
INFO 2021-04-02 02:54:44,129 gamma_engine.cc:768 Add total cost [3.43408]ms
INFO 2021-04-02 02:54:44,143 table.cc:390 Add item _id [340000], num [340000]
INFO 2021-04-02 02:54:44,144 table.cc:410 table cost [1.60791]ms
INFO 2021-04-02 02:54:46,137 gamma_engine.cc:768 Add total cost [3.46484]ms
INFO 2021-04-02 02:54:46,138 table.cc:390 Add item _id [350200], num [350000]
INFO 2021-04-02 02:54:46,140 table.cc:410 table cost [1.73804]ms
INFO 2021-04-02 02:54:48,301 gamma_engine.cc:768 Add total cost [2.99683]ms
INFO 2021-04-02 02:54:48,302 table.cc:390 Add item _id [359400], num [360000]
INFO 2021-04-02 02:54:48,303 table.cc:410 table cost [1.51587]ms
INFO 2021-04-02 02:54:50,062 gamma_engine.cc:768 Add total cost [3.76196]ms
INFO 2021-04-02 02:54:50,063 table.cc:390 Add item _id [369900], num [370000]
INFO 2021-04-02 02:54:50,064 table.cc:410 table cost [1.48193]ms
INFO 2021-04-02 02:54:51,420 gamma_engine.cc:768 Add total cost [3.31006]ms
INFO 2021-04-02 02:54:51,420 table.cc:390 Add item _id [379800], num [380000]
INFO 2021-04-02 02:54:51,422 table.cc:410 table cost [1.51099]ms
INFO 2021-04-02 02:54:53,282 gamma_engine.cc:768 Add total cost [4.01099]ms
INFO 2021-04-02 02:54:53,309 table.cc:390 Add item _id [390100], num [390000]
INFO 2021-04-02 02:54:53,311 table.cc:410 table cost [1.50195]ms
INFO 2021-04-02 02:54:55,365 gamma_engine.cc:768 Add total cost [3.24805]ms
INFO 2021-04-02 02:54:55,407 table.cc:390 Add item _id [400000], num [400000]
INFO 2021-04-02 02:54:55,408 table.cc:410 table cost [1.54199]ms
INFO 2021-04-02 02:54:57,546 gamma_engine.cc:768 Add total cost [4.60107]ms
INFO 2021-04-02 02:54:57,551 table.cc:390 Add item _id [409800], num [410000]
INFO 2021-04-02 02:54:57,552 table.cc:410 table cost [1.88696]ms
INFO 2021-04-02 02:54:59,432 gamma_engine.cc:768 Add total cost [3.2002]ms
INFO 2021-04-02 02:54:59,443 table.cc:390 Add item _id [419900], num [420000]
INFO 2021-04-02 02:54:59,444 table.cc:410 table cost [1.7168]ms
INFO 2021-04-02 02:55:01,197 gamma_engine.cc:768 Add total cost [4.33301]ms
INFO 2021-04-02 02:55:01,202 table.cc:390 Add item _id [430000], num [430000]
INFO 2021-04-02 02:55:01,204 table.cc:410 table cost [1.73096]ms
INFO 2021-04-02 02:55:02,570 gamma_engine.cc:768 Add total cost [5.27905]ms
INFO 2021-04-02 02:55:02,584 table.cc:390 Add item _id [440100], num [440000]
INFO 2021-04-02 02:55:02,586 table.cc:410 table cost [1.89917]ms
INFO 2021-04-02 02:55:03,940 gamma_engine.cc:768 Add total cost [4.70288]ms
INFO 2021-04-02 02:55:03,973 table.cc:390 Add item _id [450000], num [450000]
INFO 2021-04-02 02:55:03,974 table.cc:410 table cost [2.85889]ms
INFO 2021-04-02 02:55:05,207 gamma_engine.cc:768 Add total cost [3.58301]ms
INFO 2021-04-02 02:55:05,215 table.cc:390 Add item _id [460000], num [460000]
INFO 2021-04-02 02:55:05,216 table.cc:410 table cost [1.55811]ms
INFO 2021-04-02 02:55:06,787 gamma_engine.cc:768 Add total cost [3.53687]ms
INFO 2021-04-02 02:55:06,792 table.cc:390 Add item _id [470400], num [470000]
INFO 2021-04-02 02:55:06,794 table.cc:410 table cost [1.67603]ms
INFO 2021-04-02 02:55:08,064 gamma_engine.cc:768 Add total cost [4.65381]ms
INFO 2021-04-02 02:55:08,071 table.cc:390 Add item _id [480000], num [480000]
INFO 2021-04-02 02:55:08,072 table.cc:410 table cost [2.02808]ms
INFO 2021-04-02 02:55:09,139 gamma_engine.cc:768 Add total cost [3.51196]ms
INFO 2021-04-02 02:55:09,155 table.cc:390 Add item _id [490000], num [490000]
INFO 2021-04-02 02:55:09,156 table.cc:410 table cost [1.51318]ms
INFO 2021-04-02 02:55:10,289 gamma_engine.cc:768 Add total cost [5.61304]ms
INFO 2021-04-02 02:55:10,290 table.cc:390 Add item _id [500200], num [500000]
INFO 2021-04-02 02:55:10,292 table.cc:410 table cost [2.38794]ms
INFO 2021-04-02 02:55:11,420 gamma_engine.cc:768 Add total cost [3.87891]ms
INFO 2021-04-02 02:55:11,421 table.cc:390 Add item _id [510000], num [510000]
INFO 2021-04-02 02:55:11,423 table.cc:410 table cost [1.43896]ms
INFO 2021-04-02 02:55:12,659 gamma_engine.cc:768 Add total cost [6.00195]ms
INFO 2021-04-02 02:55:12,662 table.cc:390 Add item _id [519300], num [520000]
INFO 2021-04-02 02:55:12,663 table.cc:410 table cost [1.80688]ms
INFO 2021-04-02 02:55:13,771 gamma_engine.cc:768 Add total cost [3.53003]ms
INFO 2021-04-02 02:55:13,833 table.cc:390 Add item _id [530100], num [530000]
INFO 2021-04-02 02:55:13,834 table.cc:410 table cost [1.63306]ms
INFO 2021-04-02 02:55:15,141 gamma_engine.cc:768 Add total cost [3.45312]ms
INFO 2021-04-02 02:55:15,159 table.cc:390 Add item _id [540100], num [540000]
INFO 2021-04-02 02:55:15,161 table.cc:410 table cost [1.93408]ms
INFO 2021-04-02 02:55:16,454 gamma_engine.cc:768 Add total cost [3.71606]ms
INFO 2021-04-02 02:55:16,472 table.cc:390 Add item _id [550000], num [550000]
INFO 2021-04-02 02:55:16,474 table.cc:410 table cost [1.67188]ms
INFO 2021-04-02 02:55:17,744 gamma_engine.cc:768 Add total cost [3.21411]ms
INFO 2021-04-02 02:55:17,753 table.cc:390 Add item _id [560000], num [560000]
INFO 2021-04-02 02:55:17,755 table.cc:410 table cost [1.91602]ms
INFO 2021-04-02 02:55:19,250 gamma_engine.cc:768 Add total cost [4.31787]ms
INFO 2021-04-02 02:55:19,258 table.cc:390 Add item _id [570100], num [570000]
INFO 2021-04-02 02:55:19,259 table.cc:410 table cost [1.85303]ms
INFO 2021-04-02 02:55:20,553 gamma_engine.cc:768 Add total cost [7.7998]ms
INFO 2021-04-02 02:55:20,554 table.cc:390 Add item _id [580000], num [580000]
INFO 2021-04-02 02:55:20,556 table.cc:410 table cost [1.67114]ms
INFO 2021-04-02 02:55:21,733 gamma_engine.cc:768 Add total cost [3.85107]ms
INFO 2021-04-02 02:55:21,734 table.cc:390 Add item _id [590100], num [590000]
INFO 2021-04-02 02:55:21,735 table.cc:410 table cost [1.51685]ms
INFO 2021-04-02 02:55:22,900 gamma_engine.cc:768 Add total cost [3.89404]ms
INFO 2021-04-02 02:55:22,901 table.cc:390 Add item _id [600000], num [600000]
INFO 2021-04-02 02:55:22,903 table.cc:410 table cost [1.58496]ms
INFO 2021-04-02 02:55:24,219 gamma_engine.cc:768 Add total cost [5.11011]ms
INFO 2021-04-02 02:55:24,221 table.cc:390 Add item _id [610000], num [610000]
INFO 2021-04-02 02:55:24,222 table.cc:410 table cost [1.88989]ms
INFO 2021-04-02 02:55:25,693 gamma_engine.cc:768 Add total cost [4.97607]ms
INFO 2021-04-02 02:55:25,699 table.cc:390 Add item _id [620100], num [620000]
INFO 2021-04-02 02:55:25,701 table.cc:410 table cost [1.99609]ms
INFO 2021-04-02 02:55:27,162 gamma_engine.cc:768 Add total cost [5.81396]ms
INFO 2021-04-02 02:55:27,166 table.cc:390 Add item _id [630000], num [630000]
INFO 2021-04-02 02:55:27,167 table.cc:410 table cost [1.49805]ms
INFO 2021-04-02 02:55:28,499 gamma_engine.cc:768 Add total cost [3.80786]ms
INFO 2021-04-02 02:55:28,502 table.cc:390 Add item _id [640000], num [640000]
INFO 2021-04-02 02:55:28,503 table.cc:410 table cost [1.59888]ms
INFO 2021-04-02 02:55:29,744 gamma_engine.cc:768 Add total cost [3.18018]ms
INFO 2021-04-02 02:55:29,752 table.cc:390 Add item _id [650000], num [650000]
INFO 2021-04-02 02:55:29,754 table.cc:410 table cost [1.87109]ms
INFO 2021-04-02 02:55:31,128 gamma_engine.cc:768 Add total cost [3.93994]ms
INFO 2021-04-02 02:55:31,144 table.cc:390 Add item _id [660100], num [660000]
INFO 2021-04-02 02:55:31,146 table.cc:410 table cost [1.67383]ms
INFO 2021-04-02 02:55:32,697 gamma_engine.cc:768 Add total cost [3.69995]ms
INFO 2021-04-02 02:55:32,698 table.cc:390 Add item _id [670000], num [670000]
INFO 2021-04-02 02:55:32,700 table.cc:410 table cost [1.67212]ms
INFO 2021-04-02 02:55:34,242 gamma_engine.cc:768 Add total cost [4.31201]ms
INFO 2021-04-02 02:55:34,243 table.cc:390 Add item _id [680200], num [680000]
INFO 2021-04-02 02:55:34,245 table.cc:410 table cost [1.8291]ms
INFO 2021-04-02 02:55:35,589 gamma_engine.cc:768 Add total cost [3.63696]ms
INFO 2021-04-02 02:55:35,591 table.cc:390 Add item _id [689900], num [690000]
INFO 2021-04-02 02:55:35,592 table.cc:410 table cost [1.47485]ms
INFO 2021-04-02 02:55:37,014 gamma_engine.cc:768 Add total cost [3.84399]ms
INFO 2021-04-02 02:55:37,022 table.cc:390 Add item _id [700100], num [700000]
INFO 2021-04-02 02:55:37,024 table.cc:410 table cost [1.75293]ms
INFO 2021-04-02 02:55:38,270 gamma_engine.cc:768 Add total cost [4.61597]ms
INFO 2021-04-02 02:55:38,313 table.cc:390 Add item _id [710000], num [710000]
INFO 2021-04-02 02:55:38,315 table.cc:410 table cost [1.73389]ms
INFO 2021-04-02 02:55:39,521 gamma_engine.cc:768 Add total cost [3.90698]ms
INFO 2021-04-02 02:55:39,531 table.cc:390 Add item _id [719900], num [720000]
INFO 2021-04-02 02:55:39,532 table.cc:410 table cost [6.38306]ms
INFO 2021-04-02 02:55:40,698 gamma_engine.cc:768 Add total cost [3.28589]ms
INFO 2021-04-02 02:55:40,713 table.cc:390 Add item _id [730000], num [730000]
INFO 2021-04-02 02:55:40,714 table.cc:410 table cost [1.41699]ms
INFO 2021-04-02 02:55:41,909 gamma_engine.cc:768 Add total cost [3.34106]ms
INFO 2021-04-02 02:55:41,909 table.cc:390 Add item _id [740100], num [740000]
INFO 2021-04-02 02:55:41,911 table.cc:410 table cost [1.42822]ms
INFO 2021-04-02 02:55:43,033 gamma_engine.cc:768 Add total cost [4.14697]ms
INFO 2021-04-02 02:55:43,048 table.cc:390 Add item _id [750000], num [750000]
INFO 2021-04-02 02:55:43,050 table.cc:410 table cost [1.87305]ms
INFO 2021-04-02 02:55:44,625 gamma_engine.cc:768 Add total cost [5.58887]ms
INFO 2021-04-02 02:55:44,643 table.cc:390 Add item _id [760000], num [760000]
INFO 2021-04-02 02:55:44,644 table.cc:410 table cost [1.729]ms
INFO 2021-04-02 02:55:46,058 gamma_engine.cc:768 Add total cost [3.06396]ms
INFO 2021-04-02 02:55:46,060 table.cc:390 Add item _id [770100], num [770000]
INFO 2021-04-02 02:55:46,062 table.cc:410 table cost [1.698]ms
INFO 2021-04-02 02:55:47,240 gamma_engine.cc:768 Add total cost [3.58594]ms
INFO 2021-04-02 02:55:47,241 table.cc:390 Add item _id [780100], num [780000]
INFO 2021-04-02 02:55:47,242 table.cc:410 table cost [1.5271]ms
INFO 2021-04-02 02:55:48,607 gamma_engine.cc:768 Add total cost [4.00708]ms
INFO 2021-04-02 02:55:48,608 table.cc:390 Add item _id [790000], num [790000]
INFO 2021-04-02 02:55:48,610 table.cc:410 table cost [1.48804]ms
INFO 2021-04-02 02:55:49,827 gamma_engine.cc:768 Add total cost [4.14307]ms
INFO 2021-04-02 02:55:49,828 table.cc:390 Add item _id [799900], num [800000]
INFO 2021-04-02 02:55:49,829 table.cc:410 table cost [1.37695]ms
INFO 2021-04-02 02:55:51,161 gamma_engine.cc:768 Add total cost [3.67896]ms
INFO 2021-04-02 02:55:51,162 table.cc:390 Add item _id [809900], num [810000]
INFO 2021-04-02 02:55:51,163 table.cc:410 table cost [1.75903]ms
INFO 2021-04-02 02:55:52,344 gamma_engine.cc:768 Add total cost [3.04712]ms
INFO 2021-04-02 02:55:52,352 table.cc:390 Add item _id [820000], num [820000]
INFO 2021-04-02 02:55:52,353 table.cc:410 table cost [1.41211]ms
INFO 2021-04-02 02:55:53,482 gamma_engine.cc:768 Add total cost [3.229]ms
INFO 2021-04-02 02:55:53,483 table.cc:390 Add item _id [830000], num [830000]
INFO 2021-04-02 02:55:53,484 table.cc:410 table cost [1.36108]ms
INFO 2021-04-02 02:55:54,450 gamma_engine.cc:768 Add total cost [3.69702]ms
INFO 2021-04-02 02:55:54,452 table.cc:390 Add item _id [840000], num [840000]
INFO 2021-04-02 02:55:54,453 table.cc:410 table cost [1.45996]ms
INFO 2021-04-02 02:55:55,407 gamma_engine.cc:768 Add total cost [3.21094]ms
INFO 2021-04-02 02:55:55,409 table.cc:390 Add item _id [850000], num [850000]
INFO 2021-04-02 02:55:55,410 table.cc:410 table cost [1.55811]ms
INFO 2021-04-02 02:55:56,355 gamma_engine.cc:768 Add total cost [3.85205]ms
INFO 2021-04-02 02:55:56,373 table.cc:390 Add item _id [860100], num [860000]
INFO 2021-04-02 02:55:56,374 table.cc:410 table cost [1.50195]ms
INFO 2021-04-02 02:55:57,347 gamma_engine.cc:768 Add total cost [4.35303]ms
INFO 2021-04-02 02:55:57,348 table.cc:390 Add item _id [870000], num [870000]
INFO 2021-04-02 02:55:57,349 table.cc:410 table cost [2.10889]ms
INFO 2021-04-02 02:55:58,522 gamma_engine.cc:768 Add total cost [5.06909]ms
INFO 2021-04-02 02:55:58,533 table.cc:390 Add item _id [880000], num [880000]
INFO 2021-04-02 02:55:58,534 table.cc:410 table cost [1.55493]ms
INFO 2021-04-02 02:55:59,966 gamma_engine.cc:768 Add total cost [3.83887]ms
INFO 2021-04-02 02:55:59,967 table.cc:390 Add item _id [890200], num [890000]
INFO 2021-04-02 02:55:59,968 table.cc:410 table cost [1.43188]ms
INFO 2021-04-02 02:56:01,324 gamma_engine.cc:761 Begin indexing.
INFO 2021-04-02 02:56:01,324 gamma_engine.cc:768 Add total cost [3.71899]ms
INFO 2021-04-02 02:56:01,337 table.cc:390 Add item _id [900000], num [900000]
INFO 2021-04-02 02:56:01,338 table.cc:410 table cost [3.66699]ms
INFO 2021-04-02 02:56:02,686 gamma_engine.cc:768 Add total cost [4.01904]ms
INFO 2021-04-02 02:56:02,704 table.cc:390 Add item _id [910100], num [910000]
INFO 2021-04-02 02:56:02,706 table.cc:410 table cost [2.18506]ms
INFO 2021-04-02 02:56:04,146 gamma_engine.cc:768 Add total cost [4.32007]ms
INFO 2021-04-02 02:56:04,147 table.cc:390 Add item _id [920000], num [920000]
INFO 2021-04-02 02:56:04,149 table.cc:410 table cost [1.76099]ms
INFO 2021-04-02 02:56:05,581 gamma_engine.cc:768 Add total cost [4.1189]ms
INFO 2021-04-02 02:56:05,587 table.cc:390 Add item _id [930100], num [930000]
INFO 2021-04-02 02:56:05,589 table.cc:410 table cost [5.69312]ms
INFO 2021-04-02 02:56:06,791 gamma_engine.cc:768 Add total cost [4.22095]ms
INFO 2021-04-02 02:56:06,794 table.cc:390 Add item _id [940000], num [940000]
INFO 2021-04-02 02:56:06,796 table.cc:410 table cost [1.78906]ms
INFO 2021-04-02 02:56:07,955 gamma_engine.cc:768 Add total cost [3.81909]ms
INFO 2021-04-02 02:56:07,965 table.cc:390 Add item _id [950100], num [950000]
INFO 2021-04-02 02:56:07,967 table.cc:410 table cost [1.79224]ms
INFO 2021-04-02 02:56:09,323 gamma_engine.cc:768 Add total cost [3.12305]ms
INFO 2021-04-02 02:56:09,333 table.cc:390 Add item _id [960000], num [960000]
INFO 2021-04-02 02:56:09,334 table.cc:410 table cost [1.66479]ms
INFO 2021-04-02 02:56:10,912 gamma_engine.cc:768 Add total cost [6.04614]ms
INFO 2021-04-02 02:56:10,948 table.cc:390 Add item _id [970000], num [970000]
INFO 2021-04-02 02:56:10,949 table.cc:410 table cost [1.55713]ms
INFO 2021-04-02 02:56:12,851 gamma_engine.cc:768 Add total cost [3.7439]ms
INFO 2021-04-02 02:56:12,885 table.cc:390 Add item _id [980100], num [980000]
INFO 2021-04-02 02:56:12,886 table.cc:410 table cost [1.67993]ms
INFO 2021-04-02 02:56:14,758 gamma_engine.cc:768 Add total cost [3.82007]ms
INFO 2021-04-02 02:56:14,761 table.cc:390 Add item _id [990000], num [990000]
INFO 2021-04-02 02:56:14,762 table.cc:410 table cost [2.21021]ms
INFO 2021-04-02 02:56:15,258 gamma_index_ivfpq.cc:1287 gamma index is not trained, skip dumping
INFO 2021-04-02 02:56:15,258 vector_manager.cc:661 vector vec_IVFPQ dump gamma index success!
INFO 2021-04-02 02:56:15,258 vector_manager.cc:675 vector vec dump success!
INFO 2021-04-02 02:56:16,065 gamma_engine.cc:1078 Dumped to [/data/db/data/365/retrieval_model_index/2021-04-02-02:56:15], next dump docid [993200], last dump directory(removed)=/data/db/data/365/retrieval_model_index/2021-04-02-02:54:15
INFO 2021-04-02 02:56:16,317 gamma_engine.cc:768 Add total cost [3.83594]ms
INFO 2021-04-02 02:56:41,629 gamma_index_ivfpq.cc:540 train successed!
INFO 2021-04-02 02:56:41,639 gamma_engine.cc:939 vector manager indexing success!
INFO 2021-04-02 02:56:41,879 gamma_index_ivfpq.cc:691 Add time [0.01149]ms, count 10000
INFO 2021-04-02 02:56:42,135 gamma_index_ivfpq.cc:691 Add time [0.0147]ms, count 20000
INFO 2021-04-02 02:56:42,382 gamma_index_ivfpq.cc:691 Add time [0.0137571]ms, count 30000
INFO 2021-04-02 02:56:42,670 gamma_index_ivfpq.cc:691 Add time [0.0182969]ms, count 40000
INFO 2021-04-02 02:56:42,936 gamma_index_ivfpq.cc:691 Add time [0.0151838]ms, count 50000
INFO 2021-04-02 02:56:43,199 gamma_index_ivfpq.cc:691 Add time [0.013188]ms, count 60000
INFO 2021-04-02 02:56:43,438 gamma_index_ivfpq.cc:691 Add time [0.0137349]ms, count 70000
INFO 2021-04-02 02:56:43,692 gamma_index_ivfpq.cc:691 Add time [0.0160601]ms, count 80000
INFO 2021-04-02 02:56:43,951 gamma_index_ivfpq.cc:691 Add time [0.0197681]ms, count 90000
INFO 2021-04-02 02:56:44,196 gamma_index_ivfpq.cc:691 Add time [0.0118979]ms, count 100000
INFO 2021-04-02 02:56:44,437 gamma_index_ivfpq.cc:691 Add time [0.012363]ms, count 110000
INFO 2021-04-02 02:56:44,768 gamma_index_ivfpq.cc:691 Add time [0.0159211]ms, count 120000
INFO 2021-04-02 02:56:45,073 gamma_index_ivfpq.cc:691 Add time [0.0251519]ms, count 130000
INFO 2021-04-02 02:56:45,343 gamma_index_ivfpq.cc:691 Add time [0.011531]ms, count 140000
INFO 2021-04-02 02:56:45,597 gamma_index_ivfpq.cc:691 Add time [0.0232671]ms, count 150000
INFO 2021-04-02 02:56:45,876 gamma_index_ivfpq.cc:691 Add time [0.019981]ms, count 160000
INFO 2021-04-02 02:56:46,150 gamma_index_ivfpq.cc:691 Add time [0.012915]ms, count 170000
INFO 2021-04-02 02:56:46,454 gamma_index_ivfpq.cc:691 Add time [0.018459]ms, count 180000
INFO 2021-04-02 02:56:46,765 gamma_index_ivfpq.cc:691 Add time [0.0165222]ms, count 190000
INFO 2021-04-02 02:56:47,037 gamma_index_ivfpq.cc:691 Add time [0.0210491]ms, count 200000
INFO 2021-04-02 02:56:47,314 gamma_index_ivfpq.cc:691 Add time [0.016551]ms, count 210000
INFO 2021-04-02 02:56:47,629 gamma_index_ivfpq.cc:691 Add time [0.026741]ms, count 220000
INFO 2021-04-02 02:56:47,917 gamma_index_ivfpq.cc:691 Add time [0.0158611]ms, count 230000
INFO 2021-04-02 02:56:48,184 gamma_index_ivfpq.cc:691 Add time [0.00901489]ms, count 240000
INFO 2021-04-02 02:56:48,439 gamma_index_ivfpq.cc:691 Add time [0.019447]ms, count 250000
INFO 2021-04-02 02:56:48,748 gamma_index_ivfpq.cc:691 Add time [0.0128372]ms, count 260000
INFO 2021-04-02 02:56:49,002 gamma_index_ivfpq.cc:691 Add time [0.0166411]ms, count 270000
INFO 2021-04-02 02:56:49,287 gamma_index_ivfpq.cc:691 Add time [0.015074]ms, count 280000
INFO 2021-04-02 02:56:49,563 gamma_index_ivfpq.cc:691 Add time [0.0169299]ms, count 290000
INFO 2021-04-02 02:56:49,824 gamma_index_ivfpq.cc:691 Add time [0.012345]ms, count 300000
INFO 2021-04-02 02:56:50,087 gamma_index_ivfpq.cc:691 Add time [0.0121641]ms, count 310000
INFO 2021-04-02 02:56:50,328 gamma_index_ivfpq.cc:691 Add time [0.0169419]ms, count 320000
INFO 2021-04-02 02:56:50,615 gamma_index_ivfpq.cc:691 Add time [0.01504]ms, count 330000
INFO 2021-04-02 02:56:50,871 gamma_index_ivfpq.cc:691 Add time [0.012304]ms, count 340000
INFO 2021-04-02 02:56:51,133 gamma_index_ivfpq.cc:691 Add time [0.014531]ms, count 350000
INFO 2021-04-02 02:56:51,388 gamma_index_ivfpq.cc:691 Add time [0.0147351]ms, count 360000
INFO 2021-04-02 02:56:51,690 gamma_index_ivfpq.cc:691 Add time [0.0171189]ms, count 370000
INFO 2021-04-02 02:56:51,944 gamma_index_ivfpq.cc:691 Add time [0.01676]ms, count 380000
INFO 2021-04-02 02:56:52,205 gamma_index_ivfpq.cc:691 Add time [0.01678]ms, count 390000
INFO 2021-04-02 02:56:52,580 gamma_index_ivfpq.cc:691 Add time [0.018749]ms, count 400000
INFO 2021-04-02 02:56:52,852 gamma_index_ivfpq.cc:691 Add time [0.0139302]ms, count 410000
INFO 2021-04-02 02:56:53,118 gamma_index_ivfpq.cc:691 Add time [0.018626]ms, count 420000
INFO 2021-04-02 02:56:53,368 gamma_index_ivfpq.cc:691 Add time [0.0120652]ms, count 430000
INFO 2021-04-02 02:56:53,644 gamma_index_ivfpq.cc:691 Add time [0.013395]ms, count 440000
INFO 2021-04-02 02:56:53,920 gamma_index_ivfpq.cc:691 Add time [0.012509]ms, count 450000
INFO 2021-04-02 02:56:54,186 gamma_index_ivfpq.cc:691 Add time [0.0138159]ms, count 460000
INFO 2021-04-02 02:56:54,444 gamma_index_ivfpq.cc:691 Add time [0.013134]ms, count 470000
INFO 2021-04-02 02:56:54,723 gamma_index_ivfpq.cc:691 Add time [0.0206338]ms, count 480000
INFO 2021-04-02 02:56:54,987 gamma_index_ivfpq.cc:691 Add time [0.0163369]ms, count 490000
INFO 2021-04-02 02:56:55,262 gamma_index_ivfpq.cc:691 Add time [0.014053]ms, count 500000
INFO 2021-04-02 02:56:55,519 gamma_index_ivfpq.cc:691 Add time [0.0170999]ms, count 510000
INFO 2021-04-02 02:56:55,828 gamma_index_ivfpq.cc:691 Add time [0.015988]ms, count 520000
INFO 2021-04-02 02:56:56,083 gamma_index_ivfpq.cc:691 Add time [0.0215549]ms, count 530000
INFO 2021-04-02 02:56:56,348 gamma_index_ivfpq.cc:691 Add time [0.0195439]ms, count 540000
INFO 2021-04-02 02:56:56,626 gamma_index_ivfpq.cc:691 Add time [0.0167009]ms, count 550000
INFO 2021-04-02 02:56:56,897 gamma_index_ivfpq.cc:691 Add time [0.00986499]ms, count 560000
INFO 2021-04-02 02:56:57,143 gamma_index_ivfpq.cc:691 Add time [0.011894]ms, count 570000
INFO 2021-04-02 02:56:57,386 gamma_index_ivfpq.cc:691 Add time [0.0171721]ms, count 580000
INFO 2021-04-02 02:56:57,685 gamma_index_ivfpq.cc:691 Add time [0.022731]ms, count 590000
INFO 2021-04-02 02:56:57,952 gamma_index_ivfpq.cc:691 Add time [0.0139041]ms, count 600000
INFO 2021-04-02 02:56:58,216 gamma_index_ivfpq.cc:691 Add time [0.0122529]ms, count 610000
INFO 2021-04-02 02:56:58,456 gamma_index_ivfpq.cc:691 Add time [0.012512]ms, count 620000
INFO 2021-04-02 02:56:58,749 gamma_index_ivfpq.cc:691 Add time [0.01571]ms, count 630000
INFO 2021-04-02 02:56:59,003 gamma_index_ivfpq.cc:691 Add time [0.0119521]ms, count 640000
INFO 2021-04-02 02:56:59,264 gamma_index_ivfpq.cc:691 Add time [0.0137979]ms, count 650000
INFO 2021-04-02 02:56:59,522 gamma_index_ivfpq.cc:691 Add time [0.0122461]ms, count 660000
INFO 2021-04-02 02:56:59,807 gamma_index_ivfpq.cc:691 Add time [0.0173501]ms, count 670000
INFO 2021-04-02 02:57:00,075 gamma_index_ivfpq.cc:691 Add time [0.0175698]ms, count 680000
INFO 2021-04-02 02:57:00,330 gamma_index_ivfpq.cc:691 Add time [0.01423]ms, count 690000
INFO 2021-04-02 02:57:00,640 gamma_index_ivfpq.cc:691 Add time [0.0157839]ms, count 700000
INFO 2021-04-02 02:57:00,916 gamma_index_ivfpq.cc:691 Add time [0.018365]ms, count 710000
INFO 2021-04-02 02:57:01,178 gamma_index_ivfpq.cc:691 Add time [0.0120979]ms, count 720000
INFO 2021-04-02 02:57:01,455 gamma_index_ivfpq.cc:691 Add time [0.012364]ms, count 730000
INFO 2021-04-02 02:57:01,758 gamma_index_ivfpq.cc:691 Add time [0.018082]ms, count 740000
INFO 2021-04-02 02:57:02,047 gamma_index_ivfpq.cc:691 Add time [0.022356]ms, count 750000
INFO 2021-04-02 02:57:02,327 gamma_index_ivfpq.cc:691 Add time [0.014561]ms, count 760000
INFO 2021-04-02 02:57:02,628 gamma_index_ivfpq.cc:691 Add time [0.024718]ms, count 770000
INFO 2021-04-02 02:57:02,936 gamma_index_ivfpq.cc:691 Add time [0.0211201]ms, count 780000
INFO 2021-04-02 02:57:03,232 gamma_index_ivfpq.cc:691 Add time [0.016179]ms, count 790000
INFO 2021-04-02 02:57:03,509 gamma_index_ivfpq.cc:691 Add time [0.0146211]ms, count 800000
INFO 2021-04-02 02:57:03,825 gamma_index_ivfpq.cc:691 Add time [0.016575]ms, count 810000
INFO 2021-04-02 02:57:04,129 gamma_index_ivfpq.cc:691 Add time [0.0158252]ms, count 820000
INFO 2021-04-02 02:57:04,417 gamma_index_ivfpq.cc:691 Add time [0.0142549]ms, count 830000
INFO 2021-04-02 02:57:04,726 gamma_index_ivfpq.cc:691 Add time [0.0223511]ms, count 840000
INFO 2021-04-02 02:57:05,011 gamma_index_ivfpq.cc:691 Add time [0.0162939]ms, count 850000
INFO 2021-04-02 02:57:05,274 gamma_index_ivfpq.cc:691 Add time [0.0153931]ms, count 860000
INFO 2021-04-02 02:57:05,582 gamma_index_ivfpq.cc:691 Add time [0.023887]ms, count 870000
INFO 2021-04-02 02:57:05,933 gamma_index_ivfpq.cc:691 Add time [0.019176]ms, count 880000
INFO 2021-04-02 02:57:06,194 gamma_index_ivfpq.cc:691 Add time [0.0140579]ms, count 890000
INFO 2021-04-02 02:57:06,439 gamma_index_ivfpq.cc:691 Add time [0.012262]ms, count 900000
INFO 2021-04-02 02:57:06,710 gamma_index_ivfpq.cc:691 Add time [0.01296]ms, count 910000
INFO 2021-04-02 02:57:06,960 gamma_index_ivfpq.cc:691 Add time [0.0121838]ms, count 920000
INFO 2021-04-02 02:57:07,203 gamma_index_ivfpq.cc:691 Add time [0.015032]ms, count 930000
INFO 2021-04-02 02:57:07,437 gamma_index_ivfpq.cc:691 Add time [0.0125981]ms, count 940000
INFO 2021-04-02 02:57:07,720 gamma_index_ivfpq.cc:691 Add time [0.016554]ms, count 950000
INFO 2021-04-02 02:57:07,968 gamma_index_ivfpq.cc:691 Add time [0.012801]ms, count 960000
INFO 2021-04-02 02:57:08,229 gamma_index_ivfpq.cc:691 Add time [0.0136951]ms, count 970000
INFO 2021-04-02 02:57:08,505 gamma_index_ivfpq.cc:691 Add time [0.0165691]ms, count 980000
INFO 2021-04-02 02:57:08,828 gamma_index_ivfpq.cc:691 Add time [0.016342]ms, count 990000
INFO 2021-04-02 02:57:09,156 gamma_index_ivfpq.cc:691 Add time [0.0210149]ms, count 1000000

日志显示索引建完了，但是召回率低

你计算召回的groundtruth是用的sift1M的吧，如果是这样需要如下设置"metric_type":"L2", sift1M本身是用L2计算的groundtruth

改成L2后召回率收敛值由57%变为了77%，还是比较低，见表语句如下
{
"name":"mark",
"partition_num":1,
"replica_num":1,
"engine":{
"name":"gamma",
"index_size":900000,
"id_type":"String",
"retrieval_type":"IVFPQ",
"retrieval_param":{
"metric_type":"L2",
"ncentroids":256,
"nsubvector":32
}
},
"properties":{
"vec":{
"type":"vector",
"dimension":128,
"format":"normalization",
"store_type":"RocksDB",
"store_param":{
"cache_size":2048,
"compress":{
"rate":16
}
}
}
}
}

去掉"format":"normalization" 试试看吧

确认原因是建表空间压缩导致精度损失影响召回
"compress":{
"rate":16
}


Generate GitHub Pages

We want to generate HTML pages from the notebooks that will be hosted in this repository. Three options for doing so:

myst-nb (Sphinx ecosystem, Python-based): this is what almost all ComPWA repos use so far. Highly flexible and works well with generating an API for a Python library, which you may want to do for this repo at some stage. Disadvantage: not so user friendly.
Quarto: see e.g. RUB-EP1/amplitude-serialization#25. Very user friendly, also works well with text-based notebooks (.qmd) with good VSCode support. Everything is configured through a _quarto.yml though, which doesn't leave much control.
MyST Markdown (mystmd, JS-based): similar to Quarto, but using the same Markdown flavour as myst-nb.


Using WinGet to install package from Windows Terminal in Non-Admin mode causes Command-line alias creation to silently fail

Brief description of your issue
I tried to install mkcert from winget using the Windows Terminal in non-Admin mode. It seems that even though the package was created successfully, the line indicating that a command line alias was created didn't work. I double checked my PATH environment variable and that correctly was created and pointed to the path of where the package was installed, but when I tried to run the package using the alias 'mkcert' I got an error indicating it couldn't be found.
Eventually I tried running Windows Terminal in Administrator mode and reinstalled the package and it worked.

Steps to reproduce
In Windows Terminal (non Admin):
winget install mkcert
Install successful and command line alias added.
Restart Terminal to load new environment variables.
mkcert
(error not found)
Expected behavior
using the mkcert alias from the command line works successfully.
Actual behavior
the newly installed package is not found when using the alias, but when using the package name mkcert-xxx-windows-amd64.exe it is found.
Environment
Windows Package Manager v1.5.1572
Copyright (c) Microsoft Corporation. All rights reserved.

Windows: Windows.Desktop v10.0.22621.1848
System Architecture: X64
Package: Microsoft.DesktopAppInstaller v1.20.1572.0

This is because the exe full name is mkcert-v1.4.4-windows-amd64.exe and looks like WinGet doesn't rename portables in non-symlinked installations. Not sure if this bug is being tracked separately here in the CLI.
Similar case to https://github.com/microsoft/winget-pkgs/issues/111454
[Policy] Portable

I'll mark this as duplicate of the newly created issue for better tracking of the underlying issue


Design Question: Cross Origin Cookies / Security / Integrity / Architecture

From hitting many walls with regards to managing tokens between iframes, such as using Post Message to send data cross origin, I have finally applied cross origin cookies as the solution.
This solution delivers the cookies (soft attestation data) to the desired website (via an iframe), however this data can also be obtained by a hacker (as one of many examples) if they installed the token negotiator on their website with the same config ("phishing", "farming" attacks etc).
Oleh and I were just talking about this in more detail and wanted to check to ensure this approach is correct as per security requirements? Please can you add any comments / ask any questions / feedback. @colourful-land, @oleggrib, @jot2re (CC @AW-STJ).
Thanks guys

@nicktaras just to clarify, you mean a malicious site can pose as, say hotel-bogota.com and request ab attestation via an iframe from attestation.id? And they can then inject this on the real hotel-bogota.com website? And similar with say, devcon ticket?
If I understand correctly, then this is an issue, since this undermines the trust-model we have put in place.
As far as I remember, last time we talked about this we discussed how the iframe would actually not directly share the attestation or ticket, but instead supply a ZK proof that they know the correct attestation or ticket.
In particular this ZK proof will be made on an unpredictable number derived from the specific session and domain of the website. This prevent extraction of the attestation or ticket or reuse of the ZK proof by a malicious website.

@nicktaras , As I understand - current solution share cookies (magicLinks in the cookies) with any website? so everyone can access those magiclinks and read tickets+ticketSecrets? same for attestation.id - everyone can access attestation and read attestation+attestationSecret . correct?
in the previous iframe+postMessage flow secrets were hidden from users, even hotelBogota didnt have access to sercets, just to the proof.

@nicktaras just to clarify, you mean a malicious site can pose as, say hotel-bogota.com and request ab attestation via an iframe from attestation.id?

Because the lib is open source and based on a client side website, this is quite possible with all versions of the TN. To reduce the chances of being able to use a Devcon ticket on a non-supported site a white/black list could be used.
As far as I remember, last time we talked about this we discussed how the iframe would actually not directly share the attestation or ticket, but instead supply a ZK proof that they know the correct attestation or ticket. In particular this ZK proof will be made on an unpredictable number derived from the specific session and domain of the website. This prevent extraction of the attestation or ticket or reuse of the ZK proof by a malicious website.
Can we further discuss this, I remember you saying this and also Weiwu mentioned using something like Hurricane Cash ZKP.
To help continue the progress, I'll create a full flow of what the TN is currently doing with regards to handling the UN, signatures, Tokens and EthKey tomorrow.

Sure, but I think it is better to discuss at a meeting than over text :) It becomes quite complex, quite fast. But basically what is needed is a challenge-response approach where hotel-bogota's backend server does a challenge-response with the iframe for attestation.id via TN. Basically it sends a message to the attestation.id iframe, which uses its cookie to compute a response, and return this back.
This response will only work at this time, for this session with hotel-bogota and thus cannot be reused.

@nicktaras , As I understand - current solution share cookies (magicLinks in the cookies) with any website? so everyone can access those magic links and read tickets+ticketSecrets? same for attestation.id - everyone can access attestation and read attestation+attestationSecret . correct?
in the previous iframe+postMessage flow secrets were hidden from users, even hotelBogota didn't have access to sercets, just to the proof.

This is correct in the previous version using cross origin post message was more secure. A hacker would find it difficult to gain access to the attestations. PostMessage by default is not cross-origin compatible with Brave and Safari Web & IOS.

Sure, but I think it is better to discuss at a meeting than over text :) It becomes quite complex, quite fast. But basically what is needed is a challenge-response approach where hotel-bogota's backend server does a challenge-response with the iframe for attestation.id via TN. Basically it sends a message to the attestation.id iframe, which uses its cookie to compute a response, and return this back. This response will only work at this time, for this session with hotel-bogota and thus cannot be reused.

Thanks @jot2re - it becomes a lot of text to read through : ) I'll book a meeting to look into this as the solution for tomorrow (if you have a preferred time to catch up just let me know).

@nicktaras just to clarify, you mean a malicious site can pose as, say hotel-bogota.com and request ab attestation via an iframe from attestation.id?

if malicious site has magicLink or have access to the user email and can read OTP then it can generate attestation. but attestationSecret should stay under attestation.id domain and it can be requested by devconTickets website only. its secure flow.

@oleggrib I am not sure I understand your point in this setting. If the malicious site has access to the user's email account (or the OTP) then yes, as you say, they can simply generate a "legitimate" attestation. But I thought the issue here was that Nick suggested that the attestation (or Ticket, or whatever other object) got shared directly with TN of the malicious site?

I do recall, that we a had a very specific discussion for the specific flow of combining both a Ticket and an Anonymous Attestation through iframes to both devcon and attestation.id. In that flow the we trust the devcon iframe to receive the secret from attestation.id and create a useTicket object to pass back to hotel-bogota.

I'm sorry but my head is pretty blank when I look at the security model of iframe based attestation. It has taken a seriously long time for @nicktaras to get something working, even though violating the security assumption (I didn't know cookie method was meant to pass data as-is between iframe and host, in my mind somehow I thought it was a message STORAGE mechanism for data). In fact, Nick spent so many weeks on this and we have neglected the important aspect of developer experience.
I propose we close nick's work as an insecure but working (in terms of usability) version, let's call it the 'dev branch', finish all the test cases and write a page about the testing method, then reassign it to @oleggrib for restoring security, so in the meanwhile, Nick can continue to work on developer experience which we badly need. Nick will need a meeting to pass knowledge to @oleggrib so let's make it the topic of the next TS meeting (which happens in 36 hours).

@nicktaras , @colourful-land
I just created and tested in Brave browser solution to interact with child-tab (not iframe). It have same security level like iframe, but visible for users(It can be even helpful, so user can see actions and some company Logo's). Nick can test it in different browsers to see if it works good (I suppose that we have a chanse that some mobile browsers can freeze inactive tab scripts)

@oleggrib, @colourful-land, @jot2re - From testing this on Iphone this yesterday evening / this morning, it is as you mention above @oleggrib we have an issue. Apple and other websites are being extremely strict, not allow cross origin cookies without either:
A: Having the user explicitly enable cookies via the browser settings
B: Having the user visit the actual website page (window level), then return to the page e.g. hotel bogota.

The solution to navigate the user back and forth from a website, is a workable solution for Twitter auth. However if a user had to navigate across various token outlets each time you go to a token enabled site, would this work? It wouldn't be a great user experience, but would allow the cross origin post message to work etc.
To flesh out this concept out, this would be the flow:

The user would navigate to the outlets via magic links received via email.
When visiting Hotel Bogota, Devcon (or any token enabled website), they would see the overlay inside the bottom of the page. With a list of all the supported tokens (these would be buttons to the Outlet urls per token type)
These URLs would navigate the user to the outlet... then on load of the page it would re-direct back to the referrer

Mock design to help illustrate the first step of the flow - The connect buttons for off chain attestations would open the outlets per token.

Sure, but I think it is better to discuss at a meeting than over text :) It becomes quite complex, quite fast. But basically what is needed is a challenge-response approach where hotel-bogota's backend server does a challenge-response with the iframe for attestation.id via TN. Basically it sends a message to the attestation.id iframe, which uses its cookie to compute a response, and return this back. This response will only work at this time, for this session with hotel-bogota and thus cannot be reused.

Thanks @jot2re - it becomes a lot of text to read through : ) I'll book a meeting to look into this as the solution for tomorrow (if you have a preferred time to catch up just let me know).

Ah, sorry, I forgot to answer this. Generally any time between 9AM and 6PM CET works well for me :)

Closing this issue.
The solution applied has two alternate flows implemented the TN (Iframe and Tab) to support browsers that don't support cross origin sharing of storage data via an embedded iframe.
Just to confirm, these solutions re-enable the level of security intended, where the token attestation data and secrets are kept inside the token issuer outlet/s (out of reach of the web developers who implement the Token Negotiator).


Fixed the regex for the release builds

The stack traces do differ between the debug and release builds.
In the debug builds the column number is attached to the line number.
In the release builds the column number is omitted and therefore the
current regular expression does not detect the stack trace line.
Putting the optional colon.

Hello :)
Thanks for your PR ! I didn't know it was different in release mode and as my app is not in prod yet I didn't notice that ^^
What's the current behavior un release for now ? does it crash or just return an empty line on Crashlytics ?

It is reported but has no stack trace :)

:D not that great lol thanks a lot !


icon: iMac

Name of the icon : iMac
Tags (alternative names for this icon) : will do after the approval of icon

Full scale
100%

Something else,  I'm not sure if we should make icons for specific brands/devices.
@locness3 What do you think?

Something else, I'm not sure if we should make icons for specific brands/devices.

Definitely NOT.

@locness3 Allrighty.
@mittalyashu Sorry we close this one. We should not make icons for specific brands/devices.

Ignoring the name "iMac", it can be considered as another display icon.


Bug report for failed CircleCI job (job: test)

Auto-generated report for test job.
Link to failed build: .
Note: Information about any subsequent build failures that happen while
this issue is open, will add comments with more information to this issue.

Another occurrence

Another occurrence


Minor Bugs under Typography section

If the placeholder value in the typography section is an empty string, the fontsize token will show the default content. Fontweight and lineheight tokens, on the other hand, do not display any default text.

The default value for the fontsize and lineheights variation value input box in the edit or add modals is set to 0 and cannot be deleted when editing the input field. When entering a new value, the input is concatenated with the existing value i.e. 0, which can lead to a poor user experience. We can use a string type instead of a number type for variantValue. As in the fontweight edit(add) modal, we can simply display an error message for every non-numeric value entered. @alexdanilowicz Thoughts?

Please review and merge!


Fortify XSS issue

Attach (recommended) or Link to PDF file here:
Configuration:
it is an asp.net, C# web application. Trying to scan security vulnerabilities using fortify.
Steps to reproduce the problem:
This is the issue that fortify scan displayed :
"The method GetDocument() in RendDoc.ashx.cs sends unvalidated data to a web browser on line 160, which can result in the browser executing malicious code."
The below is C# code, it shows issues in 3 lines that are prefixed with **
public void GetDocument()
{
ServiceClient dm = null;
DocIntService.DocmanServiceSoapClient InternalService = null;
        //Use Internal/External service based on if using Windows Authentication
        if (HttpContext.Current.Request.IsAuthenticated && HttpContext.Current.User.Identity is WindowsIdentity)
            InternalService = new DocIntService.DocmanServiceSoapClient("DocManNetTcpEndpoint");
        else
            dm = new ServiceClient("DocManwsHttpEndpoint");
        //ServiceClient dm = new ServiceClient("DocManwsHttpEndpoint");

        try
        {
            byte[] DocArray;
            //DocArray = dm == null ? InternalService.GetFile(DocId) : dm.GetFile(DocId);
            if (dm == null)
            {
                //Wrapper for impersonation
                using (WindowsImpersonationContext cxt = (Thread.CurrentPrincipal.Identity as WindowsIdentity).Impersonate())
                    **DocArray = InternalService.GetFile(DocId);**
            }
            else
                **DocArray = dm.GetFile(DocId);**

            string fileExt2;
            string strConttenType = null;
            fileExt2 = fileExt;

            strConttenType = GetService.GetContenttype(fileExt2);

            HttpContext.Current.Response.ClearHeaders();
            HttpContext.Current.Response.ClearContent();
            HttpContext.Current.Response.BufferOutput = true;

            HttpContext.Current.Response.ContentType = strConttenType; // "application/msword"; // for PDF type

            HttpContext.Current.Response.AddHeader("Content-Disposition", "inline");
            **HttpContext.Current.Response.BinaryWrite(DocArray);**

            HttpContext.Current.ApplicationInstance.CompleteRequest(); //http://blogs.msdn.com/b/aspnetue/archive/2010/05/25/response-end-response-close-and-how-customer-feedback-helps-us-improve-msdn-documentation.aspx
        }
        catch (FaultException ex)
        {
            ExMgr.Publish(ex);
            dm.Abort();
        }
        catch (Exception ex)
        {
            ExMgr.Publish(ex);
        }
        finally
        {
            if (dm != null)
            {
                if (dm.State == System.ServiceModel.CommunicationState.Faulted)
                {
                    dm.Abort();              //Abort() is an ungraceful exit—when called, it immediately aborts all service calls in progress and shuts down the host. Active clients will each get an exception.
                }
                else
                {
                    if (dm.State == CommunicationState.Opened || dm.State == CommunicationState.Opening || dm.State == CommunicationState.Created)
                    {
                        dm.Close();
                    }
                    else
                    {
                        dm.Abort();
                    }
                }
            }
            if (InternalService != null)
            {
                if (InternalService.State == System.ServiceModel.CommunicationState.Faulted)
                {
                    InternalService.Abort();              //Abort() is an ungraceful exit—when called, it immediately aborts all service calls in progress and shuts down the host. Active clients will each get an exception.
                }
                else
                {
                    if (InternalService.State == CommunicationState.Opened || InternalService.State == CommunicationState.Opening || InternalService.State == CommunicationState.Created)
                    {
                        InternalService.Close();
                    }
                    else
                    {
                        InternalService.Abort();
                    }
                }
            }
        }
    }

PLease let me know solution for removing this issue and code is scanned w/o any issue

it is an asp.net, C# web application.

Please note that the PDF.js library is written in JavaScript, hence it's quite difficult to understand how your particular case is even relevant here.

Trying to scan security vulnerabilities using fortify.

Note that such tools are known for their many false positives, since the mere presence of a piece of code doesn't automatically imply a vulnerability; note the discussion in old issues such as e.g. #8737 and #9269.

This is the issue that fortify scan displayed :
"The method GetDocument() in RendDoc.ashx.cs sends unvalidated data to a web browser on line 160, which can result in the browser executing malicious code."

This sounds exactly like the situation outlined just above, where the tool simply highlights a particular coding-pattern without being "smart" enough to evaluate it properly.
Obviously it must be possible to pass data to the getDocument method in the PDF.js API, how else would you actually load/render a PDF document!?

The below is C# code, it shows issues in 3 lines that are prefixed with **

None of this is code originating in this library, which again is written in JavaScript.

When opening an issue, please provide all of the information requested in https://github.com/mozilla/pdf.js/blob/master/.github/ISSUE_TEMPLATE.md since as-is this issue unfortunately isn't actionable/valid.
Furthermore, please see https://github.com/mozilla/pdf.js/blob/master/.github/CONTRIBUTING.md (emphasis mine):

If you are developing a custom solution, first check the examples at https://github.com/mozilla/pdf.js#learning and search existing issues. If this does not help, please prepare a short well-documented example that demonstrates the problem and make it accessible online on your website, JS Bin, GitHub, etc. before opening a new issue or contacting us in the Matrix room -- keep in mind that just code snippets won't help us troubleshoot the problem. isn't actionable/valid.

All-in-all, closing as INVALID.


Improving request handling performance

I've done some testing on a large aws instance: (c6i.24xlarge, with 96 vCPU).
I've setup a grpc server that simply echoes the input. The client attempts to query the endpoint as fast as possible concurrently, logging the time taken per request. I'm also running both the client and server on the same machine, using loopback address to eliminate network issues.
It seems the maximum request rate of that the tonic GRPC server can handle is around 75K requests per second.
During max load, the cpu and memory of the server are not heavily utilised at all (max only around 10%).
I've done a quick performance profiling of the grpc server, and it shows most of the time is spent waiting on a Mutex. I've screengrabbed the callstack for the most heavily waited operation here

I wonder what the bottleneck is? Can the request rate be further improved?

Ah I believe this is a performance issue with windows based mio needing a mutex. I would imagine this mutex doesn't show up on unix. That said, this does seem like an issue to open in mio rather than here?

Understood, thanks for your response!


Resolve custom config path so plugins extracting it can work with it

If running homebridge -U ./, the plugin homebridge-config-ui-x doesn't work because it cannot find the config.json file.
Running it with an absolute path does work instead. So I suggest converting the relative path into an absolute one so when plugins request the config path, they'll get the absolute version of it.

@nfarina Can this PR be merged?
Or is there any reason why it shouldn't be?

So sorry it took me forever to get to this! I don't see any reason why this shouldn't be merged.


Suggestion: Use a longer, multi-line text on the "Overlapped" test example

...so you can better see the height difference between the fonts.

Fixed in 2e75ce5f68c


What about CMake?

I add cmake support. Now you can build and install it by cmake, and, of cause, can use find_package in other cmake scripts.

Also, you can use this as git submodule and just use add_subdir in CMakeLists.txt

What is the status of this PR? It would be nice to see CMake in master and this PR is a by-the-book example. It will make integration with other projects very simple.

Unfortunately I have not complete example of usage, but it very simple: just call add_subdirectory(Box2d) in you CMakeLists.txt and use target Box2d - that's all

Unfortunately I have not complete example of usage, but it very simple: just call add_subdirectory(Box2d) in you CMakeLists.txt and use target Box2d - that's all

Thanks for that explanation, but that was not what I was asking for :)
I was asking about whether or not this will be merged to master anytime soon, so that I can migrate from my own fork back to upstream. My CMake integration is almost identical to yours (OBJECT libraries and a shared/static lib), but I have not done any of the install/packaging stuff that you have as it was not a requirement.

Box2D now uses cmake.


Pac dev

Adding .tekton generated directory from Pipelines as Code.

@claudiol Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.


Plugin store

Is your feature request related to a problem? Please describe.

people have a hard time finding what plugins to add and how to add them
need to make it easy to add new plugins from within the app

Describe the solution you'd like

add new plugin button opens a dialog which has a list of the plugins with title + description.
When adding the plugin, it'll prepopulate the alias & command name.

Have something look similar to Obsidian.


anyway u could add a config for qb notfiy and menu , or ox_lib

could be good

I have no plans to support qb-input, notify or qb-menu going forward


add cantina audit

Review Error for kanamouru123 @ 2024-07-03 00:04:26 UTC
User must have write permissions to review


Server testing

@isaacabraham I have updated this one to match the changes you made to the client testing recipe