How to run as a binary

Hi Celogeek.
I hope you are doing well, It's been a while.
I have a question for you.
I have an CLI app that runs your comic-converter and then runs kindlegen and makes the mobis I need.
I trying to package the app and give it a simple GUI so I can pass it to my friend.
In order to do that I'll need to package a build or binary of your comic-converter that can run without GO installed on the destination machine.
Is that possible?
I tried using this https://github.com/tj/gobinaries I'm getting a 500 error. It's probably outdated.
Any help would be greatly appreciated.

go build command produces a static binary that will run on the target architecture without go being installed.

Thank you so much. I not too familiar with GO.

One more question.
I need to get a version of the binary for Windows too. Since built this one, on a Mac I'm not sure if the same binary will work on Windows. Is there a way to target multiple platforms or will the same binary work on all platforms?

One more question.
I need to get a version of the binary for Windows too. Since built this one, on a Mac I'm not sure if the same binary will work on Windows. Is there a way to target multiple platforms or will the same binary work on all platforms?

You can target any os by using GOARCH and GOOS envs.
For windows use GOOS=windows GOARCH=386 go build
Go there to find all the valid combinations of $GOOS and $GOARCH

Ok Thank you. That helps.
I'm having a bug with the converter, I'll open a new issue.


[Input] Document in ZH read-only: input not editable, buttons enabled

Document the discussion in this thread:
https://adevinta.slack.com/archives/C054Q84KPJ4/p1690358867258829

Part of the refactoring of the Input spec.


Ensure that valves are closed at initialization

This PR:

Makes the firmware use the ParametersRequest.ventilating field to determine whether to run the simulator and control loop.
Changes PI controller setpoints to 0 when the the ventilating field is set to false.
Overrides the PI controller to close the valves when flow rate setpoint is 0 L/min.
Makes the backend application turn off ventilation at startup, regardless of what the value of ParametersRequest loaded from file is. This is only a workaround and is undesirable behavior, we will need to fix this later (in fact we might just eventually want to use the value of ParametersRequest from file, since the ventilation flag should be set to off with a clean shutdown of the RPi or a clean close of the backend program).

@Tanmay06 Can you try out this PR to see if you're able to reproduce the issue you had seen with non-zero flow rate between when the backend starts and when the frontend starts? I haven't been able to reproduce the issue, but my intuition is that it should not happen after the changes from this PR.
This PR is in the critical path of development, passes clang-tidy and clang-format checks, is needed to support hardware bring-up at Bharat Forge, is of at least comparable quality to the rest of the firmware codebase, and is a bugfix. I will merge in this PR after the Bharat Forge team tests this and confirms that it fixes the problem we observed, but before a full code review is completed.
For records-keeping:

This project is licensed under Apache License v2.0 for any software, and Solderpad Hardware License v2.1 for any hardware - do you agree that your contributions to this project will be under these licenses, too? Yes
Were any of these contributions also part of work you did for an employer or a client? No
Does this work include, or is it based on, any third-party work which you did not create? No

I've tested this patch, it has fixed the initial idle flow issue. With this PR I'm not able to reproduce the issue we had seen with non-zero flow rate between when the backend starts and when the frontend starts. I've tested it out using a flow sensor at the output of the ventilator, and I didn't see any flow reading during the startup.


MeanAveragePrecision - bug in max_detection_thresholds

🐛 Bug
MeanAveragePrecision return map=-1 for any value other than 100 for max_detection_thresholds
To Reproduce
from torch import tensor
from torchmetrics.detection import MeanAveragePrecision
preds = [dict(boxes=tensor([[0, 0, 100, 100],
                      [0, 0, 50, 50]]),
              scores=tensor([1.0, 0.9]),
              labels=tensor([0, 1]))]

target = [dict(boxes=tensor([[0, 0, 100, 100],
                      [0, 0, 50, 50]]),
               labels=tensor([0, 1]),)]
metric = MeanAveragePrecision(iou_type="bbox", max_detection_thresholds=[1, 10, 50])
metric.update(preds, target)
result = metric.compute()
result  # map = -1

Expected behavior
Expected: map=1 (for 50 max detections)
Environment

TorchMetrics version (and how you installed TM, e.g. conda, pip, build from source): 1.2.1
Python & PyTorch Version (e.g., 1.0): 3.10
Any other relevant information such as OS (e.g., Linux): Ubuntu 22.04

Hi @nisyad-ms, thanks for reporting this issue.
Sadly this is due to a known bug in the official pycocotools backend, that we use for computations. Specifically, this line:
https://github.com/cocodataset/cocoapi/blob/8c9bcc3cf640524c4c20a9c40e89cb6a2f2fa0e9/PythonAPI/pycocotools/cocoeval.py#L460
should have been
stats[0] = _summarize(1, maxDets=self.params.maxDets[2])

for it to work. Sadly the repo is not really maintained anymore, but is still considered the official reference for mAP.
Instead you can install the faster-coco-eval backend (https://github.com/MiXaiLL76/faster_coco_eval) which we also supports. This backend have implemented the fix so your code calculates the correct value.
metric = MeanAveragePrecision(iou_type="bbox", max_detection_thresholds=[1, 10, 50], backend="faster_coco_eval")
metric.update(preds, target)
result = metric.compute()
print(result)

#{'map': tensor(1.), ...

Closing issue because we really cannot fix this on our side.

Thanks @SkafteNicki for the information. How to ensure the faster-coco-eval backend is used? Just installing it will do? Thanks again.

Thanks @SkafteNicki for the information. How to ensure the faster-coco-eval backend is used? Just installing it will do? Thanks again.

Sorry I should have specified that. You install the backend with pip install faster-coco-eval and then when initializing the MeanAveragePrecision class you need to set the backend argument to "faster_coco_eval".


APP-1706 - Created script to remove invalid values for the creatorId field 🐛

…field 🐛

Build triggered. sha1 is merged.

Build started sha1 is merged.

Build finished.

Codecov Report

Merging #172 into dev will decrease coverage by 0.4%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##              dev     #172      +/-   ##
==========================================
- Coverage   89.54%   89.14%   -0.41%
==========================================
  Files         144      175      +31
  Lines        2726     3482     +756
  Branches      261      327      +66
==========================================
+ Hits         2441     3104     +663
- Misses        221      285      +64
- Partials       64       93      +29

Impacted Files
Coverage Δ

...tegration/provisioning/service/KeyPairService.java
91.66% <0%> (ø)

...oning/properties/CompanyCertificateProperties.java
0% <0%> (ø)

...visioning/properties/AuthenticationProperties.java
0% <0%> (ø)

...ation/provisioning/service/ApplicationService.java
93.75% <0%> (ø)

...ation/provisioning/exception/KeyPairException.java
50% <0%> (ø)

.../integration/provisioning/service/UserService.java
94.59% <0%> (ø)

...sioning/exception/CompanyCertificateException.java
100% <0%> (ø)

.../properties/IntegrationProvisioningProperties.java
0% <0%> (ø)

.../exception/ConfigurationProvisioningException.java
100% <0%> (ø)

...ation/provisioning/IntegrationProvisioningApp.java
50% <0%> (ø)

... and 21 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bbd5148...609db44. Read the comment docs.


Calendar event is not getting added in Microsoft calendar and wrong end time is displaying in Yahoo calendar

Getting below issues while integrating addtocalendar.js in angularjs project:

Calendar event is not getting added in Microsoft calendar instead it shows only calendar page.
Wrong end time is displaying in Yahoo calendar
Downloading / opening multiple event files/tabs on requirement basis. Files not getting downloaded in IOS device. If it is downloaded, downloads only one file in devices but in desktop it's working as expected.

I added following files & dependencies but still no luck. Please let me know if I missed out anything:
addtocalendar.js - v1.3.4
moment.min.js
clickoutside.directive.js
fileSaver.js
Blob.js
Kindly help me to solve the issue.
Thanks in advance

Thanks for reporting. These issues are all fixed in v1.3.5.


Test with GitHub Actions on Ubuntu

It disables Travis-CI at the same time.
It still tests on Windows by Appveyor.
It does not upload to GitHub Package Registry, which started in #1196, and was disabled in #1223.
It disables updating https://www.embulk.org/docs/ for a while. The update would be done in a different way later.

Thanks!


Precision issue while querying the vector on the Qdrant vector benchmark test

While trying to integrate Manticore Search into the Qdrant vector database benchmark, I discovered that we have very low mean precision.
In the current test that I have attached here, Manticore Search's mean precision equals 0.16, while Qdrant's is 0.99.
We should investigate this behavior and try to understand why it happens.
I have also attached the original test set in the attachments to initialize the database.
I checked also it with https://newtum.com/calculators/maths/cosine-similarity-calculator
So Manticore top 1 result gives 0.89 similarity while the expected one is 0.91.
Path to the dump: /tmp/bench.tar.gz on dev2.

 The expected document ID 786559 was coming before 81360, as expected.
We also tried executing the request with max_matches=1000000, which produced the correct response. However, without setting max_matches to the maximum number of documents in the table, we were still unable to see the required document in the first position.
The table showed 29 chunks, so we ran an optimize operation to reduce it to a single chunk. After that, we got results even worse than before when not changing the max_matches parameter.
It seems like there's something wrong with the max_matches parameter, and it's affecting the precision of the results. 🤔

I had some buddy-related issues reproducing that behavior locally. If you have a pre-built index (preferably with one chunk), please post a link.

If you have a pre-built index

@glookka pls find it here /home/snikolaev/issue_56/bench_table.tgz on dev2. I didn't run merging just in case it may be import to have multiple chunkg for further investigations.

The fixed dump is here /home/snikolaev/issue_56/bench_dump.tgz on dev2.

I optimized the index to a single chunk and ran queries with max_matches=1000000 and with default max_matches. I don't see a difference in query results.
select id, knn_dist() from bench where knn (vector, 1000000, (0.011699999682605267, 0.011500000022351742, 0.008700000122189522, 0.009999999776482582, 0.07850000262260437, 0.10000000149011612, 0.07840000092983246, 0.05299999937415123, 0.052400000393390656, 0.08190000057220459, 0.0658000037074089, 0.057999998331069946, 0.01590000092983246, 0.017000000923871994, 0.04610000178217888, 0.02419999986886978, 0.00839999970048666, 0.006399999838322401, 0.007199999876320362, 0.010200000368058681, 0.030400000512599945, 0.06790000200271606, 0.05889999866485596, 0.057100001722574234, 0.0333000011742115, 0.07859999686479568, 0.08919999748468399, 0.04230000078678131, 0.013799999840557575, 0.013299999758601189, 0.028999999165534973, 0.021900000050663948, 0.008999999612569809, 0.012199999764561653, 0.010700000450015068, 0.01080000028014183, 0.026599999517202377, 0.03849999979138374, 0.057100001722574234, 0.052000001072883606, 0.03550000116229057, 0.04879999905824661, 0.06920000165700912, 0.05310000106692314, 0.014399999752640724, 0.014299999922513962, 0.01489999983459711, 0.02500000037252903, 0.017100000753998756, 0.016100000590085983, 0.010599999688565731, 0.03240000084042549, 0.02710000053048134, 0.04580000042915344, 0.05310000106692314, 0.06239999830722809, 0.031599998474121094, 0.06080000102519989, 0.06610000133514404, 0.08129999786615372, 0.01860000006854534, 0.022700000554323196, 0.011599999852478504, 0.03370000049471855, 0.024700000882148743, 0.011800000444054604, 0.010700000450015068, 0.06390000134706497, 0.039500001817941666, 0.040300000458955765, 0.05249999836087227, 0.0957999974489212, 0.05510000139474869, 0.06759999692440033, 0.08579999953508377, 0.17489999532699585, 0.024399999529123306, 0.02810000069439411, 0.008700000122189522, 0.05119999870657921, 0.01489999983459711, 0.00860000029206276, 0.012400000356137753, 0.035599999129772186, 0.03280000016093254, 0.03869999945163727, 0.046300001442432404, 0.048900000751018524, 0.04919999837875366, 0.0640999972820282, 0.07050000131130219, 0.11640000343322754, 0.020600000396370888, 0.016200000420212746, 0.012500000186264515, 0.04019999876618385, 0.013500000350177288, 0.011300000362098217, 0.007400000002235174, 0.011800000444054604, 0.032999999821186066, 0.02979999966919422, 0.0364999994635582, 0.041999999433755875, 0.04410000145435333, 0.051899999380111694, 0.06589999794960022, 0.05270000174641609, 0.013899999670684338, 0.016200000420212746, 0.01510000042617321, 0.02239999920129776, 0.011800000444054604, 0.008799999952316284, 0.007799999788403511, 0.010900000110268593, 0.037300001829862595, 0.07050000131130219, 0.06279999762773514, 0.03909999877214432, 0.02419999986886978, 0.055799998342990875, 0.07209999859333038, 0.04479999840259552, 0.01679999940097332, 0.013399999588727951, 0.03310000151395798, 0.020800000056624413, 0.004100000020116568, 0.008999999612569809, 0.016499999910593033, 0.011699999682605267, 0.09619999676942825, 0.13590000569820404, 0.12150000035762787, 0.08799999952316284, 0.06390000134706497, 0.11739999800920486, 0.09700000286102295, 0.07029999792575836, 0.012600000016391277, 0.021900000050663948, 0.07460000365972519, 0.0421999990940094, 0.003700000001117587, 0.00839999970048666, 0.009999999776482582, 0.013199999928474426, 0.03200000151991844, 0.06129999831318855, 0.07509999722242355, 0.09139999747276306, 0.042899999767541885, 0.0640999972820282, 0.0746999979019165, 0.04659999907016754, 0.008700000122189522, 0.028599999845027924, 0.051500000059604645, 0.027899999171495438, 0.007300000172108412, 0.01489999983459711, 0.02250000089406967, 0.018200000748038292, 0.01810000091791153, 0.01759999990463257, 0.08619999885559082, 0.06400000303983688, 0.027499999850988388, 0.028999999165534973, 0.055799998342990875, 0.060499999672174454, 0.01140000019222498, 0.014499999582767487, 0.02070000022649765, 0.02850000001490116, 0.020800000056624413, 0.019099999219179153, 0.012000000104308128, 0.04500000178813934, 0.023399999365210533, 0.023099999874830246, 0.053300000727176666, 0.08179999887943268, 0.03970000147819519, 0.024299999698996544, 0.03889999911189079, 0.10819999873638153, 0.022099999710917473, 0.030300000682473183, 0.01600000075995922, 0.05339999869465828, 0.02710000053048134, 0.01889999955892563, 0.01640000008046627, 0.08810000121593475, 0.043699998408555984, 0.017000000923871994, 0.028200000524520874, 0.13899999856948853, 0.060100000351667404, 0.03909999877214432, 0.03709999844431877, 0.20309999585151672, 0.03620000183582306, 0.0340999998152256, 0.011900000274181366, 0.07450000196695328, 0.016699999570846558, 0.0052999998442828655, 0.010400000028312206, 0.04560000076889992, 0.022199999541044235, 0.020099999383091927, 0.015599999576807022, 0.04580000042915344, 0.04309999942779541, 0.04019999876618385, 0.04439999908208847, 0.13459999859333038, 0.020600000396370888, 0.019200000911951065, 0.014999999664723873, 0.046799998730421066, 0.010300000198185444, 0.006599999964237213, 0.012000000104308128, 0.017899999395012856, 0.01850000023841858, 0.019500000402331352, 0.04399999976158142, 0.03290000185370445, 0.02979999966919422, 0.0706000030040741, 0.09549999982118607, 0.05810000002384186, 0.009700000286102295, 0.015699999406933784, 0.03189999982714653, 0.0406000018119812, 0.007400000002235174, 0.009800000116229057, 0.010300000198185444, 0.011599999852478504, 0.040699999779462814, 0.08299999684095383, 0.06750000268220901, 0.035599999129772186, 0.03519999980926514, 0.08489999920129776, 0.07400000095367432, 0.057500001043081284, 0.00800000037997961, 0.010499999858438969, 0.050200000405311584, 0.03790000081062317, 0.00989999994635582, 0.021299999207258224, 0.0364999994635582, 0.024700000882148743, 0.04809999838471413, 0.07810000330209732, 0.094200000166893, 0.08399999886751175, 0.04170000180602074, 0.05249999836087227, 0.07289999723434448, 0.06639999896287918, 0.010099999606609344, 0.023099999874830246, 0.0868000015616417, 0.06459999829530716, 0.033799998462200165, 0.02160000056028366, 0.01899999938905239, 0.03700000047683716, 0.027400000020861626, 0.044599998742341995, 0.04349999874830246, 0.10090000182390213, 0.05249999836087227, 0.06769999861717224, 0.057500001043081284, 0.04639999940991402, 0.022700000554323196, 0.04170000180602074, 0.04749999940395355, 0.024000000208616257, 0.06080000102519989, 0.033399999141693115, 0.03680000081658363, 0.0763000026345253, 0.053300000727176666, 0.021900000050663948, 0.06080000102519989, 0.11249999701976776, 0.0674000009894371, 0.021800000220537186, 0.05270000174641609, 0.13120000064373016, 0.05889999866485596, 0.03310000151395798, 0.02160000056028366, 0.06530000269412994, 0.022099999710917473, 0.017400000244379044, 0.030700000002980232, 0.030300000682473183, 0.027300000190734863, 0.029400000348687172, 0.06279999762773514, 0.033799998462200165, 0.04010000079870224, 0.043699998408555984, 0.05900000035762787, 0.07609999924898148, 0.028599999845027924, 0.01889999955892563, 0.04179999977350235, 0.06960000097751617, 0.00800000037997961, 0.007899999618530273, 0.009499999694526196, 0.011099999770522118, 0.08179999887943268, 0.10530000180006027, 0.08699999749660492, 0.05310000106692314, 0.05180000141263008, 0.08860000222921371, 0.07509999722242355, 0.06889999657869339, 0.015799999237060547, 0.018200000748038292, 0.051600001752376556, 0.028599999845027924, 0.006200000178068876, 0.004800000227987766, 0.006800000090152025, 0.009700000286102295, 0.03370000049471855, 0.06939999759197235, 0.07209999859333038, 0.061799999326467514, 0.0364999994635582, 0.08240000158548355, 0.11050000041723251, 0.04879999905824661, 0.013899999670684338, 0.01269999984651804, 0.03139999881386757, 0.020899999886751175, 0.010300000198185444, 0.008500000461935997, 0.009999999776482582, 0.00860000029206276, 0.029200000688433647, 0.03709999844431877, 0.06930000334978104, 0.05209999904036522, 0.031199999153614044, 0.04650000110268593, 0.0835999995470047, 0.04859999939799309, 0.014100000262260437, 0.011900000274181366, 0.014600000344216824, 0.02280000038444996, 0.01600000075995922, 0.012299999594688416, 0.010099999606609344, 0.030300000682473183, 0.027400000020861626, 0.04969999939203262, 0.0706000030040741, 0.0674000009894371, 0.029999999329447746, 0.05730000138282776, 0.05469999834895134, 0.08820000290870667, 0.015699999406933784, 0.020099999383091927, 0.011900000274181366, 0.03530000150203705, 0.02250000089406967, 0.01119999960064888, 0.009700000286102295, 0.05999999865889549, 0.037300001829862595, 0.04410000145435333, 0.05939999967813492, 0.08869999647140503, 0.049400001764297485, 0.06599999964237213, 0.08269999921321869, 0.18400000035762787, 0.023000000044703484, 0.02410000003874302, 0.009800000116229057, 0.05009999871253967, 0.013899999670684338, 0.006399999838322401, 0.01119999960064888, 0.033799998462200165, 0.03150000050663948, 0.03929999843239784, 0.04740000143647194, 0.04089999943971634, 0.04259999841451645, 0.06599999964237213, 0.07320000231266022, 0.125, 0.02239999920129776, 0.01549999974668026, 0.01209999993443489, 0.037300001829862595, 0.01119999960064888, 0.006300000008195639, 0.007499999832361937, 0.01119999960064888, 0.03200000151991844, 0.028300000354647636, 0.04800000041723251, 0.041200000792741776, 0.04190000146627426, 0.05829999968409538, 0.07840000092983246, 0.051600001752376556, 0.013500000350177288, 0.0142000000923872, 0.016899999231100082, 0.019500000402331352, 0.009700000286102295, 0.00570000009611249, 0.005900000222027302, 0.00800000037997961, 0.03819999843835831, 0.07479999959468842, 0.061900001019239426, 0.05050000175833702, 0.02710000053048134, 0.0608999989926815, 0.08449999988079071, 0.05510000139474869, 0.015799999237060547, 0.012000000104308128, 0.03689999878406525, 0.021299999207258224, 0.006200000178068876, 0.014399999752640724, 0.02280000038444996, 0.011300000362098217, 0.10400000214576721, 0.148499995470047, 0.14489999413490295, 0.1062999963760376, 0.06589999794960022, 0.13019999861717224, 0.11670000106096268, 0.09629999846220016, 0.015200000256299973, 0.025599999353289604, 0.08219999819993973, 0.04540000110864639, 0.004699999932199717, 0.00559999980032444, 0.010999999940395355, 0.010099999606609344, 0.03500000014901161, 0.06650000065565109, 0.09539999812841415, 0.11710000038146973, 0.045899998396635056, 0.07039999961853027, 0.09549999982118607, 0.07150000333786011, 0.009700000286102295, 0.028999999165534973, 0.05739999935030937, 0.025100000202655792, 0.008500000461935997, 0.010200000368058681, 0.019999999552965164, 0.014800000004470348, 0.024800000712275505, 0.01850000023841858, 0.09279999881982803, 0.07010000199079514, 0.029999999329447746, 0.033799998462200165, 0.07490000128746033, 0.07490000128746033, 0.01489999983459711, 0.017400000244379044, 0.020999999716877937, 0.03319999948143959, 0.018200000748038292, 0.012199999764561653, 0.008299999870359898, 0.04190000146627426, 0.025200000032782555, 0.024000000208616257, 0.05310000106692314, 0.07620000094175339, 0.048900000751018524, 0.024700000882148743, 0.032999999821186066, 0.1257999986410141, 0.017500000074505806, 0.025599999353289604, 0.013100000098347664, 0.052400000393390656, 0.026100000366568565, 0.01489999983459711, 0.015699999406933784, 0.08609999716281891, 0.04340000078082085, 0.02419999986886978, 0.03620000183582306, 0.1395999938249588, 0.05649999901652336, 0.04270000010728836, 0.04309999942779541, 0.21359999477863312, 0.03449999913573265, 0.025100000202655792, 0.014299999922513962, 0.07569999992847443, 0.014600000344216824, 0.006000000052154064, 0.00930000003427267, 0.041099999099969864, 0.025100000202655792, 0.018699999898672104, 0.03099999949336052, 0.04699999839067459, 0.035100001841783524, 0.05299999937415123, 0.054099999368190765, 0.13439999520778656, 0.016699999570846558, 0.01759999990463257, 0.01489999983459711, 0.04390000179409981, 0.009200000204145908, 0.005499999970197678, 0.01360000018030405, 0.018400000408291817, 0.01860000006854534, 0.02329999953508377, 0.06780000030994415, 0.04050000011920929, 0.024800000712275505, 0.07890000194311142, 0.11249999701976776, 0.0681999996304512, 0.008999999612569809, 0.014399999752640724, 0.033799998462200165, 0.03889999911189079, 0.006599999964237213, 0.012900000438094139, 0.011599999852478504, 0.009100000374019146, 0.041600000113248825, 0.09130000323057175, 0.08060000091791153, 0.04610000178217888, 0.037300001829862595, 0.09560000151395798, 0.07819999754428864, 0.06019999831914902, 0.009200000204145908, 0.013199999928474426, 0.0551999993622303, 0.04190000146627426, 0.009499999694526196, 0.018799999728798866, 0.040300000458955765, 0.028200000524520874, 0.052799999713897705, 0.08269999921321869, 0.11349999904632568, 0.10689999908208847, 0.04320000112056732, 0.05559999868273735, 0.08980000019073486, 0.08179999887943268, 0.011900000274181366, 0.028999999165534973, 0.10509999841451645, 0.07090000063180923, 0.03280000016093254, 0.015799999237060547, 0.019099999219179153, 0.03629999980330467, 0.02800000086426735, 0.045099999755620956, 0.048700001090765, 0.10890000313520432, 0.055799998342990875, 0.07259999960660934, 0.07270000129938126, 0.05249999836087227, 0.02070000022649765, 0.04600000008940697, 0.058400001376867294, 0.025499999523162842, 0.0551999993622303, 0.02630000002682209, 0.03319999948143959, 0.07329999655485153, 0.04749999940395355, 0.025200000032782555, 0.06620000302791595, 0.1120000034570694, 0.062199998646974564, 0.033900000154972076, 0.06599999964237213, 0.1362999975681305, 0.05119999870657921, 0.02759999968111515, 0.024900000542402267, 0.06830000132322311, 0.019899999722838402, 0.01590000092983246, 0.02930000051856041, 0.028599999845027924, 0.02810000069439411, 0.03519999980926514, 0.07850000262260437, 0.03709999844431877, 0.039000000804662704, 0.05090000107884407, 0.07649999856948853, 0.08209999650716782, 0.02500000037252903, 0.015200000256299973, 0.0471000000834465, 0.07530000060796738, 0.007699999958276749, 0.009200000204145908, 0.008500000461935997, 0.008700000122189522, 0.08950000256299973, 0.11569999903440475, 0.11339999735355377, 0.07410000264644623, 0.05290000140666962, 0.1014999970793724, 0.09799999743700027, 0.10660000145435333, 0.013299999758601189, 0.017400000244379044, 0.05990000069141388, 0.028699999675154686, 0.0066999997943639755, 0.008100000210106373, 0.008200000040233135, 0.007699999958276749, 0.03319999948143959, 0.07590000331401825, 0.08829999715089798, 0.08799999952316284, 0.040800001472234726, 0.10760000348091125, 0.1386999934911728, 0.07429999858140945, 0.01080000028014183, 0.012900000438094139, 0.03830000013113022, 0.021400000900030136, 0.006300000008195639, 0.007499999832361937, 0.011599999852478504, 0.007400000002235174, 0.030300000682473183, 0.039000000804662704, 0.08299999684095383, 0.05609999969601631, 0.031099999323487282, 0.06440000236034393, 0.11240000277757645, 0.050200000405311584, 0.011699999682605267, 0.010499999858438969, 0.016599999740719795, 0.022299999371170998, 0.013399999588727951, 0.01140000019222498, 0.012400000356137753, 0.027400000020861626, 0.02590000070631504, 0.05290000140666962, 0.08720000088214874, 0.0575999990105629, 0.02979999966919422, 0.06759999692440033, 0.09369999915361404, 0.08139999955892563, 0.013299999758601189, 0.017799999564886093, 0.01119999960064888, 0.0340999998152256, 0.020800000056624413, 0.01209999993443489, 0.012400000356137753, 0.052799999713897705, 0.029500000178813934, 0.044599998742341995, 0.09889999777078629, 0.066600002348423, 0.043699998408555984, 0.07599999755620956, 0.10930000245571136, 0.1460999995470047, 0.02319999970495701, 0.019300000742077827, 0.01360000018030405, 0.04839999973773956, 0.012000000104308128, 0.007499999832361937, 0.012199999764561653, 0.030899999663233757, 0.029999999329447746, 0.04399999976158142, 0.07959999889135361, 0.040800001472234726, 0.034299999475479126, 0.06960000097751617, 0.09059999883174896, 0.1151999980211258, 0.018300000578165054, 0.01080000028014183, 0.01769999973475933, 0.0364999994635582, 0.010200000368058681, 0.00570000009611249, 0.007899999618530273, 0.010099999606609344, 0.03220000118017197, 0.030899999663233757, 0.07159999758005142, 0.052299998700618744, 0.0340999998152256, 0.06459999829530716, 0.1177000030875206, 0.061400000005960464, 0.011699999682605267, 0.013199999928474426, 0.01899999938905239, 0.0203000009059906, 0.009800000116229057, 0.006599999964237213, 0.0066999997943639755, 0.007600000128149986, 0.04129999876022339, 0.08470000326633453, 0.09740000218153, 0.07699999958276749, 0.02969999983906746, 0.07050000131130219, 0.11559999734163284, 0.08709999918937683, 0.01209999993443489, 0.012299999594688416, 0.04129999876022339, 0.02329999953508377, 0.006099999882280827, 0.012199999764561653, 0.02280000038444996, 0.010999999940395355, 0.12189999967813492, 0.17100000381469727, 0.16410000622272491, 0.14069999754428864, 0.07479999959468842, 0.15860000252723694, 0.15629999339580536, 0.15080000460147858, 0.017100000753998756, 0.034299999475479126, 0.10440000146627426, 0.05310000106692314, 0.005200000014156103, 0.006200000178068876, 0.01209999993443489, 0.011500000022351742, 0.03929999843239784, 0.0697999969124794, 0.10480000078678131, 0.16290000081062317, 0.04520000144839287, 0.09139999747276306, 0.1193000003695488, 0.11640000343322754, 0.007199999876320362, 0.03400000184774399, 0.07320000231266022, 0.030899999663233757, 0.005400000140070915, 0.008500000461935997, 0.01979999989271164, 0.013899999670684338, 0.030700000002980232, 0.02160000056028366, 0.10180000215768814, 0.07959999889135361, 0.03449999913573265, 0.04540000110864639, 0.0957999974489212, 0.0869000032544136, 0.009600000455975533, 0.01889999955892563, 0.026399999856948853, 0.03610000014305115, 0.014700000174343586, 0.011699999682605267, 0.00839999970048666, 0.03440000116825104, 0.032499998807907104, 0.03269999846816063, 0.054999999701976776, 0.06849999725818634, 0.04899999871850014, 0.03020000085234642, 0.04089999943971634, 0.13590000569820404, 0.01590000092983246, 0.016200000420212746, 0.012799999676644802, 0.05389999970793724, 0.015399999916553497, 0.013000000268220901, 0.015200000256299973, 0.06840000301599503, 0.03099999949336052, 0.02889999933540821, 0.049300000071525574, 0.1177000030875206, 0.046799998730421066, 0.0560000017285347, 0.0674000009894371, 0.20250000059604645, 0.026900000870227814, 0.015799999237060547, 0.018400000408291817, 0.06769999861717224, 0.013299999758601189, 0.009700000286102295, 0.009600000455975533, 0.03319999948143959, 0.023499999195337296, 0.024700000882148743, 0.06599999964237213, 0.04699999839067459, 0.022299999371170998, 0.06840000301599503, 0.08169999718666077, 0.12200000137090683, 0.010700000450015068, 0.0142000000923872, 0.01860000006854534, 0.041999999433755875, 0.009200000204145908, 0.006599999964237213, 0.01549999974668026, 0.01730000041425228, 0.02410000003874302, 0.02810000069439411, 0.10459999740123749, 0.07100000232458115, 0.020400000736117363, 0.09690000116825104, 0.15240000188350677, 0.09470000118017197, 0.009499999694526196, 0.013299999758601189, 0.039400000125169754, 0.03909999877214432, 0.007000000216066837, 0.012900000438094139, 0.01360000018030405, 0.00989999994635582, 0.05000000074505806, 0.10570000112056732, 0.09459999948740005, 0.08479999750852585, 0.0430000014603138, 0.1177000030875206, 0.09070000052452087, 0.08169999718666077, 0.008799999952316284, 0.018200000748038292, 0.06419999897480011, 0.04859999939799309, 0.011699999682605267, 0.022700000554323196, 0.05469999834895134, 0.03720000013709068, 0.06459999829530716, 0.093299999833107, 0.15809999406337738, 0.15289999544620514, 0.0494999997317791, 0.05950000137090683, 0.12929999828338623, 0.11240000277757645, 0.011300000362098217, 0.03680000081658363, 0.13169999420642853, 0.07530000060796738, 0.026799999177455902, 0.013000000268220901, 0.021800000220537186, 0.03460000082850456, 0.027499999850988388, 0.04529999941587448, 0.06939999759197235, 0.12280000001192093, 0.05299999937415123, 0.08389999717473984, 0.11110000312328339, 0.06790000200271606, 0.021400000900030136, 0.05640000104904175, 0.07660000026226044, 0.023800000548362732, 0.03959999978542328, 0.01640000008046627, 0.02319999970495701, 0.055799998342990875, 0.03889999911189079, 0.0357000008225441, 0.08150000125169754, 0.10930000245571136, 0.04780000075697899, 0.06700000166893005, 0.10379999876022339, 0.14419999718666077, 0.03739999979734421, 0.021299999207258224, 0.019700000062584877, 0.060100000351667404, 0.0142000000923872, 0.01730000041425228, 0.03280000016093254, 0.02070000022649765, 0.029999999329447746, 0.05119999870657921, 0.11420000344514847, 0.05050000175833702, 0.03370000049471855, 0.07190000265836716, 0.1242000013589859, 0.10620000213384628, 0.023399999365210533, 0.012400000356137753, 0.06030000001192093, 0.08659999817609787)) option max_matches=1000000;
+--------+------------+
| id     | knn_dist() |
+--------+------------+
| 568969 | 1.13936853 |
| 568975 | 1.62225807 |
| 568976 | 2.13362741 |
| 568970 | 2.76673746 |
| 568984 | 2.87248778 |
| 568978 | 3.29825449 |
| 568972 | 3.30782843 |
| 568985 | 3.39168644 |
| 568971 | 3.67472219 |
| 568983 | 3.88973951 |
| 568980 | 4.22463274 |
| 568981 | 4.55146933 |
| 568974 |   4.772573 |
| 568986 |   5.721325 |
| 568973 |   6.276696 |
| 568987 | 6.43626547 |
| 568968 |   7.522899 |
| 568982 | 7.60857964 |
| 568979 |   7.693015 |
| 568977 |   8.359688 |
+--------+------------+
20 rows in set (4.16 sec)

select id, knn_dist() from bench where knn (vector, 1000000, (0.011699999682605267, 0.011500000022351742, 0.008700000122189522, 0.009999999776482582, 0.07850000262260437, 0.10000000149011612, 0.07840000092983246, 0.05299999937415123, 0.052400000393390656, 0.08190000057220459, 0.0658000037074089, 0.057999998331069946, 0.01590000092983246, 0.017000000923871994, 0.04610000178217888, 0.02419999986886978, 0.00839999970048666, 0.006399999838322401, 0.007199999876320362, 0.010200000368058681, 0.030400000512599945, 0.06790000200271606, 0.05889999866485596, 0.057100001722574234, 0.0333000011742115, 0.07859999686479568, 0.08919999748468399, 0.04230000078678131, 0.013799999840557575, 0.013299999758601189, 0.028999999165534973, 0.021900000050663948, 0.008999999612569809, 0.012199999764561653, 0.010700000450015068, 0.01080000028014183, 0.026599999517202377, 0.03849999979138374, 0.057100001722574234, 0.052000001072883606, 0.03550000116229057, 0.04879999905824661, 0.06920000165700912, 0.05310000106692314, 0.014399999752640724, 0.014299999922513962, 0.01489999983459711, 0.02500000037252903, 0.017100000753998756, 0.016100000590085983, 0.010599999688565731, 0.03240000084042549, 0.02710000053048134, 0.04580000042915344, 0.05310000106692314, 0.06239999830722809, 0.031599998474121094, 0.06080000102519989, 0.06610000133514404, 0.08129999786615372, 0.01860000006854534, 0.022700000554323196, 0.011599999852478504, 0.03370000049471855, 0.024700000882148743, 0.011800000444054604, 0.010700000450015068, 0.06390000134706497, 0.039500001817941666, 0.040300000458955765, 0.05249999836087227, 0.0957999974489212, 0.05510000139474869, 0.06759999692440033, 0.08579999953508377, 0.17489999532699585, 0.024399999529123306, 0.02810000069439411, 0.008700000122189522, 0.05119999870657921, 0.01489999983459711, 0.00860000029206276, 0.012400000356137753, 0.035599999129772186, 0.03280000016093254, 0.03869999945163727, 0.046300001442432404, 0.048900000751018524, 0.04919999837875366, 0.0640999972820282, 0.07050000131130219, 0.11640000343322754, 0.020600000396370888, 0.016200000420212746, 0.012500000186264515, 0.04019999876618385, 0.013500000350177288, 0.011300000362098217, 0.007400000002235174, 0.011800000444054604, 0.032999999821186066, 0.02979999966919422, 0.0364999994635582, 0.041999999433755875, 0.04410000145435333, 0.051899999380111694, 0.06589999794960022, 0.05270000174641609, 0.013899999670684338, 0.016200000420212746, 0.01510000042617321, 0.02239999920129776, 0.011800000444054604, 0.008799999952316284, 0.007799999788403511, 0.010900000110268593, 0.037300001829862595, 0.07050000131130219, 0.06279999762773514, 0.03909999877214432, 0.02419999986886978, 0.055799998342990875, 0.07209999859333038, 0.04479999840259552, 0.01679999940097332, 0.013399999588727951, 0.03310000151395798, 0.020800000056624413, 0.004100000020116568, 0.008999999612569809, 0.016499999910593033, 0.011699999682605267, 0.09619999676942825, 0.13590000569820404, 0.12150000035762787, 0.08799999952316284, 0.06390000134706497, 0.11739999800920486, 0.09700000286102295, 0.07029999792575836, 0.012600000016391277, 0.021900000050663948, 0.07460000365972519, 0.0421999990940094, 0.003700000001117587, 0.00839999970048666, 0.009999999776482582, 0.013199999928474426, 0.03200000151991844, 0.06129999831318855, 0.07509999722242355, 0.09139999747276306, 0.042899999767541885, 0.0640999972820282, 0.0746999979019165, 0.04659999907016754, 0.008700000122189522, 0.028599999845027924, 0.051500000059604645, 0.027899999171495438, 0.007300000172108412, 0.01489999983459711, 0.02250000089406967, 0.018200000748038292, 0.01810000091791153, 0.01759999990463257, 0.08619999885559082, 0.06400000303983688, 0.027499999850988388, 0.028999999165534973, 0.055799998342990875, 0.060499999672174454, 0.01140000019222498, 0.014499999582767487, 0.02070000022649765, 0.02850000001490116, 0.020800000056624413, 0.019099999219179153, 0.012000000104308128, 0.04500000178813934, 0.023399999365210533, 0.023099999874830246, 0.053300000727176666, 0.08179999887943268, 0.03970000147819519, 0.024299999698996544, 0.03889999911189079, 0.10819999873638153, 0.022099999710917473, 0.030300000682473183, 0.01600000075995922, 0.05339999869465828, 0.02710000053048134, 0.01889999955892563, 0.01640000008046627, 0.08810000121593475, 0.043699998408555984, 0.017000000923871994, 0.028200000524520874, 0.13899999856948853, 0.060100000351667404, 0.03909999877214432, 0.03709999844431877, 0.20309999585151672, 0.03620000183582306, 0.0340999998152256, 0.011900000274181366, 0.07450000196695328, 0.016699999570846558, 0.0052999998442828655, 0.010400000028312206, 0.04560000076889992, 0.022199999541044235, 0.020099999383091927, 0.015599999576807022, 0.04580000042915344, 0.04309999942779541, 0.04019999876618385, 0.04439999908208847, 0.13459999859333038, 0.020600000396370888, 0.019200000911951065, 0.014999999664723873, 0.046799998730421066, 0.010300000198185444, 0.006599999964237213, 0.012000000104308128, 0.017899999395012856, 0.01850000023841858, 0.019500000402331352, 0.04399999976158142, 0.03290000185370445, 0.02979999966919422, 0.0706000030040741, 0.09549999982118607, 0.05810000002384186, 0.009700000286102295, 0.015699999406933784, 0.03189999982714653, 0.0406000018119812, 0.007400000002235174, 0.009800000116229057, 0.010300000198185444, 0.011599999852478504, 0.040699999779462814, 0.08299999684095383, 0.06750000268220901, 0.035599999129772186, 0.03519999980926514, 0.08489999920129776, 0.07400000095367432, 0.057500001043081284, 0.00800000037997961, 0.010499999858438969, 0.050200000405311584, 0.03790000081062317, 0.00989999994635582, 0.021299999207258224, 0.0364999994635582, 0.024700000882148743, 0.04809999838471413, 0.07810000330209732, 0.094200000166893, 0.08399999886751175, 0.04170000180602074, 0.05249999836087227, 0.07289999723434448, 0.06639999896287918, 0.010099999606609344, 0.023099999874830246, 0.0868000015616417, 0.06459999829530716, 0.033799998462200165, 0.02160000056028366, 0.01899999938905239, 0.03700000047683716, 0.027400000020861626, 0.044599998742341995, 0.04349999874830246, 0.10090000182390213, 0.05249999836087227, 0.06769999861717224, 0.057500001043081284, 0.04639999940991402, 0.022700000554323196, 0.04170000180602074, 0.04749999940395355, 0.024000000208616257, 0.06080000102519989, 0.033399999141693115, 0.03680000081658363, 0.0763000026345253, 0.053300000727176666, 0.021900000050663948, 0.06080000102519989, 0.11249999701976776, 0.0674000009894371, 0.021800000220537186, 0.05270000174641609, 0.13120000064373016, 0.05889999866485596, 0.03310000151395798, 0.02160000056028366, 0.06530000269412994, 0.022099999710917473, 0.017400000244379044, 0.030700000002980232, 0.030300000682473183, 0.027300000190734863, 0.029400000348687172, 0.06279999762773514, 0.033799998462200165, 0.04010000079870224, 0.043699998408555984, 0.05900000035762787, 0.07609999924898148, 0.028599999845027924, 0.01889999955892563, 0.04179999977350235, 0.06960000097751617, 0.00800000037997961, 0.007899999618530273, 0.009499999694526196, 0.011099999770522118, 0.08179999887943268, 0.10530000180006027, 0.08699999749660492, 0.05310000106692314, 0.05180000141263008, 0.08860000222921371, 0.07509999722242355, 0.06889999657869339, 0.015799999237060547, 0.018200000748038292, 0.051600001752376556, 0.028599999845027924, 0.006200000178068876, 0.004800000227987766, 0.006800000090152025, 0.009700000286102295, 0.03370000049471855, 0.06939999759197235, 0.07209999859333038, 0.061799999326467514, 0.0364999994635582, 0.08240000158548355, 0.11050000041723251, 0.04879999905824661, 0.013899999670684338, 0.01269999984651804, 0.03139999881386757, 0.020899999886751175, 0.010300000198185444, 0.008500000461935997, 0.009999999776482582, 0.00860000029206276, 0.029200000688433647, 0.03709999844431877, 0.06930000334978104, 0.05209999904036522, 0.031199999153614044, 0.04650000110268593, 0.0835999995470047, 0.04859999939799309, 0.014100000262260437, 0.011900000274181366, 0.014600000344216824, 0.02280000038444996, 0.01600000075995922, 0.012299999594688416, 0.010099999606609344, 0.030300000682473183, 0.027400000020861626, 0.04969999939203262, 0.0706000030040741, 0.0674000009894371, 0.029999999329447746, 0.05730000138282776, 0.05469999834895134, 0.08820000290870667, 0.015699999406933784, 0.020099999383091927, 0.011900000274181366, 0.03530000150203705, 0.02250000089406967, 0.01119999960064888, 0.009700000286102295, 0.05999999865889549, 0.037300001829862595, 0.04410000145435333, 0.05939999967813492, 0.08869999647140503, 0.049400001764297485, 0.06599999964237213, 0.08269999921321869, 0.18400000035762787, 0.023000000044703484, 0.02410000003874302, 0.009800000116229057, 0.05009999871253967, 0.013899999670684338, 0.006399999838322401, 0.01119999960064888, 0.033799998462200165, 0.03150000050663948, 0.03929999843239784, 0.04740000143647194, 0.04089999943971634, 0.04259999841451645, 0.06599999964237213, 0.07320000231266022, 0.125, 0.02239999920129776, 0.01549999974668026, 0.01209999993443489, 0.037300001829862595, 0.01119999960064888, 0.006300000008195639, 0.007499999832361937, 0.01119999960064888, 0.03200000151991844, 0.028300000354647636, 0.04800000041723251, 0.041200000792741776, 0.04190000146627426, 0.05829999968409538, 0.07840000092983246, 0.051600001752376556, 0.013500000350177288, 0.0142000000923872, 0.016899999231100082, 0.019500000402331352, 0.009700000286102295, 0.00570000009611249, 0.005900000222027302, 0.00800000037997961, 0.03819999843835831, 0.07479999959468842, 0.061900001019239426, 0.05050000175833702, 0.02710000053048134, 0.0608999989926815, 0.08449999988079071, 0.05510000139474869, 0.015799999237060547, 0.012000000104308128, 0.03689999878406525, 0.021299999207258224, 0.006200000178068876, 0.014399999752640724, 0.02280000038444996, 0.011300000362098217, 0.10400000214576721, 0.148499995470047, 0.14489999413490295, 0.1062999963760376, 0.06589999794960022, 0.13019999861717224, 0.11670000106096268, 0.09629999846220016, 0.015200000256299973, 0.025599999353289604, 0.08219999819993973, 0.04540000110864639, 0.004699999932199717, 0.00559999980032444, 0.010999999940395355, 0.010099999606609344, 0.03500000014901161, 0.06650000065565109, 0.09539999812841415, 0.11710000038146973, 0.045899998396635056, 0.07039999961853027, 0.09549999982118607, 0.07150000333786011, 0.009700000286102295, 0.028999999165534973, 0.05739999935030937, 0.025100000202655792, 0.008500000461935997, 0.010200000368058681, 0.019999999552965164, 0.014800000004470348, 0.024800000712275505, 0.01850000023841858, 0.09279999881982803, 0.07010000199079514, 0.029999999329447746, 0.033799998462200165, 0.07490000128746033, 0.07490000128746033, 0.01489999983459711, 0.017400000244379044, 0.020999999716877937, 0.03319999948143959, 0.018200000748038292, 0.012199999764561653, 0.008299999870359898, 0.04190000146627426, 0.025200000032782555, 0.024000000208616257, 0.05310000106692314, 0.07620000094175339, 0.048900000751018524, 0.024700000882148743, 0.032999999821186066, 0.1257999986410141, 0.017500000074505806, 0.025599999353289604, 0.013100000098347664, 0.052400000393390656, 0.026100000366568565, 0.01489999983459711, 0.015699999406933784, 0.08609999716281891, 0.04340000078082085, 0.02419999986886978, 0.03620000183582306, 0.1395999938249588, 0.05649999901652336, 0.04270000010728836, 0.04309999942779541, 0.21359999477863312, 0.03449999913573265, 0.025100000202655792, 0.014299999922513962, 0.07569999992847443, 0.014600000344216824, 0.006000000052154064, 0.00930000003427267, 0.041099999099969864, 0.025100000202655792, 0.018699999898672104, 0.03099999949336052, 0.04699999839067459, 0.035100001841783524, 0.05299999937415123, 0.054099999368190765, 0.13439999520778656, 0.016699999570846558, 0.01759999990463257, 0.01489999983459711, 0.04390000179409981, 0.009200000204145908, 0.005499999970197678, 0.01360000018030405, 0.018400000408291817, 0.01860000006854534, 0.02329999953508377, 0.06780000030994415, 0.04050000011920929, 0.024800000712275505, 0.07890000194311142, 0.11249999701976776, 0.0681999996304512, 0.008999999612569809, 0.014399999752640724, 0.033799998462200165, 0.03889999911189079, 0.006599999964237213, 0.012900000438094139, 0.011599999852478504, 0.009100000374019146, 0.041600000113248825, 0.09130000323057175, 0.08060000091791153, 0.04610000178217888, 0.037300001829862595, 0.09560000151395798, 0.07819999754428864, 0.06019999831914902, 0.009200000204145908, 0.013199999928474426, 0.0551999993622303, 0.04190000146627426, 0.009499999694526196, 0.018799999728798866, 0.040300000458955765, 0.028200000524520874, 0.052799999713897705, 0.08269999921321869, 0.11349999904632568, 0.10689999908208847, 0.04320000112056732, 0.05559999868273735, 0.08980000019073486, 0.08179999887943268, 0.011900000274181366, 0.028999999165534973, 0.10509999841451645, 0.07090000063180923, 0.03280000016093254, 0.015799999237060547, 0.019099999219179153, 0.03629999980330467, 0.02800000086426735, 0.045099999755620956, 0.048700001090765, 0.10890000313520432, 0.055799998342990875, 0.07259999960660934, 0.07270000129938126, 0.05249999836087227, 0.02070000022649765, 0.04600000008940697, 0.058400001376867294, 0.025499999523162842, 0.0551999993622303, 0.02630000002682209, 0.03319999948143959, 0.07329999655485153, 0.04749999940395355, 0.025200000032782555, 0.06620000302791595, 0.1120000034570694, 0.062199998646974564, 0.033900000154972076, 0.06599999964237213, 0.1362999975681305, 0.05119999870657921, 0.02759999968111515, 0.024900000542402267, 0.06830000132322311, 0.019899999722838402, 0.01590000092983246, 0.02930000051856041, 0.028599999845027924, 0.02810000069439411, 0.03519999980926514, 0.07850000262260437, 0.03709999844431877, 0.039000000804662704, 0.05090000107884407, 0.07649999856948853, 0.08209999650716782, 0.02500000037252903, 0.015200000256299973, 0.0471000000834465, 0.07530000060796738, 0.007699999958276749, 0.009200000204145908, 0.008500000461935997, 0.008700000122189522, 0.08950000256299973, 0.11569999903440475, 0.11339999735355377, 0.07410000264644623, 0.05290000140666962, 0.1014999970793724, 0.09799999743700027, 0.10660000145435333, 0.013299999758601189, 0.017400000244379044, 0.05990000069141388, 0.028699999675154686, 0.0066999997943639755, 0.008100000210106373, 0.008200000040233135, 0.007699999958276749, 0.03319999948143959, 0.07590000331401825, 0.08829999715089798, 0.08799999952316284, 0.040800001472234726, 0.10760000348091125, 0.1386999934911728, 0.07429999858140945, 0.01080000028014183, 0.012900000438094139, 0.03830000013113022, 0.021400000900030136, 0.006300000008195639, 0.007499999832361937, 0.011599999852478504, 0.007400000002235174, 0.030300000682473183, 0.039000000804662704, 0.08299999684095383, 0.05609999969601631, 0.031099999323487282, 0.06440000236034393, 0.11240000277757645, 0.050200000405311584, 0.011699999682605267, 0.010499999858438969, 0.016599999740719795, 0.022299999371170998, 0.013399999588727951, 0.01140000019222498, 0.012400000356137753, 0.027400000020861626, 0.02590000070631504, 0.05290000140666962, 0.08720000088214874, 0.0575999990105629, 0.02979999966919422, 0.06759999692440033, 0.09369999915361404, 0.08139999955892563, 0.013299999758601189, 0.017799999564886093, 0.01119999960064888, 0.0340999998152256, 0.020800000056624413, 0.01209999993443489, 0.012400000356137753, 0.052799999713897705, 0.029500000178813934, 0.044599998742341995, 0.09889999777078629, 0.066600002348423, 0.043699998408555984, 0.07599999755620956, 0.10930000245571136, 0.1460999995470047, 0.02319999970495701, 0.019300000742077827, 0.01360000018030405, 0.04839999973773956, 0.012000000104308128, 0.007499999832361937, 0.012199999764561653, 0.030899999663233757, 0.029999999329447746, 0.04399999976158142, 0.07959999889135361, 0.040800001472234726, 0.034299999475479126, 0.06960000097751617, 0.09059999883174896, 0.1151999980211258, 0.018300000578165054, 0.01080000028014183, 0.01769999973475933, 0.0364999994635582, 0.010200000368058681, 0.00570000009611249, 0.007899999618530273, 0.010099999606609344, 0.03220000118017197, 0.030899999663233757, 0.07159999758005142, 0.052299998700618744, 0.0340999998152256, 0.06459999829530716, 0.1177000030875206, 0.061400000005960464, 0.011699999682605267, 0.013199999928474426, 0.01899999938905239, 0.0203000009059906, 0.009800000116229057, 0.006599999964237213, 0.0066999997943639755, 0.007600000128149986, 0.04129999876022339, 0.08470000326633453, 0.09740000218153, 0.07699999958276749, 0.02969999983906746, 0.07050000131130219, 0.11559999734163284, 0.08709999918937683, 0.01209999993443489, 0.012299999594688416, 0.04129999876022339, 0.02329999953508377, 0.006099999882280827, 0.012199999764561653, 0.02280000038444996, 0.010999999940395355, 0.12189999967813492, 0.17100000381469727, 0.16410000622272491, 0.14069999754428864, 0.07479999959468842, 0.15860000252723694, 0.15629999339580536, 0.15080000460147858, 0.017100000753998756, 0.034299999475479126, 0.10440000146627426, 0.05310000106692314, 0.005200000014156103, 0.006200000178068876, 0.01209999993443489, 0.011500000022351742, 0.03929999843239784, 0.0697999969124794, 0.10480000078678131, 0.16290000081062317, 0.04520000144839287, 0.09139999747276306, 0.1193000003695488, 0.11640000343322754, 0.007199999876320362, 0.03400000184774399, 0.07320000231266022, 0.030899999663233757, 0.005400000140070915, 0.008500000461935997, 0.01979999989271164, 0.013899999670684338, 0.030700000002980232, 0.02160000056028366, 0.10180000215768814, 0.07959999889135361, 0.03449999913573265, 0.04540000110864639, 0.0957999974489212, 0.0869000032544136, 0.009600000455975533, 0.01889999955892563, 0.026399999856948853, 0.03610000014305115, 0.014700000174343586, 0.011699999682605267, 0.00839999970048666, 0.03440000116825104, 0.032499998807907104, 0.03269999846816063, 0.054999999701976776, 0.06849999725818634, 0.04899999871850014, 0.03020000085234642, 0.04089999943971634, 0.13590000569820404, 0.01590000092983246, 0.016200000420212746, 0.012799999676644802, 0.05389999970793724, 0.015399999916553497, 0.013000000268220901, 0.015200000256299973, 0.06840000301599503, 0.03099999949336052, 0.02889999933540821, 0.049300000071525574, 0.1177000030875206, 0.046799998730421066, 0.0560000017285347, 0.0674000009894371, 0.20250000059604645, 0.026900000870227814, 0.015799999237060547, 0.018400000408291817, 0.06769999861717224, 0.013299999758601189, 0.009700000286102295, 0.009600000455975533, 0.03319999948143959, 0.023499999195337296, 0.024700000882148743, 0.06599999964237213, 0.04699999839067459, 0.022299999371170998, 0.06840000301599503, 0.08169999718666077, 0.12200000137090683, 0.010700000450015068, 0.0142000000923872, 0.01860000006854534, 0.041999999433755875, 0.009200000204145908, 0.006599999964237213, 0.01549999974668026, 0.01730000041425228, 0.02410000003874302, 0.02810000069439411, 0.10459999740123749, 0.07100000232458115, 0.020400000736117363, 0.09690000116825104, 0.15240000188350677, 0.09470000118017197, 0.009499999694526196, 0.013299999758601189, 0.039400000125169754, 0.03909999877214432, 0.007000000216066837, 0.012900000438094139, 0.01360000018030405, 0.00989999994635582, 0.05000000074505806, 0.10570000112056732, 0.09459999948740005, 0.08479999750852585, 0.0430000014603138, 0.1177000030875206, 0.09070000052452087, 0.08169999718666077, 0.008799999952316284, 0.018200000748038292, 0.06419999897480011, 0.04859999939799309, 0.011699999682605267, 0.022700000554323196, 0.05469999834895134, 0.03720000013709068, 0.06459999829530716, 0.093299999833107, 0.15809999406337738, 0.15289999544620514, 0.0494999997317791, 0.05950000137090683, 0.12929999828338623, 0.11240000277757645, 0.011300000362098217, 0.03680000081658363, 0.13169999420642853, 0.07530000060796738, 0.026799999177455902, 0.013000000268220901, 0.021800000220537186, 0.03460000082850456, 0.027499999850988388, 0.04529999941587448, 0.06939999759197235, 0.12280000001192093, 0.05299999937415123, 0.08389999717473984, 0.11110000312328339, 0.06790000200271606, 0.021400000900030136, 0.05640000104904175, 0.07660000026226044, 0.023800000548362732, 0.03959999978542328, 0.01640000008046627, 0.02319999970495701, 0.055799998342990875, 0.03889999911189079, 0.0357000008225441, 0.08150000125169754, 0.10930000245571136, 0.04780000075697899, 0.06700000166893005, 0.10379999876022339, 0.14419999718666077, 0.03739999979734421, 0.021299999207258224, 0.019700000062584877, 0.060100000351667404, 0.0142000000923872, 0.01730000041425228, 0.03280000016093254, 0.02070000022649765, 0.029999999329447746, 0.05119999870657921, 0.11420000344514847, 0.05050000175833702, 0.03370000049471855, 0.07190000265836716, 0.1242000013589859, 0.10620000213384628, 0.023399999365210533, 0.012400000356137753, 0.06030000001192093, 0.08659999817609787));
+--------+------------+
| id     | knn_dist() |
+--------+------------+
| 568969 | 1.13936853 |
| 568975 | 1.62225807 |
| 568976 | 2.13362741 |
| 568970 | 2.76673746 |
| 568984 | 2.87248778 |
| 568978 | 3.29825449 |
| 568972 | 3.30782843 |
| 568985 | 3.39168644 |
| 568971 | 3.67472219 |
| 568983 | 3.88973951 |
| 568980 | 4.22463274 |
| 568981 | 4.55146933 |
| 568974 |   4.772573 |
| 568986 |   5.721325 |
| 568973 |   6.276696 |
| 568987 | 6.43626547 |
| 568968 |   7.522899 |
| 568982 | 7.60857964 |
| 568979 |   7.693015 |
| 568977 |   8.359688 |
+--------+------------+
20 rows in set (4.16 sec)

Also CREATE TABLE looks like this:
CREATE TABLE `bench` (                                                                                `id` bigint,                                                                                `vector` float_vector knn_type='hnsw' knn_dims='960' hnsw_similarity='L2' hnsw_m='32' hnsw_ef_construction='128'                                                                                );

It uses L2 distance functions, so cosine similarity calculator doesn't seem relevant.

I identified and fixed the issue in 694cbe5

@glookka how severe was the issue? In what circumstances could it affect the search results relevancy?

@glookka how severe was the issue? In what circumstances could it affect the search results relevancy?

It affected any search with k>limit. So it was quite severe.


Night_mode and dialog_mode not supported on Playbase

I just tried changing the night_mode and dialog_mode on my Playbase, but get an error for both as not supported. I can change these modes from the Sonos app, so looks like it is a SoCo limitation.
>>> my_zone
SoCo("<IP_ADDRESS>")
>>> my_zone.get_speaker_info()
{u'display_version': '9.1', u'player_icon': '/img/icon-S11.png', u'uid': 'RINCON_XYZ', u'software_version': '45.1-56150', u'mac_address': u'MA-CA-DD-RE-S-S', u'hardware_version': '<IP_ADDRESS>-1', u'model_number': 'S11', u'serial_number': 'MA-CA-DD-RE-S-S:1', u'zone_name': 'Family Room', u'model_name': 'Sonos Playbase'}
>>> my_zone.night_mode = True
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/soco/core.py", line 813, in night_mode
    raise NotSupportedException(message)
soco.exceptions.NotSupportedException: This device does not support night mode
>>> my_zone.dialog_mode = True
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/soco/core.py", line 851, in dialog_mode
    raise NotSupportedException(message)
soco.exceptions.NotSupportedException: This device does not support dialog mode

Let me know if you need any other details

Looks like these lines in the core.py are causing the issues:
if 'PLAYBAR' not in self.speaker_info['model_name']:
Firstly, Playbase or Beam are not listed in core.py. Second, the case for speaker_info has changed - instead of all caps that is currently being used, it is Playbase. I suspect the night_mode and speech_enhance are currently broken for everyone (not just Playbase users). Changing those lines to
if self.speaker_info['model_name'] not in {'Sonos Playbase', 'Sonos Playbar', 'Sonos Beam'}:
worked in my case. If this approach is acceptable, I can create a PR.


Better visibility of custom model option in documentation

I slightly adjusted the documentation for better visibility of the option to add custom light models.

nice addition.


Update extension

Good morning from Spain.
We are Leasba Consulting S.L., Microsoft partner. We want to update an extensión to a higher version but the following error appears: "The application object of type 'Table' with the ID '50102' is defined in multiple apps. The apps are: ExtensionName by Leasba Consulting, SL <IP_ADDRESS>; ExtensionName by Leasba Consulting, SL <IP_ADDRESS>."
How could we fix it? We can't unistall the extension because it has a Table with data.
Best, Alejandro

You need to uninstall and unpublish - and then publish and run data upgrade for the app.
described here:
https://freddysblog.com/2019/12/09/having-demo-data-while-developing-business-central-apps-with-docker/
and here:
https://freddysblog.com/2019/11/24/upgrading-to-15-x-from-14-x-c-al-our-nav-techdays-2019-demo/

The uninstall and unpublish will not delete the data. The data will be preserved. There is a special switch that deletes the extension's data.


Compiling 3.7.0 won't work until base-includes.h is replaced with extension.h

Hi @chregu
I've tried compiling HHVM 3.7 against 0.16.1 (see my Docker build script here https://github.com/ckortekaas/hhvm-nginx-newrelic-docker/blob/master/Dockerfile) and it fails with:
 [ 50%] Building CXX object CMakeFiles/newrelic.dir/newrelic.cpp.o
 /usr/local/src/hhvm-newrelic-ext/newrelic.cpp:1:45: fatal error: hphp/runtime/base/base-includes.h: No   such file or directory

After a bit of searching I found that in 3.7 they removed base-includes.h from HHVM: https://github.com/facebook/hhvm/commit/91e07b951496b5bfbbb189fd5401e9dd5e711c43
Apparently they want you to use #include "hphp/runtime/ext/extension.h" instead.
Cheers,
Christiaan Kortekaas

To follow up, after making that change it fails next at:
[ 50%] Building CXX object CMakeFiles/newrelic.dir/newrelic.cpp.o
In file included from /usr/local/include/hphp/runtime/ext/extension.h:23:0,
                 from /usr/local/src/hhvm-newrelic-ext/newrelic.cpp:1:
/usr/local/include/hphp/runtime/vm/native.h: In instantiation of 'void HPHP::Native::registerBuiltinFunction(const char*, Fun) [with Fun = long int (*)(const HPHP::String&, const HPHP::String&, const HPHP::String&, const HPHP::String&, HPHP::String&)]':
/usr/local/src/hhvm-newrelic-ext/newrelic.cpp:251:9:   required from here
/usr/local/include/hphp/runtime/vm/native.h:368:3: error: static assertion failed: Arguments on builtin function were not understood types
   static_assert(
   ^

and
/usr/local/include/hphp/runtime/vm/native.h: In instantiation of 'void HPHP::Native::registerBuiltinFunction(const char*, Fun) [with Fun = HPHP::Variant (*)(const HPHP::String&, const HPHP::String&, HPHP::String, HPHP::String)]':
/usr/local/src/hhvm-newrelic-ext/newrelic.cpp:255:9:   required from here
/usr/local/include/hphp/runtime/vm/native.h:368:3: error: static assertion failed: Arguments on builtin function were not understood types

Any ideas what to do next to fix this?

hhvm-newrelic-ext git:(master) ✗ make
-- Configuring for HHVM API version 20150212
-- Configuring done
-- Generating done
-- Build files have been written to: /root/newrelic/hhvm-newrelic-ext
[ 50%] Building CXX object CMakeFiles/newrelic.dir/newrelic.cpp.o
/root/newrelic/hhvm-newrelic-ext/newrelic.cpp:1:45: fatal error: hphp/runtime/base/base-includes.h: No such file or directory
#include "hphp/runtime/base/base-includes.h"
^
compilation terminated.
make[2]: *** [CMakeFiles/newrelic.dir/newrelic.cpp.o] Error 1
make[1]: *** [CMakeFiles/newrelic.dir/all] Error 2
make: *** [all] Error 2

Please see this pull request #28

Thanks @fabiorphp - I've got a working docker container for new relic and hhvm 3.7 using your help here: https://github.com/ckortekaas/hhvm-nginx-newrelic-docker and https://registry.hub.docker.com/u/ckortekaas/hhvm-nginx-newrelic/

It's merged and I updated the README to mention 3.7.0
Thanks for all the help


Cleaning script in clean_data.py

Add cleaning scripts with

minimum mUSE similarity score
range for word ratio

done by @LalitaDeelert


Fix for Matrix -> ST -> Matrix cloning and import/export

Same problem here. It's not possible to Clone Fields they have matrix > supertable in it.

At this time, its not supported, but will be in the future. I'll keep you posted.
However, a Super Table > Matrix or Matrix > Super Table will work with cloning or import/export.

This is now fixed in version 1.5.0.


Dracula Pro not supported in the latest version of PyCharm Community

When trying to install via zip I get the following error:
Plugin 'Dracula Pro' (version '1.2.0') is not compatible with the current version of the IDE, because it requires build 202.* or older but the current build is PC-203.7717.65

My Pycharm Community build is :
PyCharm 2020.3.4 (Community Edition)
Build #PC-203.7717.65, built on March 16, 2021
Runtime version: 11.0.10+8-b1145.96 aarch64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
macOS 11.2.3
GC: ParNew, ConcurrentMarkSweep
Memory: 1981M
Cores: 8
Non-Bundled Plugins: com.markskelton.one-dark-theme, me.lensvol.blackconnect

Probably should go to the pycharm repo? @zenorocha

@mrpbennett please make sure you're using the latest version of the theme. I can confirm 1.3.0 is working just fine with my PyCharm 2020.3.4

Ok great thanks. How can I get the upgraded version without paying the $79.99 again ?

Once you login to the Gumroad account you should be able to fetch the latest version without having to pay for theme again.

Great thanks @lukaszklis after many attempts and trying to remember my email...I have the latest version and all installed ok thanks man

You're welcome! Happy coding!

@benknoble @lukaszklis you guys are the best. Thank you so much ;D


androidtests failed windows 2022 and windows 2019 fails with image state undeployable.

Description
but it did not match.
vhd:     at "$installedPackages" | Should -Match "ndk;$ndkPackage", C:\image\Tests\Android.Tests.ps1:97
vhd:     at , C:\image\Tests\Android.Tests.ps1:97
vhd: Tests completed in 19.21s
vhd: Tests Passed: 34, Failed: 1, Skipped: 0 NotRun: 0
vhd:
vhd: Containers            : {[-] C:\image\Tests\Android.Tests.ps1}
vhd: Result                : Failed
vhd: FailedCount           : 1
vhd: FailedBlocksCount     : 0
vhd: FailedContainersCount : 0
vhd: PassedCount           : 34
vhd: SkippedCount          : 0
vhd: NotRunCount           : 0
vhd: TotalCount            : 35
vhd: Duration              : 00:00:19.2144482
vhd: Executed              : True
vhd: ExecutedAt            : 1/17/2023 9:58:33 PM
vhd: Version               : 5.4.0
vhd: PSVersion             : 5.1.20348.1366
vhd: PSBoundParameters     : {[Configuration, PesterConfiguration]}
vhd: Plugins               :
vhd: PluginConfiguration   :
vhd: PluginData            :
vhd: Configuration         : PesterConfiguration
vhd: DiscoveryDuration     : 00:00:04.1323996
vhd: UserDuration          : 00:00:11.7841906
vhd: FrameworkDuration     : 00:00:03.2978580
vhd: Failed                : {[-] NDK 25 is installed}
vhd: FailedBlocks          : {}
vhd: FailedContainers      : {}
vhd: Passed                : {[+] Sdkmanager from SDK tools is available, [+] Sdkmanager from Command-line tools is
vhd:                         available, [+] Platform version platforms;android-27 is installed, [+] Platform version
vhd:                         platforms;android-28 is installed...}
vhd: Skipped               : {}
vhd: NotRun                : {}
vhd: Tests                 : {[+] Sdkmanager from SDK tools is available, [+] Sdkmanager from Command-line tools is
vhd:                         available, [+] Platform version platforms;android-27 is installed, [+] Platform version
vhd:                         platforms;android-28 is installed...}
vhd: CodeCoverage          :
vhd:
==> vhd: Test run has failed
==> vhd: At C:\Program Files\WindowsPowerShell\Modules\ImageHelpers\TestsHelpers.ps1:73 char:9
==> vhd: +         throw "Test run has failed"
==> vhd: +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~
==> vhd:     + CategoryInfo          : OperationStopped: (Test run has failed:String) [], RuntimeException
==> vhd:     + FullyQualifiedErrorId : Test run has failed
==> vhd:
==> vhd: Script exited with non-zero exit status: 1. Allowed exit codes are: [0]
==> vhd: Step "StepProvision" failed
Platforms affected

[X] Azure DevOps
[ ] GitHub Actions - Standard Runners
[ ] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 18.04
[ ] Ubuntu 20.04
[ ] Ubuntu 22.04
[ ] macOS 11
[ ] macOS 12
[ ] Windows Server 2019
[X] Windows Server 2022

Image version and build link
but it did not match.
vhd:     at "$installedPackages" | Should -Match "ndk;$ndkPackage", C:\image\Tests\Android.Tests.ps1:97
vhd:     at , C:\image\Tests\Android.Tests.ps1:97
vhd: Tests completed in 19.21s
vhd: Tests Passed: 34, Failed: 1, Skipped: 0 NotRun: 0
vhd:
vhd: Containers            : {[-] C:\image\Tests\Android.Tests.ps1}
vhd: Result                : Failed
vhd: FailedCount           : 1
vhd: FailedBlocksCount     : 0
vhd: FailedContainersCount : 0
vhd: PassedCount           : 34
vhd: SkippedCount          : 0
vhd: NotRunCount           : 0
vhd: TotalCount            : 35
vhd: Duration              : 00:00:19.2144482
vhd: Executed              : True
vhd: ExecutedAt            : 1/17/2023 9:58:33 PM
vhd: Version               : 5.4.0
vhd: PSVersion             : 5.1.20348.1366
vhd: PSBoundParameters     : {[Configuration, PesterConfiguration]}
vhd: Plugins               :
vhd: PluginConfiguration   :
vhd: PluginData            :
vhd: Configuration         : PesterConfiguration
vhd: DiscoveryDuration     : 00:00:04.1323996
vhd: UserDuration          : 00:00:11.7841906
vhd: FrameworkDuration     : 00:00:03.2978580
vhd: Failed                : {[-] NDK 25 is installed}
vhd: FailedBlocks          : {}
vhd: FailedContainers      : {}
vhd: Passed                : {[+] Sdkmanager from SDK tools is available, [+] Sdkmanager from Command-line tools is
vhd:                         available, [+] Platform version platforms;android-27 is installed, [+] Platform version
vhd:                         platforms;android-28 is installed...}
vhd: Skipped               : {}
vhd: NotRun                : {}
vhd: Tests                 : {[+] Sdkmanager from SDK tools is available, [+] Sdkmanager from Command-line tools is
vhd:                         available, [+] Platform version platforms;android-27 is installed, [+] Platform version
vhd:                         platforms;android-28 is installed...}
vhd: CodeCoverage          :
vhd:
==> vhd: Test run has failed
==> vhd: At C:\Program Files\WindowsPowerShell\Modules\ImageHelpers\TestsHelpers.ps1:73 char:9
==> vhd: +         throw "Test run has failed"
==> vhd: +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~
==> vhd:     + CategoryInfo          : OperationStopped: (Test run has failed:String) [], RuntimeException
==> vhd:     + FullyQualifiedErrorId : Test run has failed
==> vhd:
==> vhd: Script exited with non-zero exit status: 1. Allowed exit codes are: [0]
==> vhd: Step "StepProvision" failed
Is it regression?
no
Expected behavior
build to move forward, but fails with andorid tests.
retrying did not pass, fails with defender error on a retry.
Actual behavior
but it did not match.
vhd:     at "$installedPackages" | Should -Match "ndk;$ndkPackage", C:\image\Tests\Android.Tests.ps1:97
vhd:     at , C:\image\Tests\Android.Tests.ps1:97
vhd: Tests completed in 19.21s
vhd: Tests Passed: 34, Failed: 1, Skipped: 0 NotRun: 0
vhd:
vhd: Containers            : {[-] C:\image\Tests\Android.Tests.ps1}
vhd: Result                : Failed
vhd: FailedCount           : 1
vhd: FailedBlocksCount     : 0
vhd: FailedContainersCount : 0
vhd: PassedCount           : 34
vhd: SkippedCount          : 0
vhd: NotRunCount           : 0
vhd: TotalCount            : 35
vhd: Duration              : 00:00:19.2144482
vhd: Executed              : True
vhd: ExecutedAt            : 1/17/2023 9:58:33 PM
vhd: Version               : 5.4.0
vhd: PSVersion             : 5.1.20348.1366
vhd: PSBoundParameters     : {[Configuration, PesterConfiguration]}
vhd: Plugins               :
vhd: PluginConfiguration   :
vhd: PluginData            :
vhd: Configuration         : PesterConfiguration
vhd: DiscoveryDuration     : 00:00:04.1323996
vhd: UserDuration          : 00:00:11.7841906
vhd: FrameworkDuration     : 00:00:03.2978580
vhd: Failed                : {[-] NDK 25 is installed}
vhd: FailedBlocks          : {}
vhd: FailedContainers      : {}
vhd: Passed                : {[+] Sdkmanager from SDK tools is available, [+] Sdkmanager from Command-line tools is
vhd:                         available, [+] Platform version platforms;android-27 is installed, [+] Platform version
vhd:                         platforms;android-28 is installed...}
vhd: Skipped               : {}
vhd: NotRun                : {}
vhd: Tests                 : {[+] Sdkmanager from SDK tools is available, [+] Sdkmanager from Command-line tools is
vhd:                         available, [+] Platform version platforms;android-27 is installed, [+] Platform version
vhd:                         platforms;android-28 is installed...}
vhd: CodeCoverage          :
vhd:
==> vhd: Test run has failed
==> vhd: At C:\Program Files\WindowsPowerShell\Modules\ImageHelpers\TestsHelpers.ps1:73 char:9
==> vhd: +         throw "Test run has failed"
==> vhd: +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~
==> vhd:     + CategoryInfo          : OperationStopped: (Test run has failed:String) [], RuntimeException
==> vhd:     + FullyQualifiedErrorId : Test run has failed
==> vhd:
==> vhd: Script exited with non-zero exit status: 1. Allowed exit codes are: [0]
==> vhd: Step "StepProvision" failed
Repro steps
run the build as is and it will fail with the above.,

it failed for 2022, 2019 is still going on, will know in about six horus, if it does not show the undeployable message again. i will update this thread.

and i am now getting image state undeployable message with windows 2019.

@Alexey-Ayupov @mikhailkoliada @igorboskovic3
can you please look into these two issues and let me know what the next steps would be to resolve?
I have been trying for the past five weeks to get the builds going, but has been unsuccessful for windows, we are in a dead zone and cannot move forward, please help. do not convert this into a discussion please.

Hey @vamshisiram
Please, update body of the issue in "description" and "image version and build link parts" for better understanding what was done by you.
For right now I can say only that Android NDK 25 was installed normally last few times. Screenshot from our last image generation for example:

@vamshisiram but it is not an issue for our repo.
Image generation process is not currently experiencing problems with Android NDK 25 package both for Windows 2019 and Windows 2022. Just the same moment we are in process of shipping new Windows 2019 image as you can see here: https://github.com/actions/runner-images/pull/6932. And last one Windows 2022 image was generated about one hour before:

Please update your fork (as I see you have outdated version in your repo) and read again guide
I forced to close that issue because it is not reproducable and not connected with any bugs in source code. But you may continue ping us here if you have more questions about particular topic. And feel free to open issues in case you have another concerns.

@erik-bershel  @Alexey-Ayupov  @mikhailkoliada
I grabbed the latest commit by alexey and it still fails with image state undeployable.... no errors nothing.... please help.

Hey @vamshisiram!
Please provide full log from azure about an operation. It is advisable to remove personal data from the log. I'll take a look on it a bit later. And if you can then run please pipeline again just to be sure it's not a new transistent issue.


UI - Not able to present proper error messages when creation of assisstant fails.

gptscript version v0.0.0-dev+b77cd130
Steps to reproduce the problem:

Create an assistant with a script name without gpt extension.
Following error is presented to the user

This is the console error seen:
 ⨯ Error: Cannot find module '/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/.next/server/pages/_error.js'
Require stack:
- /Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/require.js
- /Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/next-server.js
- /Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/next.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1142:15)
    at /Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/require-hook.js:55:36
    at Module._load (node:internal/modules/cjs/loader:983:27)
    at Module.require (node:internal/modules/cjs/loader:1230:19)
    at mod.require (/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/require-hook.js:65:28)
    at require (node:internal/modules/helpers:179:18)
    at requirePage (/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/require.js:109:84)
    at /Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/load-components.js:98:84
    at async loadComponentsImpl (/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/load-components.js:98:26)
    at async NextNodeServer.findPageComponentsImpl (/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/next-server.js:710:36) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/require.js',
    '/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/next-server.js',
    '/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/next.js'
  ]
}

The same behavior is also seen when trying to create an assistant with the same script name
Error: Unexpected status 500: ERROR: duplicate key value violates unique constraint "idx_owner_slug" (SQLSTATE 23505)
    at h (/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/.next/server/chunks/3.js:1:10699)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async o (/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/.next/server/app/page.js:1:10063)
    at async /Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/compiled/next-server/app-page.runtime.prod.js:16:408
    at async rP (/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/compiled/next-server/app-page.runtime.prod.js:15:8043)
    at async r5 (/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/compiled/next-server/app-page.runtime.prod.js:18:1139)
    at async doRender (/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/base-server.js:1433:30)
    at async cacheEntry.responseCache.get.routeKind (/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/base-server.js:1594:28)
    at async NextNodeServer.renderToResponseWithComponentsImpl (/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/base-server.js:1502:28)
    at async NextNodeServer.renderPageComponent (/Users/sangeethahariharan/Library/Caches/gptscript/repos/977068302804850113e6a3568506f405c7df76db/tool.gpt/node21/node_modules/next/dist/server/base-server.js:1919:24) {
  digest: '2632440956'
}

This page is delete so should be deleted now @sangee2004

This workflow is not relevant anymore.


glfs: remove O_SYNC

O_SYNC makes IO a bit slow and as anyway we are ensuring the data
reaches onto nonvolatile disk with fdatasync() as part of flush op,
O_SYNC seems to be unworthy.
Signed-off-by: Prasanna Kumar Kalever<EMAIL_ADDRESS>

IMO, we should wait for perf results before getting this in.

For ibmvscsis, we actually need write-through for live migration support.  I understand that it might have a performance impact, but if that turns out to be true, can we make it configurable via targetcli instead?

Yes. Making it configurable with targetcli would be the preferred way.
If it can be translated to an existing LIO attr like device emulate_write_cache, then it would be a simple patch to target_core_user to enable that.
If we need a new setting like target_core_file's write_back, then it is a simple change to target_core_user, but will also require changes to rtslib and targetcli to take in the new setting.

@mikechristie We did our tests with and without O_SYNC on the file and we see 10X performance improvement. I think now we can start the discussion about why was O_SYNC there to begin with. I see that on qcow there are patches for Adding O_SYNC as well. So what would give us the confidence that nothing bad will happen by removing O_SYNC? Since glfs handler uses O_DIRECT, write doesn't get cached until it hits disk-cache.
@seroyer O_DIRECT is enough for transparency as well for live migration. O_SYNC is needed if the write needs to be durable as soon as the write is issued. I am curious to know the reason why you guys went for O_RDWR|O_SYNC instead of O_RDWR|O_DIRECT in https://github.com/open-iscsi/tcmu-runner/pull/153 it just kills IOPS compared to O_DIRECT.
@mikechristie When does tcmu-runner get flush command? Do let us know the kind of tests we need to make sure this is safe and we can get the results.

I do not know all the reasons a flush is sent. It depends on the OS, FS, use case, etc.
For linux for example, it is typically sent when a unmount is done, the OS is shutdown/rebooted, device is removed, FS is doing a barrier type of operation like when a journal and metadata update operation, app does a sync/flush call itself.
One question for you guys. Did you not add O_SYNC originally? Was it copied from other code like maybe the original writer of glfs.c saw it in some sort of qemu gluster code? I am just wondering where it came from originally to check if maybe it was needed for another case and it was just copied over.

@mikechristie It is there from the initial commit from Andy (bc0ba10b). We just want to make sure we can remove it safely. @agrover any pointers about why did we add O_SYNC? It seems like O_DIRECT suffices from the info we gathered. It would be really helpful for us to understand if we are missing some detail here.

@pranithk How do you know O_DIRECT is enough for Live partition migration? Just wondering since I dont know of a way to verify

Some cross reference could be our glfs driver in qemu, from [1] we can clearly see O_SYNC is not considered/used. In case if cache=none, we open with O_DIRECT.
[1] https://github.com/qemu/qemu/blob/master/block/gluster.c#L715

@bgly What I gathered from the comment from @seroyer on #153  is that the writes need to end up on the shared storage so that the changes can be witnessed from all the hosts. If we open file with just O_RDWR we are saying caching is allowed at all layers. So the writes may not reach the storage layer at all after a write. Both O_SYNC and O_DIRECT tell the layers that caching is not allowed. O_SYNC goes the extra step of doing the write on the device where as with O_DIRECT the writes can still be in the disk cache. At least doing O_DSYNC is better than O_SYNC IMHO. May be I could be completely wrong :-). May be you guys should decide based on https://lwn.net/Articles/457667/ if that helps. All that said, if you find that your solution meets the performance needs with O_SYNC, there is no need to change. The reason why I am bringing up this point in glfs.c is that the performance is just BAD!
We have seen live migrations with Ovirt+Gluster, even in that solution, kvm opens the qcow disk file which is hosted on gluster with O_DIRECT and does fsyncs at different points. Live migration of the VM works fine with this approach, this part is very well tested. I am wondering if tcmu-runner can do the same considering it also has flush() operation which does fdatasync on the backstore in glfs handler.

@mikechristie @pkalever Interesting thing is that gluster.c in qemu most probably made similar from: https://github.com/qemu/qemu/blob/master/block/file-posix.c#L362
Interesting thing is in tcmu-runner project, I see O_SYNC only for glfs.c and for nothing else. Which makes us even more curious as to why it could be :-)

@pranithk I know we haven't been real clear about how we use LIO, but it is probably different than you expect.  We are not using it for iSCSI targets.  Instead we are using it for ibmvscsis targets.  It's easier to see with a picture, but I'll try to describe it for you.  Imagine you have some shared network storage (could be NFS, could be iSCSI volumes with a cluster aware filesystem on it, etc...).  Then we have multiple Linux hosts running LIO that all see that same common storage.  Live migration for us involves moving a ibmvscsis target from one LIO instance to a different LIO instance.  So we must ensure integrity (with no cooperation from the guest VM!!!) across multiple LIO instances.  We temporarily have the target active on both systems, but have a trigger to ensure all outstanding I/O has completed (or timed out) on the source side LIO before we start actually sending I/O to the new LIO instance.
Whether we need O_SYNC or O_DIRECT (or something else) may vary depending on the details of the storage.  For example, if we use user:qcow on top of IBM GPFS, we don't need to do anything (no O_SYNC, O_DIRECT, or anything) because GPFS handles that integrity magically.  Maybe the same is true for glfs?  I don't know.

@seroyer thanks for the response. At least this clarifies some things. I guess your Last paragraph sums it best. I think to be on the safer side, it is probably better to use O_SYNC as long as you are okay with the performance impact. In case of user:glfs we need at least O_DIRECT and a setting on the volume which honours O_DIRECT. This will guarantee that the data ends up on storage which can be seen by multiple hosts. This way we don't sacrifice performance and still be safe.
@mikechristie What tests would you recommend we do so that we can close this discussion and merge the patch?

@pranithk, when the glfs backend (I think it is called a brick) gets a write and sends it to the FS it is using, what does it do to make sure the data is written to the medium? It sounds like it uses O_DIRECT to bypass the kernel page cache. How does it make sure the data in the physical device being used by the FS is synced? Does it do a sync/flush after each group of writes sent by a client, or does the FS above the device you are using do it, or do you disable the device's cache or set it to write through? Or, are you relying on the initiator side to send a SYNCHRONIZE_CACHE which calls tcmu_glfs_flush and for glfs_fdatasync you end up doing a flush/sync on brick's fs/device.

@mikechristie We want to rely on the initiator side to send a SYNCHRONIZE_CACHE. Should be fine right?

Sorry for the late response on this. @mikechristie Prasanna and I, did the following test but there are no fdatasync on glusterfs i.e. no SYNCHRONIZE_CACHE command was received by tcmu-runner. Prasanna further added that even after umount, this command is not received. How do you think we should proceed?

What did you see for the write cache settings for the discovered disk in sysfs and/or its logs?
Is it write back or through?

I think there is actually a bug in the runner code. I saw this on the initiator
Jul 14 02:21:47 rh2 kernel: sd 11:0:0:0: [sdb] Cache data unavailable
so you should look into that.

Hey @pranithk I am pretty sure our mode page handling code is just detecting that the allocated buffer is too small for the data we want to send and instead of just returning what we can we are returning a failure. Just fix up that code so it works more like the inquiry /evpd code where is sends what it can and it should work.

@mikechristie I found that what you said is what is happening using gdb. alloc_len is just 4 so it is not filling anything and as got_sense is false it says it is illegal request.
Based on your comment, I added this dummy code just to check if things will work as expected, I am not sure this is what you are expecting, but this was just for experimenting:
diff --git a/api.c b/api.c
index 1b8b7aa..3b75b34 100644
--- a/api.c
+++ b/api.c
@@ -861,12 +861,6 @@ int tcmu_emulate_mode_sense(
                        if (ret <= 0)
                                break;

-                       if  (sense_ten && (used_len + ret >= 255))
-                               break;
-
-                       if (used_len + ret > alloc_len)
-                               break;
-
                        used_len += ret;
                }
        }
@@ -879,12 +873,6 @@ int tcmu_emulate_mode_sense(
                                if (ret <= 0)
                                        break;

-                               if  (!sense_ten && (used_len + ret >= 255))
-                                       break;
-
-                               if (used_len + ret > alloc_len)
-                                       break;
-
                                used_len += ret;
                                got_sense = true;
                                break;

With this change, write-cache is enabled:
[ 4051.242032] scsi host2: iSCSI Initiator over TCP/IP
[ 4051.252396] scsi 2:0:0:0: Direct-Access     LIO-ORG  TCMU device      0002 PQ: 0 ANSI: 5
[ 4051.253979] sd 2:0:0:0: Attached scsi generic sg1 type 0
[ 4051.255285] sd 2:0:0:0: [sda] 16777216 512-byte logical blocks: (8.59 GB/8.00 GiB)
[ 4052.772025] sd 2:0:0:0: [sda] Write Protect is off
[ 4052.772034] sd 2:0:0:0: [sda] Mode Sense: 2f 00 00 00
[ 4053.597726] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 4055.564098] sd 2:0:0:0: [sda] Attached SCSI disk
We receive a flush that got translated into fdatasync on gluster
(gdb) b tcmu_glfs_flush
Breakpoint 2 at 0x7ffff48d59a0
(gdb) c
Continuing.
[Switching to Thread 0x7fffdeefe700 (LWP 5226)]
Thread 12 "tcmu-runner" hit Breakpoint 2, 0x00007ffff48d59a0 in tcmu_glfs_flush ()
from /usr/local/lib64/tcmu-runner/handler_glfs.so
(gdb) c
%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop

  0.00    3821.00 us    3821.00 us    3821.00 us              1       FSYNC
100.00  138978.19 us   16173.00 us  329099.00 us            644       WRITE

So it is working as we expected it to :-)
Thanks a lot Mike. Sorry it took a while to respond back to you, I am not well versed with this code or what it does, so had to read the scsi reference guide(Still not completely sure I understood it :-) ).
I am not completely sure about what exactly you are asking in the patch though.

On a Quick glance at the code, I'm thinking the 'alloc_len' seems to be getting a wrong representation of data, coz of a bug in tcmu_get_xfer_length() ?
diff --git a/api.c b/api.c
index 1b8b7aa..d9b0acf 100644
--- a/api.c
+++ b/api.c
@@ -85,7 +85,7 @@ uint32_t tcmu_get_xfer_length(uint8_t *cdb)
 {
        switch (tcmu_get_cdb_length(cdb)) {
        case 6:
-               return cdb[4];
+               return be16toh(*((uint16_t *)&cdb[4]));
        case 10:
                return be16toh(*((uint16_t *)&cdb[7]));
        case 12:

IMO, something like above should solve, @mikechristie @pranithk Thoughts ?

Oh, where will we have details about this representation?

MODE SENSE 6's allocation len is only 1 byte, so I do not think the patch to tcmu_get_xfer_length would not be correct.
You can find details in the SCSI specs on t10.org. Specifically in the SPC 3 spec in the MODE SENSE and Mode parameters sections.
For pranithk's patch we do not want to return error when we hit that used_len check. However, I am not sure what to return if the used_len would overflow the mode data length field. For example, mode data length is only 1 byte for MODE SENSE 6. Do you just set it to 255 if used_len > 255? I could not find anything in the spec. Do you guys see anything?
I am also not sure why the sense_ten checks are different for the different page_codes. I am trying to find that info the spec.

I googled a bit but didn't find anything useful yet :-(

In File Optical handler https://github.com/open-iscsi/tcmu-runner/blob/master/file_optical.c#L431
I have noticed
        else {
                   [...]
                                if (used_len + ret > alloc_len)
                                        break;
                   [...]
        }

is missing/not considered, in the *_emulate_mode_sense() function, but I'm not sure if it is intentional.
@bgly do you recollect why was this chunk not considered in the file optical handler ?
Thanks in advance :-)

@pkalever It was not intentional, do you want to add it in or should i?

@pkalever It was not intentional, do you want to add it in or should i?

No. I think there was a miscommunication a couple comments ago.
We are removing it from the api.c code. SPC 4r37 says:
<IP_ADDRESS> Allocation length:

The device server shall terminate transfers to the Data-In Buffer when the number of bytes or blocks specified by the ALLOCATION LENGTH field have been transferred or when all available data have been transferred, whichever is less. The allocation length is used to limit the maximum amount of variable length data (e.g., mode data, log data, diagnostic data) returned to an application client. If the information being transferred to the Data-In Buffer includes fields containing counts of the number of bytes in some or all of the data (e.g., the PARAMETER DATA LENGTH field, the PAGE LENGTH field, the DESCRIPTOR LENGTH field, the AVAILABLE DATA field), then the contents of these fields shall not be altered to reflect the truncation, if any, that results from an insuffi- cient ALLOCATION LENGTH value, unless the standard that describes the Data-In Buffer format states otherwise.

So file_optical is correct and we should not return a error for that type of issue.
However,  for the other part of the patch
-                       if  (!sense_ten && (used_len + ret >= 255))
-                               break;

I think this applies:

If the amount of information to be transferred exceeds the maximum value that the ALLOCATION LENGTH field is capable of specifying, the device server shall transfer no data and terminate the command with CHECK CONDITION status, with the sense key set to ILLEGAL REQUEST, and the additional sense code set to INVALID FIELD IN CDB.

In the test above, because MODE SENSE 6's allocation len is 1 byte we are limited to 255 bytes. So if used_len + ret is greater than that we must return ILLEGAL REQUEST since it is impossible to handle.
The last issue we have been trying to figure out is why for the sense_ten check is different when the page code is 0x3f. I have not had found any reason in the spec.

@mikechristie Based on your comment, this is what I understood.
We should have used !sense_ten instead of sense_ten in both the branches, and
When page_code is not 0x3f these are the changes.
diff --git a/api.c b/api.c
index 1b8b7aa..8ec4fe9 100644
--- a/api.c
+++ b/api.c
@@ -861,7 +861,7 @@ int tcmu_emulate_mode_sense(
                        if (ret <= 0)
                                break;

-                       if  (sense_ten && (used_len + ret >= 255))
+                       if  (!sense_ten && (used_len + ret >= 255))
                                break;

                        if (used_len + ret > alloc_len)
@@ -882,9 +882,6 @@ int tcmu_emulate_mode_sense(
                                if  (!sense_ten && (used_len + ret >= 255))
                                        break;

-                               if (used_len + ret > alloc_len)
-                                       break;
-
                                used_len += ret;
                                got_sense = true;
                                break;

But from the spec, it seems like we need to error out if the response is more than 255 for !sense_ten even when the page_code is 0x3f, so I think, this is what we should do? May be that's what you meant, but I am not completely sure. Just this last doubt if you could clarify, then I think I am all good to submit a new PR with the change.
diff --git a/api.c b/api.c
index 1b8b7aa..2ba95c5 100644
--- a/api.c
+++ b/api.c
@@ -855,15 +855,15 @@ int tcmu_emulate_mode_sense(
        /* Don't report block descriptors */

        if (page_code == 0x3f) {
-               got_sense = true;
                for (i = 0; i < ARRAY_SIZE(modesense_handlers); i++) {
                        ret = modesense_handlers[i].get(&buf[used_len], sizeof(buf) - used_len);
                        if (ret <= 0)
                                break;

-                       if  (sense_ten && (used_len + ret >= 255))
+                       if  (!sense_ten && (used_len + ret >= 255))
                                break;

+                        got_sense = true;
                        if (used_len + ret > alloc_len)
                                break;

@@ -882,9 +882,6 @@ int tcmu_emulate_mode_sense(
                                if  (!sense_ten && (used_len + ret >= 255))
                                        break;

-                               if (used_len + ret > alloc_len)
-                                       break;
-
                                used_len += ret;
                                got_sense = true;
                                break;

we are executing a MODE_SENSE 6 and used_len + ret >= 255, we need to always return failure. This is for both the 0x3f and non 0x3f case.

for both the 0x3f case and non 3f case, if use_len + ret > alloc_len, then we do not return failure unless we hit also

there is also the case where our buffer, buf, is too small. In this case, we probably should be returning a ASC_INTERNAL_TARGET_FAILURE. This is hit when the mode sense handler get() function returns -1. We should probably just avoid this case and alloc the buf alloc_len bytes.

I attached a patch which is completely untested and probably does not compile to show what I mean.
fix-mode-sense.txt

@mikechristie I will test it and let you know if there are any issues.

@mikechristie I had to make some small changes to the patch and I tested it and it worked fine. I am just finishing up testing with address sanitizer, just to make sure there aren't any other issues we might have introduced. Give me some more time and I will attach the modified patch for you to submit.

@mikechristie Done with my testing. Here is the patch. Let me know if you want to make any other changes. Happy to test. So it doesn't matter if we don't send the values handler popualtes in buf? Even then it is working fine :-). I was a bit surprised to find this actually.
Here is the diff between the patch you sent and the changes I made:
20:18:47⚡ diff ~/Downloads/fix-mode-sense.txt fix-mode-sense.txt
2c2
< index b53fd1c..cf4305c 100644
---
> index b53fd1c..11bba92 100644
26c26
< +	       to_len > from_len ? from_len : from_len - to_len);
---
> +	       to_len > from_len ? from_len : to_len);
28c28
< +
---
>
32c32
<
---
> +
41c41,42
<  int handle_cache_page(uint8_t *buf, size_t buf_len)
---
> -int handle_cache_page(uint8_t *buf, size_t buf_len)
> +int handle_cache_page(uint8_t *ret_buf, size_t ret_buf_len)
56c57,58
<  static int handle_control_page(uint8_t *buf, size_t buf_len)
---
> -static int handle_control_page(uint8_t *buf, size_t buf_len)
> +static int handle_control_page(uint8_t *ret_buf, size_t ret_buf_len)
80c82
< @@ -828,6 +848,43 @@ static struct {
---
> @@ -828,6 +848,42 @@ static struct {
86c88
< +				 uint8_t *used_len, bool sense_ten)
---
> +				 size_t *used_len, bool sense_ten)
88d89
< +	uint8_t *buf;
124c125
< @@ -846,13 +903,22 @@ int tcmu_emulate_mode_sense(
---
> @@ -846,13 +902,23 @@ int tcmu_emulate_mode_sense(
130a132
> +        uint8_t *orig_buf = NULL;
134c136
< +		return SAM_SAT_GOOD;
---
> +		return SAM_STAT_GOOD;
170c172
< +			if (ret < 0);
---
> +			if (ret < 0)
223c225
< +	tcmu_memcpy_into_iovec(iovec, iov_cnt, orig_buf, sizeof(orig_buf));
---
> +	tcmu_memcpy_into_iovec(iovec, iov_cnt, orig_buf, alloc_len);

fix-mode-sense.txt

@mikechristie Thanks for the Patch.
@pranithk Thanks for testing this through.
Mike, If you think this patches are ready, will it be possible for you to merge your modesense patch along with O_SYNC patch as head ?

@pkaleevr, yes I will merge everything at the same time.
@pranithk, could you send the final patch you tested? And yeah, it is weird, but the initiator side can just read the final length value that is calculated and then resend the command with a buffer/allocation length with the correct value. It does not have to continually guess until it finally works.
I just had one review comment for myself. I did not like the temp buf use. tcmu_memcpy_into_iovec would do the right thing so I can just use that directly. I am going to convert to using that. I will adapt that to your patch and test it.

@mikechristie the patch I attached in the last comment is the updated patch, I attached it with same name. Sorry for the confusion :-(

@pranithk that was my fault. I did not check it :)
I made a PR with the mode sense patch here:
https://github.com/open-iscsi/tcmu-runner/pull/220
I will merge that PR with your patch here when it is all reviewed. I did not convert the patch to use the iovec directly instead of the tmp buffer. Because we need to put the use_len at the very front of all the other page's data, it became too much of a pain. I could just sneak a peak at the first iovec's buffer but that was too hacky. I want to fix up the buffer handling for all these types of commands, but will do that in another patchset, so we can get the protocol fixes in sooner.


Update README.md

Description

Code sample

Add link to AngularFire2.

I signed it! For maybe the 100th time. Really.

Closed in favor or #739.

Coverage remained the same at 93.886% when pulling 1d7f0b2e717005acabc461965f4df55abfd45190 on katowulf-af2-readme into afa0788793d38edcb1f2c8c82ec9cb29803efddb on master.