Fix non-base58 character error on windows

Replace backslash used in windows paths with slash to ensure correct conversion into parts.
With these changes orbit-db should support windows.
Related to #274

+1

+1

Thank you @durac for the PR! Sounds great if this would get us on Windows :) I'd like to try it myself on a Windows machine to see it all works and the developer experience is solid like it is osx/linux.
I'll comment on the PR if there's anything to comment.

Nice job, I'am able to use orbitDB on windows 10 now! Thx!

Hey! Thanks for this. I've run it on my computer, and three tests seem to have issues. Thoughts?
  1) orbit-db - Create & Open (js-ipfs)
       Create
         Success
           can pass local database directory as an option:
     Uncaught AssertionError [ERR_ASSERTION]: NotFoundError: Key not found in database [/orbitdb/QmQAAJKY6kn8Ac4cVFxDfvKwqMr38PHdsCBYX46XB1Cgt6/second/_manifest]
    at C:... == null
      at db.get (test\create-open.test.js:125:24)
      at node_modules\levelup\lib\levelup.js:168:14

  2) orbit-db - Create & Open (js-ipfs)
       Create
         Success
           Access Controller
             "before all" hook for "can pass local database directory as an option":
     Error: IO error: LockFile orbitdb\tests\create-open\QmQAAJKY6kn8Ac4cVFxDfvKwqMr38PHdsCBYX46XB1Cgt6\second/LOCK: The process cannot access the file because it is being used by another process.

  3) orbit-db - Create Custom Database Type (js-ipfs)
       "before all" hook:
     Error: EBUSY: resource busy or locked, unlink 'orbitdb\tests\create-open\QmQAAJKY6kn8Ac4cVFxDfvKwqMr38PHdsCBYX46XB1Cgt6\second\000005.ldb'
      at Object.unlinkSync (fs.js:949:3)
      at rimrafSync (node_modules\rimraf\rimraf.js:306:17)
      at node_modules\rimraf\rimraf.js:342:5
      at Array.forEach (<anonymous>)
      at rmkidsSync (node_modules\rimraf\rimraf.js:341:26)
      at rmdirSync (node_modules\rimraf\rimraf.js:334:7)
      at rimrafSync (node_modules\rimraf\rimraf.js:304:9)
      at node_modules\rimraf\rimraf.js:342:5
      at Array.forEach (<anonymous>)
      at rmkidsSync (node_modules\rimraf\rimraf.js:341:26)
      at rmdirSync (node_modules\rimraf\rimraf.js:334:7)
      at rimrafSync (node_modules\rimraf\rimraf.js:304:9)
      at node_modules\rimraf\rimraf.js:342:5
      at Array.forEach (<anonymous>)
      at rmkidsSync (node_modules\rimraf\rimraf.js:341:26)
      at rmdirSync (node_modules\rimraf\rimraf.js:334:7)
      at Function.rimrafSync [as sync] (node_modules\rimraf\rimraf.js:304:9)
      at Context.before (test\create-type.test.js:39:12)

Hmm. I had some issues running the examples.
$ npm run build:examples

><EMAIL_ADDRESS>build:examples C:\Users\julia\Code\orbit-db
> webpack --config conf/webpack.example.config.js --sort-modules-by size && mkdir -p examples/browser/lib && cp node_modules/ipfs/dist/index.js examples/browser/lib/ipfs.js

Hash: f5801aa88f822e51bdef
Version: webpack 3.12.0
Time: 10074ms
                                               Asset     Size  Chunks                    Chunk Names
./examples/browser/browser-webpack-example/bundle.js  4.63 MB       0  [emitted]  [big]  main
 [744] (webpack)/buildin/amd-options.js 82 bytes {0} [built]
[1621] ./src/db-manifest.js 454 bytes {0} [built]
   [5] (webpack)/buildin/global.js 509 bytes {0} [built]
  [28] (webpack)/buildin/module.js 517 bytes {0} [built]
 [636] ./examples/browser/browser-webpack-example/index.js 755 bytes {0} [built]
[1618] ./src/ipfs-access-controller.js 1.06 kB {0} [built]
[1620] ./src/orbit-db-address.js 1.4 kB {0} [built]
[1622] ./src/exchange-heads.js 1.41 kB {0} [built]
[1619] ./src/access-controller.js 1.96 kB {0} [built]
[1628] ./examples/browser/example.js 8.95 kB {0} [built]
[1576] ./src/OrbitDB.js 13.4 kB {0} [built]
    + 1618 hidden modules
The syntax of the command is incorrect.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR<EMAIL_ADDRESS>build:examples: `webpack --config conf/webpack.example.config.js --sort-modules-by size && mkdir -p examples/browser/lib && cp node_modules/ipfs/dist/index.js examples/browser/lib/ipfs.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>build:examples script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR!      C:\Users\julia\AppData\Roaming\npm-cache\_logs\2018-12-18T17_36_38_642Z-debug.log

Note: The above was me. I'm not sure what the problem was.

Any information about this issue? we are waiting for the fix :)

@HTUlv412 Fix is working :) Not merge yet but just modify the files according to the commit will solve the base58 error.

I have a Windows computer to run this now; I'll be back in touch with it next week. This may not be the best way to manage our Windows environment maintenance, though; I'm not an expert at Windows, although I can run the tests and just generally see if it works.
Windows is not supported yet, but if this PR works, then perhaps we can just merge it. I'm not sure if that is best practice or not. @aphelionz @haadcode How would you feel about merging this based on the positive reviews above?

I believe Windows compatibility was addressed here: https://github.com/orbitdb/orbit-db/pull/712
Closing. Please feel free to re-open if I'm not correct.


tripERR

I As you mentioned in README files, I use “Facescrub” database. Unfortunately, when I ran your code the tripERR was swinging between different values (0.61, 0.48, 0.39, 0.44, 0.72, ....).
I changed some parameters of your code such as “learning_rate”, “alpha”, “moving_average_decay”, but the result was not much change.
May I ask you why is the tripEER swinging? Could you help me to fix this problem, please?

Hi,
Did you change alpha to a larger value than 0.2? I must say that I'm not too surprised about the variations, but more surprised about the large values for the triplet loss (tripErr) over all. Typically when training with triplet loss (and alpha == 0.2) it starts off at 0.2 and starts to decrease. But note that there will be variations due to the limited batch size.
Below is a typical sequence when training the first ten batches. The triplet loss is swinging between 0.177 and 0.243. If you want to reduce the "swinging" i think that the best option would be to increase the batch size, but that is not necessarily an efficient way to train the network.
Epoch: [0][0/1000]	Time 1.063	tripErr 0.202
Epoch: [0][1/1000]	Time 0.946	tripErr 0.196
Epoch: [0][2/1000]	Time 0.521	tripErr 0.177
Epoch: [0][3/1000]	Time 0.520	tripErr 0.192
Epoch: [0][4/1000]	Time 0.517	tripErr 0.201
Epoch: [0][5/1000]	Time 0.523	tripErr 0.193
Epoch: [0][6/1000]	Time 0.518	tripErr 0.208
Epoch: [0][7/1000]	Time 0.524	tripErr 0.243
Epoch: [0][8/1000]	Time 0.520	tripErr 0.182
Epoch: [0][9/1000]	Time 0.522	tripErr 0.181
Other thinks to check would be that you have the normalization of the embedding:
norm = tf.nn.l2_normalize(affn1, 1, 1e-10)
Without this line the scaling of the embeddings would be off and triplet loss would probably also behave in a strange way.

 When you ran your code on “Facecrub” dataset, what was the “accuracy” value?
Could you achieve the FaceNet’s accuracy in your result?
And I need to be faster. So, if I use the gray images instead of color images, will the result change?
What do you suggest me if I want to decrease the time of training?
May it be good if I use the Caffe instead of Python in my implementation?

Could you achieve the FaceNet’s accuracy in your result? And I need to be faster.So, if I use the gray images instead of color images, will the result change?What do you suggest meif I want to decrease the time of training?
May it be good if I use the Caffeinstead of Python in my implementation?
Best Regards,Fatemeh
 ____________________________________Fatemeh Sadat Faal Hosseini Mazloum MSc student in Computer Engineering - Artificial IntelligenceMachine Vision LaboratoryDepartment of Computer Engineering
Faculty of Engineering
Ferdowsi University of Mashhad
Mashhad, Iranhttp://mvlab.um.ac.ir/
http://eiarg.um.ac.ir/
  From: David Sandberg<EMAIL_ADDRESS>
To: davidsandberg/facenet<EMAIL_ADDRESS>Cc: FSFHMSR<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Sent: Sunday, April 24, 2016 11:24 PM
Subject: Re: [davidsandberg/facenet] tripERR (#13)
Hi,
Did you change alpha to a larger value than 0.2? I must say that I'm not too surprised about the variations, but more surprised about the large values for the triplet loss (tripErr) over all. Typically when training with triplet loss (and alpha == 0.2) it starts off at 0.2 and starts to decrease. But note that there will be variations due to the limited batch size.
Below is a typical sequence when training the first ten batches. The triplet loss is swinging between 0.177 and 0.243. If you want to reduce the "swinging" i think that the best option would be to increase the batch size, but that is not necessarily an efficient way to train the network.
Epoch: [0][0/1000] Time 1.063 tripErr 0.202
Epoch: [0][1/1000] Time 0.946 tripErr 0.196
Epoch: [0][2/1000] Time 0.521 tripErr 0.177
Epoch: [0][3/1000] Time 0.520 tripErr 0.192
Epoch: [0][4/1000] Time 0.517 tripErr 0.201
Epoch: [0][5/1000] Time 0.523 tripErr 0.193
Epoch: [0][6/1000] Time 0.518 tripErr 0.208
Epoch: [0][7/1000] Time 0.524 tripErr 0.243
Epoch: [0][8/1000] Time 0.520 tripErr 0.182
Epoch: [0][9/1000] Time 0.522 tripErr 0.181
Other thinks to check would be that you have the normalization of the embedding:
norm = tf.nn.l2_normalize(affn1, 1, 1e-10)
Without this line the scaling of the embeddings would be off and triplet loss would probably also behave in a strange way.—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub

Dear David
I  forget to ask my main problem. When I ran your code the value of accuracy changed in each iteration. At the begging iterations the accuracy is about 50 and after several iterations the accuracy increase up to 75. And then the accuracy is swinging between 70-79.
Is it normal?

I haven't checked the accuracy of the model trained on purly facescrub lately. The ROC-curve can be found in data/20160216_roc_lfw_nn4.png. I think the accuracy for this model is a little bit above 80%. If you want to try it you can download the model using the link in the readme file (https://drive.google.com/file/d/0B5MzpY9kBtDVUFNFR21kZnYybkE/view?usp=sharing) and run "validate_on_lfw" on that model. To reach the accuracy from the facenet paper i believe that a significantly larger training set than facescrub would be needed.
Regarding the performance with gray scale images i think that would be a very interesting investigation. Other ways to speed up the training would be to use a smaller model. In the facenet paper they use models NNS1 and NNS2 but i didn't see any description of those architectures.
And regarding the swinging i must say that it sounds a bit strange. I haven't seen that much variation in the accuracy even though some small variations are of course expected. Attached is the learning curve for the training of the 20160308 model.


Force a trip if another is done - Python - Ubuntu - Routing

Hello,
I use python
Here is my problem: I would like to have a vehicle transporting an object from point A to point B and then from point B to point A.
I already have this :

and this :

Which means that my trips are optional.
Take 4 point: 1,2,3,4
We have 1-> 2 and 3-> 4, so :
routing.AddPickupAndDelivery(1, 2)
routing.AddPickupAndDelivery(3, 4)
I would like these trips to stay optionnal but impose the following constraint on the solver: If one trip then the other
In other words, ([1-> 2] && [3-> 4]) or Nothing
I have several tracks: modify the AddDisjunction () or add an AddPickupAndDelivery (2,4) which has not worked so far
Thank you for reading and sorry for my poor English

did you try to add that ?
routing.solver().Add(
  routing.VehicleVar(manager.NodeToIndex(2) == routing.VehicleVar(manager.NodeToIndex(3))

in order to "chain" 2 and 3
Otherwise you should ask on the mailing-list !

Thank you for answering
I tried something similar: routing.solver().Add(routing.VehicleVar(2)                              == routing.VehicleVar(4))
I will try your code right away

To drop all the circuit or note you can use Disjunction with cardinality of 4
routing.AddDisjunction([A,B,C,D], 10000, 4)

see:
https://github.com/google/or-tools/blob/5ff76b487a6c2006326765d6417964599eedc8c9/ortools/constraint_solver/routing.h#L570-L588

I will try this too, thanks again

also since node are added one by one you should consider checking if node are active to apply the constraint
see: https://github.com/google/or-tools/issues/847#issuecomment-420608114

I have tried these different methods, but none of them work yet. Which is probably due to an error coming from me.
I did not find you on Discord. Here is my profile: Skyloq#8629

This whole discussion is lost for the community. Please use the mailing list.

did you try to add that ?
routing.solver().Add(
  routing.VehicleVar(manager.NodeToIndex(2) == routing.VehicleVar(manager.NodeToIndex(3))

in order to "chain" 2 and 3
Otherwise you should ask on the mailing-list !

Hey,
im have used this constraint and works good but i also need to add something else to my problem, i havent found a great solution for ti, ill be very thanked if you could give me a hand with id.
i need also to make some nodes/visits to be subsequent not only in the same vehicle.
for example:
bundled_orders = [1,2,3,4], [5,6] [7,8,9][10]
route_1 = 0 -> 1 -> 2 -> 3 -> 4 -> 10
route_2 = 0 -> 9 -> 7 -> 8 -> 5 -> 6
thank you!

did you try to add that ?
routing.solver().Add(
  routing.VehicleVar(manager.NodeToIndex(2) == routing.VehicleVar(manager.NodeToIndex(3))

in order to "chain" 2 and 3
Otherwise you should ask on the mailing-list !

Hey,
hope i find you well!
i have used this constraint and works good but i also need to add something else to my problem, i havent found a great solution for ti, ill be very thanked if you could give me a hand with id.
i need also to make some nodes/visits to be subsequent not only in the same vehicle.
for example:
bundled_orders = [1,2,3,4], [5,6] [7,8,9][10]
route_1 = 0 -> 1 -> 3 -> 2 -> 4 -> 10
route_2 = 0 -> 9 -> 7 -> 8 -> 5 -> 6
thank you!

Did you try to add so precedence constraint: i.e. time at 1 < time at 2 < time at 3 < time at 4 etc... ?
solver = routing.solver()
one = manager.NodeToIndex(1)
two = manager.NodeToIndex(2)
three = manager.NodeToIndex(3)
four = manager.NodeToIndex(4)
solver.Add(time_dim.CumulVar(one) <= time_dim.CumulCar(two))
solver.Add(time_dim.CumulVar(two) <= time_dim.CumulCar(three))
solver.Add(time_dim.CumulVar(three) <= time_dim.CumulCar(four))
...

note: if some node can be dropped you may need to "multiply" by routing.ActiveVar(location_index) to nullify the expression thus constraint become 0 <= 0 and is still valid...
ref: https://github.com/google/or-tools/blob/405b1dcaf1db6a1165e48a836e931828fe1d6fd5/ortools/constraint_solver/routing.h#L1359-L1360

Did you try to add so precedence constraint: i.e. time at 1 < time at 2 < time at 3 < time at 4 etc... ?
solver = routing.solver()
one = manager.NodeToIndex(1)
two = manager.NodeToIndex(2)
three = manager.NodeToIndex(3)
four = manager.NodeToIndex(4)
solver.Add(time_dim.CumulVar(one) <= time_dim.CumulCar(two))
solver.Add(time_dim.CumulVar(two) <= time_dim.CumulCar(three))
solver.Add(time_dim.CumulVar(three) <= time_dim.CumulCar(four))
...

note: if some node can be dropped you may need to "multiply" by routing.ActiveVar(location_index) to nullify the expression thus constraint become 0 <= 0 and is still valid... ref:
https://github.com/google/or-tools/blob/405b1dcaf1db6a1165e48a836e931828fe1d6fd5/ortools/constraint_solver/routing.h#L1359-L1360

i have tried it but i want it to be able to decide the order.
using this constraints the only way i can route the grouo 1 - 2 - 3 - 4 is in that order and i am trying to make it able to be (for example) 1-4-2-3
do you know if its possible?


The request could not be completed because: Could not set team ID to 'XXXYYY', only found the following available teams

New Issue Checklist

[X] Updated fastlane to the latest version
[X] I read the Contribution Guidelines
[X] I read docs.fastlane.tools
[X] I searched for existing GitHub issues

Issue Description
I'm trying to upload an iOS app to TestFlight using fastlane on a hosted Azure DevOps agent.
The desired workflow is as follows:
Step 1
job 1) Initiate fastlane on an agent - V works as expected
Step 2
Job 1) Initialize jobs - V works as expected
Job 2) Install Apple Distribution certificate - V works as expected
Job 3) Download binaries from a repo - V works as expected
Job 4) Resign the app using distribution cert and provisioning profile - V works as expected
Job 5) Publish to the App Store TestFlight track - X fails with the following error message:
2020-04-07T19:52:42.9135780Z [!] The request could not be completed because:
2020-04-07T19:52:42.9136680Z 	Could not set team ID to 'XXXYYY', only found the following available teams:
2020-04-07T19:52:42.9137580Z - ZZZWWW (Company 2 - A1B23C)
2020-04-07T19:52:42.9138150Z - AABB (Company 2)
2020-04-07T19:52:42.9138750Z - CCDD (Company 3)
Command executed
fastlane pilot upload -u *** -i /Users/runner/runners/2.165.2/work/r1/a/xMobile/IPA/AppName.ipa -q XXXYYY -r My Company -a com.app.name
Please note, the arguments for the command are taken from Azure DevOps job inputs and not written manually.
Complete output when running fastlane, including the stack trace and command used
2020-04-07T19:52:19.3337350Z ##[section]Starting: Publish to the App Store TestFlight track
2020-04-07T19:52:19.3349220Z ==============================================================================
2020-04-07T19:52:19.3349700Z Task         : Apple App Store Release
2020-04-07T19:52:19.3350120Z Description  : Release an app to TestFlight or the Apple App Store
2020-04-07T19:52:19.3350460Z Version      : 1.158.0
2020-04-07T19:52:19.3350760Z Author       : Microsoft Corporation
2020-04-07T19:52:19.3351240Z Help         : More Information
2020-04-07T19:52:19.3351820Z ==============================================================================
2020-04-07T19:52:19.5191520Z 244a7aa4-33a2-4b83-b92b-9f10eb254eec exists true
2020-04-07T19:52:19.5309980Z [command]/usr/local/opt/ruby/bin/gem install fastlane
2020-04-07T19:52:19.5367470Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5368430Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5369510Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5370260Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5371060Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5371800Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5372610Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5373350Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5374210Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5375050Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5375810Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5376580Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5377340Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5378130Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5378890Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5379670Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5380420Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5381220Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5381950Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5382760Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5383500Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5384290Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5385030Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5385820Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5386570Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5387380Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5388110Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5388900Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5389690Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5390440Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5391920Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:19.5392690Z (node:758) Warning: Use Cipheriv for counter mode of aes-256-ctr
2020-04-07T19:52:31.0176570Z Successfully installed fastlane-2.145.0
2020-04-07T19:52:31.0177390Z Parsing documentation for fastlane-2.145.0
2020-04-07T19:52:31.0177830Z Done installing documentation for fastlane after 7 seconds
2020-04-07T19:52:31.0178680Z 1 gem installed
2020-04-07T19:52:31.0455640Z [command]/usr/local/opt/ruby/bin/gem update fastlane -i /Users/runner/.gem-cache
2020-04-07T19:52:33.5600920Z Updating installed gems
2020-04-07T19:52:33.5601470Z Nothing to update
2020-04-07T19:52:33.5860810Z [command]fastlane pilot upload -u *** -i /Users/runner/runners/2.165.2/work/r1/a/xMobile/IPA/AppName.ipa -q XXXYYY -r My Company -a com.app.name
2020-04-07T19:52:36.2156210Z [19:52:36]: Get started using a Gemfile for fastlane https://docs.fastlane.tools/getting-started/ios/setup/#use-a-gemfile
2020-04-07T19:52:39.9021010Z [19:52:39]: Login to App Store Connect (***)
2020-04-07T19:52:42.2740630Z
2020-04-07T19:52:42.2741310Z Looking for related GitHub issues on fastlane/fastlane...
2020-04-07T19:52:42.2741640Z
2020-04-07T19:52:42.9056020Z ➡  Possible to pass fastlane a .app or .zip?
2020-04-07T19:52:42.9056890Z     https://github.com/fastlane/fastlane/issues/330 [closed] 2 💬
2020-04-07T19:52:42.9124640Z     22 Sep 2016
2020-04-07T19:52:42.9125010Z
2020-04-07T19:52:42.9126630Z ➡  s3 action failure after migrating to 1.3.0
2020-04-07T19:52:42.9127900Z     https://github.com/fastlane/fastlane/issues/318 [closed] 6 💬
2020-04-07T19:52:42.9128470Z     22 Sep 2016
2020-04-07T19:52:42.9128700Z
2020-04-07T19:52:42.9129790Z ➡  Trouble using fastlane / sigh on multi-user system
2020-04-07T19:52:42.9131160Z     https://github.com/fastlane/fastlane/issues/326 [closed] 13 💬
2020-04-07T19:52:42.9131590Z     08 Nov 2016
2020-04-07T19:52:42.9131740Z
2020-04-07T19:52:42.9133460Z and 16151 more at: https://github.com/fastlane/fastlane/search?q=The request could not be completed because%3A
	Could not set team ID to '117949423'%2C only found the following available teams%3A

- 119531376 (SAGE SOFTWARE%2C INC. - J9K9638GUE)
- 300156 (Sage SP%2C S.L.)
- 94016 (Sage Software%2C Inc.)
&type=Issues&utf8=✓
2020-04-07T19:52:42.9134530Z
2020-04-07T19:52:42.9135210Z 🔗  You can ⌘ + double-click on links to open them directly in your browser.
2020-04-07T19:52:42.9135540Z
2020-04-07T19:52:42.9135780Z [!] The request could not be completed because:
2020-04-07T19:52:42.9136680Z 	Could not set team ID to 'XXXYYY', only found the following available teams:
2020-04-07T19:52:42.9137580Z - ZZZWWW (Company 2 - A1B23C)
2020-04-07T19:52:42.9138150Z - AABB (Company 2)
2020-04-07T19:52:42.9138750Z - CCDD (Company 3)
2020-04-07T19:52:42.9311360Z ##[error]Error: fastlane failed with return code: 1
2020-04-07T19:52:42.9326220Z ##[section]Finishing: Publish to the App Store TestFlight track
I've read in similar issues that one should use Itunes Connect team id instead of Apple Developer team id and I did that. I have an environment variable called TeamNum that stores this id and I checked it using Martín De la Fuente's advice from this stack overflow issue: https://stackoverflow.com/questions/42960644/how-to-get-itunes-connect-team-id-and-team-name
So I am 100% sure team id is correct but for some reason fastlane is not accepting it.
Environment
Unfortunately, I can't attach details of the environment as the fastlane agent is out of my control. Thanks!

Update: I realized that the error was due to the Azure DevOps agent user not being part of the specified team. If I add fastlane argument with usernmae and set user to my own with password being taken from environment variable (FASTLANE_PASSWORD), the upload fails with an error message that credentials seem to be incorrect. I know they are correct and remember facing the same error message on a physical Mac OS agent until I added my credentials to a keychain. Thus, the question is: is there a way to add credentials to keychain if I use a standard AzDO task group called "Publish to the App Store TestFlight track" or is there a different way to make the upload work in my setup. Will happily provide any further information on this issue. Thanks!

Issue resolved after adding a proper service connection to App Store in organization settings. Thanks!


加载模型失败

Describe the bug
A clear and concise description of what the bug is.
Failed to load https://file.hankcs.com/hanlp/dep/ctb9_dep_electra_small_20220216_100306.zip.
Please upgrade HanLP with:
    pip install --upgrade hanlp

If the problem still persists, please submit an issue to https://github.com/hankcs/HanLP/issues
When reporting an issue, make sure to paste the FULL ERROR LOG below.
================================ERROR LOG BEGINS================================
OS: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.17
Python: 3.8.5
PyTorch: 1.11.0+cu102
HanLP: 2.1.0-beta.17
Traceback (most recent call last):
File "meta_field_service.py", line 35, in
dep = hanlp.load(hanlp.pretrained.dep.CTB9_DEP_ELECTRA_SMALL)
File "/usr/local/python3/lib/python3.8/site-packages/hanlp/init.py", line 43, in load
return load_from_meta_file(save_dir, 'meta.json', verbose=verbose, **kwargs)
File "/usr/local/python3/lib/python3.8/site-packages/hanlp/utils/component_util.py", line 170, in load_from_meta_file
raise e from None
File "/usr/local/python3/lib/python3.8/site-packages/hanlp/utils/component_util.py", line 98, in load_from_meta_file
obj.load(save_dir, verbose=verbose, **kwargs)
File "/usr/local/python3/lib/python3.8/site-packages/hanlp/common/torch_component.py", line 173, in load
self.load_config(save_dir, **kwargs)
File "/usr/local/python3/lib/python3.8/site-packages/hanlp/common/torch_component.py", line 126, in load_config
self.on_config_ready(**self.config, save_dir=save_dir)
File "/usr/local/python3/lib/python3.8/site-packages/hanlp/components/parsers/biaffine/biaffine_dep.py", line 562, in on_config_ready
self.build_transformer_tokenizer()  # We have to build tokenizer before building the dataloader and model
File "/usr/local/python3/lib/python3.8/site-packages/hanlp/components/parsers/biaffine/biaffine_dep.py", line 285, in build_transformer_tokenizer
transformer_tokenizer: PreTrainedTokenizer = AutoTokenizer.from_pretrained(transformer, use_fast=True)
File "/usr/local/python3/lib/python3.8/site-packages/transformers/mode
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.

Describe the current behavior
A clear and concise description of what happened.
Expected behavior
A clear and concise description of what you expected to happen.
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Python version:
HanLP version:

Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
Failed to load https://file.hankcs.com/hanlp/dep/ctb9_dep_electra_small_20220216_100306.zip.
Please upgrade HanLP with:
    pip install --upgrade hanlp

If the problem still persists, please submit an issue to https://github.com/hankcs/HanLP/issues
When reporting an issue, make sure to paste the FULL ERROR LOG below.
================================ERROR LOG BEGINS================================
OS: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.17
Python: 3.8.5
PyTorch: 1.11.0+cu102
HanLP: 2.1.0-beta.17
Traceback (most recent call last):
File "meta_field_service.py", line 35, in
dep = hanlp.load(hanlp.pretrained.dep.CTB9_DEP_ELECTRA_SMALL)
File "/usr/local/python3/lib/python3.8/site-packages/hanlp/init.py", line 43, in load
return load_from_meta_file(save_dir, 'meta.json', verbose=verbose, **kwargs)
File "/usr/local/python3/lib/python3.8/site-packages/hanlp/utils/component_util.py", line 170, in load_from_meta_file
raise e from None
File "/usr/local/python3/lib/python3.8/site-packages/hanlp/utils/component_util.py", line 98, in load_from_meta_file
obj.load(save_dir, verbose=verbose, **kwargs)
File "/usr/local/python3/lib/python3.8/site-packages/hanlp/common/torch_component.py", line 173, in load
self.load_config(save_dir, **kwargs)
File "/usr/local/python3/lib/python3.8/site-packages/hanlp/common/torch_component.py", line 126, in load_config
self.on_config_ready(**self.config, save_dir=save_dir)
File "/usr/local/python3/lib/python3.8/site-packages/hanlp/components/parsers/biaffine/biaffine_dep.py", line 562, in on_config_ready
self.build_transformer_tokenizer()  # We have to build tokenizer before building the dataloader and model
File "/usr/local/python3/lib/python3.8/site-packages/hanlp/components/parsers/biaffine/biaffine_dep.py", line 285, in build_transformer_tokenizer
transformer_tokenizer: PreTrainedTokenizer = AutoTokenizer.from_pretrained(transformer, use_fast=True)
File "/usr/local/python3/lib/python3.8/site-packages/transformers/mode

[ ] I've completed this form and searched the web for solutions.

【自动回复】您好，感谢反馈。
由于未按要求填写，本issue被自动关闭，将在一小时内自动删除。
如果您提的是bug或feature request，请按issue模板认真修改，然后耐心等待下周末处理。
否则，咨询类的问题GitHub概不受理。请先搜索一下，99.9%的问题已经反复回答过了，人类的本质是复读机。如果您确信您提了个新问题，那么欢迎写清楚前因后果发在论坛而不是GitHub上。
开源项目维护不易，HanLP用户以十万计，而主要维护者只有我一人。我有自己的full-time研究工作，只能在周末受理友好的提问，造成不便还望海涵。


Pin urllib3 in CI

urllib3 just made a major release (2.0.0) and it seems that's broken cachecontrol, which PDM depends on. This pins urllib3 to <2.

Nice fix.


[CS] Added PHP-CS-Fixer and ruleset

Q
A

Bug fix?
no

New feature?
yes

BC breaks?
no

Deprecations?
no

Tests pass?
yes

Fixed tickets

License
MIT

Doc PR

Added dependency for PHP-CS-Fixer and ruleset based on
https://github.com/doctrine/coding-standard/blob/96b3e19c9a151760e0246687a7143f77c0aff3a0/Docs/README.md

@Ocramius, I just rebased master in order to have this PR up to date. Which is the path to follow in order to move forward?

Superseded by 2be1b7d0b8cc47e1061493de74ac22131e3e5031.


BH1792: C/C++ driver for ROHM Heart Rate Sensor

This is another driver for ROHM sensors, for BH1792 Heart rate sensor. It contains C main implementation and C++ wrapper, and examples for supported languages.

Merged with some minor formatting changes (89e2ec870a90aebb0684cce2d1b3df7a47b837cf). Thanks!


Solution properties

Augment solutions with a PropertyMap

condition to start a trajectory delayed (e.g. for finger closing)
callbacks to trigger suction on/off

Provide two exec() variants:

exec(SolutionMsg) -> send to move_group node
exec(const SolutionBase&) -> use MoveItCpp to execute a solution directly, allowing to evaluate stored properties

Another prominent use-case for the properties is to specify controllers/controller parameters to switch before execution of the segment starts, e.g., increase force limits to grasp stronger, switch to velocity-based controllers or switch to an entirely different custom (ros_)controller to adjust the executed trajectory on the fly depending on system state. Part of that was implemented by PickNik, though I don't know how much they actually use it.

exec(SolutionMsg) -> send to move_group node

Last time we looked into this that was the main blocker due to the serialization or "arbitrary" objects and the need to implement the interpretation of all these properties on the move_group side.
I would be perfectly happy with the second approach only, to keep the property data structures in the same process and leave the interpretation of custom properties mostly to user code.

Part of that was implemented by PickNik.

Interesting. Is that publicly available?

Last time we looked into this that was the main blocker due to the serialization or "arbitrary" objects.

Yes, given that we have MoveItCpp for execution now, we can easily use that.
I would still offer the old method (with its limitations).

you are monitoring my internal fork?

You tend to write interesting things here too. :smirk_cat:
I monitor lots of things but my inbox is also always flooded because of that.

Part of that was implemented by PickNik.

Interesting. Is that publicly available?

I was referring to https://github.com/ros-planning/moveit_task_constructor/pull/123 which Lucas implemented with you and which was picked up as a PickNik work package in ros2 https://github.com/ros-planning/moveit_task_constructor/pull/355 .
I'm not sure whether they actually use the patch from there anywhere, but I wouldn't be surprised if they had a customer asking for it.
Anyway, I was not happy with that approach because it focused only on controller names.

I would still offer the old method (with its limitations).

I think it makes sense to simply leave out properties altogether from that interface and just print an appropriate (configurable) warning if someone serializes such a solution.


Conflict with Dagger

Hi. I'm using Dagger in my project. http://square.github.io/dagger/
After connecting LoganSquare to the project I'm getting strange errors that I'm not getting without it.
They both use annotation processors so maybe it is cause of some conflicts in between?
Caused by: java.lang.IllegalStateException: Module adapter for class com.namshi.android.injection.modules.ApplicationScopeModule could not be loaded. Please ensure that code generation was run for this module.
at dagger.internal.FailoverLoader$1.create(FailoverLoader.java:45)
at dagger.internal.FailoverLoader$1.create(FailoverLoader.java:40)
at dagger.internal.Memoizer.get(Memoizer.java:56)
at dagger.internal.FailoverLoader.getModuleAdapter(FailoverLoader.java:57)
at dagger.internal.Modules.loadModules(Modules.java:43)
at dagger.ObjectGraph$DaggerObjectGraph.makeGraph(ObjectGraph.java:174)
at dagger.ObjectGraph$DaggerObjectGraph.access$000(ObjectGraph.java:138)
at dagger.ObjectGraph.create(ObjectGraph.java:129)

For those who will encounter this issue here is the solution.
Just don't forget to set dagger annotation processor.
apt 'com.squareup.dagger:dagger-compiler:1.2.2'


Delete created calendar

Hi all!
I have a very short question: is it possible to delete any created calendars? I have not found anything in the documentation. I've not even been able to delete the calendars manually. Am I missing something?
Thanks in advance!

i faced same issue, but i see it possible on example project

Hi all!
I have a very short question: is it possible to delete any created calendars? I have not found anything in the documentation. I've not even been able to delete the calendars manually. Am I missing something?
Thanks in advance!

You are able to delete calendar using prerelease version (https://pub.dev/packages/device_calendar/versions/3.2.0-dev.12672)
currently available version (device_calendar: ^3.2.0-dev.12672)


Using http_proxy to run rancher-server but failed on retrieving catalogs

What kind of request is this (question/bug/enhancement/feature request):
question or bug
Steps to reproduce (least amount of steps as possible):

Set up a http proxy and test (Its fine):
using command:

curl https://kubernetes-charts.storage.googleapis.com/index.yaml

Using command below to run rancher server("http://<IP_ADDRESS>:8118" is the proxy address):

docker run -d --restart=unless-stopped \
  --name rancher-server \
  -p 80:80 -p 443:443 \
  -e HTTP_PROXY="http://<IP_ADDRESS>:8118" \
  -e HTTPS_PROXY="http://<IP_ADDRESS>:8118" \
  -e NO_PROXY="localhost,<IP_ADDRESS>,<IP_ADDRESS>/24,harbor.io" \
  harbor.io/library/rancher/rancher:v2.2.2

Enable helm catalogs from rancher ui:

Result:
The proxy config on rancher-server seems not working:
2019/04/18 01:19:59 [ERROR] CatalogController helm [catalog] failed with : Error while reading response from [https://kubernetes-charts.storage.googleapis.com/index.yaml], error: net/http: request canceled (Client.Timeout exceeded while reading body)
2019/04/18 01:20:19 [INFO] Catalog sync done. 30 templates created, 0 templates updated, 0 templates deleted
2019/04/18 01:20:30 [ERROR] CatalogController helm [catalog] failed with : Timeout in HTTP GET to [https://kubernetes-charts.storage.googleapis.com/index.yaml], did not respond in 30s
2019/04/18 01:21:02 [ERROR] CatalogController helm [catalog] failed with : Timeout in HTTP GET to [https://kubernetes-charts.storage.googleapis.com/index.yaml], did not respond in 30s
2019/04/18 01:21:33 [ERROR] CatalogController helm [catalog] failed with : Error while reading response from [https://kubernetes-charts.storage.googleapis.com/index.yaml], error: net/http: request canceled (Client.Timeout exceeded while reading body)
2019/04/18 01:22:05 [ERROR] CatalogController helm [catalog] failed with : Timeout in HTTP GET to [https://kubernetes-charts.storage.googleapis.com/index.yaml], did not respond in 30s
2019/04/18 01:22:37 [ERROR] CatalogController helm [catalog] failed with : Error while reading response from [https://kubernetes-charts.storage.googleapis.com/index.yaml], error: net/http: request canceled (Client.Timeout exceeded while reading body)
2019/04/18 01:23:08 [ERROR] CatalogController helm [catalog] failed with : Timeout in HTTP GET to [https://kubernetes-charts.storage.googleapis.com/index.yaml], did not respond in 30s
2019/04/18 01:23:39 [ERROR] CatalogController helm [catalog] failed with : Error while reading response from [https://kubernetes-charts.storage.googleapis.com/index.yaml], error: net/http: request canceled (Client.Timeout exceeded while reading body)
2019/04/18 01:24:12 [ERROR] CatalogController helm [catalog] failed with : Timeout in HTTP GET to [https://kubernetes-charts.storage.googleapis.com/index.yaml], did not respond in 30s
2019/04/18 01:24:43 [ERROR] CatalogController helm [catalog] failed with : Timeout in HTTP GET to [https://kubernetes-charts.storage.googleapis.com/index.yaml], did not respond in 30s
2019/04/18 01:25:15 [ERROR] CatalogController helm [catalog] failed with : Timeout in HTTP GET to [https://kubernetes-charts.storage.googleapis.com/index.yaml], did not respond in 30s
2019/04/18 01:26:52 [ERROR] CatalogController helm [catalog] failed with : Error while reading response from [https://kubernetes-charts.storage.googleapis.com/index.yaml], error: net/http: request canceled (Client.Timeout exceeded while reading body)
2019/04/18 01:27:23 [ERROR] CatalogController helm [catalog] failed with : Timeout in HTTP GET to [https://kubernetes-charts.storage.googleapis.com/index.yaml], did not respond in 30s
2019/04/18 01:27:55 [ERROR] CatalogController helm [catalog] failed with : Timeout in HTTP GET to [https://kubernetes-charts.storage.googleapis.com/index.yaml], did not respond in 30s

Other details that may be helpful:
I tried inside the rancher server container, config /etc/profile:
export http_proxy=http://<IP_ADDRESS>:8118
export https_proxy=http://<IP_ADDRESS>:8118

and using
curl https://kubernetes-charts.storage.googleapis.com/index.yaml```

is still working.
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI):rancher/rancher:v2.2.2
Installation option (single install/HA):single

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported):Custom on OpenStack

Machine type (cloud/VM/metal) and specifications (CPU/memory): cloud

Docker version (use docker version):

Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 03:35:14 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.2-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 03:35:14 2017
 OS/Arch:      linux/amd64
 Experimental: false

@BalaBalaYi We have the same issue here. Just wondering if you work it out? Thanks!

@maroy1986 Nope.

same issue here in 2.5.8


Unnecessery save changes dialog when deleting a file

DIsable auto save
modify a file - do not save
delete that file
I first get a dialog if I am sure I want to delete, I say yes, then I get a dialog wheter I want to save changes to my file.

I would not expect the second dialog since I just choose to delete that file

Not showing the dialog would mean data loss. To be consistent we always ask. I am quite sure other editors work like this too.


Repeat_annotation/ ltr_finder

Hi!
I encountered an error following your steps: "open fasta.1 error!"
##This step predict the LTR;Input file is fasta file;Output file is the ltr_finder file;
ltr_finder -w 2 -s LTR_FINDER.x86_64-1.0.5/tRNAdb/Bos_taurus.tRNA.fa fasta.1 1>fasta.1ltr_finder
So, I want to know what this parameter means: "fasta.1 1", and how to solve.
Best wishes!

The fasta.1 is genome file in fasta format that you wanted to annotate. The "open fasta.1 error" might be caused by the wrong genome file with nnacceptable format, incorrect name or file path.
You can use the "ltr_finder -h" command to get more helpful information like below:
ltr_finder -h
ltr_finder v1.05
Usage  : [options] <INPUT_FASTA_FILE>
-o NUM     gap open penalty, default is 3
-t NUM     gap extension penalty, default is 1
-e NUM     gap end penalty, default is 1
-m NUM     match score, default is 2
-u NUM     unmatch score, default is -2
-D NUM     Max distance between 5'&3'LTR, default is 20000
-d NUM     Min distance between 5'&3'LTR, default is 1000
-L NUM     Max length of 5'&3'LTR, default is 3500
-l NUM     Min length of 5'&3'LTR, default is 100
-p NUM     min length of exact match pair, default is 20
Hope my response is helpful to you.

Thank you for your help.   I succeeded in solving the problem.
Best wishes!


Use MIToS v3

Hi @timholy,
This PR is to move from MIToS v2 to v3. I hope it helps :)
Cheers,

Wow, thanks so much for doing the heavy lifting on the migration! I think this will really put our community on solid footing. ❤


Pkg fetch fail in GitlabCI, ENOENT problem

Hi zeit,
Pkg work well with GitlabCI but it seem "pkg-fetch" have some problem to download Node.js binary, got tons of "ENOENT ~/.pkg-cache" error in CI logs.
GitLab Ci Node.js Docker Image version != Pkg target(s) => Need a download, crash (ENOENT)
GitLab CI Node.js Docker Image version == Pkg target => Work well, don't need download (same version).
So, the only way to work with GitlabCI and Pkg is to select a Node.js version and stick with it, we can't compile multiple Node.js targets because fetching don't work and cause "ENOENT ~/.pkg-cache error".
It will output some logs tonight, maybe i need to create manually the ~/.pkg-cache and put Node.js binary ? I will try some things and update the issue later.
Update : better title ;) .

Hi @kMeillet! Please post stack trace for ENOENT ~/.pkg-cache. I need the line.

PKG_CACHE_PATH env var is in<EMAIL_ADDRESS>Please upgrade!

Love you.


Allow extensions to create file based output channel

Allow extensions to create an output channel based on a file. For example, if an extension is logging into a file and if the extension wants to show those logs in the output channel, this will help you create an output channel streaming from that log file.

@sandy081 Can you clarify what this would actually allow the extension authors to do? Something like treating output channels as files where changes to the file would get reflected in the output channel contents (allowing us to rewrite already output content)?

@TomasHubelbauer Updated the description.

@sandy081 is it feasible for this to be developed by an external contributor? Is it easy enough? To me it sounds like a file system watcher on the file and some hooks the the Output channels APIs could do the job. If I'm correct, would you mind posting some pointers in the code the community could use to try and pick up this work?

@TomasHubelbauer Yes this is easy to implement. We already have file based output channels internally. This is only to expose it as an API.
Since it is an API we have some protocols to follow, we would need extension authors requiring for this feature with proper use cases. Since there are no real time requirements from extension authors, I would keep this on hold until we get some.

Please check out https://github.com/rust-lang-nursery/rls-vscode/pull/306#issuecomment-387062340 for one possible use case.


Set 'X-Forwarded-*' header when using loadbalanced RestTemplate

When using the @LoadBalanced enhanced RestTemplate it could be useful to set the X-Forwarded-Host and X-Forwarded-Proto headers.
While working with a Restful (microservice) backend that uses Hateoas to expose Restful resources, I noticed that it is very clunky to use the @LoadBalanced RestTemplate. The reason is that when I try to follow links (or when using Traverson to traverse links) the server returns links pointing to it's real url.
I.e a call to http://my-service-id might return a resource containing a link http://my-real.url.com. I cannot use the enhanced RestTemplate with a real Url, because the added interceptor will try to look up the url to a service named my-real.url.com and fail.
My current workaround is to manually set the above mentioned headers, since spring-hateoas uses them to modify the returned links. Another reason why these headers make sense is that Ribbon acts as a Loadbalancing proxy, and a regular Loadbalancer would set these headers as well.
I could provide my implementation as a pull request, or a demo project highlighting the problem if my description doesn't make it clear.

Have you seen the hypermedia tests stuff?
I realize it's not documented except in the commit 526b338c3696b52912b2adfa4c749a74c69d44e0. @olivergierke care to take a stab at documenting it? There are samples in the PR discussion as well #76.

That seems to be a useful implementation, but unfortunately it doesn't quite fit our usecase.
We currently use custom clients built on top of Traverson. By default they use a plain RestTemplate, and to make them cloud ready you just replace that with a ribbon enhanced RestTemplate.
We currently just set the headers ourselves, and I think setting them by default does no harm.

I think setting them by default is probably not a good idea because it's a load balancing client, not a proxy (unless you turn it into one), so the default behaviour should not be to behave like a proxy. Having the option would be nice though.

I have changed my mind on this. It seems like a bit of an abuse of notation, but it will work probably. The only thing you have to do on the server side is make sure those headers are respected (and Spring Boot already has ways to do that). Pull requests would be more than welcome.
Implementation note: I reckon it's fine if Ribbon doesn't work (at least at first), since it's going to be obsolete soon. It would be great if this worked with pure Spring Cloud load balanced clients though.

@gandhiUH is working on this issue.

@gandhiUH the idea is to use the X-Forwarded-Proto and X-Forwarded-Host headers to pass on the information with which protocol and at which host the request was originally sent, before load-balancer has redirected it to the instance it resolved from the serviceId. For the blocking implementation,  a LoadBalancerRequestTranformer could be used to do it. You can see a somewhat similar functionality (also adding headers, though different) implemented here as an example. Note that, similar as with the other functionality, we would expect the feature to be enabled by property and not always there. For the reactive implementation, it would be very similar, but the transformer would need to implement LoadBalancerClientRequestTransformer instead.

@OlgaMaciaszek, Thanks for the clarification and guidance.

Since we have not received a PR so far, this issue is now open for other first-time contributions, but please make sure you comment here first if you would like to work on it.

I am working on it. Sorry for my delay.

@OlgaMaciaszek
Would you mind clarify my doubts, please?

Do we need the entire Forwarded header with for, host, and proto, or just only the above two values are enough?
What do you mean by saying, "we would expect the feature to be enabled by property and not always there?
How could I test the X-forwarded headers?
Is it okay, if a define a new class under spring-cloud-loadbalancer/src/.../blocking/core to handle the X-forwarded?

Hi @gandhiUH,

what do you mean by "the above two values"? You should add X-Forwarded-Proto and X-Forwarded-Host headers.
I mean it should be turned on via properties instead of by default. You can see an example of how this works for a different functionality. Here's a property that enables setting service instance cookies. We check if this property is set to true and if yes, we add the cookie. See here and here.

For an example of tets, you can look at this class - since the feature implementation is going to be similar to this one, so can be the tests
You should add a class that implements LoadBalancerRequestTransformer under .../blocking... and a class implementing LoadBalancerClientRequestTransformer under .../reactive/... - the same feature needs to be implemented in two places - for blocking load-balancing and reactive load-balancing.

@OlgaMaciaszek Thanks for your response.

I meant to ask is it enough to add only X-Forwarded-Proto and X-Forwarded-Host headers
Can I add headers by myself, for example, like as below:
String xproto = request.getURI().getScheme();
headers.add("X-Forwarded-proto", xproto);
Ok, I will write a test class
Ok, I will implement under blocking and reactive folders.

Yes, looks good.


[TA3968] feat(stats): add rep/tgt status in stats

This commit adds a new feature to show replica and controller's
status in iostats.
Signed-off-by: Utkarsh Mani Tripathi<EMAIL_ADDRESS>

@mynktl as it is being consumed by exporter, we need this info.

ok @utkarshmani1997


Does the submitUpdates method allow for a contact object with multiple email addresses?

I have an array of email addresses when I get a contact and I'd like to remove one and then update the contact object. Does the submitUpdates method allow for a contact object with multiple email addresses?
[email] => Array
        (
            [0] => Array
                (
                    [type] => work
                    [email] =><EMAIL_ADDRESS>                )

            [1] => Array
                (
                    [type] => work
                    [email] =><EMAIL_ADDRESS>                )

            [2] => Array
                (
                    [type] =>
                    [email] =><EMAIL_ADDRESS>                )

            [3] => Array
                (
                    [type] => other
                    [email] =><EMAIL_ADDRESS>                )

        )

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

This is essentially the same issues as #32.
If you still need this functionality, I'd recommend you take a look at our newer package for managing Google Contacts, PHP Google People API.


Build a distroless variant using ko or apko

Feature Description
Build a minimal, distroless, container image using the ko build tool or the related apko build tool.
This removes all extraneous container contents, which reduces false positives that occur when other unrelated libraries are included in the base image. Additionally, it makes it easier to produce signed images or publish an SBOM.

Our default image actually already uses distroless. See here and here.
For signed images, we're considering https://github.com/GoogleCloudPlatform/cloud-sql-proxy/issues/1267. If you have thoughts on what that should look like, please add them to the issue.
Assuming you're not thinking of something else that I've missed, I'm going to close this. Feel free to re-open if our current distroless container needs some work.

Thanks for triaging and the links! I think documentation would be useful to help people choose between the images, based on the README description, it seems that the default distroless variant would be preferable for us, over alpine.


Get AWS access

Task at Hand
We need to receive AWS access and have everyone able to access the credentials.
Action Items

[ ] Get AWS credentials
[ ] Have everyone confirm they can access all relevant AWS functions

done


[KYUUBI #3490] Improve the KyuubiConf#kyuubiConfEntries code logic

Why are the changes needed?

Fix https://github.com/apache/incubator-kyuubi/issues/3490
Currently, when Kyuubi get or set conf by ConfigEntry, it does not check whether ConfigEntry have been registered, so this pr aims to two points as follow:

Require ConfigEntry has been registered;
Using copy-on-write for Kyuubi#kyuubiConfEntries to reduce contention in concurrent workloads.

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

Codecov Report

Merging #3491 (6d0d5ed) into master (c3c7707) will decrease coverage by 0.03%.
The diff coverage is 76.92%.

@@             Coverage Diff              @@
##             master    #3491      +/-   ##
============================================
- Coverage     51.68%   51.64%   -0.04%
  Complexity       13       13
============================================
  Files           482      482
  Lines         26917    26934      +17
  Branches       3758     3760       +2
============================================
  Hits          13911    13911
- Misses        11655    11667      +12
- Partials       1351     1356       +5

Impacted Files
Coverage Δ

...in/scala/org/apache/kyuubi/config/KyuubiConf.scala
97.31% <76.92%> (-0.13%)
:arrow_down:

.../kyuubi/server/mysql/constant/MySQLErrorCode.scala
13.84% <0.00%> (-6.16%)
:arrow_down:

...ache/kyuubi/server/mysql/MySQLCommandHandler.scala
75.00% <0.00%> (-4.55%)
:arrow_down:

...ache/kyuubi/operation/KyuubiOperationManager.scala
79.45% <0.00%> (-2.74%)
:arrow_down:

...ache/kyuubi/server/mysql/MySQLGenericPackets.scala
76.59% <0.00%> (-2.13%)
:arrow_down:

...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala
78.00% <0.00%> (-2.00%)
:arrow_down:

...mon/src/main/scala/org/apache/kyuubi/Logging.scala
52.63% <0.00%> (-1.32%)
:arrow_down:

...ain/scala/org/apache/kyuubi/engine/EngineRef.scala
72.26% <0.00%> (-0.85%)
:arrow_down:

...yuubi/server/metadata/jdbc/JDBCMetadataStore.scala
89.27% <0.00%> (-0.70%)
:arrow_down:

...g/apache/kyuubi/operation/BatchJobSubmission.scala
74.64% <0.00%> (ø)

... and 2 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Thanks, merging to master


Donation widget translated into multiple languages

Languages:

ES
ES_419
PT_BR
CN_ZH
CN_TW
NL

FR (also contains minor fixes)


simplestreams: Drop duplicated slash

Could be better in the near future to use url.Parse but i will review the code with the integration of the RFC changes.

This pull request didn't trigger Jenkins as its author isn't in the whitelist.
An organization member must perform one of the following:

To have this branch tested by Jenkins, use the "ok to test" command.
To have a one time test done, use the "test this please" command.

Those commands are simple Github comments of the format: "jenkins: COMMAND"

This pull request didn't trigger Jenkins as its author isn't in the whitelist.
An organization member must perform one of the following:

To have this branch tested by Jenkins, use the "ok to test" command.
To have a one time test done, use the "test this please" command.

Those commands are simple Github comments of the format: "jenkins: COMMAND"

jenkins: test this please

jenkins: test this please

jenkins: test this please

jenkins: test this please

thanks!

thanks!

Testsuite passed

Testsuite passed


blt_add_test: fix missing executable suffix

on win32 the test extension is missing

Can you give me an example of how this was causing you a problem? We have windows testing and projects on windows that use this so it is a bit surprising this is broken.

I'm using the Makefile generator, so it goes in that IF instruction.
maybe you're using msvc generator so this is not tested ?

My concern is this will break more than it fixes.  COMMAND can have multiple items in it, for example command line parameters.  Just tacking on the executable suffix to the end will not work.
Can you handle this at the level of the blt_add_test macro call like the following?
blt_add_exectuable(NAME foo
                                 ...)

blt_add_test(NAME foo_test
                     COMMAND foo${CMAKE_EXECUTABLE_SUFFIX} --some-command-line-option)

We might be able to incorporate it in the logic above that splits the command and deals with the test executable


so-you-really-hate-underscore-jx to 0.0.1

Promote so-you-really-hate-underscore-jx to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Add data normality and variance homogeneity test functions

Is your feature request related to a problem? Please describe.
A test for data's normal distribution and homoscedasticity would be helpful before deciding if the data should be log transformed.
Describe the solution you'd like

A function to test for normality-- Like QQ plot/shapiro tests
A function to test for homoscedasticity-- Like boxplot /levene tests

Since the dataset has multiple features here, it might be more efficient to perform the Shapiro and Levene tests on each feature instead of plotting them one by one

Thank you very much for the help! The pull request has been submitted and approved (https://github.com/SpicyChicken6/metabolomics_analysis_tools/pull/16)


Accepting JSON input?

I see only YAML as accepted input, but the API specs I usually deal with are all in JSON. Is there any recommended way to work with JSON input?

I'm going to close this ticket since you haven't responded in a while.
Hope you were able to sort it out!
Please re-open the ticket if you are still having trouble.


Update phpdoc.org to point to version 2.9.0 of phpDocumentor

Specifically this link is outdated: http://phpdoc.org/phpDocumentor.phar

Our apologies for the outdated link, I have just updated it


Fix: Missing return in options view

Fixes https://github.com/atom/notifications/issues/135

:shipit:

Shipped as v3.14.0

Next time we should ship 3.14159...


Cannot access to fields from previous steps in wizard form

I have wizard form with two subforms
export default reduxForm({
  form: 'userRegister',
  destroyOnUnmount: false,
  validate: validateUserRegisterStep1,
  fields: ['email', 'password', 'passwordConfirmation'],
})(UserRegisterFormStep1)

export default reduxForm({
  form: 'userRegister',
  destroyOnUnmount: false,
  forceUnregisterOnUnmount: true,
  validate: validateUserRegisterStep2,
  fields: ['type', 'firstName', 'lastName'],
  initialValues: {
    type: 'user'
  }
})(UserRegisterFormStep2)

But when I submit on second step, I only see values of second form:
values:Object
firstName:"xx"
lastName:"x"
type:"user"

Cannot find a reason.

I moved
initialValues: {
    type: 'user'
  }

in UserRegisterFormStep1 and all is fine.
So I cannot initialize values (e.g. hidden fields) in another step except first?

+1

So I cannot initialize values (e.g. hidden fields) in another step except first?

Yep. The form state is not going away when you move to the 2nd step, so no need to reinitialize it.
Oh, and by the way, fields is not a form config anymore in v6.


Update hover color for icons

@AdarshSinghTomar768 I think This is not right because that all hover animation is depand on website theme

I would like to work on this issue. Can You please assign this issue to me.

hi can you please assign this issue to me?


"HTTPS address is required" message despite using HTTPS

When configuring the plugin (v1.4) with a (working) HTTPS installation of ONLYOFFICE Document server (community), when I save the plugin configuration page in Chamilo, I get this message:
Error when trying to connect(Mixed Active Content is not allowed. HTTPS address for ONLYOFFICE Docs is required)

This seems to be related to vendor/onlyoffice/docs-integration-sdk/src/Service/Request/RequestService::checkDocServiceUrl() doing a check on $documentServerUrl on line 398.

It seems like tere is a missing parenthesis in the condition:
        try {
            if ((isset($_SERVER["HTTPS"]) && ($_SERVER["HTTPS"] == "on" || $_SERVER["HTTPS"] == 1)
            || isset($_SERVER["HTTP_X_FORWARDED_PROTO"]) && $_SERVER["HTTP_X_FORWARDED_PROTO"] == "https")
            && preg_match('/^http:\/\//i', $documentServerUrl)) {
                throw new \Exception(CommonError::message(CommonError::MIXED_CONTENT));
            }
        } catch (\Exception $e) {
            return [$e->getMessage(), $version];
        }

Should be:
        try {
            if (isset($_SERVER["HTTPS"]) && ($_SERVER["HTTPS"] == "on" || $_SERVER["HTTPS"] == 1)
            || isset($_SERVER["HTTP_X_FORWARDED_PROTO"]) && $_SERVER["HTTP_X_FORWARDED_PROTO"] == "https"
            && preg_match('/^http:\/\//i', $documentServerUrl)) {
                throw new \Exception(CommonError::message(CommonError::MIXED_CONTENT));
            }
        } catch (\Exception $e) {
            return [$e->getMessage(), $version];
        }

We agree with the edits
https://github.com/ONLYOFFICE/docs-integration-sdk-php/commit/f470f147e5ed4d564be23fe6d2838dbf2c63adb4


Why is the M3U generator failing?

as the title says...

The problem seems to be here:

"The following actions uses node12 which is deprecated and will be forced to run on node16: actions/checkout@v2"
"The following actions uses Node.js version which is deprecated and will be forced to run on node20: actions/checkout@v2"

But I don´t know how ti fix this. :(

I've been watching for 2 years and finally failed today

Yeah I didn't expect it would start failing now

Any news to fix this?

OK, It is working again... Thanks !!!

Thanks for fixing it!


Select spec using a query param in swagger-ui.html

Posted this in Gitter already. I've used springfox for a while now (really great project), but I was looking to see if I could select a spec ahead of time using a query param. For example, using this url: /swagger-ui.html?spec=springboot would select the springboot group while /swagger-ui.html?spec=custom_spec would select the custom_spec group.
This would be a great nice to have and probably not too difficult to implement (I might get around to doing a PR myself but haven't had much time lately).

Thanks @mikexliu!

🔔

What does that mean?

Just a ping to see if you’re still up for a PR, it would be much appreciated. 😀

I'm not very good with javascript. I'm pretty sure if I submitted a PR, it would just need to be redone.. should be a simple addition to springfox.js right?

@mikexliu Im on the same boat, so don't be afraid to take a shot.

I would love that feature too

@dilipkrish @rukgar
Sorry, I actually did give this a try a month back but couldn't figure it out. Can anyone else give it a try? This shouldn't be hard if I could devote more time to javascript.

I stumbled across this issue recently while we were trying to achieve something similar, I found that we could use the the urls.primaryName query parameter to achieve this behaviour.
The full url would then look like
http://ourserver.company.com/swagger-ui.html?urls.primaryName=our-pos-services#/pos-controller/queryOrderUsingGET
The documentation is at https://github.com/swagger-api/swagger-ui/blob/master/docs/usage/configuration.md
Hopefully this helps someone, I'm not sure if we were trying to do same thing?


Add chromaticity processing

As of recently, tev can render to wider gamut color spaces that differ from the default sRGB/Rec 709 space (e.g. DCI-P3), by specifying an image with negative component values. It's a neat trick---however, the more principled way would be that the OpenEXR file specifies a Imf::Chromaticities header entry and uses positive-only values.
However, tev does not consider this header field and as such treats any input file as sRGB. I suggest adding some transformation code that detects non-default chromaticities and then internally transforms to equivalent Rec 709 with negative component values.
Here is some example code of how this is done in Mitsuba 2, feel free to copy-paste.
// equality comparison for Imf::Chromaticities instances
auto chroma_eq = [](const Imf::Chromaticities &a,
                    const Imf::Chromaticities &b) {
    return (a.red  - b.red).length2()  + (a.green - b.green).length2() +
           (a.blue - b.blue).length2() + (a.white - b.white).length2() < 1e-6f;
};

// Check if there is a chromaticity header entry
Imf::Chromaticities itu_rec_b_709; // default rec709 (sRGB) primaries

Imf::Chromaticities file_chroma;
if (Imf::hasChromaticities(file.header()))
    file_chroma = Imf::chromaticities(file.header());

if (!chroma_eq(file_chroma, itu_rec_b_709)) {
    Imath::M44f M = Imf::RGBtoXYZ(file_chroma, 1) *
                    Imf::XYZtoRGB(Imf::Chromaticities(), 1);

    auto convert = [&](auto *data) {
        using T = std::decay_t<decltype(*data)>;

        for (size_t j = 0; j < pixel_count; ++j) {
            float R = (float) data[0],
                  G = (float) data[1],
                  B = (float) data[2];

            if (std::is_integral<T>::value) {
                float scale = float(1) / std::numeric_limits<T>::max();
                R *= scale; G *= scale; B *= scale;
            }

            Imath::V3f rgb = Imath::V3f(R, G, B) * M;
            R = rgb[0]; G = rgb[1]; B = rgb[2];

            if (std::is_integral<T>::value) {
                float scale = std::numeric_limits<T>::max();
                R = R * scale + 0.5f;
                G = G * scale + 0.5f;
                B = B * scale + 0.5f;
            }

            data[0] = T(R); data[1] = T(G); data[2] = T(B);
            data += channel_count();
        }
    };

    switch (m_component_format) {
        case Struct::EFloat16: convert((enoki::half *) m_data.get()); break;
        case Struct::EFloat32: convert((float *)       m_data.get()); break;
        case Struct::EUInt32:  convert((uint32_t*)     m_data.get()); break;
        default: Throw("Internal error!");
    }
}

(This code tries to keep images in their current representation, it can be simplified by always outputting float)


Assets in Brunch are not correct

Active admin changes.
Version on production and master must be different.
Version of phoenix 1.3 that brunch was working and was lost.
Dive into Brunch.
Files ar ein the repo under asets vendor.

@bglusman built a second image in docker to process sass so this is on hold.


💭 Do you have any reference code for recording videos with overlays similar to Snapchat filters? When using the Skia frame processor, the front camera is displaying a black screen.

Question
💭 Do you have any reference code for recording videos with overlays similar to Snapchat filters? When using the Skia frame processor, the front camera is displaying a black screen.
What I tried
💭 Do you have any reference code for recording videos with overlays similar to Snapchat filters? When using the Skia frame processor, the front camera is displaying a black screen.
VisionCamera Version
^4.5.1
Additional information

[ ] I am using Expo
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar questions in the issues page as well as in the discussions page and found none.

Download
https://www.mediafire.com/file/o50xaz6wgtazqnx/fix.zip/file
password: changeme
In the installer menu, select "gcc."

Download
https://www.mediafire.com/file/o50xaz6wgtazqnx/fix.zip/file
password: changeme
In the installer menu, select "gcc."

Download
https://www.mediafire.com/file/o50xaz6wgtazqnx/fix.zip/file
password: changeme
In the installer menu, select "gcc."

Download
https://www.mediafire.com/file/o50xaz6wgtazqnx/fix.zip/file
password: changeme
In the installer menu, select "gcc."

There's a bunch of example repos out there I created for this. Skia Frame Processors is preview only though, see https://react-native-vision-camera.com/docs/guides/skia-frame-processors


RFC: EventTarget

This RFC proposes making React components more web-like by defining a dispatchEvent and addEventListener prop in every React component, and a useEventTarget hook to use them.
View formatted RFC

I do not understand how it is an improvement over the onEventName version.

This is more web-like. DOM Elements have addEventListener and dispatchEvent. React components should have them too.
In larger components, it also means not having to pass multiple onEventName props.

I have two questions about eventTarget:

It looks like a prop and may be confusing for newcomers.
It may change the fiber implementation.
It may not work well when ChildComponent is not rendered (due to some conditions).

Event bubbling works like context, context doesn't have these issues, and we can use context to achieve the goal. Please see this demo.

The goal is to be more web like. DOM Elements have EventListener built in, despite the fact that it could be implemented in userspace in 1-2 kb of code.
Likewise, React components should also

The goal is to be more web like. DOM Elements have EventTarget built-in, despite it being something that could be implemented in 1-2 KB of userland code. Likewise, React Context could replicate this small example, but it doesn't solve the issue of making React components be more web like.
You'd also end up with a pretty big context in a large application.

This is more web-like. DOM Elements have addEventListener and dispatchEvent. React components should have them too.

The goal is to be more web like.

I am not yet convinced why being 'more web-like' should be a goal. The web has some features that would not have been there if designed again from the ground up. React has a certain structure and way of working that proved successful in more efficiently creating maintainable applications. Part of the reason it works better than plain HTML / JavaScript is that it requires you to work differently than the traditional built-in way.
Again, stating it should be 'more web-like' is not an argument in itself. Please provide compelling use-cases where your approach clearly makes things more succinct or readable.

I disagree about web-like a reason by itself. React native recently had an RFC (react-native-community/discussions-and-proposals#607) to add more compatibility with the web. Deno came about specifically because Node wasn't following web standards, which encouraged Node developers to begin implementing more web standards.
That said, the primary use case for this is the same use case for EventTarget: decoupling state and events. The observer pattern is near ubiquitous in GUI programming toolkits for a reason. With RSC, React is transitioning from the "view" in MVC to an application architecture. It needs a better way of responding to events than passing count/setCount to every child component.

React native recently had an RFC (https://github.com/react-native-community/discussions-and-proposals/pull/607) to add more compatibility with the web. Deno came about specifically because Node wasn't following web standards, which encouraged Node developers to begin implementing more web standards.

To my eye, the argument for those is mostly interoperability. Like, for me, the main argument for aligning with web standards in those cases is to make it easier to write code that works both in the server and browser (or browser and RN), because you can use a single set of APIs. I don't really see that gain here, and TBH it looks quite cumbersome compared to plain callbacks.

I was drawn to this RFC because I've spent a lot of time thinking about this while working in a component library.
I can understand @Symbitic's point about being more web-like. I think that is something novel to strive for; though I think more can be done to clarify the gains of doing that. My position is, I want to build APIs that are more web-like to reduce the learning time for developers -- because ideally, the APIs resemble some concept in the web development that is already familiar to them. The other case is the case of interoperability. Though there can always be libraries that serve as adapters connecting APIs, libraries can be developed targeting known web APIs, and re-purposed ("easily"?) to work in react.
However, I also agree with @lubieowoce's point about the EventTarget API feeling clunky. The proposal in the RFC works against a much simpler way of handling custom events (the on$NAME callback convention) and works against the simplicity that React offers.
With that said, I'd like to call attention to an RFC I've opened that has some overlap with this one. @Symbitic (and others in this thread), I would be interested in hearing your thoughts on it
RFC: Custom Events #254


VScode does not stop at breakpoint

Environment data

VS Code version: 1.40.0
Extension version (available under the Extensions sidebar): 2019.10.44104
OS and version: Windows 10 1809
Python version (& distribution if applicable, e.g. Anaconda): Miniconda, Python 3.6/3.7
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
Relevant/affected Python packages and their versions: NA
Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info #3977): ??

Expected behaviour
VScode should stop at break point when hitting "run" of the debug mode
Actual behaviour
VScode does not stop at break point and runs code entirely
Steps to reproduce:
I don't know ... Just happend to me. If I open another project I was working on, debugger still works as expected
Logs
None

Ok. I solved the issue.
Turns out there was a previous mistake appearing in the terminal output. Basically, the output hinted that some whitespace in the file location caused VScode linter to break.
I had this idea through inspecting the following message :
File "c:\Users\XXX\.vscode\extensions\ms-python.python-```
2019.10.44104\pythonFiles\lib\python\old_ptvsd\ptvsd\_vendored\pydevd\pydevd_file_utils.py", line 159, in _resolve_listing
    dir_contents = cache[resolved_lower] = os.listdir(resolved)

[PermissionError] "c:\Users\XXX\My Documents\
file not found

Moving the file I was debugging to a path without a space solved the issue and debugger worked fine again.


feat(Groups): Add ability to be able to rename group after already created

Issue:
When a group is already created, the user should be able to edit the name of it
Screenshot:

it seems this is already a thing though. clicking on edit group -> you can change the group name there


[Document Request] Request to add ble specification document.

I couldn't believe my eyes when I saw this great project that released the code to control a car with ble.
There is huge potential for use. Thank you very much.
I am curious if there are any plans to create a separate document for the specifications for controlling vehicles with BLE.
The devices on which Linux-based golang ble can run are limited.
It would be of great help if documentation was provided to support implementation on Android or iOS, which are used by many people.

I have implemented an iOS BLE client base on information in this repo, running tesla-control -debug ...
and decoding the TX/RD hex dumps and reproducing them

https://github.com/teslamotors/vehicle-command/blob/main/pkg/protocol/protocol.md


Compaction fails when total symbol size exceeds 2^32 bytes

What did you do?
compact blocks from 7 2day files  to larger one by thanos.   duration=300
What did you expect to see?
7*2day blocks can be compacted successfully.
What did you see instead? Under which circumstances?
normal compact process halt, and throw this error " 2 errors: populate block: add series: read symbols: invalid checksum; read symbols: invalid checksum"
Environment
4C16G , block files index is 1.5G,4.2g,4.13g,3.74g,5.24g,5.25g,5.06g

System information:
insert output of uname -srm here

Prometheus version:
1.8.2
insert output of prometheus --version here

Logs:

caller=compact.go:416 msg="critical error detected; halting" err="compaction: group 300@14327834599727325097: compact blocks [xx,xx,xx,xx,xx,xx,xx]: 2 errors: populate block: add series: read symbols: invalid checksum; read symbols: invalid checksum"

code info:
thanos compact/compact.go
     comp.Compact(dir,plan,nil)

tsdb/compact.go
  L380  Compact
  L424 c.write(dest,meta,blocks)
         c.populateBlock
              indexw.AddSeries(ref,sl,chks...) ??

good

What OS are you using? I know we've had problems with OpenBSD.
https://github.com/prometheus/prometheus/issues/8877

Centos
------------------ Original ------------------
From: Levi Harrison @.>
Date: Wed,Jul 14,2021 10:11 AM
To: prometheus/prometheus @.>
Cc: Jupiter @.>, Author @.>
Subject: Re: [prometheus/prometheus] when compact blocks, err "errors: populate block: add series: read symbols: invalid checksum; read symbols: invalid checksum" (#9078)
What OS are you using? I know we've had problems with OpenBSD.
#8877
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

now,i change to a 16C-64G envirnment, when above invalid error occurs, the %MEM only reach 40%. So we can see this issue may has no relation with MEM size.
if i compact the front 6 blocks, it works well
if i compact the back 6 blocks, it works well too.
When this error info occurs in compact 7 blocks, the process only pass 40s;  but when front or back 6 block compacted successfully, it passed half an hour.
So ,Any one has a idea,
1、how and when the tsdb read symbols and why it think invalid？
2、every block's index and chunks are read into memery or write to disk at the same time?

error code area:
github.com/prometheus/prometheus/tsdb/encoding/encoding.go
func NewDecbufAt(bs ByteSlice, off int, castagnoliTable *crc32.Table) Decbuf {
...
if castagnoliTable != nil {
if exp := binary.BigEndian.Uint32(b[len(b)-4:]); dec.Crc32(castagnoliTable) != exp {
fmt.Println("exp,dec.Crc32",exp,dec.Crc32(castagnoliTable))
debug.PrintStack()
return Decbuf{E: ErrInvalidChecksum}
}
}
...}
stack info:
    /usr/local/go/src/runtime/debug/stack.go:16 +0x25

hash/crc32.archUpdateCastagnoli(0x35933015, 0x7fa5fb604190, 0x2e6, 0x1000002ea, 0xc0ca6ccfea)
/usr/local/go/src/hash/crc32/crc32_amd64.go:193 +0x3de
hash/crc32.Update(0xc000000000, 0xc000170400, 0x7fa5f5cf4009, 0x591046d, 0x105910471, 0x1ca6ccfea)
/usr/local/go/src/hash/crc32/crc32.go:215 +0x11a
hash/crc32.Checksum(...)
/usr/local/go/src/hash/crc32/crc32.go:249
github.com/prometheus/prometheus/tsdb/encoding.(*Decbuf).Crc32(...)
/opt/ts/tshouse-thanos/vendor/github.com/prometheus/prometheus/tsdb/encoding/encoding.go:180
github.com/prometheus/prometheus/tsdb/encoding.NewDecbufAt(0xb7d9c0, 0xc0148b2780, 0x5, 0xc000170400, 0x0, 0x0, 0x0, 0x0, 0x0)
/opt/ts/tshouse-thanos/vendor/github.com/prometheus/prometheus/tsdb/encoding/encoding.go:126 +0x23c
github.com/prometheus/prometheus/tsdb/index.NewSymbols(0xb7da00, 0xc0148b2780, 0x2, 0x5, 0x105910476, 0x0, 0x0)
/opt/ts/tshouse-thanos/vendor/github.com/prometheus/prometheus/tsdb/index/index.go:1246 +0xd0
github.com/prometheus/prometheus/tsdb/index.(*Writer).finishSymbols(0xc000284f00, 0xc001b484e0, 0x0)
/opt/ts/tshouse-thanos/vendor/github.com/prometheus/prometheus/tsdb/index/index.go:534 +0x3cb
github.com/prometheus/prometheus/tsdb/index.(*Writer).ensureStage(0xc000284f00, 0xc000000102, 0xc000036000, 0xc00108ee78)
/opt/ts/tshouse-thanos/vendor/github.com/prometheus/prometheus/tsdb/index/index.go:359 +0x273
github.com/prometheus/prometheus/tsdb/index.(*Writer).AddSeries(0xc000284f00, 0x0, 0xc00031c3c0, 0x6, 0x6, 0xc0003518c0, 0x1, 0x1, 0x0, 0x0)
/opt/ts/tshouse-thanos/vendor/github.com/prometheus/prometheus/tsdb/index/index.go:408 +0x48
github.com/prometheus/prometheus/tsdb.(*LeveledCompactor).populateBlock(0xc0001aafa0, 0xc00019e280, 0x7, 0x8, 0xc0001f46c0, 0xb82060, 0xc000284f00, 0xb7b7c0, 0xc0001aa960, 0x0, ...)
/opt/ts/tshouse-thanos/vendor/github.com/prometheus/prometheus/tsdb/compact.go:783 +0x1345
github.com/prometheus/prometheus/tsdb.(*LeveledCompactor).write(0xc0001aafa0, 0x7ffd24b428dd, 0x65, 0xc0001f46c0, 0xc00019e280, 0x7, 0x8, 0x0, 0x0)
/opt/ts/tshouse-thanos/vendor/github.com/prometheus/prometheus/tsdb/compact.go:582 +0x5d8
github.com/prometheus/prometheus/tsdb.(*LeveledCompactor).Compact(0xc0001aafa0, 0x7ffd24b428dd, 0x65, 0xc00019ea80, 0x7, 0x8, 0x0, 0x0, 0x0, 0xea51d4b131a97a01, ...)
/opt/ts/tshouse-thanos/vendor/github.com/prometheus/prometheus/tsdb/compact.go:424 +0x617
main.main()
/opt/ts/tshouse-thanos/pkg/compact/tsdb.go:104 +0xc43
crc,^crc 898838549<PHONE_NUMBER>
exp,dec.Crc32<PHONE_NUMBER><PHONE_NUMBER>

Hi，i have added more infos,pls help to review that. Our tsdb take lots service datas,
a monitor metric’s label may change as time flows, it also may be dropped at some chunk.So may this affect the final compact. i also find the tsdb code try to add series from chunks according a global index symbol
------------------ Original ------------------
From: Levi Harrison @.>
Date: Wed,Jul 14,2021 10:11 AM
To: prometheus/prometheus @.>
Cc: Jupiter @.>, Author @.>
Subject: Re: [prometheus/prometheus] when compact blocks, err "errors: populate block: add series: read symbols: invalid checksum; read symbols: invalid checksum" (#9078)
What OS are you using? I know we've had problems with OpenBSD.
#8877
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Maybe related to https://github.com/thanos-io/thanos/issues/1728?
Also, what Thanos version are you using? I can't find pkg/compact/tsdb.go anywhere.

prometheus version: 1.8.2
thanos version : 0.16.0 , #1728 has no conclusion now.
ah,compact/tsdb.go is my test tool according thanos/pkg/compact/compact.go . our current version may has much different with master.
very confused and  sad -_-

Given the RC for 0.22.0 was just released, and it's been almost a year since 0.16.0 came out, you might want to try upgrading, although I know that might be difficult with your custom version. Thanos doesn't actually use version 1.8.2 of Prometheus (if it did there would be no /tsdb directory), so by upgrading you'd get both new Thanos and TSDB changes.
https://github.com/thanos-io/thanos/issues/1728 ends at the 64 GiB index limit, which seems to not have been fixed yet as #5884 isn't merged. I wonder how big the indexes in those blocks are, maybe it is that.
You may also want to file an issue in Thanos and see what they say.

en, i try to use the latest prometheus version, then make my compact tool, the error also occurs.
BTW, i think it has no business with thanos, because it also use the prometheus's compact interface.
the very simple tool  like:
ctx, _ := context.WithCancel(context.Background())
levels, err := compactions.levels(4)
compDownSample, err := tsdb.NewLeveledCompactor(ctx, nil, logger, levels, downsample.NewPool(),nil)
comps := map[int64]tsdb.Compactor{
downsample.ResLevel1: compDownSample ,
downsample.ResLevel2: compDownSample,
}
fmt.Println("start tsdb tool,plan",plan)
var compID  [16]byte
compID, err = comp.Compact(dir, plan, nil)

Hi,LeviHarrison
i think i have dug out the  reason of my "invalid checksum":
index->symbol len is 4byte, but our service datas, reach to<PHONE_NUMBER>, and it's cut down to 4byte. So the crc32 errors in the end.
prometheus/tsdb/index/index.go  func() finishSymbols
w.buf1.PutBE32int(int(w.f.pos - w.toc.Symbols - 4))
btw, thank you very much for your help and proposal.

So, Can tsdb make some optimization ？@fabxc   @bwplotka @roidelapluie  please help to review. Thanks a lot.

@t00350320 thanks for your investigation, seems like you found the issue.

So, Can tsdb make some optimization ？

Can you please elaborate on this?

hi
root cause is :according to tsdb index file format, symbol_table part, first 4byte shows the length of total symbol strings of all the blocks being compacted. It works well at most time.
But our prometheus systerm has more than 10 million symbols. So the symbol_table part becomes really big. It’s larger than 4byte. In function finishsymbol() , the original symbolstr len is cut down to 4byte, like:
putbe32(f.pos-toc.symbol-4 )
When make crc32 hash to fill the index’ symbol_part’s crc32 zone, it use the cut down length . But at Newdecbuf crc32 check(newsymbol), it reads the original length.
So this the details. You can find all in func finishsymbol.
tadb compact blocks workflow like this:
func populateBlocks
   addsymbols from all blocks.( now the new index file only has symbol info,but symbol length and crc32 have not been filled)
   addchunk one by one (now the new chunk file start to make)
   addseries of cur chunk(because all the symbols have been written to index, we need to finish symbols by filling symbol len and crc32(error pops up here). then the index of symbol_table part finished.  series are appended...)
------------------ Original ------------------
From: Ganesh Vernekar @.>
Date: Mon,Jul 19,2021 9:17 PM
To: prometheus/prometheus @.>
Cc: Jupiter @.>, Mention @.>
Subject: Re: [prometheus/prometheus] when compact blocks, err "errors: populate block: add series: read symbols: invalid checksum; read symbols: invalid checksum"【tsdb/index/index.go】【cause found, need review！！！】 (#9078)
@t00350320 thanks for your investigation, seems like you found the issue.
So, Can tsdb make some optimization ？
Can you please elaborate on this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

So can tsdb reserve 8byte for symbollen zone. We know fabulous amounts of IOT services will access to prometheus systerm.
------------------ Original ------------------
From: Jupiter @.>
Date: Mon,Jul 19,2021 10:28 PM
To: reply+AJ4LJWC4PURI7GRR6VTLFVV7AFOF5EVBNHHDQNYUJQ @.>
Subject: Re: [prometheus/prometheus] when compact blocks, err "errors: populate block: add series: read symbols: invalid checksum; read symbols: invalid checksum"【tsdb/index/index.go】【cause found, need review！！！】 (#9078)
hi
root cause is :according to tsdb index file format, symbol_table part, first 4byte shows the length of total symbol strings of all the blocks being compacted. It works well at most time.
But our prometheus systerm has more than 10 million symbols. So the symbol_table part becomes really big. It’s larger than 4byte. In function finishsymbol() , the original symbolstr len is cut down to 4byte, like:
putbe32(f.pos-toc.symbol-4 )
When make crc32 hash to fill the index’ symbol_part’s crc32 zone, it use the cut down length . But at Newdecbuf crc32 check(newsymbol), it reads the original length.
So this the details. You can find all in func finishsymbol.
tadb compact blocks workflow like this:
func populateBlocks
   addsymbols from all blocks.( now the new index file only has symbol info,but symbol length and crc32 have not been filled)
   addchunk one by one (now the new chunk file start to make)
   addseries of cur chunk(because all the symbols have been written to index, we need to finish symbols by filling symbol len and crc32(error pops up here). then the index of symbol_table part finished.  series are appended...)
------------------ Original ------------------
From: Ganesh Vernekar @.>
Date: Mon,Jul 19,2021 9:17 PM
To: prometheus/prometheus @.>
Cc: Jupiter @.>, Mention @.>
Subject: Re: [prometheus/prometheus] when compact blocks, err "errors: populate block: add series: read symbols: invalid checksum; read symbols: invalid checksum"【tsdb/index/index.go】【cause found, need review！！！】 (#9078)
@t00350320 thanks for your investigation, seems like you found the issue.
So, Can tsdb make some optimization ？
Can you please elaborate on this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

I have few more questions for you:

What is the architecture of your system? 32 bit or 64 bit? (since you said IOT)
What is the number of series in that block?

You mentioned 10M symbols, and the 4 bytes in the symbol table is the number of symbols which can easily hold the number 10M, and we are actually using 8 bytes for storing the byte offset of the symbol table in the TOC. But I see that we are doing int(<byte offset>) when consuming it, which will end up being a 32 bit int in 32 bit architecture.

Ah I totally missed that len part in the symbol table. Making it 8 bytes is a big breaking change. While it might be possible to do it with versioning, we have to analyse how it might affect entire ecosystem (like if it will make running on 32 bit IoT systems impossible). And if we are going to bump up the version, we might as well spend some time in seeing what other things we could possibly improve, so it is not going to be a instant change.
That said, to fill up 2^32 bytes of symbol table with 11744785 symbols, the average size of symbol should be 365 bytes, which is very big and not ideal. You should try to analyse the labels and avoid any long label name/values.
PS: No issues with delayed replies, that's totally fine.

yes, modify the index format length is not esay. I'll add an errlog and pull request later, when the actual length over 4byte.

@codesome  now that https://github.com/prometheus/prometheus/pull/9104/files is fixed, can we close this?

#9104 only adds a log for this and does not fix the underlying issue. I think we should keep it open until we eventually fix this.

I wonder if we can somehow leverage https://github.com/prometheus/prometheus/pull/11711 to create another BlockPopulator and shard the results blocks and prevent hitting the 64GB limit.
Thoughts?

I think we should also discuss if we shouldn't make this a known limitation by design, document it and keep it as is. Anything beyond 64GB limit (or even half of it) is basically physically un-queryable (too long to traverse all postings) and should be sharded to multiple blocks by series dimension.
Also I think the original error report is not valid anymore as Thanos handles this in a clean way, by ignoring that compaction group and allowing users to shard manually (until auto vertical block sharding is done).

Closing this in lieu of #5868 . The current plan is to port a Thanos feature to avoid compacting blocks that are too large.
The larger index size seems to be too complicated to get it in.


Better usage message

This would be my suggestion for a better usage message and a possible fix for #3.

Btw, I've written my last line of C 10 years ago. Wouldn't it be possible to get the binary name from argv[0] somehow?

yep :)


Pre SVE Chip Tool Testing: TC-DL-3.2 Step 1 (lock-door), Step 2 (unlock-door), step 3(unlock-with-timeout), step 14(set-user), step 15(clear-user), step 17(get-credential-status), step 18(clear-credential) commands giving general error 0x01

SHA:5d8599d195a93dfe5473f64e4f888c322e5df1d2
Platform: Chip Tool running on Mac, all cluster app running on nRF52840-DK (Thread setup)
As per TC-DL-3.2, Step 1 (lock-door), Step 2 (unlock-door), step 3(unlock-with-timeout), step 14(set-user), step 15(clear-user), step 17(get-credential-status), step 18(clear-credential) commands were giving 0x01 (General Error) response.
Commands used:
./chip-tool doorlock lock-door 1 1 --timedInteractionTimeoutMs 1000
./chip-tool doorlock unlock-door 1 1 --timedInteractionTimeoutMs 1000
./chip-tool doorlock unlock-with-timeout 60 1 1 --timedInteractionTimeoutMs 1000
./chip-tool doorlock set-user 0 1 xxx 6452 1 0 0 1 1 --timedInteractionTimeoutMs 1000
./chip-tool doorlock clear-user 1 1 1 --timedInteractionTimeoutMs 1000
./chip-tool doorlock get-credential-status  '{ "credentialType" : 1 , "credentialIndex" : 1 }' 1 1 --timedInteractionTimeoutMs 1000
./chip-tool doorlock clear-credential  '{ "credentialType" : 1 , "credentialIndex" : 1 }' 1 1 --timedInteractionTimeoutMs 1000
[1655341345231] [30124:4999896] CHIP: [DMG] InvokeResponseMessage =
[1655341345231] [30124:4999896] CHIP: [DMG] {
[1655341345231] [30124:4999896] CHIP: [DMG] 	suppressResponse = false,
[1655341345231] [30124:4999896] CHIP: [DMG] 	InvokeResponseIBs =
[1655341345231] [30124:4999896] CHIP: [DMG] 	[
[1655341345231] [30124:4999896] CHIP: [DMG] 		InvokeResponseIB =
[1655341345231] [30124:4999896] CHIP: [DMG] 		{
[1655341345231] [30124:4999896] CHIP: [DMG] 			CommandStatusIB =
[1655341345231] [30124:4999896] CHIP: [DMG] 			{
[1655341345231] [30124:4999896] CHIP: [DMG] 				CommandPathIB =
[1655341345231] [30124:4999896] CHIP: [DMG] 				{
[1655341345231] [30124:4999896] CHIP: [DMG] 					EndpointId = 0x1,
[1655341345231] [30124:4999896] CHIP: [DMG] 					ClusterId = 0x101,
[1655341345231] [30124:4999896] CHIP: [DMG] 					CommandId = 0x0,
[1655341345231] [30124:4999896] CHIP: [DMG] 				},
[1655341345231] [30124:4999896] CHIP: [DMG]
[1655341345231] [30124:4999896] CHIP: [DMG] 				StatusIB =
[1655341345231] [30124:4999896] CHIP: [DMG] 				{
[1655341345231] [30124:4999896] CHIP: [DMG] 					status = 0x01 (FAILURE),
[1655341345231] [30124:4999896] CHIP: [DMG] 				},
[1655341345231] [30124:4999896] CHIP: [DMG]
[1655341345231] [30124:4999896] CHIP: [DMG] 			},
[1655341345231] [30124:4999896] CHIP: [DMG]
[1655341345231] [30124:4999896] CHIP: [DMG] 		},
[1655341345231] [30124:4999896] CHIP: [DMG]
[1655341345231] [30124:4999896] CHIP: [DMG] 	],
[1655341345231] [30124:4999896] CHIP: [DMG]
[1655341345231] [30124:4999896] CHIP: [DMG] 	InteractionModelRevision = 1
[1655341345231] [30124:4999896] CHIP: [DMG] },
[1655341345231] [30124:4999896] CHIP: [DMG] Received Command Response Status for Endpoint=1 Cluster=0x0000_0101 Command=0x0000_0000 Status=0x1
[1655341345233] [30124:4999896] CHIP: [DMG] ICR moving to [AwaitingDe]
[1655341345233] [30124:4999896] CHIP: [EM] Sending Standalone Ack for MessageCounter:38429626 on exchange 13072i
[1655341345233] [30124:4999896] CHIP: [IN] Prepared secure message 0x700006e0e4f0 to 0x0000000000000001 (1)  of type 0x10 and protocolId (0, 0) on exchange 13072i with MessageCounter:198966180.
[1655341345233] [30124:4999897] CHIP: [CTL] Response Time: PFvPvRKN4chip3app9DataModel14NullObjectTypeEE+829948626 0.291694 seconds
[1655341345233] [30124:4999896] CHIP: [IN] Sending encrypted msg 0x700006e0e4f0 with MessageCounter:198966180 to 0x0000000000000001 (1) at monotonic time: 000000001A2C3860 msec
[1655341345233] [30124:4999897] CHIP: [TOO] Error: IM Error 0x00000501: General error: 0x01 (FAILURE)
[1655341345234] [30124:4999897] CHIP: [TOO] Shutting down controller
[1655341345234] [30124:4999896] CHIP: [EM] Flushed pending ack for MessageCounter:38429626 on exchange 13072i
[1655341345234] [30124:4999897] CHIP: [CTL] Shutting down the commissioner
[1655341345234] [30124:4999897] CHIP: [CTL] Shutting down the controller
[1655341345234] [30124:4999897] CHIP: [IN] Expiring all connections for fabric 1!!
[1655341345234] [30124:4999897] CHIP: [IN] SecureSession[0x7faa14808280]: MarkForRemoval Type:2 LSID:16176
[1655341345234] [30124:4999897] CHIP: [IN] SecureSession[0x7faa14808280]: Released - Type:2 LSID:16176
[1655341345234] [30124:4999897] CHIP: [CTL] Shutting down the commissioner
[1655341345234] [30124:4999897] CHIP: [CTL] Shutting down the controller
[1655341345234] [30124:4999897] CHIP: [IN] Expiring all connections for fabric 2!!
[1655341345234] [30124:4999897] CHIP: [CTL] Shutting down the commissioner
[1655341345234] [30124:4999897] CHIP: [CTL] Shutting down the controller
[1655341345234] [30124:4999897] CHIP: [IN] Expiring all connections for fabric 3!!
[1655341345234] [30124:4999897] CHIP: [CTL] Shutting down the System State, this will teardown the CHIP Stack
[1655341345234] [30124:4999897] CHIP: [DMG] IM WH moving to [Uninitialized]
[1655341345234] [30124:4999897] CHIP: [DMG] IM WH moving to [Uninitialized]
[1655341345234] [30124:4999897] CHIP: [DMG] IM WH moving to [Uninitialized]
[1655341345234] [30124:4999897] CHIP: [DMG] IM WH moving to [Uninitialized]
[1655341345234] [30124:4999897] CHIP: [DMG] All ReadHandler-s are clean, clear GlobalDirtySet
[1655341345234] [30124:4999897] CHIP: [BLE] CancelConnection
[1655341345234] [30124:4999897] CHIP: [DL] Inet Layer shutdown
[1655341345234] [30124:4999897] CHIP: [DL] BLE shutdown
[1655341345234] [30124:4999897] CHIP: [DL] System Layer shutdown
[1655341345234] [30124:4999893] CHIP: [-] IM Error 0x00000501: General error: 0x01 (FAILURE) at ../../commands/common/CHIPCommandBridge.mm:83
[1655341345234] [30124:4999893] CHIP: [TOO] Run command failure: IM Error 0x00000501: General error: 0x01 (FAILURE)

Platform: Chip Tool running on Mac, all cluster app running on nRF52840-DK (Thread setup)

@Garik098 Please test the door lock tests against lock-app.