Error in Transposed Conv2d

Refer to comment here.
Even when trying to write test cases for #174 and the transposed conv2d layer, I found a similar error and i reported it there. I think we need to add preconditions and check the layer logic thoroughly.

The error in question :

Trying to use the TransposedConv2D layer but getting strange error. Below is a small example that when I run either in Colab or locally with a recent build, I get the following error. Any ideas??
[1, 4, 4, 256]
0.69032013
Fatal error: Conv2DCustomBackpropFilter: filter and out_backprop must have the same out_depth: file /swift-base/swift/stdlib/public/TensorFlow/CompilerRuntime.swift, line 2123
Current stack trace:
0    libswiftCore.so                    0x00007f06347314a0 swift_stdlib_reportFatalErrorInFile + 115
1    libswiftCore.so                    0x00007f063467930c  + 3035916
2    libswiftCore.so                    0x00007f06346793fe  + 3036158
3    libswiftCore.so                    0x00007f06344c06c2  + 1230530
4    libswiftCore.so                    0x00007f0634646292  + 2826898
5    libswiftCore.so                    0x00007f06344bfba9  + 1227689
6    libswiftTensorFlow.so              0x00007f0631aae572  + 599410
7    libswiftTensorFlow.so              0x00007f0631aaccc0 checkOk(:file:line:) + 508
8    libswiftTensorFlow.so              0x00007f0631ad1ad0 TFCCheckOk(:) + 81
9    libswiftTensorFlow.so              0x00007f0631ad1ac0 _swift_tfc_CheckOk + 9
Current stack trace:
frame #12: 0x00007f0661d90ac9 $__lldb_expr76AD__$s14__lldb_expr_7510TensorFlow0C0VySfGAB16TransposedConv2DVXEfU___adjoint_src_0_wrt_0 at <Cell 14>:14:15 	frame #20: 0x00007f0661d8e9a0 $__lldb_expr76main at <Cell 14>:13:38
frame #33: 0x00007f0661d98be7 $lldb_expr72AD__$s14__lldb_expr_7110TensorFlow0C0VySfGAB16TransposedConv2DVXEfU___primal_src_0_wrt_0(layer=<unavailable>) at <Cell 13>:14:15 	frame #34: 0x00007f0661d99b84 $__lldb_expr72AD$s14__lldb_expr_7110TensorFlow0C0VySfGAB16TransposedConv2DVXEfU___vjp_src_0_wrt_0 at <Cell 13>:13:56
frame #42: 0x00007f0661d979a0 $lldb_expr72main at <Cell 13>:13:38 	frame #55: 0x00007f0661da2be7 $__lldb_expr68AD$s14__lldb_expr_6710TensorFlow0C0VySfGAB16TransposedConv2DVXEfU___primal_src_0_wrt_0(layer=) at <Cell 12>:14:15
frame #56: 0x00007f0661da3b84 $__lldb_expr68AD__$s14__lldb_expr_6710TensorFlow0C0VySfGAB16TransposedConv2DVXEfU___vjp_src_0_wrt_0 at <Cell 12>:13:56 	frame #64: 0x00007f0661da19a0 $__lldb_expr68main at <Cell 12>:13:38
import TensorFlow
let tensorShape = TensorShape([4, 4, 256, 64])
let seed = (Int64.random(in: Int64.min..<Int64.max), Int64.random(in: Int64.min..<Int64.max))
let transposedConv2D = TransposedConv2D(
filter: Tensor(glorotUniform: tensorShape, seed: seed),
bias: Tensor(zeros: TensorShape([128])),
activation: identity,
strides: (1, 1),
padding: .valid)
let tensor = Tensor(randomUniform: TensorShape([1,1,1,64]))
let (loss, grads) = transposedConv2D.valueWithGradient { layer -> Tensor in
let out = layer(tensor)
print(out.shape)
let loss = out.sum()
print(loss)
return loss
}

@Shashi456 do you think it has to do with this. Shouldn't channels (c) be filter.shape[3]. I'm looking at the original merged code, so not sure???
@differentiable
    public func applied(to input: Tensor<Float>, in _: Context) -> Tensor<Float> {
        let batchSize = input.shape[0]
        let w = (input.shape[1] - (1 * paddingIndex)) * strides.0 + (filter.shape[0] * paddingIndex)
        let h = (input.shape[2] - (1 * paddingIndex)) * strides.1 + (filter.shape[1] * paddingIndex)
        let c = filter.shape[2]
        let newShape = Tensor<Int32>([batchSize, w, h, c])
        return activation(input.conv2DBackpropInput(shape: newShape, filter: filter,
                                                         strides: (1, strides.0, strides.1, 1),
                                                         padding: padding) + bias)
    }

Note this other issue with this layer. https://bugs.swift.org/browse/TF-540

The solution also seems to be here, https://groups.google.com/a/tensorflow.org/forum/m/#!msg/swift/UUPwV01sZrE/LszG6T7dBQAJ
@sjaz24, would you be up to fix the error and write a test for the layer? I think it's the perfect opportunity for a first contribution to the repo :) , if not I'll put a PR up later today.

I've already done what he suggests locally to get by that issue but then I run into this issue so I believe them to be different issues but wanted to point out the issue in case it's related. I'll play around with it and see. I really find this line below weird. Hmmm.. I'll give it a go at fixing it.
let c = filter.shape[2]

@sjaz24 maybe you could take a look at the tests I wrote in #174, the filter dimensions and other dimensions were inspired by running a transposed layer in tensor flow, I ran into the filter depth and out backprop error there as well and just played around with the dimensions until they worked.  Maybe that could be another starting point.

@rxwei I had some doubts, when i'm trying to solve this, Should i open a PR and we could continue the conversation and discuss the solution there?

@Shashi456 Feel free to open a PR and continue the conversation there.


[simplify-cfg] When determining/searching for the end of a switch_enu…

…ms diamond using single successors, make sure we do not have single block self-loop.
Otherwise, we infinite loop.
rdar://52131975

@swift-ci test

preset=buildbot,tools=RA,stdlib=RD
@swift-ci Please test with preset macOS Platform

One of the tests I modified only runs when asserts are disabled on the stdlib. So I am doing a preset test as well just to be careful.

Build failed
Swift Test OS X Platform
Git Sha - 9a40e9c5ca21b36ff5c8f1da8dbcf1b1a8b3ebd1

@swift-ci smoke test and merge

@swift-ci smoke test and merge

@swift-ci smoke test and merge

@swift-ci smoke test and merge

When this code wil be available in production or in Xcode11, we have Xcode 11 GM and still have this problem? Here they said that this code is fixing a very annoying problem with Carthage and this library?
scinfu/SwiftSoup#116 (comment)

We also would like to know whether this fix is incorporated into the official release of Xcode 11? We're currently seeing a problem with Xcode 11 in which the build system stuck when building our project with -Osize. Changing the optimization setting to -Onone unstuck the build process, but it's not a solution for us.

@andree-surya @catluc @imWildCat I can confirm that this fix is /not/ in the GM release. I can not confirm when a fix will be released.
That being said, I would strongly suggest filing a radar or a bug on bugs.swift.org with a project that reproduces the behavior. My reason for suggesting this is that:

The actual test case that was given previously that exposed this problem was a corner case example, so what you are seeing may be a different issue in a similar part of the compiler.
The snapshot that claimed to fix the bug may be fixing the bug for other reasons (for instance due to changing codegen).

By posting a project that reproduces, we can get to the bottom of this.

@gottesmm Thanks for your reply!
The related radar has been submitted on 8/20/2019 (FB7084207).

Hi @imWildCat @andree-surya @catluc – we are working on getting this resolved in a future Xcode release. Could you try downloading the toolchain that you can find here and see if this resolves the issue? This would be very helpful in establishing whether the issues all have the same root cause, or if there are different issues with similar symptoms.

@gottesmm I've submitted FB7347879 with a project attached that has similar symptoms as what's discussed in this thread. The toolchain you linked fixed the issue for that project.

I was trying to test this toolchain but got this error:

ld: file not found: /Applications/Xcode.app/Contents/Developer/Toolchains/swift-PR-27423-387.xctoolchain/usr/lib/arc/libarclite_iphoneos.a
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)

@imWildCat arclite isn't in open source toolchains, it should be found from the xcode default toolchain. Normally xcode finds the right path itself. Do you have any unusual linker flags in the project?

actually @imWildCat before we get to linker flags, I was just looking at the path of the toolchain. I think that is the wrong path and that may be leading to the issue here.
If you look at the linked instructions it says:

(sorry fat finger)
Install command
tar -zxf swift-PR-27423-387-osx.tar.gz --directory ~/

This will install the toolchain in $HOME/Library/Developer/Toolchains/ where Xcode will automatically pick it up.

To use the toolchain in Xcode

Open Xcode’s Preferences, navigate to Components > Toolchains and select the installed Swift toolchain (Swift PR - 27423 (387)).

Can you install the toolchain into $HOME/Library/Developer/Toolchains/ as mentioned in the instructions?


Screen with Color

Hi,
I add a Screen in HA and I want this Screen in a specific Text Color.
Actual I call to Services. Add_Screen and Text_Color.
That doesn't work very fine. Sometimes it work, sometimes not!
Is it possible to Add_Screen with a specific COLOR in RGB?

Hi, take a look at https://github.com/lubeda/EspHoMaTriXv2 in the 2023.9.1 branch. This is the evolution of your version, so you can expect more functionality but also the need of changing your service calls. e.g.
here you can set the color per icon:
icon_screen => {"icon_name", "text", "lifetime", "screen_time", "default_font", "r", "g", "b"}

Thank U. I try it with my new Ulanzi Clock.


Compatibility with PHPUnit 6

This is a backwards-incompatible change.

Thanks for your contribution!

You're welcome. And sorry, I just realized I pretty much ignored all your contribution guidelines. In the rush of getting this done I didn't even think to read them through. I promise I will respect them next time! ;)

It's nothing to apologize for. The guidelines weren't available at the time, we just added them today :)


Option to stop training if a lower limit of selected features is reached

Hello,
I do not see a method to stop training when the lower limit of selected features is reached. Is there? If not, that would be a handy feature instead of going all the way to 0 selected features. For example, I am dealing with a reconstruction/regression of all features problem, and my lower limit of selected features is around 500 out of around 8000.
Regards
Aditya

Hi and thank you for reaching out! I think this use case is very specific and is a convenience optimization so we won't implement it.
If you really need to, you can force the training to stop whenever you want by manually setting the weights to 0 using the callback argument.


Customization of the Modell

Hi
I am in desperate need of a fast Hand-Detection model.
Although I wasn't able to reach the 24 fps (I am stuck at around 10 with overclocking, 7 without - loading the model-file in python), this is still the fastest detection I got to run on my Pi 4.
But its for the "wrong" objects - I need only the detection of hands.
While comparing your model with a pretrained PyTorch implementation of the ssd mobile net v3 I noticed that even the official models are too slow. Even on an 2023 MacBook I reached only 5 fps.
Could you point me in the somewhat right direction for how to customize the model for my own use case?
I was thinking of using the EgoHands dataset
Ff my request is considered inappropriate here, feel free to delete it without comment.
Best regards,
Jan

Dear @ningelsohn ,
Please take a look at the repo https://github.com/Qengineering/Hand-Pose-ncnn-Raspberry-Pi-4.
It consist of two parts. The first detect hands in a scene. The second part recognize the position of fingers.
If you only use the first part, You have a fast detection of hands.
At the time of writing, the second part doesn't work well with the latest ncnn framework. No bother to you.
Please note, using Python does indeed slow down you performance as you have seen with tf-lite.
In the nanodet-hand.cpp, skip the lines 337 - 383, as being the detection of finger position.
In fact all code related to nanopnt can be pruned.

Dear @Qengineering,
Thank you for the quick and helpful reply, I will look into it!
Just out of curiosity: I thought that the detection of different every-day-objects and detection of hands are very similar tasks. Why do the projects differ so much?
Would the model used in the current repo (TensorFlow_Lite_SSD_RPi_64-bits – when trained with appropriate data for hand detection – perform just like it does right now?
In the meantime, I have also found the underlying paper from you. If I understand correctly, the speed is mainly (or even exclusively) achieved by adjusting the width-multiplier and the input resolution. Is that correct?
Sorry for the possibly clumsy question, I have little experience with such scientific papers.
Thanks again! This is all very helpful.

Dear @ningelsohn,
You're right. After transfer training with the appropriate data, you could recognize hands with this network.
Keep in mind, the used network in this repo is a quantize 8-bit version special tailored for TF-lite.
Speed depends on the input resolution and the number of layers. In case of MobileNet the input size is fixed. You can't change it without needing another network.
Modern models like YoloV5 can handle dynamic sizes. The same model can works with inputs ranging from let say 320x320 to 1280x1280.

@Qengineering thank you for your help!

Sorry to open this again, but this is really the best lead I had so far and other approaches just don't seem to work as good.
This repo hat the quantized model, but not the code it was created with, is this correct?
If so, is that intentional?
I have, among other things, to your repos and papers learned about how to optimize with quantization for example.
But putting everything together is not trivial.
I will close this in a few days as you have already helped me more than I could have expected

Dear Jan,
What exactly do you want to achieve?
I understand that you want to recognize hands from an image. But in what context?
For example, if you want to follow a shoplifter's hands from a surveillance camera. Or if you like to signal a "hands" of an FC Bayern München player, this will not work. At the very first stage, images are resized to the (small) format used by the deep learning model. By then, hands are reduced to a few pixels.
In this situation, you will have to use modern models like YoloV5 with a large input resolution. Needless to say, your speed has completely evaporated as a result. 0.5 FPS will already be a lot on an RPi 4.
If you want to recognize the hands of a person waving directly in front of the camera, then you have at least a reasonable amount of pixels per hand to recognize.
If you can also handle 10-12 FPS, then a NanoDet or YoloV5-tiny model is the solution. I'd be happy to open a repo for you on GitHub with the code.
But remember, the hands will have to be fairly prominent in the picture and will not always be recognized (think of a fist). Fingers are also not recognized with this model.
Training a network yourself is indeed not trivial.
You will have to determine what is feasible. MobileNet with Tensorflow Lite is easy to train, but only works fast if you are able to convert the floats of the weights to int8 numbers. Often this is accompanied by a significant loss of accuracy. In addition, as already mentioned, the input size of the images is also not large (320x320).
Another possibility is the Yolo road. However, you quickly get stuck here when it comes to transferring a PyTorch model to a model that is better supported by an RPi, such as the C++ ncnn framework. It is doable, but then you have to count on speeds of 5-10 FPS.
In both cases you will have to provide images yourself that can be used during the training. A time consuming and boring job.

Dear Qengineering,
in my use case the hand-detection will be used as a touches navigator/selector and will be very prominent in the frame.
I am currently already looking at tiny yolo. As only flat hands are to be recognized, there is little to no variation in the appearance of the object which should get recognized.
Thanks again for your detailed response!
I will take a deeper look into NanoDet and Yolov5-Tiny.
If it is not too much trouble, I would also like to have a look at the code mentioned.
PS:
ultralytics (well known I guess) has stated that quantization is not really a big factor regarding speed: https://github.com/ultralytics/yolov5/issues/211
But with their smallest model I get "only" 1.5 fps


Use libsodium to encrypt private files

Use pwhash key derivation to encrypt private files? This would require rekeying files on password change.
Necessary?

Regarding doing this client-side..
https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2011/august/javascript-cryptography-considered-harmful/

So I suppose we'll have to prompt for password when viewing a private paste in order to decrypt?

I can think of a few points here:

Prompt the user for a password which is then sent server-side.

Drawback: server sees password, but more convenient.

On the password prompt page, also include a download button and/or command the user can copy and issue to download and decrypt the file locally using tools on their own machine - if they desire.

Prompt the user for a password which is then sent server-side.

Another pertinent question is how do we handle them reloading the page or editing and so on without them re-entering the password?
Store the password in memory? Decrypt the file on the server for a period of time?

Another pertinent question is how do we handle them reloading the page or editing and so on without them re-entering the password?

Store the password in memory? Decrypt the file on the server for a period of time?

This is a good question, and I think you've already suggested the approaches I'd have mentioned. I'm not sure I like the idea of storing the password in memory... but what's the alternative? It could go in window.sessionStorage and then be sent to the server for each operation (fetch, refresh, edit) and then the UX would be a bit better. The downside of course is that this is then accessible if there's an XSS.
You could store it in a HttpOnly, SameSite=strict, Path-constrained session cookie (e.g. could be only available to stuff under /private/ to limit attack surface1). This wouldn't be accessible from anything client-side and would only be accessible to the server in the context of the relevant requests. You'd want some way to expire the cookie - maybe have it only valid for ten minutes at a time.

What's the risk? It's unlikely, but if there was a vuln. in something under e.g. an /uploads/ directory or elsewhere on the site (maybe an SSRF that forwarded the client request headers) the cookie would not be sent.

Alright, I've started work on this. I have some notes and questions, but I'd like to detail the process for encryption and decryption and get @lol768's opinion.
Encrypting

The plaintext password is provided by the user at the time of paste creation.
The password is hashed in the same manner as account passwords, and it is stored as part of the paste in the database.
The password is used to create a secret key using the salt that was used to hash the password in step 2.
A nonce is generated.
The nonce is used with the derived key and plaintext file data to create a secret box.
The nonce is written to the disk, followed by the encrypted data.
Repeat steps 4-6 for each file.

Decrypting

The plaintext password is provided by the user at the time of paste retrieval.

If using the web UI, the user is prompted to provide the password if a password is set on the paste. Upon submitting the password, it is checked, and if it is valid, it is stored in an HttpOnly, Secure, Path-constrained (constrained to that paste specifically), encrypted, authenticated, cookie that expires in 15 minutes. The expiration time is updated every time a request to the paste is made.
If using the web UI with a cookie set, the cookie is read and used without user intervention. If the password stored in the cookie is incorrect, the user is prompted to provide the correct password.
If using the API, requesting an encrypted paste without specifying a password is an error, and the client will have to try again.

The password is checked against the hashed password.
If correct, the password is used to derive a secret key, using the salt stored in the database (step 2 of encryption).
The encrypted file is read, and the nonce and data are separated.
The secret box is opened, using the key, nonce, and encrypted data.
Repeat steps 4-5 for each file.
Serve the decrypted data to the user.

No plaintext data is ever saved or stored. It only exists in memory for as long as necessary.
Notes/questions

Currently, the paste title and description are plaintext only. Should they also be encrypted?
Revisions are more difficult with encrypted data. Git never sees the plaintext, so each change would require a diff to be generated and encrypted, then stored on the disk. I vote for disabling revisions on encrypted pastes.
Each edit to a paste requires the password, which will require some internal refactoring.

After some time away from actively working on this and reflection, I think that this is not a necessary feature.

Anyone who is concerned about this can use GPG (symmetric and asymmetric) and upload ASCII armoured output or any other encryption method.

paste supports the ability to programmatically upload pre-encrypted data and download it to decrypt client-side. The API for upload and download or using raw files for download to decrypt works just fine.

Personally, I don't like handling encryption and decryption on the server side, since a malicious server can read the data and it introduces an extra party into the exchange of encrypted data.
The only benefit of encrypted pastes that is lost by having users encrypt before upload is ease of access. Considering this is a tool made primarily for programmers, I don't envision this being a real issue. However, encrypting before upload (potentially) adds several security benefits which makes up for the loss of ease of access.

If anyone has objections to this not being implemented, let me know. Otherwise, I'll close this and delete the encryption branch eventually.

Not mentioned in this issue is the extensive discussion on IRC around this. Following said discussion, I support not implementing this. Perhaps too many technical details had been discussed instead of effort answering "What problem does this solve, what is the threat model?".
However, you may wish to create a documentation issue explaining how folks can use e.g. keybase or gpg and pipe the output to get a paste they can share.

Perhaps too many technical details had been discussed instead of effort answering "What problem does this solve, what is the threat model?".

In fact, it was mostly you bringing this up that made me realise that this is mostly a useless feature.

However, you may wish to create a documentation issue explaining how folks can use e.g. keybase or gpg and pipe the output to get a paste they can share.

My thoughts exactly!


[Feature]: Filter genuine email

Description
Security Feature
In your application anyone can enter using temporary or disposable email.
I will add a feature so that no one could create an account using temporary or disposable email.
It will allow only genuine users in your database.
It will help to avoid any bot.
Suggested Solution
I'll implement a feature that will not allow such emails for login or register.
Assign me this work under GSSoC24, I will implement it.
Alternatives
No response
Additional Context
No response

@pankaj-bind our application does not need an authentication system. it is a cli tool.

@pankaj-bind our application does not need an authentication system. it is a cli tool and If I am unable understand what you want to say. Join our discord server and ping me.

It is not only work for authentication. It will allow only those emails which are genuine.


Update commercial paper doc to use enrollUser.js

Signed-off-by: NIKHIL E GUPTA<EMAIL_ADDRESS>Type of change

Documentation update

Description
Currently, the commercial paper tutorial uses a file system transfer program to create the wallet used by the application. I added an enrollUser.js file to use the test network CA's to enroll an application user. This change moves the tutorial from the transfer program to enrollUser.js, and updates the tutorial for the CAs that are now deployed.
Other updates to the tutorial to reflect that CAs are now deployed as well.

@Mergifyio backport release-2.1


Improve plugin

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 16 (0.0%)  changed or added relevant lines in 1 file are covered.
4 unchanged lines in 2 files lost coverage.
Overall coverage increased (+2.9%) to 22.862%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/tracing-collector.ts
0
16
0.0%

Files with Coverage Reduction
New Missed Lines
%

src/newrelic.ts
2
32.75%

src/tracing-collector.ts
2
28.24%

Totals

Change from base Build<PHONE_NUMBER>:
2.9%

Covered Lines:
121

Relevant Lines:
527

💛  - Coveralls


Add a more tolerant vs runtime replacement

This vs runtime replacement for the openssl mak files (required for Visual Studio/nmake) are more tolerant then the existing implementation (which leads to replacement errors in 1.0.2l).
The new function try to replace any existing vs runtime in the openssl mak files, if one is found, it will exit normally, but if no vs runtime is found in the file it will throw a Exception to guarentee that we have replaced the vs runtime.
Greetings
Tonka

The CI fails are confusing, appveyor fails with MinGW because of a missing parameter for gcc and travis build always throw error because of wrong username, password, but it is not the release branch.
At the moment it is not possbile to install poco 1.7.9:stable (from conan center) from source, because OpenSSL 1.0.2l fails.
Is the 1.0.2l branch to old and not supported anymore?

Hi @tonka3000 , yes, it looks like a problem with the package tools, let me take a look.

I see, it's about new security policy with travis:  https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
I have to think about how to solve this.

Solved, it was a bug in the latest conan package tools affecting the pull requests only. Let's see how it goes with your PR

Hey @lasote ,
please can you check the CI again, seems to work in most cases, but some error messages from CI are weird again.
I would need that fix in order to use this version with conan.
Thanks in advance

Yeah, it's weird, something is happening. By the way, as a user suggested here: https://github.com/lasote/conan-openssl/issues/33 it's easier to use the strict=False parameter to solve this problem, maybe we can try if it is related or not to the error. (I don't think so)

I've pushed the strict=False, should be ok now

The strict=False approach is buggy, i get a /MDdd flag (which will be ignored by the compiler) when i use compiler.runtime=MDd, so it fallback to the default settings, which could be bad!


Add Semantic Attacks PyTorch

Description
Implementation of single semantic attacks on PyTorch image classifiers. The attacks are generated by perturbing hue, saturation, contrast, brightness, and rotation.
Paper: https://openaccess.thecvf.com/content/CVPR2023/papers/Hsiung_Towards_Compositional_Adversarial_Robustness_Generalizing_Adversarial_Training_to_Composite_Semantic_CVPR_2023_paper.pdf
Fixes #2126
Type of change
Please check all relevant options.

[ ] Improvement (non-breaking)
[ ] Bug fix (non-breaking)
[x] New feature (non-breaking)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Testing
Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

[x] Unit tests
[x] Notebook example

Test Configuration:

OS
Python version
ART version or commit number
TensorFlow / Keras / PyTorch / MXNet version

Checklist

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

Codecov Report

Merging #2200 (de3208f) into dev_1.15.0 (4bfed67) will decrease coverage by 62.46%.
The diff coverage is 17.85%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@               Coverage Diff               @@
##           dev_1.15.0    #2200       +/-   ##
===============================================
- Coverage       85.68%   23.22%   -62.46%
===============================================
  Files             306      311        +5
  Lines           27022    27633      +611
  Branches         4962     5029       +67
===============================================
- Hits            23153     6418    -16735
- Misses           2599    21102    +18503
+ Partials         1270      113     -1157

Impacted Files
Coverage Δ

...on/semantic_attacks/brightness_gradient_pytorch.py
17.19% <17.19%> (ø)

...sion/semantic_attacks/contrast_gradient_pytorch.py
17.19% <17.19%> (ø)

...s/evasion/semantic_attacks/hue_gradient_pytorch.py
17.19% <17.19%> (ø)

...on/semantic_attacks/saturation_gradient_pytorch.py
17.19% <17.19%> (ø)

art/attacks/evasion/__init__.py
98.24% <100.00%> (+0.03%)
:arrow_up:

art/attacks/evasion/semantic_attacks/__init__.py
100.00% <100.00%> (ø)

... and 245 files with indirect coverage changes


Use hipMemcpyAsync for validation

replace hipMemcpy with hipMemcpyAsync + hipStreamSynchronize in ReferenceValidator

This is a trial fix for random mismatch with device result 0.
I will run CI test multiple times with this change.

I will fix the build error...

Reference rocblas commit for hipMemcpyAsync:
https://github.com/ROCm/rocBLAS/commit/c070df9bdc31cce30e0d4d732ad8a60f3b2ee332

I ran precheckin test 11 times and extended test 7 times.
I did not see mismatch issue with device result=0 so far.
I will merge this change.


create timer

timer operates need to be able to add time and end quiz with check for success

timer is functional


Feat: add Skeleton effect

Add a new Skeleton component and examples to show a loading state while content is loading. The Skeleton component can be used to display a placeholder while the actual content is being fetched. The component API includes an optional CSS class for styling the skeleton.

DEMO:
https://github.com/user-attachments/assets/956b8ec0-8fd7-4a88-a0f8-9a7f85acd5ca


WEB-3936-homepage-styling-fixes

Build succeeded (check pipeline).

https://checklist.intgdc.com/job/compliance/job/gooddata-docs-theme-oss-check/63/ : SUCCESS in 3s

Build succeeded (gate pipeline).

https://checklist.intgdc.com/job/compliance/job/gooddata-docs-theme-fossa-scanning-oss-zuul/38/ : SUCCESS in 45s (non-voting)
https://checklist.intgdc.com/job/compliance/job/gooddata-docs-theme-merge-compliance-check/23/ : SUCCESS in 9s


custom scheme

why use fake domain when it can be custome url scheme (is it even possible)?

LCU responds with header Referrer Policy: strict-origin-when-cross-origin.
Then all cross-orgin and cross-scheme (https -> pengu://) requests will be blocked.
Since Chromium v87, there's no way to bypass CORS policy, disabling whole web security is not a good idea.

@nomi-san as far as i can see, it manageable
  virtual bool AddCustomScheme(const CefString& scheme_name, int options) = 0;
 CEF_SCHEME_OPTION_CORS_ENABLED = 1 << 4

Yesh, this method allows cross-scheme import, but cant bypass resource requests.
Also tried CefAddCrossOriginWhitelistEntry().
// This function cannot be used to bypass the restrictions on local or display
// isolated schemes. See the comments on CefRegisterCustomScheme for more
// information.

@nomi-san it can, i playing around and now at the point where i just not good enough to implement cef_resource_handler_t and i just get crashes, but request doesnt get blocked by cors

Fetch API should not work on non HTTP scheme, even with CEF_SCHEME_OPTION_CORS_ENABLED.

but esm imports works fine

Fetch API should not work on non HTTP scheme, even with CEF_SCHEME_OPTION_FETCH_ENABLED.

then whats the point of thet option ...

In case of switching to custom scheme, what will you do

Replacing https://plugins with pengu://plugins, that has no meaning
Replacing https://plugins with plugins://<name> (name is TLD, alias of pkg), you will do more work

And both cases need to redirect ALL requests that point to https://plugins/. Also exising plugins have used fetch to acquire their resouces will not work.
I will not handle this case.


There should only be one offscreen canvas

We only use one at a time, so dont bother rendering all of them?
Or maybe multiple ones will come in handy for when we want to switch between them!?!?
Hmmmm.... it doesn't cause any issue having multiple canvases really, so let's just keep them there inactive for now!

Yeah I'll keep the multiple ones for now!


Don't publish empty patches

For #655 #294
This update explicitly checks if generatePatch indicates a change, and if not, will skip publishing that patch.
e.g. If the current data looks like: { a: 1, b: 2, c: 3 } and there's an open query such as { a, b }, if c changes and we invalidate on the server, it would previously publish an empty patch to that user.
Also, I'm not super familiar with Typescript, so the correct types might need to be added.

Hey @n1ru4l, when you get a chance would you mind taking a look over these changes?

@dburles Hey, sorry for the delay I just pushed some minor adjustments, for using a symbol instead of undefined. I will now rebase the branch and release it :)

@dburles Thank you for taking the initiative on this! Highly appreciated ❤

No problem, thanks!


Dbz 3772 update oracle connector deployment instructions

DBZ-3772
This change harmonizes the deployment instructions for the Oracle connector with the instructions that we provide for the other Debezium connectors, including conditionalizing steps for exclusive upstream or downstream use.
Tested in a local Antora build. Also tested that the annotations for splitting the file for downstream use return the expected results, and that the conditionalization behaves as expected.
Please backport these changes to the 1.5 branch.

Applied, thanks @roldanbob !


[$500]  Referral - Copy referral link missing in Referral overaly when navigated via FAB

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number:  1.4.6-2
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Action Performed:

Navigate to staging.new.expensify.com & login
Click FAB button
Select Start Chat
Click Start a chat, get $250 banner

Expected Result:
Copy Referral link to be displayed in referral overlay when navigated via FAB
Actual Result:
Copy referral link missing in Referral overaly when navigated via FAB
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] Windows: Chrome
[x] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence

https://github.com/Expensify/App/assets/78819774/a5748aab-a6e7-4c99-82c4-f14bfffbab5b
View all open jobs on GitHub

Proposal
Please re-state the problem that we are trying to solve in this issue.
Copy referral link missing if we navigate from the FAB > New chat
What is the root cause of that problem?
On the NewChatPage Page we are using the OptionsSelector and here we pass the shouldShowReferralCTA prop to display the referral buttton on the new chat page and we are passing the referralContentType={CONST.REFERRAL_PROGRAM.CONTENT_TYPES.START_CHAT} here.
https://github.com/Expensify/App/blob/4fc20d3f93a3693f91021901a760923cc5d913af/src/pages/NewChatPage.js#L257-L258
But On ReferralDetailsPage we only display the button if contentType === CONST.REFERRAL_PROGRAM.CONTENT_TYPES.REFER_FRIEND , so here contentType is not matching and button is hidden
https://github.com/Expensify/App/blob/4fc20d3f93a3693f91021901a760923cc5d913af/src/pages/ReferralDetailsPage.js#L55
What changes do you think we should make in order to solve the problem?
To Display the button on the ReferralDetailsPage, I would suggest we should check all the four below types, and if contentType is any of the below type then we should display the button
[
    CONST.REFERRAL_PROGRAM.CONTENT_TYPES.SEND_MONEY,
    CONST.REFERRAL_PROGRAM.CONTENT_TYPES.MONEY_REQUEST,
    CONST.REFERRAL_PROGRAM.CONTENT_TYPES.START_CHAT,
    CONST.REFERRAL_PROGRAM.CONTENT_TYPES.REFER_FRIEND
]

We can update like this
const shouldShowClipboard = [
    CONST.REFERRAL_PROGRAM.CONTENT_TYPES.SEND_MONEY,
    CONST.REFERRAL_PROGRAM.CONTENT_TYPES.MONEY_REQUEST,
    CONST.REFERRAL_PROGRAM.CONTENT_TYPES.START_CHAT,
    CONST.REFERRAL_PROGRAM.CONTENT_TYPES.REFER_FRIEND
].includes(contentType) || isShareCode;

Logic behind adding these four types is because we are showing the referralCTA for all these four types from the NewChatPage, SearchPage, and MoneyRequestParticipantsSelector

https://github.com/Expensify/App/blob/4fc20d3f93a3693f91021901a760923cc5d913af/src/pages/SearchPage.js#L207-L208

https://github.com/Expensify/App/blob/4fc20d3f93a3693f91021901a760923cc5d913af/src/pages/NewChatPage.js#L257-L258

https://github.com/Expensify/App/blob/4fc20d3f93a3693f91021901a760923cc5d913af/src/pages/iou/steps/MoneyRequstParticipantsPage/MoneyRequestParticipantsSelector.js#L323-L324

Result:
https://github.com/Expensify/App/assets/35371050/a9441bd9-4325-44f4-96c1-c6fbe82390c7

What alternative solutions did you explore? (Optional)
N/A

Let's see the current UI of referral pages

If the type is one of START_CHAT, SEND_MONEY, MONEY_REQUEST

If the type is REFER_FRIEND

If the type is SHARE_CODE

@laurenreidexpensify @shawnborton @puneetlath   Currently

Copy to clipboard option display in SHARE_CODE and REFER_FRIEND
Description body 2 option only display in SHARE_CODE

Could you help to confirm which page we should display Copy to clipboard option Description body 2 option ?
Proposal
Please re-state the problem that we are trying to solve in this issue.
Referral - Copy referral link missing in Referral overaly when navigated via FAB
What is the root cause of that problem?
https://github.com/Expensify/App/blob/2e8de3ffd53f40139a927bc60461feadefe88d78/src/pages/ReferralDetailsPage.js#L90-L103
we are using shouldShowClipboard and shouldShowBody2 as condition to display Copy to clipboard option Description body 2 option
What changes do you think we should make in order to solve the problem?
We need to update as the final expected

cc @jamesdeanexpensify and @dubielzyk-expensify for thoughts here too

So here's where we're at overall with these:

START_CHAT, SEND_MONEY, MONEY_REQUEST - we purposely left out the "Copy referral link" here. These are more informational modals infused into flows where you're already in the middle of an action (like starting a chat, sending money, or requesting money). And those actions, if the recipient is a new Expensify account, will already be tracked automatically as part of our referral program so there's no need to copy/share your link at all.
SHARE_CODE and REFER_FRIEND - we're actually updating these two to use the exact same copy, so we won't need to worry about the description body 2 once the change is implemented.
Also just FYI, for all five of the above, we're updating them visually to a new layout based on design comments (reference).

Does this all make sense?

@DylanDylann does this answer your Q ^^

@laurenreidexpensify Got it. So I think this is intentional, not a bug as confirmed here


Backend - Add The Reputation System

Requirements:

<IP_ADDRESS>
<IP_ADDRESS>

<IP_ADDRESS>.1
<IP_ADDRESS>.2
<IP_ADDRESS>.3

Description:
Make the reputation system available to the front teams in its entirety as described in the requirements.

<IP_ADDRESS> Vote on other users based on their profile, behaviour and activity on the platform which will affect the target's reputation scores (See "<IP_ADDRESS> Reputation system")
<IP_ADDRESS> Reputation system:

<IP_ADDRESS>.1 The system shall allow users to vote on other users based on their qualities and it shall keep record of these votes.
<IP_ADDRESS>.2 Votes shall represent either positive or negative side of a quality of the user.
<IP_ADDRESS>.3 The system shall assign badges to users depending on how many positive and negative votes they received on a certain quality.

A user might be good at fact-checking and receive many positive fact-checker votes from other users. Conversely, a user might do false fact-checks constantly and receive negative fact-checker votes from other users. When the user accumulates enough net positive or negative votes as a fact-checker, the system shall assign them with the badge that represents this good or bad quality.

spent: 1.5h

spent: 1h


#50: Feature: added extra view to see if someone filled in their wishlist on admin view

Fix for issue #50

Thanks!


Allow for const json importing

Description
Having constant information about an unwavering JSON file that is developer-controlled is very useful for certain bouts of information, as well as providing great autocomplete help on datasets.
Suggested solution
Allow for import queries (or something backwards-compatible of the sorts) that allows for importing JSON objects as constant type objects.
Alternative
When I need constant values, I either:

end up converting the json file to typescript and making it a default export.
Using runtime typescript validation (e.g. zod).

Additional context
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

I guess you are taking about typescript's feature: https://github.com/microsoft/TypeScript/issues/32063


Documentation review

For traceability purposes. Pinging @ansys/documentation team.
We would kindly request a review to our posters whenever you have the chance. Source code format is .tex files. POsters can be downloaded easily from https://scipy2023.pyansys.com/
You can either submit a PR or provide feedback on the PDFs directly. Attach the PDFs on your comments in this issue in that case. GitHub should allow it.
Thanks in advance! Deadline for review should be next Tuesday 4th of July.

Closed by #25 and #24


Implement the specific import pipeline for DSpace stats

[ ] Implement the specific import pipeline for DSpace stats

DSpace did not ultimately feature as a datasource for MVP

Following the design of the import pipeline and its relationship to the event API, it was determined that no specific import pipelines would be required for any external services.  This issue is therefore no longer relevant.


Check allowed_builders for Bodhi updates triggered by builds

Fixes #2312
RELEASE NOTES BEGIN
You can now configure a list of Fedora accounts of users whose successful Koji builds will trigger automatic Bodhi updates (by default only packit).
RELEASE NOTES END

recheck


map.getLayers().setAt issue

Hi. I have the problem when i send to order my layers according i am loading the second time if i set the function setAt with the same position it give me this error:
Uncaught TypeError: Array.prototype.forEach called on null or undefined
at forEach (native)
at Sa (http://geoactiva.dev:8080/geoactivaV2.0/js/plugins/ol3/ol-3.6.js:23:625)
at G.l.Vj (http://geoactiva.dev:8080/geoactivaV2.0/js/plugins/ol3/ol-3.6.js:231:45)
at ad (http://geoactiva.dev:8080/geoactivaV2.0/js/plugins/ol3/ol-3.6.js:43:187)
at nf.l.dispatchEvent (http://geoactiva.dev:8080/geoactivaV2.0/js/plugins/ol3/ol-3.6.js:41:328)
at nf.l.Vn (http://geoactiva.dev:8080/geoactivaV2.0/js/plugins/ol3/ol-3.6.js:80:72)
at OrdenaCapas (http://geoactiva.dev:8080/geoactivaV2.0/:497:23)
at :2:1
at Object.InjectedScript._evaluateOn (:895:140)
at Object.InjectedScript._evaluateAndWrap (:828:34)

Have you tried ol-debug.js? What is the error then?

Hi. I am traying to change order the layers.
var Pos=0;
var Pos2=1;
var LayerTarget=map.getLayers().getArray()[Pos];
var LayerSource=map.getLayers().getArray()[Pos2];
map.getLayers().setAt(Pos,LayerTarget);
map.getLayers().setAt(Pos2,LayerSource);
i got the this error some times and some times dosent:
Uncaught TypeError: Array.prototype.forEach called on null or undefinedSa @ ol-3.6.js:23l.Vj @ ol-3.6.js:231ad @ ol-3.6.js:43l.dispatchEvent @ ol-3.6.js:41l.Vn @ ol-3.6.js:80OrdenaCapas @ administrador:105MueveCapa.$.ajax.success @ VM6323:166jQuery.Callbacks.fire @ jquery-2.2.1.js:3073jQuery.Callbacks.self.fireWith @ jquery-2.2.1.js:3185done @ jquery-2.2.1.js:8253jQuery.ajaxTransport.send.callback @ jquery-2.2.1.js:8600

You need to provide more information. I just tried with http://openlayers.org/en/master/examples/vector-layer.html in things work as expected.
var layer0 = map.getLayers().getArray()[0];
var layer1 = map.getLayers().getArray()[1];
map.getLayers().setAt(0, layer0);
map.getLayers().setAt(1, layer1);

and I don't get any error.

not sure if this is the problem here, but if you try and setAt beyond the current length of the collection, it inserts undefined entries in the collection, which will then cause errors for anything iterating over the collection https://github.com/openlayers/ol3/blob/master/src/ol/collection.js#L269 for example map.getLayers().setAt(3, layer0); in the vector-layer example.
That shouldn't cause a problem in the simple replace case given here, but it would if you were removing layers and then trying to set at a position that no longer exists.

Here is my site.  http://geoactiva.no-ip.biz/geoactivaV2.0/index.php
press ENTRAR.
Move the layer order.
The first time it did. The second time send the error.
This is the function that order the layers.
for(var Grupo=Grupos.length-1 ;Grupo >=0; Grupo--){
var Capas=Grupos[Grupo].children;
for(var n= Capas.length-1 ;n >=0;n--){
LayerSource=map.getLayers().getArray()[Pos];
Pos2=null;
map.getLayers().getArray().forEach(function(lyr,Position){
if(lyr.get('name')==Capas[n].Capa.idcapa){
LayerTarget=lyr; Pos2=Position;
return;
}
});
if(LayerSource != LayerTarget){
map.getLayers().setAt(Pos,LayerTarget);
map.getLayers().setAt(Pos2,LayerSource);
}
Pos++;
}
}
for(var n= Tematicos.length-1; n >=0 ; n--){
LayerSource=map.getLayers().getArray()[Pos]; Pos2=null;
map.getLayers().getArray().forEach(function(lyr,Position){
if(lyr.get('name')=="Tematico_"+Tematicos[n].Tematico.idtematico){
LayerTarget=lyr; Pos2=Position;
return;
}
});
if(LayerSource != LayerTarget){
map.getLayers().setAt(Pos,LayerTarget);
map.getLayers().setAt(Pos2,LayerSource);
}
Pos++;
}
for(var n= MapasCalor.length-1; n>=0; n--){
LayerSource=map.getLayers().getArray()[Pos]; Pos2=null;
map.getLayers().getArray().forEach(function(lyr,Position){
if(lyr.get('name')=="MapaCalor_"+MapasCalor[n].id){
LayerTarget=lyr; Pos2=Position;
return;
}
});
if(LayerSource != LayerTarget){
map.getLayers().setAt(Pos,LayerTarget);
map.getLayers().setAt(Pos2,LayerSource);
}
Pos++;
}

I'm having the same problem with 3.5.0 and 3.6.0. There is no error with 3.4.0. See this jsfiddle. Clicking the "Swap" button works once but not twice. The example above by @elemoine doesn't actually change the layer positions which may be why the error isn't seen.
The first time the layers are swapped it works fine. But the second time the following error is thrown
TypeError: can't convert undefined to object

When ol-debug.js is used the error thrown is
AssertionError: Assertion failed: listeners already registered

yes, this can be reliably reproduced in the vector-layer example too.
function chg(){
  var layers=map.getLayers();
  var layer0 =layers.item(0);
  var layer1 = layers.item(1);
  layers.setAt(0, layer1);
  layers.setAt(1, layer0);
}
chg();
chg();

The debug msg is from ol.layer.Group.prototype.handleLayersAdd_. Because the listener key is based on the layer Uid, which will have been created by the previous action. Presumably the assumption is that you won't add the same layer twice to the same collection?

no, I think I see what the problem is. When you swap the layers like this, you are adding a layer that is already in the collection. I think the solution is to remove both layers, then insert/push them both.

so changing my function to the following works fine:
function chg(){
  var layers=map.getLayers();
  var layer1 = layers.pop();
  var layer0 = layers.pop();
  layers.setAt(0, layer1);
  layers.setAt(1, layer0);
}
chg();
chg();

The current docs for ol.Collection.remove "Remove the first occurrence of an element from the collection" implies that an element, i.e. an object instance, can be in a collection more than once. But the change in #3528, which creates the event listener based on Uid, means that this is no longer the case. Are there valid use cases for this? If not, I'd suggest changing the docs to make it clear that an instance can only be present once in a collection. It might also be a good idea to change the assert "listeners already registered" to the more helpful "object already in collection".

Okay. Thanks for looking into it. I worked around it by removing one of the layers to be swapped and then inserting it at the desired location. I was trying to avoid this because I am monitoring the change:length event and removing/inserting layers was causing undesired side effects. But, I found a work around by having my application track if a layer is being moved and have it avoid the undesired behavior. If this is intended behavior then so be it.

But what about if you want swap position 4 and position 6 in a map that has 15 positions ?

have a dummy layer to set temporarily? create refs for 4 and 6; put dummy in 6, put 6 in 4, put 4 in 6. Bit of a pain, but it should work.

To swap non-adjacent layers, you could move the invisible layer up first then move the visible layer down second. See here. If you are doing something more complicated like partially visible layers, spy layers, etc, it might get more complicated and have to do the dummy layer suggested by @probins.

alternatively use removeAt/insertAt (making sure you adjust the insertAt position to reflect the fact that you've just removed one :smiley: )


FlattenLens Postgres Arrays

Added support for Postgres array types T[] in FlattenLenses.
Also rewrote the lightweight test cases for nested data, so that they cover a wider range of corner cases and have fewer JSON-related examples.

Thanks Damian!


Add DisplayLine.registerHovered

Triggers once on first mouse over with true as input and once on mouse leave with false as input

Added with 22d40bad3c833d3320fdb1b69365bb14a2920e76


allow base MQTT topic to be configurable (instead of hardcoded to teslamate/)

What would you like to be added:
Allow (optional) configuration of the base MQTT topic, instead of being hardcoded to teslamate/
Why is this needed:
I would like teslamate data to live under an existing base topic, along with all my other home automation data/feeds.
Additional context:

Just had a look at the code, one solution could be to remove teslamate from the hardcoded topic (line 79 in here) and just default namespace to teslamate if there is nothing explicitly defined in the env vars. This would keep the current behaviour (so long as namespace isn't set) and would mean no new config variables are needed.
This would break for anyone who currently uses namespace however, as they would need to prepend telsamate/ to keep current behaviour.


About p2-weighting

weight = _extract_into_tensor(1 / (self.p2_k + self.snr)**self.p2_gamma, t, target.shape)

Why is the numerator 1 in the code, not the lambda mentioned in the equation 8?

@HarperHao @jychoi118
I have the same question, why does the lambda_t equal to 1?
Do you find out the reason?

Sorry,Finally I modify the code to replace 1 with lambda

@HarperHao @jychoi118 I have the same question, why does the lambda_t equal to 1? Do you find out the reason?

Sorry,Finally I modify the code to replace 1 with lambda

@HarperHao @LinWeiJeff @jychoi118
The way I understand it is as follows:
The original VLB loss $L_t$ can be rewritten as the expected value of a weighted MSE of the noise prediction and the actual noise. This is described in equation 5:

However, we are using $L_{simple}$, which drops this weighting factor (so the weighting factor is one). In terms of VLB this means that we are multiplying equation 5 with the inverse of the weighting factor. The authors refere to this inverse as $\lambda_t$ or $baseline$

later on in equation 8 the authors present a new weighting scheme that replaces $\lambda_t$ with $\lambda_t'$

They then state the following:

Note how the authors are using $L_t$ and NOT $L_{simple}$ in the highlighted text. Remember, $L_t$ still has the weighting factor attached to it and to get rid of it you need to multiply by $\lambda_t$. In the code however we are using $L_{simple}$ which is inherently already multiplied by $\lambda_t$. This means we can simply multiply $L_{simple}$ with $\frac{1}{(k + SNR(t))^\gamma}$ to get the P2 weighting scheme.
I hope my explanation was clear.
@jychoi118 Please correct me if I misinterpreted any of your claims.


[cocoon] Dashboard is not updating retried luci tasks

If a luci task fails on prod schedulers, it can be retried using the buildbucket ui or bb command but the dashboard is not updating the new status.
The following image shows the retried tasks as green:

The flutter dashboard refreshed at the same time:

Links to tasks:

[failing] https://ci.chromium.org/p/flutter/builders/prod/Linux/11926
[passing] https://ci.chromium.org/p/flutter/builders/prod/Linux/11927

The correct behavior is if there are many results for the same commit then it should be ordered by time and show the status for the most recent one.

/cc @jonahwilliams @keyonghan

List of builds should be filtered to one build per branch and commit. If there is more than one build the one created more recently should be kept: https://github.com/flutter/cocoon/blob/master/app_dart/lib/src/service/luci.dart#L92
Make getRecentBuilds to return a map to remove the second loop here: https://github.com/flutter/cocoon/blob/master/app_dart/lib/src/request_handlers/refresh_chromebot_status.dart#L82

Updating logic is working well. The problem is that this rerun here missed commitSha and ref:
taskName: mac_bot,
commitSha: unknown, ref: unknown, status: Succeeded,
commitSha: bd6ccb606a11dacf7c72970a9d623e6b89668d70, ref: refs/heads/master, status: Failed

https://github.com/flutter/cocoon/blob/master/app_dart/lib/src/request_handlers/refresh_chromebot_status.dart#L85 needs that info to update luci status.
@godofredoc do you have any idea why those info was missing? Or can we have rerun with those info?

Reruns from the scheduler do not pass commit sha. They just run whatever is at head.

Rerun mac and windows with a hash, let's see if the status gets updated:
bb add -commit "https://chromium.googlesource.com/external/github.com/flutter/flutter/+/bd6ccb606a11dacf7c72970a9d623e6b89668d70" "flutter/prod/Mac"
bb add -commit "https://chromium.googlesource.com/external/github.com/flutter/flutter/+/bd6ccb606a11dacf7c72970a9d623e6b89668d70" "flutter/prod/Windows"

the updating process works as expected. Closing.


Update tests to handle changes in job cancellation.

Testing fails until this is fixed.  A number of other PRs have this change so that they can pass testing.
The jobs.cancel is event is consumed by the remote worker plugin, so we can't rely on it here.  Since we only use it in testing, it is easier to change the tests than to change the order that events are processed in Girder.

Change seems fine, but I'm wondering if the behavior of eating the event in the worker is correct?

This has to do with how Girder handles events.  If any process bound to an event calls stopPropagation, then the next handler won't get the event (of course).  Further, the test as I had originally written it was expecting the default event handler to fire, but remote worker now prevents the default and instead issues an update.
Since the test is just preventing the job from changing state, so that I can verify that certain conditions are handled properly, I am not concerned that remote worker changed what it was doing.

The use of stopPropagation() also means that plugins essentially compete for the event, the plugin loaded first wins. I think this is a feature of the Girder event system.

Since it inspects the handler, I guess it does make sense for it to use stopPropagation.


Always crash when running selenium/standalone-chrome

run:
docker run -d -p 4444:4444 selenium/standalone-chrome

It crash on last layer.

that's expected.  those types of workloads are not allowed in the environment.


[torqued] Fix high speed oscillations

Changes

Reduce low speed factor during high speeds (scales down the error)

Friction: Currently full friction is applied at 0.2 error (~4% of instances). Increase to 0.3 (~2% of cases) to reduce oscillations at higher errors.

FWIW, this change significantly improves torque control performance for my Honda Passport.
Significantly less oscillation.
Looking forward to seeing torqued rolled out to more brands!


bin/brew: fix stat on linux

This PR fixes the following error:
$ sudo brew
stat: invalid option -- '%'
Try 'stat --help' for more information.

stat in GNU coreutils uses -c for format, where BSD stat uses -f.

Linuxbrew aims to be portable to both Linux and Mac OS, and stat -c %u does not work on Mac OS.
I went with ls -ld |awk '{print $3}'. I'm not keen that it requires awk.
See http://unix.stackexchange.com/questions/7730/find-the-owner-of-a-directory-or-file-but-only-return-that-and-nothing-else


Added QRDataListener.onReadQrError(Exception) callback for error handling

Currently it is called on failed to open camera, which will happen when:

Any other application did not close the camera properly.
or
You have API22 and below compiled application on a API23+ device and you force revoke the camera permission in the application settings despite the Android warning.

PS Do we need to call it in any other places?

PPS without this fix the app just crashes on opening camera in 2 cases I mentioned above.

@vitaliyistomov Thanks for the PR, let me verify the same and then revert back.


21.7.1: Unbound fails to start up automatically / from GUI

As per https://forum.opnsense.org/index.php?topic=24264.0:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Description
Unbound fails to start up (and stay running) when booting OPNsense - it seems to start initially and then fails/stops.
Unbound fails to start up when pressing "start" in the OPNsense GUI.
Background:

On 21.1, I used Adguard Home  as DNS server on port 53 and Unbound on Port 5553 for local resolves. Setup worked fine.
On a fresh install + import config of 21.7, Unbound shows the above behaviour, i.e. it fails to keep running (Adguard does start and keep running).
upd: Manually running "unbound -c /var/unbound/unbound.conf" works and unbound is operative thereafter (until reboot).

To Reproduce
Steps to reproduce the behavior:

Use Adguard Home on port 53 and unbound on port 5553.
Boot OPNsense.
Unbound enters failed state.

Expected behavior
Unbound to start, bind to port 5553 and keep running.
Describe alternatives you considered
Reboots did not change anything.
Relevant log files
2021-08-08T13:28:07	unbound[2474]	daemonize unbound dhcpd watcher.
2021-08-08T13:28:05	unbound[2663]	daemonize unbound dhcpd watcher.
2021-08-08T13:28:00	unbound[38473]	daemonize unbound dhcpd watcher.
2021-08-08T13:27:52	unbound[60119]	daemonize unbound dhcpd watcher.
2021-08-08T13:27:52	unbound[25826]	[25826:0] info: mesh has 0 recursion states (0 with reply, 0 detached), 0 waiting replies, 0 recursion replies sent, 0 replies dropped, 0 states jostled out
2021-08-08T13:27:52	unbound[25826]	[25826:0] info: server stats for thread 3: requestlist max 0 avg 0 exceeded 0 jostled 0
2021-08-08T13:27:52	unbound[25826]	[25826:0] info: server stats for thread 3: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
2021-08-08T13:27:52	unbound[25826]	[25826:0] info: mesh has 0 recursion states (0 with reply, 0 detached), 0 waiting replies, 0 recursion replies sent, 0 replies dropped, 0 states jostled out
2021-08-08T13:27:52	unbound[25826]	[25826:0] info: server stats for thread 2: requestlist max 0 avg 0 exceeded 0 jostled 0
2021-08-08T13:27:52	unbound[25826]	[25826:0] info: server stats for thread 2: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
2021-08-08T13:27:52	unbound[25826]	[25826:0] info: mesh has 0 recursion states (0 with reply, 0 detached), 0 waiting replies, 0 recursion replies sent, 0 replies dropped, 0 states jostled out
2021-08-08T13:27:52	unbound[25826]	[25826:0] info: server stats for thread 1: requestlist max 0 avg 0 exceeded 0 jostled 0
2021-08-08T13:27:52	unbound[25826]	[25826:0] info: server stats for thread 1: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
2021-08-08T13:27:52	unbound[25826]	[25826:0] info: mesh has 0 recursion states (0 with reply, 0 detached), 0 waiting replies, 0 recursion replies sent, 0 replies dropped, 0 states jostled out
2021-08-08T13:27:52	unbound[25826]	[25826:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0
2021-08-08T13:27:52	unbound[25826]	[25826:0] info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
2021-08-08T13:27:51	unbound[25826]	[25826:0] info: service stopped (unbound 1.13.1).
2021-08-08T13:27:51	unbound[25826]	[25826:0] info: control cmd: dump_cache
2021-08-08T13:27:45	unbound[25826]	[25826:0] info: start of service (unbound 1.13.1).
2021-08-08T13:27:45	unbound[25826]	[25826:0] notice: init module 2: iterator
2021-08-08T13:27:45	unbound[25826]	[25826:0] notice: init module 1: validator
2021-08-08T13:27:45	unbound[25826]	[25826:0] notice: init module 0: dns64
2021-08-08T13:27:39	unbound[25826]	[25826:0] notice: Restart of unbound 1.13.1.
2021-08-08T13:27:39	unbound[25826]	[25826:0] info: mesh has 0 recursion states (0 with reply, 0 detached), 0 waiting replies, 0 recursion replies sent, 0 replies dropped, 0 states jostled out
2021-08-08T13:27:39	unbound[25826]	[25826:0] info: server stats for thread 3: requestlist max 0 avg 0 exceeded 0 jostled 0
2021-08-08T13:27:39	unbound[25826]	[25826:0] info: server stats for thread 3: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
2021-08-08T13:27:39	unbound[25826]	[25826:0] info: mesh has 0 recursion states (0 with reply, 0 detached), 0 waiting replies, 0 recursion replies sent, 0 replies dropped, 0 states jostled out
2021-08-08T13:27:39	unbound[25826]	[25826:0] info: server stats for thread 2: requestlist max 0 avg 0 exceeded 0 jostled 0
2021-08-08T13:27:39	unbound[25826]	[25826:0] info: server stats for thread 2: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
2021-08-08T13:27:39	unbound[25826]	[25826:0] info: mesh has 0 recursion states (0 with reply, 0 detached), 0 waiting replies, 0 recursion replies sent, 0 replies dropped, 0 states jostled out
2021-08-08T13:27:39	unbound[25826]	[25826:0] info: server stats for thread 1: requestlist max 0 avg 0 exceeded 0 jostled 0
2021-08-08T13:27:39	unbound[25826]	[25826:0] info: server stats for thread 1: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
2021-08-08T13:27:39	unbound[25826]	[25826:0] info: mesh has 0 recursion states (0 with reply, 0 detached), 0 waiting replies, 0 recursion replies sent, 0 replies dropped, 0 states jostled out
2021-08-08T13:27:39	unbound[25826]	[25826:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0
2021-08-08T13:27:39	unbound[25826]	[25826:0] info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
2021-08-08T13:27:38	unbound[25826]	[25826:0] info: service stopped (unbound 1.13.1).
2021-08-08T13:27:38	unbound[25826]	[25826:0] info: start of service (unbound 1.13.1).
2021-08-08T13:27:38	unbound[25826]	[25826:0] notice: init module 2: iterator
2021-08-08T13:27:38	unbound[25826]	[25826:0] notice: init module 1: validator
2021-08-08T13:27:38	unbound[25826]	[25826:0] notice: init module 0: dns64
2021-08-08T13:27:31	unbound[66648]	daemonize unbound dhcpd watcher.
2021-08-08T13:27:31	unbound[2967]	[2967:0] info: mesh has 0 recursion states (0 with reply, 0 detached), 0 waiting replies, 0 recursion replies sent, 0 replies dropped, 0 states jostled out
2021-08-08T13:27:31	unbound[2967]	[2967:0] info: server stats for thread 3: requestlist max 0 avg 0 exceeded 0 jostled 0
2021-08-08T13:27:31	unbound[2967]	[2967:0] info: server stats for thread 3: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
2021-08-08T13:27:31	unbound[2967]	[2967:0] info: mesh has 0 recursion states (0 with reply, 0 detached), 0 waiting replies, 0 recursion replies sent, 0 replies dropped, 0 states jostled out
2021-08-08T13:27:31	unbound[2967]	[2967:0] info: server stats for thread 2: requestlist max 0 avg 0 exceeded 0 jostled 0
2021-08-08T13:27:31	unbound[2967]	[2967:0] info: server stats for thread 2: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
2021-08-08T13:27:31	unbound[2967]	[2967:0] info: mesh has 0 recursion states (0 with reply, 0 detached), 0 waiting replies, 0 recursion replies sent, 0 replies dropped, 0 states jostled out
2021-08-08T13:27:31	unbound[2967]	[2967:0] info: server stats for thread 1: requestlist max 0 avg 0 exceeded 0 jostled 0
2021-08-08T13:27:31	unbound[2967]	[2967:0] info: server stats for thread 1: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
2021-08-08T13:27:31	unbound[2967]	[2967:0] info: mesh has 0 recursion states (0 with reply, 0 detached), 0 waiting replies, 0 recursion replies sent, 0 replies dropped, 0 states jostled out
2021-08-08T13:27:31	unbound[2967]	[2967:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0
2021-08-08T13:27:31	unbound[2967]	[2967:0] info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
2021-08-08T13:27:31	unbound[2967]	[2967:0] info: service stopped (unbound 1.13.1).
2021-08-08T13:27:31	unbound[2967]	[2967:0] info: control cmd: dump_cache
2021-08-08T13:27:19	unbound[2967]	[2967:0] info: control cmd: list_local_data
2021-08-08T13:27:19	unbound[2967]	[2967:0] info: start of service (unbound 1.13.1).
2021-08-08T13:27:19	unbound[2967]	[2967:0] notice: init module 2: iterator
2021-08-08T13:27:19	unbound[2967]	[2967:0] notice: init module 1: validator
2021-08-08T13:27:19	unbound[2967]	[2967:0] notice: init module 0: dns64
2021-08-08T13:27:12	unbound[87717]	daemonize unbound dhcpd watcher.
[*** REBOOT ***]
Additional context
Pressing "start" button in the GUI will only generate the following log entry (nothing more!), but not start unbound up:
2021-08-08T13:28:07	unbound[2474]	daemonize unbound dhcpd watcher.
Environment
Software version used and hardware type if relevant, e.g.:
OPNsense 21.7.1-amd64
FreeBSD 12.1-RELEASE-p19-HBSD
OpenSSL 1.1.1k 25 Mar 2021

hi
can confirm unstableu unbound behavior on OPN restart
(in my test VM unbound is "active" but does not serve requests and does not respond to unbound-control commands)
unbound-control can stop working (the connection starts, but there is not even a tls handshake) right after unbound start and executing a couple of unbound-control commands (list_local_data and load_cache).
need to kill -KILL process and start again
@asche77 can you try with "Flush DNS cache during reload" in SERVICES: UNBOUND DNS: GENERAL?

upd: sorry,  "Flush DNS cache during reload" did not help - on the third reboot the unbound "hangs" again
will dig some more

Just to add/clarify that manually running "unbound -c /var/unbound/unbound.conf" does start Unbound up (have edited original post). Until the next reboot of course ...

@asche77  not sure that issues related but have you configure DoT on unbound?

Hi @kulikov-a, no, no DoT:
<unbound>
    <enable>1</enable>
   <dnssecstripped>1</dnssecstripped>
  <domainoverrides/>
 <custom_options/>
    <regdhcp>1</regdhcp>
   <regdhcpstatic>1</regdhcpstatic>
  <txtsupport>1</txtsupport>
    <hideidentity>1</hideidentity>
   <hideversion>1</hideversion>
  <prefetch>1</prefetch>
    <prefetchkey>1</prefetchkey>
   <cache_max_ttl>28800</cache_max_ttl>
  <cache_min_ttl>600</cache_min_ttl>
 <incoming_num_tcp>10</incoming_num_tcp>
    <infra_cache_numhosts>10000</infra_cache_numhosts>
   <infra_host_ttl>900</infra_host_ttl>
  <jostle_timeout>200</jostle_timeout>
 <log_verbosity>2</log_verbosity>
<msgcachesize>4</msgcachesize>
    <num_queries_per_thread>4096</num_queries_per_thread>
   <outgoing_num_tcp>10</outgoing_num_tcp>
  <unwanted_reply_threshold/>
 <dnssec>1</dnssec>
<port>5553</port>
<hosts>
[...]
</hosts>
    <dns64>1</dns64>
    <extended_statistics>1</extended_statistics>
    <acls>
[...]
    </acls>
    <qnameminstrict>1</qnameminstrict>
    <active_interface>lan,opt1,opt2,opt4,opt3</active_interface>
    <cacheflush>1</cacheflush>
  </unbound>

Is it worth getting the logging up from 2?

So unbound is not starting or stops shortly after boot? Have you checked dmesg to see if it crashed?
Cheers,
Franco

@cookiemonsteruk

Is it worth getting the logging up from 2?

I think so.  and may be general log shows some?
and and I apologize to everyone. the problem on my test vm is not related to this one: it seems that the use of the quad9 (<IP_ADDRESS>) dot-forwarder led to unstable work of the unbound (still need to check. has something to do with tcp reuse?) on reboot

I followed up on the lead that this might be caused by <dns64>1</dns64> but that is not the case. I tried squeezing in <active_interface>lan</active_interface> but that also doesn't help reproduce.
I'm pretty sure this is some kind of configuration overlap, but someone with the issue needs to please post the actual steps to reproduce this before we can jump into the fixing.
Cheers,
Franco

So unbound is not starting or stops shortly after boot? Have you checked dmesg to see if it crashed?
dmesg does not show errors other than "module_register_init: MOD_LOAD (vesa, 0xffffffff812907f0, 0) error 19".

Attaching my dmesg.txt.
dmesg.txt
Since it's my home gateway, I am constrained by family re: rebooting at will ...

I think so. and may be general log shows some?
system.log
My system log attached. I tried to anonymise my domain+WiFi name and IP.

Not seeing anything out of the ordinary except maybe:
Aug  8 17:51:25 OPNsense opnsense[62675]: /services_unbound.php: Warning! dhcpd_radvd_configure(auto) found no suitable IPv6 address on igb1
Do you have working IPv6? If not try to disable it on WAN and LAN interfaces..
Cheers,
Franco

Don't think IPv6 is the problem:
IPv6 "None" on WAN and "track interface" on LAN (have now changed to "none").
Firewall setting "Allow IPv6" unchecked.

Just trying to make sure...

Have now rebooted.
Unbound came up fine ...
(I disabled DNS64 and IPv6 on LAN)
Will post logs later tonight.

@bedub1 - thanks for reporting in; your issues sound like those I had seen.
However, I do also use blocklists in unbound and unbound came up fine on reboot yesterday. Things I changed from last reboot:

enable cacheflush
disable DNS64
disable IPv6 by setting "LAN" to "IPv6: none"

Could you check/try if those settings rectify your issue?

IPv6 was already disabled for a long time.  I enabled the CacheFlushing, and it rebooted properly.  When I have more time I'll reboot several times, with the CacheFlushing enabled/disabled and see if it's 100% repro.

Okay I did 7 reboots.  Enabling DNS Caching Flushing seems to fix the problem, but seems to cause other problems.
Flush Cache Enabled (Box Checked)
1st reboot - unbound starts, openvpn client shows offline
2nd reboot - unbound starts, openvpn client connects, ntp doesn't start
3rd reboot - unbound starts, openvpn client shows offline
Flush Cache Disabled (Box Unchecked)
0 - disable Flush Cache
1st reboot - Unbound fails to start, openvpn client connects, ntp starts
2nd reboot - Unbound fails to start, openvpn client connects, ntp starts
2nd reboot - Unbound starts, openvpn client connects, ntp starts
0 - Enable Flush Cache
1st reboot - unbound starts, openvpn client shows offline

@bedub1 @asche77 unbound logs (verb 4 or higher) would be very helpful in trying to understand what's going on when unbound fails to start

@bedub1 @asche77 unbound logs (verb 4 or higher) would be very helpful in trying to understand what's going on when unbound fails to start

Do I need to scrub the logs before attaching?  I looked through it, and the only personal information I see is my LAN IP Address.

it's up to you imho )

Logs: I had my top level domain in there, as well as my public IP. I removed both.

Reboot: I have just rebooted and unbound has come up fine.
Going to try re-enabling my prior settings (cumulatively):
Turning DNS64 back on: unbound eats a lot of cpu initially but comes up fine.
Turning IPv6 on LAN back on (track interface): unbound comes up fine. Unlike before, DHCPv6 server does not show up in services dashboard (before, it did with failed state).
Turning cacheflush back off:  TA-DAH! Unbound fails. Logs attached.
Turning DNS64 and IPv6 track interface back off but leaving cacheflush disabled: TA-DAH! Unbound fails. Logs attached.
Turning cacheflush back on: unbound starting up fine.

Can you run the following after a reboot when it doesn't start:
sh -x /usr/local/opnsense/scripts/unbound/start.sh
And this is 100% 27.1.1 and not 27.1?

This is:

Versions
OPNsense 21.7.1-amd64FreeBSD 12.1-RELEASE-p19-HBSDOpenSSL 1.1.1k  25 Mar 2021

root@OPNsense:~ # sh -x /usr/local/opnsense/scripts/unbound/start.sh

set -e
chroot -u unbound -g unbound / /usr/local/sbin/unbound-anchor -a /var/unbound/                                                                              root.key
[ ! -f /var/unbound/unbound_control.key ]
find /var/unbound/etc -depth 1
rm -rf /var/unbound/etc/dot.conf
rm -rf /var/unbound/etc/dnsbl.conf
rm -rf /var/unbound/etc/miscellaneous.conf
find /usr/local/etc/unbound.opnsense.d -depth 1 -name '*.conf'
cp /usr/local/etc/unbound.opnsense.d/dot.conf /var/unbound/etc/
cp /usr/local/etc/unbound.opnsense.d/miscellaneous.conf /var/unbound/etc/
cp /usr/local/etc/unbound.opnsense.d/dnsbl.conf /var/unbound/etc/
chown -R unbound:unbound /var/unbound
/usr/local/sbin/unbound -c /var/unbound/unbound.conf
/usr/local/opnsense/scripts/unbound/cache.sh load
ok
root@OPNsense:~ #

It's funny because that's the script it's supposed to run in the first place.

It's probably the same behaviour as on boot: it runs briefly and then stops/crashes after a few secs.

reboot with cacheflush = OFF, unbound enters failed state shortly after boot, running manual startup:
root@OPNsense:/var/log # sh -x /usr/local/opnsense/scripts/unbound/start.sh                                                                            + set -e + chroot -u unbound -g unbound / /usr/local/sbin/unbound-anchor -a /var/unbound/root.key + [ ! -f /var/unbound/unbound_control.key ] + find /var/unbound/etc -depth 1 + rm -rf /var/unbound/etc/dot.conf + rm -rf /var/unbound/etc/dnsbl.conf + rm -rf /var/unbound/etc/miscellaneous.conf + find /usr/local/etc/unbound.opnsense.d -depth 1 -name '*.conf' + cp /usr/local/etc/unbound.opnsense.d/dot.conf /var/unbound/etc/ + cp /usr/local/etc/unbound.opnsense.d/miscellaneous.conf /var/unbound/etc/ + cp /usr/local/etc/unbound.opnsense.d/dnsbl.conf /var/unbound/etc/ + chown -R unbound:unbound /var/unbound + /usr/local/sbin/unbound -c /var/unbound/unbound.conf [1629020885] unbound[43905:0] debug: creating udp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating tcp4 socket <IP_ADDRESS> 5553 [629020885] unbound[43905:0] debug: creating udp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating tcp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating udp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating tcp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating udp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating tcp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating udp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating tcp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating udp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating tcp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating udp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating tcp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating udp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating tcp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating udp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating tcp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating udp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating tcp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating udp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating tcp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating udp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating tcp4 socket <IP_ADDRESS> 5553 [1629020885] unbound[43905:0] debug: creating tcp4 socket <IP_ADDRESS> 953 [1629020885] unbound[43905:0] debug: setup SSL certificates [1629020885] unbound[43905:0] debug: switching log to syslog + /usr/local/opnsense/scripts/unbound/cache.sh load ok root@OPNsense:/var/log #`
Unbound is up

resolver.log looks quite normal. just a few commands to restart are visible (on .bootip, on newwanip etc).
there is no corresponding entries in system.log for 11:52:07 when unbound gracefully stopped

then it enters red "failed" state
sorry, any chance that its just GUI?  unbound needs some time to start (even more with bla(o)ck-lists) and GUI may show "red stop" until page refresh

Unfortunately, not only GUI - unbound also stops resolving (i.e. I cannot resolve local DHCP registered hosts).

sorry, completely copied your config from https://github.com/opnsense/core/issues/5150#issuecomment-894844062 (without cacheflash) and can't reproduce the behavior
any chance that there is an event in the system log explaining the unbound stop at 11:52:07?
Aug 15 11:52:07 OPNsense unbound[51361]: [51361:0] info: service stopped (unbound 1.13.1).  Aug 15 11:52:07 OPNsense unbound[51361]: [51361:0] debug: stop threads  Aug 15 11:52:07 OPNsense unbound[51361]: [51361:0] debug: join 1  Aug 15 11:52:07 OPNsense unbound[51361]: [51361:2] debug: got control cmd quit  Aug 15 11:52:07 OPNsense unbound[51361]: [51361:3] debug: got control cmd quit  Aug 15 11:52:07 OPNsense unbound[51361]: [51361:1] debug: got control cmd quit

I apologize for being off-topic, but also related to unbound problems at start..
@fichtner is it possible to take https://github.com/NLnetLabs/unbound/commit/d88f554503cbb31b924087f82ecd774145506a8f into account when you will update the unbound? thnks!

It looks like the problem is that the cache-load command of unbound hangs:
root@opnsense-host:~ # ps aux -d | grep unbound
-- /usr/local/bin/flock -n -E 0 -o /tmp/unbound_start.lock /usr/local/opnsense/scripts/unbound/start.sh
...\-- /bin/sh /usr/local/opnsense/scripts/unbound/start.sh
.......\-- /bin/sh /usr/local/opnsense/scripts/unbound/cache.sh load
...........\-- /usr/local/sbin/unbound-control -c /var/unbound/unbound.conf load_cache
I could imagine that unbound-control connects to the remote-control port of unbound via localhost to update the cache and in the meantime the state reset function for dial-up connections kills all network connections (including loopback connections). We also see this of broken ssh connections to the console.
By the way, the script "/usr/local/opnsense/scripts/unbound/cache.sh" does not check any return codes of executed commands. This is not a good idea.

Good news: on 21.7.2 even with cache flush disabled, unbound came up fine.
So can't really add anything to this issue right now.

Good news: on 21.7.2 even with cache flush disabled, unbound came up fine.

Unfortunately, this issue is not solved. The start of unbound still hung after first boot of Opnsense version 21.7.2.

By the way, the script "/usr/local/opnsense/scripts/unbound/cache.sh" does not check any return codes of executed commands. This is not a good idea.

Politically correct would be to say that if unbound-control fails to dump its cache and then fails to load its dump that is a pretty unusable situation for any caller.
We still need to know why it dumps partial or invalid data or how we could possibly verify that data... ideally hoping the process that knows the data could tell us. ;)
Last but not least cache.sh sets "set -e" which aborts on errors throws by unbound-control, so I guess it doesn't... so we can't check for errors? It's a good suggestion to check for errors, but it's not very well grounded in reality.
Cheers,
Franco

Maybe we can link this issue to #5367, because it looks like issues with the dynamic state reset function interfere with the start of unbound.

I experience this issue and have never had Dynamic state reset enabled, and my Unbound fails to start on boot.

@evulhotdog so not your bug, maybe you should take a look at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
Thanks,
Franco

Going to close this via  #5367


Advanced options: load any model from HF or allow local models

Right now, the model from HF is hard-coded and supports only English. While it is just one line of code and one could easily recompile the project, it would be nicer to have advanced options for custom models, either from HF or local ones in the GUI e.g. to support other languages too.
However, usability shouldn't be sacrificed, so I'd stick to all-MiniLM-L6-v2 as default because it works great with English and is super small. Under the advanced options section there should be either a drop down or a combined input text field for a custom URL or for file uploads (local models).
Not sure what's best though, but the latter seems more versatile and would clutter the UI less than having two different inputs, drop-down and input text box. But a drop-down would be more convenient for the average user and offer some space for notes (link to model explanation, supported languages etc.) maybe...

This gets to a good point which will probably be my next task. WebML needs more comprehensive benchmarking and a better grasp on how "large" we can actually get in-browser. I'll benchmark some of the existing webml models in terms of memory performance and some canonical measurements for their task. Then I'll basically verify they work on various browsers (will probably test chrome, safari, mobile.). Hopefully I can make something resembling HF leaderboards:
https://huggingface.co/spaces/optimum/llm-perf-leaderboard
I'd guess Xenova has a good internal intuition for how big is possible but the data should be more readily available.

I started working on this repo as basis for an  npm package that would allow anyone injecting their content and searching on it.
Something with an API along these lines
async init(modelName: string, modelConfig: ModelConfig) -> bool
async embedContent(content: string, config: EmbeddingConfig) -> EmbeddingMap
async search(query: string, threshold: number, maxResults: number) -> Array<SearchResult> // This could also return an Observable and stream results as they are found

Does this align in any way with your plans? Do you envision something else for this repo?

That's a really cool idea to create an npm package with an easy-to-use API - I support this 100%!
Thinking about it, it sounds like the right way to go as well, decoupling the semantic search logic and the frontend. Also this would benefit the browser plugin idea.
I was already thinking about a more efficient logic considering the indexing part / vector storage. I pinged the folks from Qdrant here (initial question) and here (answer) whether they were interested in providing a WASM version of Qdrant for faster and more efficient vector storage / querying. They said it's not that easy but at some point they might give it a try, so no ETA yet. The reason why I was interested are all the optimizations that would make quite a difference for larger texts / more segments.
It would be interesting to port semantic.js to Rust, then compile to WASM and see whether there are speed and memory benefits. To my understanding there should be as Rust has more fine-grained control about data structures. I'm not an expert on Rust though, just what friends told me. :D

I created the semantic-js library.
It exposes the methods I mentioned - init, embedContent and search.
I also created a version of SemanticFinder that uses it (it's currently in a branch, but I could merge it back into yours).
Let me know what you think!

That's pretty cool! Let me have a closer look at the library tomorrow, but as said earlier I do appreciate the modular idea :)

With the latest changes SemanticFinder supports all models tagged with feature-extraction on HF. TBH I don't know how frequent it would happen that people would really like to test their own models locally - there is always the option of hosting it on HF and adding an option in index.html like I did for do-me/test for testing some models. Closing for now but if there's still interest in using local models we might reopen any time.


Remove all Opbeat Logging

We still have Opbeat on the server, we should remove it as it will not be very helpful next week.
This issue is a good fit for anyone that can delete code :grin:

On it B)

@Bouncey I can work on it, does it already picked up by @tamj0rd2 ?
let me know

Yep I have @rishiip PR coming shortly

@tamj0rd2 no worries.


Improve consistency of the gpu-arch-specs table.

There was inconsistency in the CU count where 250X lists per GCD, but 250 does not. In keeping with the per GCD convention, it would make sense to list CU and L2 cacher per XCD for MI300 as well.

Thanks for the suggested fixes.  Since you added the XCD acronym, please add that to the Glossary at the bottom of the page as well (for completeness).
XCD = Accelerated Complex Die.


watch mode is broken

Describe the bug
No console messages while watching for file changes, earlier it used to be there.
Also if --reloadHandler is not passed to serverless offline, then reload doesn't happen at all
Versions (please complete the following information):

OS: [e.g. Linux, Mac, Windows] linux
Serverless Framework Version: [e.g. 3.0.0]
"serverless-esbuild": "^1.33.0",
"serverless-offline": "^10.3.2"
Plugin Version: [e.g. 1.25.0]

Additional context
Add any other context about the problem here.

--verbose flag in to serverless fixed this


Unable to verify Thirdweb smart contract on Celo

According to the docs:
import { ThirdwebSDK } from "@thirdweb-dev/sdk";

const sdk = new ThirdwebSDK("celo", {
    secretKey: "XXX",
})
const explorerAPIUrl = "https://api.celoscan.io/api"
const explorerAPIKey = "XXX"

await sdk.verifier.verifyThirdwebContract(
    "ManagedAccountFactory",
    explorerAPIUrl,
    explorerAPIKey,
);

Output:
/tmp/thirdweb/node_modules/.pnpm/@thirdweb-dev+sdk@4.0.50_ethers@5.7.2_typescript@5.4.3/node_modules/@thirdweb-dev/sdk/dist/index-1e72d576.cjs.dev.js:5109
      throw new Error("Can't resolve constructor arguments");
            ^

Error: Can't resolve constructor arguments
    at /tmp/thirdweb/node_modules/.pnpm/@thirdweb-dev+sdk@4.0.50_ethers@5.7.2_typescript@5.4.3/node_modules/@thirdweb-dev/sdk/dist/index-1e72d576.cjs.dev.js:5109:13
    at Array.map (<anonymous>)
    at encodeConstructorParamsForImplementation (/tmp/thirdweb/node_modules/.pnpm/@thirdweb-dev+sdk@4.0.50_ethers@5.7.2_typescript@5.4.3/node_modules/@thirdweb-dev/sdk/dist/index-1e72d576.cjs.dev.js:5075:70)
    at Object.computeDeploymentInfo (/tmp/thirdweb/node_modules/.pnpm/@thirdweb-dev+sdk@4.0.50_ethers@5.7.2_typescript@5.4.3/node_modules/@thirdweb-dev/sdk/dist/index-1e72d576.cjs.dev.js:5041:29)
    at Object.getDeploymentInfo (/tmp/thirdweb/node_modules/.pnpm/@thirdweb-dev+sdk@4.0.50_ethers@5.7.2_typescript@5.4.3/node_modules/@thirdweb-dev/sdk/dist/deployContractDeterministic-31e620a3.cjs.dev.js:140:76)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async predictThirdwebContractAddress (/tmp/thirdweb/node_modules/.pnpm/@thirdweb-dev+sdk@4.0.50_ethers@5.7.2_typescript@5.4.3/node_modules/@thirdweb-dev/sdk/dist/sdk-3a69b331.cjs.dev.js:120:28)
    at async getThirdwebContractAddress (/tmp/thirdweb/node_modules/.pnpm/@thirdweb-dev+sdk@4.0.50_ethers@5.7.2_typescript@5.4.3/node_modules/@thirdweb-dev/sdk/dist/sdk-3a69b331.cjs.dev.js:147:27)
    at async verifyThirdwebPrebuiltImplementation (/tmp/thirdweb/node_modules/.pnpm/@thirdweb-dev+sdk@4.0.50_ethers@5.7.2_typescript@5.4.3/node_modules/@thirdweb-dev/sdk/dist/sdk-3a69b331.cjs.dev.js:203:27)
    at async ContractVerifier.verifyThirdwebContract (/tmp/thirdweb/node_modules/.pnpm/@thirdweb-dev+sdk@4.0.50_ethers@5.7.2_typescript@5.4.3/node_modules/@thirdweb-dev/sdk/dist/sdk-3a69b331.cjs.dev.js:525:18)

Node.js v21.7.1 with sdk version ^4.0.50

Edit:
I tried to verify on the Dashboard. It reports the verification as successful but on Celoscan it isn't verfied. That is a separate bug related to the Thirdweb SaaS.

Hey @kamikazechaser we're having some trouble currently with contract verification on Celo.
We're looking into it. Will let you know once this is resolved.


hosts field not exposed in the AuthPolicy

Motivation
Authorino will reject (with 404 Not Found) any request for which there is no matching authconfig object. Exposing the hosts in the spec.authScheme object can lead to unwanted scenarios for the end user and it is, at the very least, error prone. One example to illustrate:
kind: HTTPRoute:
spec:
  hostnames: [`*.company.com`]
---
kind: AuthPolicy
spec:
  authScheme:
    hosts: ["api.petstore.company.com"]

The route allows traffic for *.company.com, but the authconfig object have rules only for api.petstore.company.com, then a request for other.company.com will hit authorino and authorino will reject that traffic. Something clearly unwanted.
Kuadrant core should provide means to configure automatically the authorization process to only protect the wanted domains and pass through the remaining ones.
What

Regarding managed AuthConfig

The proposal is to remove hosts from spec.authScheme. The Authconfig object still needs a hosts field, so it will be the kuadrant core, the owner of the authconfig object, who will be filling the hosts field.
The hosts field will be computed in the following way:

When there are no rules, authconfig's hosts field will be read from the network resource (HTTPRoute or Gateway).
When there is at least one rule with the hosts field empty/missing, authconfig's hosts field will be read from the network resource (HTTPRoute or Gateway).
authconfig's hosts field will be the list of hosts appearing in the authpolicy rules

For example:
kind: HTTPRoute:
spec:
  hostnames: [`*.petstore.com`]
----
kind: AuthPolicy
spec:
  rules:
  - hosts: ["api.petstore.com"]
    methods: ["GET"]
  - methods: ["POST"]

in this example, there is a rule with the hosts missing. Then the authconfig's hosts field will be [*.petstore.com]
For example:
kind: HTTPRoute:
spec:
  hostnames: [`*.petstore.com`]
----
kind: AuthPolicy
spec:
  rules:
  - hosts: ["api.petstore.com"]
    methods: ["GET"]
  - hosts: ["admin.petstore.com"]
    paths: ["/admin*"]

in this example, there is a rule with the hosts missing. Then the authconfig's hosts field will be [api.petstore.com, admin.petstore.com]

Regarding managed Istio's AuthorizationPolicy

If there is a rule in the kuadrant's auth that does not have hosts specified, kuadrant will add the hostnames from the network resource when reconciling the AuthorizationPolicy.
For example:
kind: AuthPolicy
spec:
  rules:
  - paths: ["/toy*"]

The reconciled Istio's AuthorizationPolicy will include the network resource's (route) hostnames
piVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
spec:
  action: CUSTOM
  provider:
    name: kuadrant-authorization
  rules:
  - to:
    - operation:
        hosts:
        - '*.toystore.com'
        paths:
        - /toy*
  selector: {}

The Istio's authorization policies configure a common (shared) envoy's authorization filter. The rules coming from a kuadrant's policy targeting a given route should all be scoped to the route's traffic workloads.

For completeness, I think it is valid.


BinderCustomizer issues

See this SO issue: https://stackoverflow.com/questions/77597349/how-to-pass-bindercustomizer-to-spring-cloud-streams-binderconfig

Fixed via: https://github.com/spring-cloud/spring-cloud-stream/commit/b09892904864f1fe120a7d33ab3deb3a4d581063