Consistent auth failed

Similar to https://github.com/samm-git/aws-vpn-client/issues/5
I am also getting AUTH_FAILED and cannot overcome it for the life of me. It is a 100% failure rate
Getting SAML redirect URL from the AUTH_FAILED response (host: 54.189.XXX.YYY:443)
Opening browser and wait for the response file...
Running OpenVPN with sudo. Enter password if requested
2021-12-31 01:28:28 OpenVPN 2.5.5 [git:makepkg/e3bac09f6a128260+] x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Dec 31 2021
2021-12-31 01:28:28 library versions: OpenSSL 1.1.1l  24 Aug 2021, LZO 2.10
2021-12-31 01:28:28 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2021-12-31 01:28:28 TCP/UDP: Preserving recently used remote address: [AF_INET]54.189.XXX.YYY:443
2021-12-31 01:28:28 Socket Buffers: R=[212992->212992] S=[212992->212992]
2021-12-31 01:28:28 UDP link local: (not bound)
2021-12-31 01:28:28 UDP link remote: [AF_INET]54.189.XXX.YYY:443
2021-12-31 01:28:28 TLS: Initial packet from [AF_INET]54.189.XXX.YYY:443, sid=100f471a e691210f
2021-12-31 01:28:28 VERIFY OK: depth=1, CN=Mason America Inc
2021-12-31 01:28:28 VERIFY KU OK
2021-12-31 01:28:28 Validating certificate extended key usage
2021-12-31 01:28:28 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
2021-12-31 01:28:28 VERIFY EKU OK
2021-12-31 01:28:28 VERIFY OK: depth=0, CN=VPN-Server
2021-12-31 01:28:28 Control Channel: TLSv1.2, cipher TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256
2021-12-31 01:28:28 [VPN-Server] Peer Connection Initiated with [AF_INET]54.189.XXX.YYY:443
2021-12-31 01:28:29 SENT CONTROL [VPN-Server]: 'PUSH_REQUEST' (status=1)
2021-12-31 01:28:29 AUTH: Received control message: AUTH_FAILED,Invalid username or password
2021-12-31 01:28:29 SIGTERM[soft,auth-failure] received, process exiting

My conf looks like this:
client
dev tun
proto udp
resolv-retry infinite
nobind
remote-cert-tls server
cipher AES-256-GCM
verb 3
<ca>
snipped cert
</ca>

auth-nocache
reneg-sec 0

We're also using Okta for SAML. I also found through experimenting that the port has to be 443 for anything to happen.
I'm trying this on manjaro linux. Here's the script for how I compiled openvpn 2.5.5: https://github.com/AnilRedshift/awsvpn-saml/blob/main/PKGBUILD
In that repository, I also forked my own code for the server & script, but I get the same error regardless if I try my code or the code in this repository.
Things I've tried

Changing ports
Changing the host to use the URL
Ensured that the SAML token coming back is valid (decoded the base64 and it looks like valid xml)
Ran tcpdump on the socket to verify that there is traffic
Tried with openvpn 2.4.11 instead of 2.5.5 (using the correct corresponding patch)

If you want to repro my exact setup, then either in arch or manjaro, you can run git clone https://github.com/AnilRedshift/awsvpn-saml and then makepkg -i
That will install openvpn, and then you just need to use OVPN_BIN="/usr/share/awsvpn-saml/bin/openvpn" to run the aws-connect.sh script in this repo.
(or you can run the awsvpn shell script installed by my makepkg if you want to use 100% of my code

Aha! Figured out my issue. I was compiling openvpn with these options:
--enable-pkcs11
--enable-plugins
--enable-systemd
--enable-x509-alt-username
Removing this in the ./configure step fixed the issue

Awesome, glad I looked through the closed ticket because this now works for me even with the latest openvpn 2.6.3 and a small patch tweak. I just had to remove --enable-pkcs11 in my newer openvpn-aws.rb based on the 2.6.3 recipe.


Timesheet - 29/12/2021 - Edward aka SwedishFisherman

Task 1
Creating 3 social graphics for the 12 Days of Bitty campaign
Proof of Work

Time Spent
6 hours

Task 2
Created social graphics for the Dec Chess tourney campaign
Proof of Work

Time Spent
2 hours

Total Time Spent
8 hours
Account Number
a3e779bdb64f28307177244ac2b42e209519c163cdd18c92c303a93642dcc00f

Hey @SwedishFisherman, do you have any kind of version history for these?

@Kycermann Unfortunately, Photoshop and Illustrator does not have a version history. The creation of these graphical art pieces always varies.


package 'vscDebugger' not installed because it is not built for UCRT

Error in unpackPkgZip(pkgs[i], pkgnames[i], lib, libs_only, lock, quiet) :
package 'vscDebugger' not installed because it is not built for UCRT
Calls: install.packages -> .install.winbinary -> unpackPkgZip

Any solution for this error? It's happening to me too in VS Code but I'm not sure to what is due.

A workaround is to build it from source using R 4.2 on Windows.
The error relates to the UCRT runtime that is used starting from version 4.2 to support native UTF-8 encoding for Windows.
This is my binary build from simply cloning the repo and building the binary packages via RStudio:
vscDebugger_0.4.7.zip
I have not seen the error since.

Thanks @mvwestendorp, it worked perfectly.

thanks you are the real hero

@mvwestendorp  you are so cool！！！😎

does anyone know what command to run in github codespaces

does anyone know what command to run in github codespaces

I don't think this issue is related to Linux on Codespaces as it only concerns Windows.
You should be able to use the debugger-configured container using the template below.
https://github.com/rocker-org/devcontainer-templates/tree/main/src/r-ver


v2.1 autosnat support

Fixes issues w/ 2.1 and autosnat configuration parameter. (#102)

This is not ready to be merged; it doesn't reliably enable auto snat on 2.1 series devices.

This should work on versions 2.7.2 V3, indicate that's required for ip in ip in the docs.


Update Set-SPServerScaleOutDatabaseDataRange.md

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

sharepoint/sharepoint-ps/sharepoint/Set-SPServerScaleOutDatabaseDataRange.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Add Antenore Guaccero

@VitantonioS First stage of verification passed, official domain is icantenoreguaccero.edu.it as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.

@VitantonioS Thank you for your patience while waiting for a review and processing your request. Pull request merged.

Hi @philipto
sorry I have a problem, all alumni emails are<EMAIL_ADDRESS>but the registered school domain is different.
During registration so I cannot proceed as it is not verified by the system.
Could you please help me?
Thank you very much,
Best regards


Delete WeekSubscribe method

Description of Change
As discussed in #1001, deleted WeekSubscribe since it's most likely will be misused.
It's not released yet, so this is not a breaking change.
This method is now released in 1.1.0. Maybe it's deletion should be reevaluated.
Bugs Fixed

Fixes #1001

API Changes
Removed:

void WeakSubscribe<T>(this INotifyPropertyChanged target, T subscriber, Action<T, object?, PropertyChangedEventArgs> action)

Behavioral Changes
None
PR Checklist

[ ] Has tests (if omitted, state reason in description)
[ ] Has samples (if omitted, state reason in description)
[x] Rebased on top of main at time of PR
[x] Changes adhere to coding standard

[ ] Updated documentation

We would need to first mark it as [Obsolete] in the next release.
Then we could remove it in the next major release.

@brminnick oh yeah, We already released the 1.1 in a stable channel! I forgot it. @maxkoshevoi please restore that file and add the Obsolete attribute as suggested by @brminnick, thanks.

@pictos Done

Thanks, @maxkoshevoi, now we just need to fix the CI to merge this puppy (:

@pictos This is not related to localization. WeekSubscribe to subscribe to any property change event.

@pictos This is not related to localization. WeekSubscribe to subscribe to any property change event.

but this isn't created because of LocalizedString feature? I didn't found any better label for that

but this isn't created because of LocalizedString feature? I didn't find any better label for that

Yes, but it's still a general purpose method.
Yeah, there isn't really a label for it 😄 What about Breaking change? Most users have Treat warnings as errors set, so this will be a breaking change for them.


Add x25519 based image encryption

Working support for the simulator and bootutil. Still missing imgtool x25519 key generation and encryption, zephyr+mynewt support, THIS WAS NOT YET TESTED IN HW.

Working support for the simulator and bootutil. Still missing imgtool x25519 key generation and encryption, zephyr+mynewt support, THIS WAS NOT YET TESTED IN HW.

Now it has imgtool support for generating x25519 private keys and encrypting images; Zephyr support was added and tested (nrf52840_pca10056) and docs are "updated".
TODO: will add another commit with Mynewt support once tested.

This looks good. Let me know if you want to merge this before the MyNewt support comes in, and I can approve it.

Mynewt added.


Run the conformance tests against CAPA

Task
Run the conformance tests against CAPA clusters.
Background
The conformance tests haven't been executed against CAPA clusters in a very long time. The tests passed last time they were executed.
TODOs

[ ] Execute the conformance tests;
[ ] Create follow-up issues for any problem encountered.

References:

https://github.com/cncf/k8s-conformance/blob/master/instructions.md

{"msg":"FAILED [sig-cli] Kubectl client Guestbook application should create and stop a working application  [Conformance]","total":356,"completed":219,"skipped":4247,"failed":3,"failures":["[sig-auth] ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance]","[sig-scheduling] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run  [Conformance]","[sig-cli] Kubectl client Guestbook application should create and stop a working application  [Conformance]"]}

{"msg":"FAILED [sig-scheduling] SchedulerPredicates [Serial] validates that NodeSelector is respected if not matching  [Conformance]","total":356,"completed":304,"skipped":5780,"failed":4,"failures":["[sig-auth] ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance]","[sig-scheduling] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run  [Conformance]","[sig-cli] Kubectl client Guestbook application should create and stop a working application  [Conformance]","[sig-scheduling] SchedulerPredicates [Serial] validates that NodeSelector is respected if not matching  [Conformance]"]}

{"msg":"FAILED [sig-scheduling] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run  [Conformance]","total":356,"completed":109,"skipped":2114,"failed":2,"failures":["[sig-auth] ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance]","[sig-scheduling] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run  [Conformance]"]}

{"msg":"FAILED [sig-auth] ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance]","total":356,"completed":90,"skipped":1727,"failed":1,"failures":["[sig-auth] ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance]"]}

The last one is common for grizzly,golem and goat. Other three were seen both at grizzly and goat.

I will create separate issues to track them.


Handle internationalisation (i18n) with proper pluralisation

For example, if only 1 user is registered, this is shown (code here):

+1
Support for I18n would be great.

Also see #248


Multiple element selector only hiding first match

Describe your question with as much detail as possible
Hiding only one of the elements matching a selector when updating to puppeteer. It seems the removal/hidden elements is passed through all the way to the client script but I have been having trouble debugging where exactly it might break. My question is basically do I need to debug a visual Chrome execution to get access to the client script logic?
My selector is something like browser.$$('[data-thing="MULTIPLE"]') and I make sure they are all displayable elements before sending them in. When in puppeteer we are getting HTMLElements or are they still WebElements?
If it's about a specific piece of code, try and include some of it to support your question.
https://github.com/wswebcreation/webdriver-image-comparison/blob/master/lib/clientSideScripts/hideRemoveElements.ts#L30
Environment (please complete the following information):

Node.js version: 10.16.3
NPM version: locally: 6.14.4, docker: 6.9.0
Browser name and version: locally: Chrome 81.0.4044.113, docker: Chrome 79
Platform name and version: locally: OSX, docker: linux
WebdriverIO version: 6.1.0
wdio-image-comparison-service version: 1.13.0

Additional context
Add any other context about the problem here.

Thanks for filing an issue, I'll take a look at it in the coming days

Hi @kevinlacotaco
I've been checking this with this version:
  "devDependencies": {
    "@wdio/cli": "^6.1.3",
    "@wdio/devtools-service": "^6.0.16",
    "@wdio/jasmine-framework": "^6.1.3",
    "@wdio/local-runner": "^6.1.3",
    "@wdio/spec-reporter": "^6.0.16",
    "@wdio/sync": "^6.1.0",
    "devtools": "^6.1.3",
    "webdriverio": "^6.1.3",
    "webdriver-image-comparison": "0.14.1"
  },

and this script
    describe('compare screen', () => {
        it('should compare successful with a baseline', () => {
            browser.url('https://www.saucelabs.com')
            browser.saveScreen('with-links-hl')
            browser.saveScreen('wihtout-links-hl', {
                hideElements: [$$(('.link'))],
            })
        });
    });

And this is the result, first image without hiding the elements, with-links-hl

And the second image with hiding the links

It looks ok, can you check with the latest versions? If it still is a problem, can you provide a repo with a reproducible example?

Ok, let me try to get a reproducible example. I am wondering if its due to [data-x] attributes instead of classes?

I threw this together: https://github.com/wswebcreation/webdriver-image-comparison/pull/68 which seems to have solved my issue.

Should be fixed now with https://github.com/wswebcreation/wdio-image-comparison-service/releases/tag/v1.13.2


Error: spawn ffmpeg ENOENT

Hello,
First, thank you for your effort. When I try to run my serverside api.js I always get 'Error: spawn ffmpeg ENOENT' error. I couldn't find a solution. I use Windows pc. Does it matter ? If you can respond I wil be really glad.

And my api.js code is this :
Stream = require("node-rtsp-stream");
stream = new Stream({
name: "name",
streamUrl: myurl,
wsPort: 9999,
ffmpegOptions: {
// options ffmpeg flags
"-stats": "", // an option with no neccessary value uses a blank string
"-r": 30 // options with required values specify the value after the key,
}
});

ENOENT usually means that node is trying to run a command that doesn't exist. Are you sure you have ffmpeg installed?
On Tue, Mar 26, 2019, at 10:03 AM, anllyksell wrote:

And my api.js code is this :

Stream = require("node-rtsp-stream");
stream = new Stream({
name: "name",
streamUrl: myurl,
wsPort: 9999,
ffmpegOptions: {
// options ffmpeg flags
"-stats": "", // an option with no neccessary value uses a blank string
"-r": 30 // options with required values specify the value after the key,
}
});

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/kyriesent/node-rtsp-stream/issues/28#issuecomment-476656738, or mute the thread https://github.com/notifications/unsubscribe-auth/ABatsi56nmMnefZkKw8HR16bFIY_Mw3vks5vaiiqgaJpZM4cLhiW.

It was and its path was clear. I'm using Windows. On Linux without needing anything else it worked. But on Windows I needed to put ffmpeg.exe into project folder. I didn't understand why it was different on differend os. But it worked. You can close issue if you want. Thank you for your response :)

Same problem，using windows and cannot find the ffmpeg，but I dont wanna copy ffmpeg.exe to the project folder. How could I set the config to let the project find the ffmpeg?

how to get ffmpeg.exe?

how to get ffmpeg.exe?

how to get ffmpeg.exe?

I got it, thank you

Should be closed by #50


Add new FindByPrefix Command

closes #45

Codecov Report

Merging #103 (70f754a) into master (c216ec6) will decrease coverage by 1.19%.
The diff coverage is 25.96%.

@@             Coverage Diff              @@
##             master     #103      +/-   ##
============================================
- Coverage     43.84%   42.65%   -1.20%
- Complexity      522      543      +21
============================================
  Files           130      138       +8
  Lines          2689     2893     +204
  Branches        299      333      +34
============================================
+ Hits           1179     1234      +55
- Misses         1437     1565     +128
- Partials         73       94      +21

Impacted Files
Coverage Δ

...rc/main/java/seedu/address/logic/LogicManager.java
60.00% <0.00%> (-2.50%)
:arrow_down:

...du/address/logic/commands/FindByPrefixCommand.java
0.00% <0.00%> (ø)

...model/person/AddressContainsKeywordsPredicate.java
0.00% <0.00%> (ø)

...rc/main/java/seedu/address/model/person/Email.java
90.00% <ø> (+10.00%)
:arrow_up:

...s/model/person/EmailContainsKeywordsPredicate.java
0.00% <0.00%> (ø)

...main/java/seedu/address/model/person/NusNetId.java
0.00% <ø> (ø)

...rc/main/java/seedu/address/model/person/Phone.java
90.00% <ø> (+10.00%)
:arrow_up:

...s/model/person/PhoneContainsKeywordsPredicate.java
0.00% <0.00%> (ø)

...del/person/StudentIdContainsKeywordsPredicate.java
0.00% <0.00%> (ø)

...ess/model/person/TagContainsKeywordsPredicate.java
0.00% <0.00%> (ø)

... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c216ec6...70f754a. Read the comment docs.


Invalid argument(s): Failed to load dynamic library ''/data/data/PACKAGE/lib/libsqlite3.so'': dlopen failed: library "/data/data/PACKAGE/lib/libsqlite3.so" not found

In my production app I am facing an error with Moor that only happens on some Android 5 and Android 6 devices when opening the database:
Here is how I open the database:
Future<MoorIsolate> _createMoorIsolate() async {
  final dir = await paths.getApplicationDocumentsDirectory();
  final dirOfOldAndroidDatabase = Directory(dir.path + "/databases");
  if(Platform.isAndroid) {
    if (!await dirOfOldAndroidDatabase .exists()) {
      await dirOfOldAndroidDatabase .create(recursive: true);
    }
  }
  final path = Platform.isIOS ? p.join(dir.path, 'db.db') : p.join(dir.path.replaceAll("/app_flutter", "/databases"), 'db.db');
  final receivePort = ReceivePort();

  await Isolate.spawn(
    _startBackground,
    _IsolateStartRequest(receivePort.sendPort, path),
  );

  return (await receivePort.first as MoorIsolate);
}

The reason why I am replacing the path on Android is that I upgraded from a native Android app to Flutter and wanted to hold the old database.

In _startBackground, can you try awaiting applyWorkaroundToOpenSqlite3OnOldAndroidVersions before opening the database? There's more information here.

Thanks for your quick response, in your readme you are suggesting 2 options. So to get the highest chance that it will work can I use both options together?

If I remember correctly, the applyWorkaroundToOpenSqlite3OnOldAndroidVersions workaround works without android.bundle.enableUncompressedNativeLibs=false.
cc @knaeckeKami, do you remember whether opening the library in Java fixed the other loader issues too? Which one did you end up using?

I will try it. I changed the code like this is that the correct place for waiting for that call?
Future<MoorIsolate> _createMoorIsolate() async {

  final dir = await paths.getApplicationDocumentsDirectory();
  final dirOfOldAndroidDatabase= Directory(dir.path + "/databases");
  if(Platform.isAndroid) {
    if (!await dirOfOldAndroidDatabase.exists()) {
      await dirOfOldAndroidDatabase.create(recursive: true);
    }
  }
  final path = Platform.isIOS ? p.join(dir.path, 'pro_darts.db') : p.join(dir.path.replaceAll("/app_flutter", "/databases"), 'pro_darts.db');
  final receivePort = ReceivePort();

  await applyWorkaroundToOpenSqlite3OnOldAndroidVersions(); // <- Inserted here

  await Isolate.spawn(
    _startBackground,
    _IsolateStartRequest(receivePort.sendPort, path),
  );
  return (await receivePort.first as MoorIsolate);
}

I think that should work, but just in case I'd probably put it inside the _startBackground function somewhere since that's where the database will be opened in the end.

I ended up using a whole combination of workarounds, but I think the applyWorkaround method fixes the issue and the other (older, partial) workarounds are not necessary anymore. But I would need to check my crash reporting logs if anything popped up.
But here's the function that I'm using currently:

// Do this once, before opening a database
// see https://github.com/simolus3/moor/issues/876
Future<void> _ensureSqlite3Initialized() async {
  //TODO maybe ensure synchronized?
  if (Platform.isAndroid) {
    await applyWorkaroundToOpenSqlite3OnOldAndroidVersions();
    final tempdir = (await getTemporaryDirectory()).path;
    final nativeLibDir = await NativeLibDir.nativeLibraryDir;
    open.overrideFor(
        OperatingSystem.android, () => _androidOpenSqlite(nativeLibDir));
    sqlite_lib.sqlite3.tempDirectory = tempdir;
  }
}

///https://github.com/simolus3/moor/issues/895
DynamicLibrary _androidOpenSqlite(String nativeLibDir) {
  if (Platform.isLinux || Platform.isAndroid) {
    try {
      return DynamicLibrary.open('libsqlite3.so');
    } catch (_) {
      debugPrint("fail opening sqlite, trying workarounds...");

      if (Platform.isAndroid) {
        try {
          final lib = DynamicLibrary.open("$nativeLibDir/libsqlite3.so");
          debugPrint(
              "successfully loaded sqlite3 with strange workaround at $nativeLibDir");
          return lib;
        } catch (_) {
          // On some (especially old) Android devices, we somehow can't dlopen
          // libraries shipped with the apk. We need to find the full path of the
          // library (/data/data/<id>/lib/libsqlite3.so) and open that one.
          // For details, see https://github.com/simolus3/moor/issues/420
          final appIdAsBytes = File('/proc/self/cmdline').readAsBytesSync();

          // app id ends with the first \0 character in here.
          final endOfAppId = max(appIdAsBytes.indexOf(0), 0);
          final appId =
              String.fromCharCodes(appIdAsBytes.sublist(0, endOfAppId));

          return DynamicLibrary.open('/data/data/$appId/lib/libsqlite3.so');
        }
      }

      rethrow;
    }
  }
  if (Platform.isIOS) {
    return DynamicLibrary.process();
  }
  if (Platform.isMacOS) {
    return DynamicLibrary.open('/usr/lib/libsqlite3.dylib');
  }
  if (Platform.isWindows) {
    return DynamicLibrary.open('sqlite3.dll');
  }

  throw UnsupportedError('Unsupported platform: ${Platform.operatingSystem}');
}

NativeLibDir Plugin for getting the native lib directory:

  @Override
  public void onMethodCall(@NonNull MethodCall call, @NonNull Result result) {
    if (call.method.equals("getNativeLibraryDir")) {
      final ApplicationInfo info = context.getApplicationInfo();
      if(info != null) {
        result.success(info.nativeLibraryDir);
      }else{
        result.success(null);
      }
    } else {
      result.notImplemented();
    }
  }

Thanks for all the information! Btw, I think on iOS it's safer to open sqlite3.framework/sqlite3 since DynamicLibrary.process() did not resolve to the version from sqlite3_flutter_libs in all cases. This is what sqlite3 is doing by default on iOS:
if (Platform.isIOS) {
  try {
    return DynamicLibrary.open('sqlite3.framework/sqlite3');
    // Ignoring the error because its the only way to know if it was sucessful
    // or not...
    // ignore: avoid_catching_errors
  } on ArgumentError catch (_) {
    // In an iOS app without sqlite3_flutter_libs this falls back to using the version provided by iOS.
    // This version is different for each iOS release.
    //
    // When using sqlcipher_flutter_libs this falls back to the version provided by the SQLCipher pod.
    return DynamicLibrary.process();
  }
}

@julienduchow It looks like you don't need to enable uncompressed native libs if you're employing the other workaround.

Alright, thank you both for your help! :)
I will try it and see in 2-3 days when the update is in the Google Store if it worked and will comment then again.

@simolus3
I tried to move the call in _startBackground, but that is causing an error on this line (Null check operator used on null value):
await _platform.invokeMethod('doesnt_matter');
So i kept the call where I wrote in my prev comment and this is working fine, just wanted to let you know.

Ah right, good call. Yeah the workaround uses a platform channel which aren't available on background isolates. The workaround should still work (it involves opening the library in Java, opening the same name in Dart later will give us the opened library regardless on which thread we're opening it).

@simolus3 thanks, I should delete the non-android related code anyway, as it is unreachable (see
open.overrideFor(
        OperatingSystem.android, () => _androidOpenSqlite(nativeLibDir));

On iOS, this is never called


ci: collect failed test logs as artifacts.

This allows us to avoid relying on bazel output behavior (possibly being eaten in #5104).
Turned on for a bunch of builds, but not all. E.g. coverage is skipped as it's a bit more
complicated.
Risk Level: Low
Testing: Tested in PR with CircleCI.
Signed-off-by: Harvey Tuch<EMAIL_ADDRESS>

OK, looks like this is working! See https://circleci.com/gh/envoyproxy/envoy/129421#artifacts/containers/0. Will push a PR to revert the debug stuff and we can review/merge.


Delete AadJwtClaimValidator, use JwtClaimValidator instead.

Delete AadJwtClaimValidator, use JwtClaimValidator instead.
Delete AadJwtAudienceValidator, use JwtClaimValidator instead.
Rename AadTokenClaim to AadJwtClaimNames, and enrich java doc.

Hi, @backwind1233 , @saragluna , please help to review this PR when you have time.


Missing PR template

Addition of Pull Request Template to improve clarity and to make it easier for new contributors.
Please assign this issue to me under GSSoC'23 @Shreyaar12

Assign to  @Anshika-Jain02 Complete it asap in 2-3 Days.
Include Screenshot to assesss your PR and follow contributing.md guidelines


[v1.14] fqdn: Add Protocol to DNS Proxy Cache

DNS Proxy indexes domain selectors by port
only. In cases where protocols collide on port
the DNS proxy may have a more restrictive selector than it should because it does not merge port
protocols for L7 policies (only ports).
Refactor all users of the DNS Proxy are updated
to add protocol to any DNS Proxy entries, and all
tests are updated to test for port-protocol
merge errors.
fqdn: Fixed bug that caused DNS Proxy to be overly restrictive on allowed DNS selectors.

This is a backport of https://github.com/cilium/cilium/pull/31328

/test-backport-1.14

@nathanjsweet looks like the integration tests caught a persistent fail: https://github.com/cilium/cilium/actions/runs/8574930543/job/23517128951. I'll switch to draft for now, could you please have a look?

/test-backport-1.14


File Actions

Describe the feature and the current behavior/state.
What Raycast is missing to be complete are file actions. Afred aces this. Currently there is no extension or core functionality providing this capability. You need to be able to act on items and send them to another app, or action.
With Alfred you can move, copy files to recent locations or even search for a new location by triggering a shortcut:

Will this change the current API? How?
The current API has not such capabilities.
Who will benefit from this feature?
Power users and developers.

Wanted to voice my support for this as an Alfred user testing out a switch 🙂  A very common scenario for me is moving items out of Downloads shortly after they get dumped in there, so I: search for a file by name, open the action palette, choose "Move to...", then search for the directory I want and action it.

Hi @zenopopovici @helen
I have created a Common Directory extension that has been submitted to PR and is under review
The extension has two main features

Quick access to common folder directories.

Quickly Send files to the common directory, open directory or folder chosen manually

This is the best I could do with the existing API, although it's still inconvenient compared to Alfred's search mobile, but it's an effort.
Of course there are advantages to this extension (it can move or copy files/folders in bulk)
https://user-images.githubusercontent.com/36128970/162682823-8fa9bb32-4cd5-4f52-bf1a-827bb6fc1bb3.mp4

@koinzhang Very very cool, I'll install it right away, this is actually a huge step forward into moving 100% on RayCast from Alfred.


Signup and LogIn should happen in the simulator itself.

Is your feature request related to a problem? Please describe.
Consider me a new user , I opened the website and just started playing without noticing the "Sign in" button . Suddenly when I will want to save the project , I will click the button and it will take me out of the page , believe me it's frustating and not okay.
Describe the solution you'd like
I want that there should be a cool "modal" , bootsrap's modal is pretty fine . In the modal there should be all sort of login methods , and when the user is signed in , the modal should just go away , updating the simulator without refreshing the page at all.
Are you working on this? (Yes/No)
Yes , I have deignned the look of the modal . As understanding the login flow of CircuitVerse will take time , I can't finish it completely as PR , But I will surely show the mentors how I want it to look in proposal with a picture . I will create a basic modal with dummy inputs and a create a PR for sure.

#1421 shall address this I guess @nitin10s

Wow, someone noticed this before, didn't know, Closing this #1421 explains the bug better along with the new design


May yield* send the last yield value?

YieldExpression:yield*AssignmentExpression
...
4. Let received be NormalCompletion(undefined).

May it update to 4. Let received be NormalCompletion(LastYieldValue) so we can satisfy:
function *g1() {
  yield function.sent
}
function *g2() {
  yield* g1()
}

g2().next(42) // => {value:42}

But allow delegate generator access the last yield value also have dark side:
function* service() {
  const token = function.sent
  if (!auth(token)) fail()
  yield* serviceOf3rdParty()
}

const serv = service()
serv.next('my-secret-token')

I'd rather not see that happen by default given the second case, perhaps a variant of yield* or a function that forwards a value e.g.:
function* g2() {
    yield** g1()
}

// Or just a function (which can even be written in userland)

function* g2() {
    yield* withInitial(function.sent, g1())
}


"Understanding Kubernetes Objects" docs should reference field validation

The docs on Understanding Kubernetes Objects currently make no mention of field validation. Now that server-side field validation of unknown and duplicate fields is graduating to GA, we should mention it briefly here.

/cc @sftim

/triage accepted

/wg api-expression
/language en
/priority important-longterm
/sig api-machinery

/assign


Update compiler version

@enricosada This updates the compiler to the latest build we have on NuGet.
/cc @KevinRansom

@cartermp the System.ValueTuple package version required by compiler package is not published on nuget.org yet.
we should use v4.0.0-rc3-24212-01 in visualfsharp. is that version ok too? i'll open an issue in visualfsharp repo about that.
with a clean build (and no System.ValueTuple on the nuget cache)
running build.cmd
error: Unable to resolve 'System.ValueTuple (>= 4.0.1-beta-24405-03)' for '.NETCoreApp,Version=v1.0'.

as note: it doesnt matter if it's specified to use only the nuget.org as source feed, because if nuget find the package in the local cache (on windows C:\Users\%USER%\.nuget\packages) the package restore complete successufully.
if you can enable appveyor or jenkins in this repo, and run build.cmd it's easier to see the error and build/pr are safer

@enricosada looks like it's already getting used: https://github.com/Microsoft/visualfsharp/blob/master/src/fsharp/FSharp.Compiler/project.json#L22
@KevinRansom would we need to just publish a new build of the comiler?

@cartermp @enricosada
It's a bug in the nuspec file ... I will fix it and republish today
project.json
{
  "dependencies": {
   ...
    "System.ValueTuple": "4.0.0-rc3-24212-01",
  },

nuspec:
        <dependencies>
            <group targetFramework=".NETStandard1.6">
                ....
                <dependency id="System.ValueTuple" version="4.0.1-beta-24405-03" />
                ...
            </group>
        </dependencies>

Kevin

@enricosada @cartermp
Done:
https://www.nuget.org/packages/Microsoft.FSharp.Compiler.Host.netcore/1.0.0-alpha-160831

Updated to latest.

nice, let me try to build some libraries (suave, argu) to test the new compiler!


CPP-ify sdram

Created SdramHandler class.
Updated all of those typedef structs to enum classes.
Renamed and slightly tweaked functions to work within class.
Updated seed board support to use cpp-ified sdram.

 SDRAM C++ wrapper

Simply CPP-ifies SDRAM. There will be some more changes made on the GPIO branch (specifically changing the GPIO usage to the cpp gpio). Saved that for that pr, since it's outside the scope of this one.  Tested working on the patch multidelay example.

Realistically, I think we can scrap the single-configured pin and hard code it. There was a difference on one old version of hardware board.
Debatably, this doesn't need a config struct at all. I have not personally "disabled" the SDRAM, and I can't think of a reason that you'd need to do that at the config level.
Plus, in the near future, I'd like to move the SDRAM initialization into the startup file. So that the text section can be used in some capacity, or for alternate linkers where the SDRAM is the primary memory for things where tons of memory is needed, but performance doesn't matter.

It looks like there's a small bug with the timing configuration that's stuck around since the base C version. In sdram.cpp, line 80:
SdramTiming.RPDelay              = 0;

the row precharge delay parameter is set to 0. If you dig through the HAL, you'll find that the function that actually writes this value to the FMC SDTR register subtracts one before shifting and ORing:
MODIFY_REG(Device->SDTR[FMC_SDRAM_BANK1],
               SDTR_CLEAR_MASK,
               (((Timing->LoadToActiveDelay) - 1U)                                      |
                (((Timing->ExitSelfRefreshDelay) - 1U) << FMC_SDTRx_TXSR_Pos) |
                (((Timing->SelfRefreshTime) - 1U)      << FMC_SDTRx_TRAS_Pos) |
                (((Timing->RowCycleDelay) - 1U)        << FMC_SDTRx_TRC_Pos)  |
                (((Timing->WriteRecoveryTime) - 1U)    << FMC_SDTRx_TWR_Pos)  |
                (((Timing->RPDelay) - 1U)              << FMC_SDTRx_TRP_Pos)  |
                (((Timing->RCDDelay) - 1U)             << FMC_SDTRx_TRCD_Pos)));

meaning that it actually underflows and writes 0xF to RPDelay and RCDDelay. I'm guessing it would only make a small different to performance, but we'll probably want to correct that. The HAL function is expecting the number of cycles of delay, not the actual register value, so a 1 should be written instead.

closing because I think most/all of the changes were absorbed into #390


⚠ St. Lawrence Global Observatory CKAN has degraded performance

In dee08c1, St. Lawrence Global Observatory CKAN (https://catalogue.ogsl.ca/en/) experienced degraded performance:

HTTP code: 200
Response time: 23872 ms

Resolved: St. Lawrence Global Observatory CKAN performance has improved in 2f98b6e after 7 minutes.


ModuleNotFoundError: No Module named 'ipadapter.ip_adapter'

hi!
im getting this when pressing the train button after selecting images to train on. any advice? i installed the requirements with pip with no problems before (only warnings of old pip version). thanks!
 File "H:\ImageModel_Training\FluxLocalTraining\ai-toolkit\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 859, in run
    result = context.run(func, *args)
  File "H:\ImageModel_Training\FluxLocalTraining\ai-toolkit\venv\lib\site-packages\gradio\utils.py", line 826, in wrapper
    response = f(*args, **kwargs)
  File "H:\ImageModel_Training\FluxLocalTraining\ai-toolkit\flux_train_ui.py", line 229, in start_training
    job = get_job(config_path)
  File "H:\ImageModel_Training\FluxLocalTraining\ai-toolkit\toolkit\job.py", line 28, in get_job
    from jobs import ExtensionJob
  File "H:\ImageModel_Training\FluxLocalTraining\ai-toolkit\jobs\__init__.py", line 1, in <module>
    from .BaseJob import BaseJob
  File "H:\ImageModel_Training\FluxLocalTraining\ai-toolkit\jobs\BaseJob.py", line 5, in <module>
    from jobs.process import BaseProcess
  File "H:\ImageModel_Training\FluxLocalTraining\ai-toolkit\jobs\process\__init__.py", line 8, in <module>
    from .TrainSliderProcess import TrainSliderProcess
  File "H:\ImageModel_Training\FluxLocalTraining\ai-toolkit\jobs\process\TrainSliderProcess.py", line 26, in <module>
    from .BaseSDTrainProcess import BaseSDTrainProcess
  File "H:\ImageModel_Training\FluxLocalTraining\ai-toolkit\jobs\process\BaseSDTrainProcess.py", line 26, in <module>
    from toolkit.custom_adapter import CustomAdapter
  File "H:\ImageModel_Training\FluxLocalTraining\ai-toolkit\toolkit\custom_adapter.py", line 11, in <module>
    from toolkit.models.ilora import InstantLoRAModule
  File "H:\ImageModel_Training\FluxLocalTraining\ai-toolkit\toolkit\models\ilora.py", line 12, in <module>
    from ipadapter.ip_adapter.resampler import  Resampler
ModuleNotFoundError: No module named 'ipadapter.ip_adapter'

You need to clone all other repositories, that are stored inside repositories dir, inside this one. This helped me to solve it
# inside ai-toolkit dir
git submodule update --init --recursive


Run linkage checks in brew audit

So remaining question is should we keep brew linkage --test to check undeclared deps.

Maybe change it to linkage --broken and linkage --undeclared?

I think there needs to be a way to suppress the undeclared linkages from causing audit failure on mere formula revision (not version) bumps, (but possibly flipped back on if a dependency is being added), so I think this needs to be exposed with some kind of command line option to either explicitly opt in or explicitly opt out of the undeclared check.
Also, should we fail audit if something is declared as a dependency but it's NOT linked && NOT marked => :run? I think we're going to end up starting to accumulate depends_on`s that are no longer actually used if that's not being checked, too.

I think there needs to be a way to suppress the undeclared linkages from causing audit failure on mere formula revision

I disagree; you can merge a PR even if CI is red if you know better that it does.

you can merge a PR even if CI is red

But that violates my religion :(

@ilovezfs All fall short of the glory of @God @BrewTestBot

This is causing a regression
Error: /usr/local/Cellar/amazon-ecs-cli/0.3.1 is not a directory

where it doesn't let you run audit at all unless the formula is installed now.
iMac-TMP:local joe$ brew uninstall amazon-ecs-cli
Error: No such keg: /usr/local/Cellar/amazon-ecs-cli
iMac-TMP:local joe$ git reset --hard origin/master
HEAD is now at 86b1df9 Autoupdate tweaks (#442)
iMac-TMP:local joe$ brew audit --strict amazon-ecs-cli
iMac-TMP:local joe$ brew pull https://github.com/Homebrew/brew/pull/470
==> Fetching patch
Patch: https://github.com/Homebrew/brew/pull/470.patch
==> Applying patch
Applying: move LinkageChecker to standalone file
Applying: formula_cellar_checks: add check_linkage
==> Patch closes issue #470
==> Patch changed:
 Library/Homebrew/dev-cmd/linkage.rb       | 114 +----------------------------
 Library/Homebrew/formula_cellar_checks.rb |  28 +++++++
 Library/Homebrew/linkage_checker.rb       | 118 ++++++++++++++++++++++++++++++
 3 files changed, 147 insertions(+), 113 deletions(-)
iMac-TMP:local joe$ brew audit --strict amazon-ecs-cli
Error: /usr/local/Cellar/amazon-ecs-cli/0.3.1 is not a directory

For the background on my comments, please see https://github.com/Homebrew/brew/commit/080ddd8804be14f4b18f9558b58270456ff313c2#commitcomment-18165984.

So remaining question is should we keep brew linkage --test to check undeclared deps.

I think we should revert --test to its previous meaning of only checking broken linkage (and possibly rename it to --linkage as suggested by @ilovezfs). Undeclared dependencies should not cause a failure unless explicitly requested. To summarize:

brew linkage <formula> should print all the traditional information and always exit with success. This is basically verbose linkage information on a per-formula basis and for human consumption only.

brew linkage <formula> --reverse should also print its usual thing and always exit with success. This is even more verbose linkage information on a per-file basis and also for humans only.

brew linkage <formula> [--linkage] [--undeclared-deps] (where at least one of the options is given), should run either of the tests or both and exit with failure if any of them report failure. Maybe a clearer interface would be:
brew linkage <formula> --test[=<tests>]

where <tests> accepts a comma-separated list of checks to perform (currently linkage and undeclared-deps) and defaults to linkage.

Also, should we fail audit if something is declared as a dependency but it's NOT linked && NOT marked => :run (or it goes without saying => :build)? I think we're going to end up starting to accumulate depends_ons that are no longer actually used if that's not being checked, too.

Unfortunately the relationship between dependencies and dynamic linkage is not an equivalence. Dynamic linkage implies dependency, but not the other way round. There are so many different reasons why a formula may need a certain dependency on run time beyond just dynamic linkage. I don't think expanding our use of :run for this is going to scale.
The problem with possibly no longer needed depends_on lines is already a reality. I think it's out of scope for this PR to address this and I fear it's a challenging task to try to tackle.

I think it's out of scope for this PR to address this

It's not out of the scope insofar as we're talking about rigorously "dragging in" all linked but undeclared dependencies. The undeclared dependencies exist only because a declared dependency caused them to be installed, or one of the dependencies of a declared dependencies (potentially recursively) caused the them to be installed. Then the undeclared dependency got linked during the build, but that linkage was not (until now) formalized with an actual declaration. However, when/if upstream later removes the original direct dependency that caused the undeclared dependency to occur, there is nothing this PR does to unwind the formalized declaration of the originally undeclared yet linked dependency. And indeed it may end up continuing to be linked at that point ONLY because it is declared or it may end up NOT LINKED AT ALL and yet still declared ("dangling dependency"). Neither of those cases is acceptable and yet we will guarantee they occur all over the place because so many build systems opportunistically link against what's available (one of the major points of superenv hiding, or at least making a best effort to hide, things we didn't ask it to expose) and because the dependencies we're adding solely because they were undeclared are in no way designated as such and are not going to be pointed out by upstream as no longer being needed because upstream never explicitly said they were needed in the first place, so there would be no reason to call them out when later the originating dependency is dropped. As a result, the only way these dependencies will ever get removed is if we go in and reassess every single dependency in the formula to determine whether or not it is actually required directly, actually required indirectly, only opportunistically linked but not actually needed (except perhaps sometimes optionally), or not linked at all.
We already have this problem, but deliberately combining all-things-that-are-linked with all-things-upstream-says-are-required under the same umbrella of "depends_on" with no distinction whatsoever in the DSL between them is going to cause this problem grow at faster rater, and cause it to accumulate over time.
And no I don't think :run actually solves any of this, but the point would be to at least call out things that are not marked build and are not linked as part of audit, which addresses the dangling half but not the opportunistic half. Marking with :run would be the way to "override" that audit error, but of course :dangling-but-that's-fine would work too.
Moreover, the opportunistic half of the problem is the worse of the two anyway because it's much harder to see even if you're looking for it. This is why undeclared dependencies are not actually bugs but features unless they're distinguished from the actual intended direct dependencies in some way in the DSL itself.

Marking with :run would be the way to "override" that audit error, but of course :dangling-but-that's-fine would work too.

I think having :linked or something may be what we want here but, for changes like this, we really want/need a way to be able to run it over all our packages. I too 😬 a bit at our approaches sometimes of "add a new rule, make the new poor soul to bump the package fix it"; it's pretty unfriendly to first-time contributors.

@UniqMartin

(and possibly rename it to --linkage as suggested by @ilovezfs)

I think I said --broken and --undeclared. brew linkage --linkage sounds slightly redundant.

I think I said --broken and --undeclared. brew linkage --linkage sounds slightly redundant.

You're right. Sounds silly. Sorry for the misquote!

You're right. Sounds silly. Sorry for the misquote!

No problem. It made me laugh.

Only more comments before shipping this?

I'd still like to see there be some standard way of designating things that are added solely because "audit says it's undeclared but linked" before we start enforcing this. Even if we punt and make that a standard Ruby comment, that's better than nothing.
Undeclared dependencies are a feature not a bug, but we're killing them off without preserving the information about what was required directly and what got brought in due to something else. (The only case where they're actually a bug is if brew install doesn't install them unless they're declared and yet they're truly necessary, but that is explicitly never the case here since they're linked but undeclared precisely because something else caused them to be installed. If they were undeclared, unlinked, and yet necessary, audit would not report them.)
If the thing that was originally required directly no longer is, an undeclared dependency will naturally drop away. A declared dependency will not.

If the thing that was originally required directly no longer is, an undeclared dependency will naturally drop away. A declared dependency will not.

I think this is still a problem. Perhaps a depends_on "foo" => :linked or similar (which we could suggest in the audit text will throw an audit failure if it isn't linked against in future?

@MikeMcQuaid even better it would also say exactly what formula(e) brought it in originally (e.g., depends_on "foo", :linked => ["needy_thing1", "needy_thing2"] since the most problematic cases are opportunistic linking to formulae that are not actually required anymore (or even "worse" weren't ever required in the first place (!) but got linked simply because they were found by configure) and yet now is still getting linked anyway, even after the original thing that brought it in has been deliberately removed as a depends_on of the formula, due to opportunistic linking/discovery in configure, and not due to some explicit option we passed to configure asking for it to enable that option.

@MikeMcQuaid even better it would also say exactly what formula(e) brought it in originally (e.g., depends_on "foo", :linked => ["needy_thing1", "needy_thing2"] since the most problematic cases are opportunistic linkages to formulae that are not actually required anymore (or even "worse" weren't ever required in the first place (!) but got linked simply because they were found by configure) and yet now are still getting linked anyway, even after the original thing(s) that brought them in have been deliberately removed as depends_ons of the formula, due to opportunistic linking/discovery in configure, and not due to some explicit option we pass to configure asking for configure to enable them.

That's a neat idea but I think it's a feature request to address opportunistic linking better through superenv (once we have these dependencies tagged we can start filtering out more with superenv) and the opportunistic linking will likely be revealed by future brew audit runs.

@MikeMcQuaid The point is to make it easier for someone removing a dependency to know which :linked to reassess as part of the removal procedures, but, yes, we can always require people to do more work in the future and have them sit there and check every single one of the :linked even though we could have left a proper message in a bottle (or in this case in a formula) so-to-speak (pun intended). Hopefully the list of :linked is short enough that it's as feature-requesty as you suggest.

The other reason that we ought to say what's bring it in, is that will allow audit to puke up an error if in the future the needy_thing1 is no longer a depends_on and yet the :linked is still sitting there. A :linked that doesn't say whose entourage it belonged to leaves audit in the dark.

Another thought about this PR. Should we phase this in and start out by always doing the check and printing the undeclared, but only make the presence of undeclared an audit failure if --strict is passed? That might soften the blow to CI a bit.

I'm still not sure I understand by what you mean by "what's bringing it in"; if it links directly (not recursively) to something else I'm not sure that's machine identifiable so it seems fine to omit.

Another thought about this PR. Should we phase this in and start out by always doing the check and printing the undeclared, but only make the presence of undeclared an audit failure if --strict is passed? That might soften the blow to CI a bit.

I'm 👍 on keeping it as a failure for now.

I'm still not sure I understand by what you mean by "what's bringing it in"; if it links directly (not recursively) to something else I'm not sure that's machine identifiable so it seems fine to omit.

"foo brings in bar" means, for instance, all of the following are true:
a) my_formula has a depends_on "foo" and does not have a depends_on "bar"
b) brew audit my_formula says that bar is linked but undeclared
c) the reason that bar got installed during brew install --build-bottle my_formula is that foo has a depends_on "bar"

c) the reason that bar got installed during brew install --build-bottle my_formula is that foo has a depends_on "bar"

This is the bit I don't think we can necessarily identify automatically through code but if you can: go for it (but I don't think it needs to block this PR).

This is the bit I don't think we can necessarily identify automatically through code

It's the existingdepends_on (one or more) in my_formula for which it is true that brew deps "foo" already lists bar.

Example:
iMac-TMP:Formula joe$ brew linkage file-roller
System libraries:
  /usr/lib/libSystem.B.dylib
Homebrew libraries:
  /usr/local/opt/atk/lib/libatk-1.0.0.dylib (atk)
  /usr/local/opt/cairo/lib/libcairo-gobject.2.dylib (cairo)
  /usr/local/opt/cairo/lib/libcairo.2.dylib (cairo)
  /usr/local/opt/gdk-pixbuf/lib/libgdk_pixbuf-2.0.0.dylib (gdk-pixbuf)
  /usr/local/opt/gettext/lib/libintl.8.dylib (gettext)
  /usr/local/opt/glib/lib/libgio-2.0.0.dylib (glib)
  /usr/local/opt/glib/lib/libglib-2.0.0.dylib (glib)
  /usr/local/opt/glib/lib/libgobject-2.0.0.dylib (glib)
  /usr/local/opt/glib/lib/libgthread-2.0.0.dylib (glib)
  /usr/local/opt/gtk+3/lib/libgdk-3.0.dylib (gtk+3)
  /usr/local/opt/gtk+3/lib/libgtk-3.0.dylib (gtk+3)
  /usr/local/opt/json-glib/lib/libjson-glib-1.0.0.dylib (json-glib)
  /usr/local/opt/libarchive/lib/libarchive.13.dylib (libarchive)
  /usr/local/opt/libmagic/lib/libmagic.1.dylib (libmagic)
  /usr/local/opt/pango/lib/libpango-1.0.0.dylib (pango)
  /usr/local/opt/pango/lib/libpangocairo-1.0.0.dylib (pango)
iMac-TMP:Formula joe$ perl -p -i -e 's/depends_on "atk"//' file-roller.rb
iMac-TMP:Formula joe$ brew linkage file-roller
System libraries:
  /usr/lib/libSystem.B.dylib
Homebrew libraries:
  /usr/local/opt/atk/lib/libatk-1.0.0.dylib (atk)
  /usr/local/opt/cairo/lib/libcairo-gobject.2.dylib (cairo)
  /usr/local/opt/cairo/lib/libcairo.2.dylib (cairo)
  /usr/local/opt/gdk-pixbuf/lib/libgdk_pixbuf-2.0.0.dylib (gdk-pixbuf)
  /usr/local/opt/gettext/lib/libintl.8.dylib (gettext)
  /usr/local/opt/glib/lib/libgio-2.0.0.dylib (glib)
  /usr/local/opt/glib/lib/libglib-2.0.0.dylib (glib)
  /usr/local/opt/glib/lib/libgobject-2.0.0.dylib (glib)
  /usr/local/opt/glib/lib/libgthread-2.0.0.dylib (glib)
  /usr/local/opt/gtk+3/lib/libgdk-3.0.dylib (gtk+3)
  /usr/local/opt/gtk+3/lib/libgtk-3.0.dylib (gtk+3)
  /usr/local/opt/json-glib/lib/libjson-glib-1.0.0.dylib (json-glib)
  /usr/local/opt/libarchive/lib/libarchive.13.dylib (libarchive)
  /usr/local/opt/libmagic/lib/libmagic.1.dylib (libmagic)
  /usr/local/opt/pango/lib/libpango-1.0.0.dylib (pango)
  /usr/local/opt/pango/lib/libpangocairo-1.0.0.dylib (pango)
Possible undeclared dependencies:
  atk
iMac-TMP:Formula joe$ brew deps -1 file-roller | while read f; do brew deps $f | grep -q atk && echo $f ; done
gtk+3

So for file-roller you'd have
  depends_on "atk", :linked => "gtk+3"
  depends_on "cairo", :linked => ["gdk-pixbuf", "gtk+3", "json-glib", "pango"]
  depends_on "gdk-pixbuf", :linked => "gtk+3"
  depends_on "gettext", :linked => ["glib", "gtk+3", "json-glib", "pango"]
  depends_on "glib", :linked => ["gtk+3", "json-glib", "pango"]
  depends_on "pango", :linked => "gtk+3"

Ok, I see. If you can make the code do that: 👍 (and throw a similar audit if there's trailing linked values that shouldn't be).

Yeah, I'll think about it, sure.
Also, it's probably even more useful if deps that are themselves listed as :linked be excluded from the lists, in which case the above reduces to:
depends_on "atk", :linked => "gtk+3"
depends_on "cairo", :linked => ["gtk+3", "json-glib"]
depends_on "gdk-pixbuf", :linked => "gtk+3"
depends_on "gettext", :linked => ["gtk+3", "json-glib"]
depends_on "glib", :linked => ["gtk+3", "json-glib"]
depends_on "pango", :linked => "gtk+3"

which I think makes it pretty clear what's actually going on with these previously-undeclared ones and "who is bringing them in?"
I think it might be useful to have a simple :linked sans right hand side lists be part of this PR so that we're at least not "throwing away" that information, or at minimum do as we've already done in file-roller here: https://github.com/Homebrew/homebrew-core/blob/7a9338a751f458ef3fff3759ab8d5b91c7d84a2c/Formula/file-roller.rb#L25-L31 with the # Add linked-library dependencies comment that you'd suggested at the time. If we do that, then the :linked could be added at a later time if/when deemed desirable if @xu-cheng's trigger finger is itchy to ship this right away.

I think it might be useful to have a simple :linked sans right hand side lists be part of this PR

Agreed 👍

I think it might be useful to have a simple :linked sans right hand side lists be part of this PR

I won't have time to do this in this PR. And it would be a too large change for a single PR. So I think we can either ship this with # Add linked-library dependencies comment or wait for someone else to implement it.

I won't have time to do this in this PR.

It's just adjusting the output text to have depends_on ... => :linked and verifying that doesn't break anything to use that as a depends_on tag. It's OK if you don't have time now but I do think it should be changed before this included.

@MikeMcQuaid you may not realize that depends_on "pango", :linked => "gtk+3" is in fact in compatible with our current DSL. We current on support depends_on <a value of string/symbol> or depends_on <a hash> not depends_on <a value>, <a hash>

@MikeMcQuaid you may not realize that depends_on "pango", :linked => "gtk+3" is in fact incompatible with our current DSL. We currently support depends_on <a value of string/symbol> or depends_on  not depends_on ,

I hadn't checked either way but thanks 👍. In that case I think just depends_on "pango" => :linked is fine for now and we can investigate a better format later. In that case it should just be a matter of updating the text that's output by the audit to tell people what depends_on to add.

might be contributor friendly to literally print the blob of formula text we expect inserted

Does this seem redundant to anyone else?
iMac-TMP:Formula joe$ brew audit file-roller
file-roller:
  * Undeclared dependencies found:
      glib
      pango
      atk
      cairo
      gdk-pixbuf
      gettext

    Formulae are required to declare all linked dependencies.
    Please check whether these dependencies are indeed needed.
    If so, please add all missing dependencies to the formula with:
      depends_on "glib" => :linked
      depends_on "pango" => :linked
      depends_on "atk" => :linked
      depends_on "cairo" => :linked
      depends_on "gdk-pixbuf" => :linked
      depends_on "gettext" => :linked
Error: 1 problem in 1 formula

Also that order seems rather random. Should it be alphabetized or something or is there meaning to the order? Maybe other taps last, tool, perhaps using the same sort that brew list uses?

This now looks much better to me:
iMac-TMP:local joe$ brew audit file-roller
file-roller:
  * Formulae are required to declare all linked dependencies.
    Please add all linked dependencies to the formula with:
      depends_on "atk" => :linked
      depends_on "cairo" => :linked
      depends_on "gdk-pixbuf" => :linked
      depends_on "gettext" => :linked
      depends_on "glib" => :linked
      depends_on "pango" => :linked
Error: 1 problem in 1 formula

However, this looks like a bit of a "mess":
iMac-TMP:local joe$ brew uninstall glib
Uninstalling /usr/local/Cellar/glib/2.48.1... (425 files, 22.3M)
iMac-TMP:local joe$ brew audit file-roller
file-roller:
  * Broken dylib links found:
      /usr/local/opt/glib/lib/libgthread-2.0.0.dylib
      /usr/local/opt/glib/lib/libgio-2.0.0.dylib
      /usr/local/opt/glib/lib/libgobject-2.0.0.dylib
      /usr/local/opt/glib/lib/libglib-2.0.0.dylib

    It is likely the installation was broken.
    Please check formula and make sure it is built from source.
    Formulae are required to declare all linked dependencies.
    Please add all linked dependencies to the formula with:
      depends_on "atk" => :linked
      depends_on "cairo" => :linked
      depends_on "gdk-pixbuf" => :linked
      depends_on "gettext" => :linked
      depends_on "pango" => :linked
Error: 1 problem in 1 formula
iMac-TMP:local joe$

And is that really only "one problem"? Seems to imply that adding the depends_ons is going to fix everything, but the glib issue is entirely separate and won't be fixed by that.

iMac-TMP:local joe$ brew audit file-roller
file-roller:
  * Broken dylib links found:
      /usr/local/opt/glib/lib/libgthread-2.0.0.dylib
      /usr/local/opt/glib/lib/libgio-2.0.0.dylib
      /usr/local/opt/glib/lib/libgobject-2.0.0.dylib
      /usr/local/opt/glib/lib/libglib-2.0.0.dylib

    It is likely the installation was broken.
    Please check formula and make sure it is built from source.

    Formulae are required to declare all linked dependencies.
    Please add all linked dependencies to the formula with:
      depends_on "atk" => :linked
      depends_on "cairo" => :linked
      depends_on "gdk-pixbuf" => :linked
      depends_on "gettext" => :linked
      depends_on "pango" => :linked
Error: 1 problem in 1 formula

Getting better. Maybe it can be a separate * for Undeclared dependencies and register 2 problems?

Maybe it can be a separate * for Undeclared dependencies and register 2 problems?

It would mean calling audit_check_output inside check_linkage which would be not fit with rest of code style. OR we split the check into two methods, but that would cause LinkerChecker run twice.

It would mean calling audit_check_output inside check_linkage which would be not fit with rest of code style.

I can live with that if it means it's easier for people to read the output ;)

iMac-TMP:Formula joe$ brew audit file-roller
iMac-TMP:Formula joe$ brew audit --strict file-roller
file-roller:
  * `desc` (line 3) should be put before `homepage` (line 2)
  * Broken dylib links found:
      /usr/local/opt/glib/lib/libgthread-2.0.0.dylib
      /usr/local/opt/glib/lib/libgio-2.0.0.dylib
      /usr/local/opt/glib/lib/libgobject-2.0.0.dylib
      /usr/local/opt/glib/lib/libglib-2.0.0.dylib

    It is likely the installation was broken.
    Please check formula and make sure it is built from source.
  * Formulae are required to declare all linked dependencies.
    Please add all linked dependencies to the formula with:
      depends_on "atk" => :linked
      depends_on "cairo" => :linked
      depends_on "gdk-pixbuf" => :linked
      depends_on "gettext" => :linked
      depends_on "pango" => :linked
Error: 3 problems in 1 formula

This looks reasonable to me.
Only complaint is that the blank line above "It is likely the installation was broken" looks a little weird. Also I guess that text should say "is broken" not "was broken."

Also, I'm not sure what
Please check formula and make sure it is built from source.

means.

Also, I'm not sure what
Please check formula and make sure it is built from source.
means.

Because other than deleting one of dependencies' installation, I think most common cause is the formula fetches binaries from upstream instead of building from source.

Do you mean bottles?

No, I mean formulae in homebrew/bintary

If audit checks are run on a configured formula, rather than a bottle, the dependencies for the enabled options are identified by the linkage checker as not being declared in the formula, throwing an error.
For example:
mkvtoolnix 9.3.1 --with-gettext --with-lzo --with-qt5 ;
brew audit mkvtoolnix ;
gives https://gist.github.com/JCount/076ca82c969653f89f91933c1081b821
@ilovezfs @tdsmith both felt that this specific behavior was likely undesirable.

@JCount should be fixed now.

Thanks!

I'm increasingly considering this to be a mistake outside of --strict.
I'm frankly fed up at the amount of my time this check is taking up & with the greatest respect for the work they do I'm frustrated that maintainers responsible for implementing this check are barely present around homebrew/core to deal with dancing around it like a slightly drunk turtle.
IMO, this still isn't particularly ready to be something exposed to every contributor. Xu's changes here made it better, but it's still frustrating, unclear & sometimes plain wrong.

FWIW, I think the second part of this check (namely the one that ultimately suggests to add lines like depends_on "dog" => :linked) should be completely reverted and formulae that had such lines added should be stripped of them (hopefully fairly painless and easily done).
There have been major concerns with this check and while it solves a problem, it does so in a way that had no consensus at all. (The only concern addressed was to make the output more actionable—something that IMO made things worse as there's no “one size fits all” solution for those possibly undeclared dependencies.) Yes, I realize I was initially in part a supporter of this check, but re-reading the following discussion made me realize that the pain far outweighs the benefits. Sorry I failed (due to lack of time) to chime in with a revised opinion later on!

There have been major concerns with this check and while it solves a problem, it does so in a way that had no consensus at all.

While I think we still need to tweak the way this is handled: I think not solving problems unless there is a consensus is a bit of a dangerous path to go down because it implies we let users suffer because we can't agree. I'd like to humbly propose that in this case (and in any others) that if we want it reverted you (not @UniqMartin specifically but anyone) are prepared to do the work yourself to work on an alternative (e.g. make PRs rather than issue comments). Perhaps that involves fixing uses --recursive but we need something to handle this.

Again the "you" below does not refer to any individual as it applies to me plenty of the time too:
On the flip side, there's a reason we say There are no obligations, but we'd appreciate your continuing help in keeping on top of contributions. when we invite people to be core contributors. It's pretty noticeable that there's folks who are maintainers but not following homebrew-core (or checking the backlog periodically) and this means this work falls back to other maintainers who are (who may prefer to be working on Homebrew/brew but have to do disproportionate work on it). If you're shipping something that affects formula contributors: you should be around to watch how they react to your changes and tweak things after the fact. This does not mean that you need to be available in the next minute/hour/day but if you're not on vacation and not noticing comments related to your changes a week later: you are not really following due diligence.

I can handle :linked in theory but I think our practical implementation leaves a fair bit to be desired for now and it should be nested under --strict in the audit.
I can get over my gripes with :linked as an idea, which primarily revolve around it making formulae that link into heavy dependency trees (Cough, GTK/GTK+3) a mess & us currently having no way to track when a dependency is listed as :linked but no link is present.
However, on a practical level right now I think it's unsuitable to be exposed to regular contributors & I'm tired of dealing with the fallout. I am grateful to maintainers for putting in the work on this PR & related ones; I'd like to spend more time with you than I do in the other place in an ideal world, I'm not ungrateful for the work here at all, I just think we need to refine the system before we expose it this widely.

However, on a practical level right now I think it's unsuitable to be exposed to regular contributors & I'm tired of dealing with the fallout.

As one of the main people dealing with that (and @ilovezfs who has also 👍d above): I think that's good enough to just make this --strict for now.

(Yes, I realize that most of what I'm replying to wasn't necessarily addressing me.)

While I think we still need to tweak the way this is handled: I think not solving problems unless there is a consensus is a bit of a dangerous path to go down because it implies we let users suffer because we can't agree.

The current solution trades a possibly improved experience for our users with apparently major grief for maintainers (those active in homebrew/core) and contributors. I'm not sure that's the right balance in this particular case, not to mention that this adds :linked to our formula API despite it only being a marker for a workaround that we have no truly satisfying solution to yet. I'm not trying to block change/progress here; my subjective feeling is just that the change as-is didn't have enough support from those participating in the discussion in this PR.
(I guess I made a mistake by using the word “consensus” that represents a more fundamental level of agreement. I didn't mean to imply we always need a consensus for a change like this.)

I'd like to humbly propose that in this case (and in any others) that if we want it reverted you (not @UniqMartin specifically but anyone) are prepared to do the work yourself to work on an alternative (e.g. make PRs rather than issue comments). Perhaps that involves fixing uses --recursive but we need something to handle this.

I agree that expecting a PR that tweaks the behavior, e.g. only enable the check for --strict, instead of idle words in comment is preferable. But working on an alternative is a completely different thing and feels a bit unfair, as it sounds like someone gets to “break” something and someone who's annoyed/affected by the outfall of that change is forced to investigate and implement a better solution. Or am I missing the point?

If you're shipping something that affects formula contributors: you should be around to watch how they react to your changes and tweak things after the fact.

Fully agreed.

However, on a practical level right now I think it's unsuitable to be exposed to regular contributors & I'm tired of dealing with the fallout.

As one of the main people dealing with that (and @ilovezfs who has also 👍d above): I think that's good enough to just make this --strict for now.

I was close to submitting a PR for this when I realized: That would mainly address the problem of CI builds marked as failed due to this check. We're still asking contributors in our PR template to run brew audit --strict --online, so they would still be exposed to this undeclared dependency check and its output, if they accurately follow our PR check list. Is that tolerable? Should we hide the check behind a dedicated option like --undeclared-deps? Any other ideas?

Without getting into the politics of how decisions are made, work is distributed, or any of the rest of that, my current belief is that this change, including the :linked decorator and anywhere it has been used, should be rolled back entirely not simply confined to --strict. It was both theoretically and practically more sound as a non-fatal, informational output, only exposed via brew linkage, which requires HOMEBREW_DEVELOPER be set, and involved neither with brew audit (strict and non-strict alike), nor with brew linkage --test.

dancing around it like a slightly drunk turtle

I still haven't stopped laughing about that.

I'm not trying to block change/progress here; my subjective feeling is just that the change as-is didn't have enough support from those participating in the discussion in this PR.

Yep, that's fair.

(I guess I made a mistake by using the word “consensus” that represents a more fundamental level of agreement. I didn't mean to imply we always need a consensus for a change like this.)

👍

I agree that expecting a PR that tweaks the behavior, e.g. only enable the check for --strict, instead of idle words in comment is preferable. But working on an alternative is a completely different thing and feels a bit unfair, as it sounds like someone gets to “break” something and someone who's annoyed/affected by the outfall of that change is forced to investigate and implement a better solution. Or am I missing the point?

I think we agree here; I guess I mean even just a PR to revert the original commit (or delete the added code) is preferable to discussion on a merged PR.

I was close to submitting a PR for this when I realized: That would mainly address the problem of CI builds marked as failed due to this check. We're still asking contributors in our PR template to run brew audit --strict --online, so they would still be exposed to this undeclared dependency check and its output, if they accurately follow our PR check list. Is that tolerable? Should we hide the check behind a dedicated option like --undeclared-deps? Any other ideas?

I think we can't/shouldn't rely on people necessarily running things that our CI doesn't. That said, I'd happily put this behind another audit flag that CI always runs (but I think from @DomT4's comments that is the main problem, here). I guess I'm hesitant to turn off something annoying that does catch/fix a problem when we don't have another solution to the problem or someone else 🙌 to work on it.

It was both theoretically and practically more sound as a non-fatal, informational output, only exposed via brew linkage, which requires HOMEBREW_DEVELOPER be set, and involved neither with brew audit (strict and non-strict alike), nor with brew linkage --test.

I guess the main argument here is: does it help to have things that flag problems if users/maintainers never see them because CI is never red as a result?
It feels like the best solution is a fatal check that allows direct or indirect linkage (not just direct like we have currently) and for brew uses  --recursive to get fixed up (which, I believe, it will be just by us using Ruby 2).

I think we agree here; I guess I mean even just a PR to revert the original commit (or delete the added code) is preferable to discussion on a merged PR.

Yes, I agree. Thanks for picking this up before I managed to set aside some time for that!

It feels like the best solution is a fatal check that allows direct or indirect linkage (not just direct like we have currently) and […]

Agreed. If we could teach that check to take indirect dependencies into account when checking for apparently undeclared dependencies due to detected linkage, it would be more useful.

[…] and for brew uses --recursive to get fixed up (which, I believe, it will be just by us using Ruby 2).

Sadly, only using Ruby 2 will lessen the pain and make the performance of that more tolerable, but a proper fix and a more appropriate performance for this kind of operation probably involves rewriting significant parts of our dependency resolution code.

I'm deep sorry that such feature causes more trouble then it solves. :+1: on partially revert it.
I think for long term, if we want to move bottle block outside formula to a separate file with many other meta information (like binaries list), we could consider to put linkage information there automatically by the bot.


Tests covering more functionalities

Greetings,
while adapting autocmake for Windows I see need to have more complex tests, what means tests covering all possible Python/CMake functionalities of the autocmake project.
For example, I would welcome all possible entries into test/fc and test/cc autocmake.cfg files. This is closely related to modifying tests to include math libs (blas/lapack), parallel compilation, Fortran-C-C++ languge mixing etc.
The more autocmake's functionalities we test, the better for us.

You are perfectly right, we need more tests.
However a tests should ideally test one thing and one thing only
in the spirit of unit tests. Later we may introduce combination tests.
The reason why there are only few tests at this moment is
that I was not sure (and still am not sure) about all aspects
of the API and first wanted to stabilize the API before introducing
many tests which use the API.

IMO, as long as is autocmake employed in "bigger", more complex projects, we should consequently include more testing.
For example, when autocmake-for-DIRAC is using math libs detection, we are supposed to have one (simple) test for math detection.

Absolutely.

Hi,
as the next simple test, I am suggesting a fc_openblas test , where simple a Fortran code would call some  BLAS routine located in the famous OpenBLAS library.
The reason for OpenBLAS is that it's parallel and is availalble also for MS Windows (we have it in DIRAC, see https://testboard.org/cdash/buildSummary.php?buildid=16874).  Unfortunately, there is no GNU /usr/lib/libblas equivalent on MS Windows.

A good idea but the math module will go through at least one more iteration so expect API changes.

I am closing this issue because it is a bit too diffuse. But it is of course a good point
and will be pursued.

Hi,
could I please get green light for implementing the https://github.com/scisoft/autocmake/issues/16#issuecomment-116199678 ?


mgmt: mcumgr: Improve fs_mgmt file upload/download

Is your feature request related to a problem? Please describe.
At present, the fs_mgmt upload/download functions are self contained, this means that for each command, the underlying file is opened, seeked to the desired offset, and data read/written, then the file is closed. This works, but has an enormous overhead, reducing the throughput by roughly 6 times. This can be improved by keep the file handle open (optionally, via Kconfig) so that if the same file is written to/read from, it uses the existing object without needing to open or close it, this will vastly speed up file transfers.
Also the file access hook can be simplified to run once upon file being opened or closed, instead of for each read/write.
Note that this will require locking the file object to a specific transport, to prevent data leaking or access by a different transport/connection.
Describe alternatives you've considered
Leaving it as is.

NFS does the same thing. The problem is that it is hard to tell when operation will end and how long should the file handle be opened. This will add requirements for additional timeout mechanism to close the file handle when connection breaks too long.
Currently when you can resume uploading file, as long as device does not get reset, anytime you want.


Discussion on combination of result sets from different query types

Recently, from k-NN, we have received interest in combining results from text matching queries with k-NN queries (ref). So, I wanted to start a discussion about result combination and the problems doing it when scores are computed at the shard level.
From my understanding, the only way to currently combine scores from different queries is through a boolean query. In a boolean query, multiple query clauses are provided in the should or must section and the scores are combined via addition. On top of this, with function scoring, it is possible to manipulate the score of the results further before the addition of scores.
That being said, I see a few limitations with this approach.
First, it is difficult to combine result sets whose scores are produced via different scoring methods. In order to effectively combine results, the different queries' scores would need to be put on the same scale. By this, I mean that a score would need to meet 2 requirements: (1) indicates its relative relevance between it and the other documents scored in the query and (2) also be comparable with the relative relevance of results from other queries. For example, for k-NN, the score range may be 0-1 while BM25 scoring would be 0-Float.MAX_INT (I think). With this, it would be difficult to figure out an effective way to weight each result appropriately. One way to do this would be to normalize the scores before combining them. Normalization might be possible through rescoring, but this would happen at the shard level, which could cause problems when results are combined. For instance, if one shard has better results than another, normalization may skew the importance so that the top results from the latter shard are better than the former shard.
Second, it is not possible to consider global hits for re-ranking. Because scores are assigned at the shard level, any rescoring has to be done at the shard level. I see a problem with this in two cases: first, for score combination, if an index has a significant number of shards, a user may want only the top results to be combined instead of combining for the results of all shards in the index and then aggregating at the coordinator; second, in the future, a user may have a model that they want to run the results through to re-rank them that is expensive and they dont want to run for each shard in the index.
That being said, my preliminary thoughts to creating a solution for this would be to create some kind of search and merge api that might look like:
GET /{index_name}/_search_and_merge
{
    "queries": [
      {
        "query": {
          "knn": {
            "my_vector2": {
              "vector": [2, 3, 5, 6],
              "k": 2
            }
          }
        }
      },
      {
        "query": {
          "match_all": {}
        }
      }
    ],
    "merge": {
      "norm": true,
      "disjunctive": false,
      "strategy": "linear_combo"
      "params": {
        "weight_1": 10.2,
        "weight_2": 0.25
      }
    },
    "size": 100
}

Here the queries would be a list of queries to be executed independently and merge would contain the logic to combine the results at the coordinator level.
All that being said, I want to see what people think of doing this? Is there a way to accomplish this with the current search interface?

Could we extend rescoring to run on a higher level than shard (on merge)?

In general, I think it would be similar to rescore on coordinator level. The workflow I have in mind would:

Execute multiple queries that would return results to coordinator (almost like _msearch API)
Pre-process results from each query (i.e. normalization)
Expose rescore functionality based on these results (take into account score combination)
Reduce results based on score and execute fetch phase

I think extending rescore might be confusing from user perspective (i.e. when does it happen shard or coordinator).

Would it make sense to extend rescoring to run on a higher level than shard (i.e. merge)? Another idea is that merge could be seen as any map/reduce, so rescore (map) and merge (reduce)? That could scale like map/reduce as well.

Yes this can be seen as a possible solution and I think we should do some brainstorming around how we want to do this.
But as from requirements standpoint, what Jack mentioned on the top is the exact requirements.

Take a look at this paper from the Pinecone team:  https://arxiv.org/abs/2210.11934
Convex Combination shows promise as a simple/effective way to combine the two hybrid scores.  (Normalize each, and regress to find a linear scaling factor between the two distributions).
Distributed normalization of course is the challenge.  Shard-level might be workable.
I’d suggest starting with simply allowing the scores from each query type to be returned and combined as the developer sees fit.
For example convex combination could be implemented by two script_score queries inside a dismax tie=1 (where each script score employed a precomputed normalization function and boost for each query type).

What will be the order of execution of the BM25(may be multi-match) query and KNN query if we use a hybrid search query?

We have not decided yet as of now. I am working on building some proposal around the same. Will update this issue once I have some research done. But on high level I was thinking to run the queries in parallel. So that we can reduce the latency and then do the normalizations of the scores before we combine them.
But this is still very early thought. Will need to see what is best.

What will be the order of execution of the BM25(may be multi-match) query and KNN query if we use a hybrid search query?

@aishwaryabajaj-54 do we see a difference if the queries are performed in different order? As per my understanding if we are doing normalization of scores for each query and then combining the scores then it doesn't matter in which order the queries are executed.
Please let us know if you think otherwise and is there can be a case where this assumption doesn't hold true.

Would it make sense to extend rescoring to run on a higher level than shard (i.e. merge)? Another idea is that merge could be seen as any map/reduce, so rescore (map) and merge (reduce)? That could scale like map/reduce as well.

The idea of extending the rescoring at higher level that shard looks very promising. But on a different level I was thinking to build another phase in the whole Query and Fetch phases flow. This can be used by different plugins to do operations at Manager Level before the control goes to Fetch phase. I will try to create a proposal for this to start some discussion.

Thanks @navneet1v. Since we're talking additional phases in query and fetch, cc-ing @nknize who has been thinking about refactoring aggregation, which I believe is either on the same path or at least related.

I think ElasticSearch has a solution for this https://www.elastic.co/guide/en/elasticsearch/reference/master/knn-search.html#_combine_approximate_knn_with_other_features, maybe this could be replicated in OpenSearch? I've created a feature request for this too with the same recommendation https://github.com/opensearch-project/k-NN/issues/717
I realised that the query below simply retrieves based on keyword matching and then re-ranks based on cosinesimil, which leaves out many relevant candidates in vector space.
body = {
 "size": 4,
 "query": {
   "script_score": {
     "query": {
       "query_string": {
           "fields": ["category"],
           "query": "hairdresser"
       }
     },
     "script": {
       "source": "knn_score",
       "lang": "knn",
       "params": {
         "field": "text_embeddings",
         "query_value": [0.1, 1.0, 0.3, 0.0],
         "space_type": "cosinesimil"
       }
     }
   }
 }
}

I've tried to adapt the recommended answer on a test index from this https://stackoverflow.com/questions/67701323/full-text-and-knn-vector-hybrid-search-for-elastic and it seems to work for me. Any thoughts on this solution? Perhaps latency concerns?
body = {
  "size": 5,
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "should" : [
            {
              "multi_match" : {
                "query": "electrician",
                "fields": ["category"]
              }
            },
            {
              "match_all": { }
            }
          ],
          "minimum_should_match" : 0
        }
      },
      "functions": [
      {
        "script_score" : {
          "script" : {
            "source": "doc['my_vector'].size() == 0 ? 0 : cosineSimilarity(params.query_value, doc['my_vector']) + _score",
            "params": {
              "query_value": [1.0, 1.0, 0.0, 0.0]
            }
          }
        },
        "weight": 1
      }
      ],
      "score_mode": "sum",
      "boost_mode": "sum"
    }
  }
}

I think ElasticSearch has a solution for this https://www.elastic.co/guide/en/elasticsearch/reference/master/knn-search.html#_combine_approximate_knn_with_other_features, maybe this could be replicated in OpenSearch? I've created a feature request for this too with the same recommendation opensearch-project/k-NN#717

I have responded on the feature request. Lets use that issue to talk more about the feature request.
Coming back the problem statement for this particular github issue, all the current score combinations work at Shard Level. We don't want to combine the scores at shard level, but at the Coordinator node level. Also the scores which are generated by keyword match and k-NN are not on the same scale. I can have K-NN scores between 0 to 1 and BM-25 and floating point values > 1. Quoting from description

First, it is difficult to combine result sets whose scores are produced via different scoring methods. In order to effectively combine results, the different queries' scores would need to be put on the same scale. By this, I mean that a score would need to meet 2 requirements: (1) indicates its relative relevance between it and the other documents scored in the query and (2) also be comparable with the relative relevance of results from other queries. For example, for k-NN, the score range may be 0-1 while BM25 scoring would be 0-Float.MAX_INT (I think). With this, it would be difficult to figure out an effective way to weight each result appropriately. One way to do this would be to normalize the scores before combining them. Normalization might be possible through rescoring, but this would happen at the shard level, which could cause problems when results are combined. For instance, if one shard has better results than another, normalization may skew the importance so that the top results from the latter shard are better than the former shard.

I can have K-NN scores between 0 to 1 and BM-25 and floating point values > 1.

Yeah that makes sense. One way could be to use a sigmoid function on the BM-25 score to map it between 0 and 1. The parameters of the function could be hyperparameters that the user selects. Example,
1 / (1 + Math.exp( - a * (BM25_score - b)))
This a and b can be used to control the slope and point of transition of the curve.

We want to use something similar to get results from kNN field along with exact match. Currently the scores received from both queries are not on the same scale as mentioned in the discussion above. Following is the query we expect to work to achieve so.
body = {
 "size": 10,
 "query": {
    "bool": {
      "should": [
        {
          "multi_match": {
              "query": "smart tv",
              "fields": ["summary"],
              "_name": "multi_match",
              "type": "phrase"
          }
        },
        {
          "knn": {
            "text_vector": {
              "vector": [2, 3, 5, 6],
              "k": 10,
                "_name": "knn"
            }
          }
        }
      ]
    }
  }
}

Will the queries above run in parallel or series? Search speed of BM25 is relatively slow.

Hi @ankitas3,
The queries present in the should clause doesn't run in parallel.
Consider there is a shard on the data node which has lets say 3 segments. So, the sub-queries(k-nn and multi-match) will be keep on executing on the segments one by one in sequential fashion and at the Bool query level for 1 shards the scores will be combined.
The speed of BM-25 depends on many factors. I would recommend running the mult-match query separately to find is BM-25 really running slow, if yes please try to optimize the BM-25 query.

@navneet1v, As in continuation to what @ankitas3 is doing is it possible to execute the queries in parallel, or may be specifying the execution order to reduce the latency?

@navneet1v I tried running BM25 separately, which is slow. Can you share any references to improve its speed.

@navneet1v, As in continuation to what @ankitas3 is doing is it possible to execute the queries in parallel, or may be specifying the execution order to reduce the latency?

Right now there is no way to execute them in parallel in a single query. You can see _msearch api but I am not sure if you are looking for that. It works something like _bulk api.
For improving the speed of the query I would recommend cutting a separate Github issue to talk, as this Github issue is not related to improving the BM-25 speed. In the new issue please make sure to provide details like OS version, Operating System, indices shards and number of nodes etc. On that GH issue we can talk around what things you can do to reduce the latency.

@navneet1v, I have tried _msearch and it has its own cons. Can we consider the speed and relevance benchmarking of parallel execution and normalization of scores of both queries (BM-25 and ANN) with the same GH issue?

Can we consider the speed and relevance benchmarking of parallel execution and normalization of scores of both queries (BM-25 and ANN) with the same GH issue?

@aishwaryabajaj-54 can you please elaborate? what is your ask here? and it is related to what?

I am closing this issue, as the problem defined above is getting taken care via : https://github.com/opensearch-project/neural-search/issues/123. Please do a +1 on the issue to support the work. I will keep that other issue updated and will use that to track the progress.


[RESOLVED] High ICMP ping response time

Problem started at 02:07:14 on 2023.08.02
Problem name: High ICMP ping response time
Host: ZO-FG
Severity: Warning
Operational data: Value: 58ms
Original problem ID: 34531
Event details in Zabbix: http://<IP_ADDRESS>/tr_events.php?triggerid=23125&eventid=34531

Problem has been resolved in 1m 0s at 02:08:14 on 2023.08.02
Problem name: High ICMP ping response time
Host: ZO-FG
Severity: Warning
Original problem ID: 34531


add bc checks to the repo

Subject

I am targeting this branch, because {reason}.

Closes #{put_issue_number_here}.
Changelog

### Added
- Added some `Class::newMethod()` to do great stuff.

### Changed

### Deprecated

### Removed

### Fixed

### Security

Codecov Report

Merging #4 (d756df6) into 3.x (1dea724) will increase coverage by 0.08%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                3.x       #4      +/-   ##
============================================
+ Coverage     80.36%   80.45%   +0.08%
  Complexity     2727     2727
============================================
  Files           173      173
  Lines          9141     9131      -10
============================================
  Hits           7346     7346
+ Misses         1795     1785      -10

Impacted Files
Coverage Δ
Complexity Δ

src/Util/AdminAclManipulator.php
0.00% <0.00%> (ø)
17.00% <0.00%> (ø%)

src/EventListener/AssetsInstallCommandListener.php
0.00% <0.00%> (ø)
37.00% <0.00%> (ø%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1dea724...d756df6. Read the comment docs.

Codecov Report

Merging #4 (d756df6) into 3.x (1dea724) will increase coverage by 0.08%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                3.x       #4      +/-   ##
============================================
+ Coverage     80.36%   80.45%   +0.08%
  Complexity     2727     2727
============================================
  Files           173      173
  Lines          9141     9131      -10
============================================
  Hits           7346     7346
+ Misses         1795     1785      -10

Impacted Files
Coverage Δ
Complexity Δ

src/Util/AdminAclManipulator.php
0.00% <0.00%> (ø)
17.00% <0.00%> (ø%)

src/EventListener/AssetsInstallCommandListener.php
0.00% <0.00%> (ø)
37.00% <0.00%> (ø%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1dea724...d756df6. Read the comment docs.


Fix unending loop when rtl

Fixes #245
I'm not sure if the RTL support is now complete because I don't write RTL. But it doesn't break the page now.

Hi @frissdiegurke, thanks for the patch!
Please remove the changes in dist directory from the commit. I'll merge it when you fix. :)


Application Restarts when Closing Sign In Screen after Orientation Change

DESCRIPTION:
Closing the Sign In screen after rotating the phone's orientation results in the app restarting.

REPRO STEPS:

Install and Launch LBRY 0.16.11 on Android 8.1.0 in Portrait orientation
Go to 'Sign In'
Rotate to Landscape orientation
Rotate back to Portrait orientation
Tap the X in upper right to close screen

Results: App restarts
Expected: User is returned to previous screen

ENVIRONMENT:

LG V20
Android 8.1.0
LBRY 0.16.11

MEDIA:

LBRY ADDRESS:
bZSqrVdpxt5KJdn2eGMUzT9Jn3gEiRet7b
<EMAIL_ADDRESS>

Thanks for the issue. I don't think this is a common user flow and not a big deal to fix at this time.


docker build fails FATAL

C:\GitHub\femtech-reminder>docker compose up iris -d
[+] Running 0/1

iris Error
Error response from daemon: pull access denied for intersystems/iris
repository does not exist or may require 'docker login':
denied: requested access to the resource is denied

works in branch MAIN, failed in MASTER

ok

ok in MAIIN


chore: updating pagination to match the gravity response

Super small follow up PR which is based off of this comment
Not sure if you had anything else in mind here @jpotts244 ? Happy to chat about it!

Sorry @Mitch-Henson ! Will close due to the dupe + additional discovery faffing that went on here


Minecraft 1.13 Stairs is not being connected!

Describe the bug, provide any errors
The problem is the stairs is not being displayed correctly like in older versions.
The stairs is displaying as not connected together with other stairs.
Example if you set a stair at 90 degress and another one at 180 degress they will normally display as connected, but if you login in 1.13.2 it's displaying as not connected just like a single stair, that is placed randomly.
How can we reproduce it?
Steps to reproduce the behavior:

Login on 1.13'
Look at a stair that should been connected with another stairs

Expected behaviour
There is no errors logs in the console.
ViaVersion Dump:
https://dump.viaversion.com/e6b596b331a1582a4f02cff72ae6f6ac6afa17ec26d815cbde66d3491659a541
Additional server info
I use Spigot 1.8.8 (Latest version)
The server system is connected up with the proxy system fork: Waterfall

Look at #856 and #1061

Alright, so a cache bug?
It doesn't tell me how to clear the cache and also fix the issue.

No, this is a bug where clients no longer auto link the blocks like fences and stairs together. In the 1.13 update, or aka the technical update or idc, servers will tell the clients how the blocks are linked. So ViaVersion needs to send stuff to tell the clients to link them. #1061 is a WIP prototype but it has been reported that it consumes ram like hell.


x86/MinnowBoard MAX: Adjust gpio pins based on kernel version

This should adjust the pinouts, at runtime, based on the current
kernel that's in use.  The Linux kernel made a change between
3.17 and 3.18 ( specifically ea584595fc85e65796335033dfca25ed655cd0ed )
that changed the numbering for GPIOs.  This obviously breaks everything
for mraa on the MinnowBoard MAX if you have a 3.18 or newer kernel.
Signed-off-by: John 'Warthog9' Hawley<EMAIL_ADDRESS>

Thanks alot for this, I wasn't aware of this change, it seems all gpiolibs implementation start from ARCH_NR_GPIOS (does anyone know how to 'query' this value at runtime?) so we'll likely need to treat this at a more global level at a later date. I've pushed it to the tocompile branch along with a few tweaks to make the code easier to change. I'll merge to master when I get a chance to give this a quick whirl on my minnow, probably monday.

All merged & tested, works a treat on a random 3.19 from git :). Thanks again!

Awesome!  Glad the fix made it in!