Support Global Principal Attribute from ldap for MFA

I have recently started using CAS and i wanted to pass the MFA provider do use via LDAP.  however it seems that values retrieved via ldap always end up in a collection even if the ldap attribute is a SINGLE-VAULE[1] .  This means i always seem to hit the following error
2019-08-14 17:56:04,202 DEBUG [org.apereo.cas.authentication.MultifactorAuthenticationUtils] - <Attribute value [[mfa-gauth]] is not a single-valued attribute>
This patch attempts to fix this so that collections of size one are also considered.  Apologizes in advance for the quality, its been some time since i looked at java
More details of my specific issue can be found on the cas-users google group[2]
[1]https://ldapwiki.com/wiki/SINGLE-VALUE
[2]https://groups.google.com/forum/#!topic/jasig-cas-user/6R4hFG74MG0

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.John Bond seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

Thank you very much!
I'd like to confirm this case with a test case first, and then we should be able merge quickly. Stay tuned please.

Could you also provide a sample project that demonstrates the problem?

@mmoayyed thanks for your comments.  We are currently using a this cas overlay
and my config is below
server.port=443
cas.server.name=https://idp.wikimedia.org
cas.server.prefix=https://idp.wikimedia.org/

cas.authn.mfa.gauth.json.location=file:///etc/cas/config/gauthdevices.json
cas.authn.mfa.u2f.json.location=file:///etc/cas/config/u2fdevices.json

logging.config: file:/etc/cas/config/log4j2.xml

cas.serviceRegistry.json.location=file:/etc/cas/services

cas.authn.mfa.globalPrincipalAttributeNameTriggers=preferredLanguage

cas.authn.ldap[0].providerClass=org.ldaptive.provider.unboundid.UnboundIDProvider
cas.authn.ldap[0].principalAttributeList=cn,memberOf,mail,preferredLanguage
cas.authn.ldap[0].type=AUTHENTICATED
cas.authn.ldap[0].connectionStrategy=ACTIVE_PASSIVE
cas.authn.ldap[0].ldapurl=ldaps://ldap-ro.eqiad.wikimedia.org:636 ldaps://ldap-ro.codfw.wikimedia.org:636
cas.authn.ldap[0].useStartTLS=false
cas.authn.ldap[0].basedn=dc=wikimedia,dc=org
cas.authn.ldap[0].searchFilter=cn={user}
cas.authn.ldap[0].binddn=cn=proxyagent,ou=profile,dc=wikimedia,dc=org
cas.authn.ldap[0].bindcredential=password
cas.authn.accept.users=
logging.level.org.apereo=DEBUG

My ldap entry looks like
dn: uid=jbond,ou=people,dc=wikimedia,dc=org
objectClass: ....
memberOf: ....
....
preferredLanguage: mfa-gauth

Ill try to create a vagrant project with all this configured later this evening

Thanks for the reference.
Your overlay is based on 6.0.3, while master here targets 6.1.x. Could you switch to 6.1.x and see if you can duplicate the problem? You could also try switching 6.0.4 though I am somewhat certain you might see the same issue.
I am unable to see failures as you report on master (6.1.x) via a few unit tests. See: https://github.com/apereo/cas/blob/master/core/cas-server-core-authentication-mfa/src/test/java/org/apereo/cas/authentication/DefaultMultifactorAuthenticationProviderResolverTests.java#L74

@mmoayyed thanks for all your help, i have tried using 6.1.0-RC4 and still see the same problem.  I have started a vagrant project to demo this but struggling to get the ldap stuff working.  Will hopefully be able to get a colleague to help tomorrow

@mmoayyed i have updated the cagrant project and one should [hoepfully] be able to recreate the issue with the following
# git clone https://github.com/b4ldr/cas-test
# cd cas-test
# vagrant up

This should spin up the cas server with a local ldap server on https://localhost:8443.  Once its started (which will be a couple of minutes after the vagrant up command finishes) you can login with:
username: test
password: changeme
This should succeed without any MFA pipeline.  One can then use vagrant ssh to login to the VM and check the logs
vagrant@buster:~$ grep 'is not a single-valued attribute' /var/log/cas.log
2019-08-16 13:47:13,662 DEBUG [org.apereo.cas.authentication.MultifactorAuthenticationUtils] - <Attribute value [[mfa-gauth]] is not a single-valued attribute>

Thanks

@b4ldr - Can you try with 6.1.0-RC5-SNAPSHOT?

@headman I get the following error when i try to build with rc5
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':bootWar'.
> Could not resolve all files for configuration ':runtimeClasspath'.
   > Could not resolve org.apereo.cas:cas-server-webapp-tomcat:6.1.0-RC5.
     Required by:
         project :
      > Could not resolve org.apereo.cas:cas-server-webapp-tomcat:6.1.0-RC5.
         > Could not get resource 'https://oss.jfrog.org/artifactory/oss-snapshot-local/org/apereo/cas/cas-server-webapp-tomcat/6.1.0-RC5/cas-server-webapp-tomcat-6.1.0-RC5.pom'.
            > Could not GET 'https://oss.jfrog.org/artifactory/oss-snapshot-local/org/apereo/cas/cas-server-webapp-tomcat/6.1.0-RC5/cas-server-webapp-tomcat-6.1.0-RC5.pom'. Received status code 409 from server:
   > Could not resolve org.apereo.cas:cas-server-support-ldap:6.1.0-RC5.
     Required by:
         project :
      > Could not resolve org.apereo.cas:cas-server-support-ldap:6.1.0-RC5.
         > Could not get resource 'https://oss.jfrog.org/artifactory/oss-snapshot-local/org/apereo/cas/cas-server-support-ldap/6.1.0-RC5/cas-server-support-ldap-6.1.0-RC5.pom'.
            > Could not GET 'https://oss.jfrog.org/artifactory/oss-snapshot-local/org/apereo/cas/cas-server-support-ldap/6.1.0-RC5/cas-server-support-ldap-6.1.0-RC5.pom'. Received status code 409 from server:
   > Could not resolve org.apereo.cas:cas-server-support-json-service-registry:6.1.0-RC5.
     Required by:
         project :
      > Could not resolve org.apereo.cas:cas-server-support-json-service-registry:6.1.0-RC5.
         > Could not get resource 'https://oss.jfrog.org/artifactory/oss-snapshot-local/org/apereo/cas/cas-server-support-json-service-registry/6.1.0-RC5/cas-server-support-json-service-registry-6.1.0-RC5.pom'.
            > Could not GET 'https://oss.jfrog.org/artifactory/oss-snapshot-local/org/apereo/cas/cas-server-support-json-service-registry/6.1.0-RC5/cas-server-support-json-service-registry-6.1.0-RC5.pom'. Received status code 409 from server:
   > Could not resolve org.apereo.cas:cas-server-support-saml:6.1.0-RC5.
     Required by:
         project :
      > Could not resolve org.apereo.cas:cas-server-support-saml:6.1.0-RC5.
         > Could not get resource 'https://oss.jfrog.org/artifactory/oss-snapshot-local/org/apereo/cas/cas-server-support-saml/6.1.0-RC5/cas-server-support-saml-6.1.0-RC5.pom'.
            > Could not GET 'https://oss.jfrog.org/artifactory/oss-snapshot-local/org/apereo/cas/cas-server-support-saml/6.1.0-RC5/cas-server-support-saml-6.1.0-RC5.pom'. Received status code 409 from server:
   > Could not resolve org.apereo.cas:cas-server-support-u2f:6.1.0-RC5.
     Required by:
         project :
      > Could not resolve org.apereo.cas:cas-server-support-u2f:6.1.0-RC5.
         > Could not get resource 'https://oss.jfrog.org/artifactory/oss-snapshot-local/org/apereo/cas/cas-server-support-u2f/6.1.0-RC5/cas-server-support-u2f-6.1.0-RC5.pom'.
            > Could not GET 'https://oss.jfrog.org/artifactory/oss-snapshot-local/org/apereo/cas/cas-server-support-u2f/6.1.0-RC5/cas-server-support-u2f-6.1.0-RC5.pom'. Received status code 409 from server:
   > Could not resolve org.apereo.cas:cas-server-support-gauth:6.1.0-RC5.
     Required by:
         project :
      > Could not resolve org.apereo.cas:cas-server-support-gauth:6.1.0-RC5.
         > Could not get resource 'https://oss.jfrog.org/artifactory/oss-snapshot-local/org/apereo/cas/cas-server-support-gauth/6.1.0-RC5/cas-server-support-gauth-6.1.0-RC5.pom'.
            > Could not GET 'https://oss.jfrog.org/artifactory/oss-snapshot-local/org/apereo/cas/cas-server-support-gauth/6.1.0-RC5/cas-server-support-gauth-6.1.0-RC5.pom'. Received status code 409 from server:

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

You will likely need to make sure the spring snapshot repository is included and enabled in the build.

You will likely need to make sure the spring snapshot repository is included and enabled in the build.

unfortunately i was already in over my head and im not sure how to do that.  If you are happy to provide instructions I'm happy to try, however i appreciate this is not an issue tracker or support forum.

You probably just need to add these two lines your build.gradle for your overlay:
https://github.com/apereo/cas-overlay-template/blob/bd648186dcbf4e46af3ee266276625e09a0eb803/build.gradle#L7
https://github.com/apereo/cas-overlay-template/blob/bd648186dcbf4e46af3ee266276625e09a0eb803/build.gradle#L25

thanks hdeadman, i have updated the vagrant project to use RC5-SNAPSHOT now and i do see a difference but it is now picking the wrong MFA
2019-08-16 18:34:06,091 DEBUG [org.apereo.cas.authentication.DefaultMultifactorAuthenticationProviderResolver] - <Located attribute value [[mfa-gauth]] for [[preferredLanguage]]>
2019-08-16 18:34:06,091 DEBUG [org.apereo.cas.authentication.MultifactorAuthenticationUtils] - <Attribute value [[mfa-gauth]] is not a single-valued attribute>
2019-08-16 18:34:06,092 DEBUG [org.apereo.cas.authentication.MultifactorAuthenticationUtils] - <Attribute value [[mfa-gauth]] is a multi-valued attribute>
2019-08-16 18:34:06,102 DEBUG [org.apereo.cas.authentication.DefaultMultifactorAuthenticationProviderResolver] - <Resolved set of events based on the attribute [preferredLanguage] are [[mfa-u2f]]>

For reference i do see
2019-08-16 18:31:23,078 DEBUG [org.apereo.cas.authentication.DefaultAuthenticationEventExecutionPlan] - <Candidate/Registered authentication handlers for this transaction are [[org.apereo.cas.authentication.handler.support.HttpBasedServiceCredentialsAuthenticationHandler@255d9277, org.apereo.cas.authentication.LdapAuthenticationHandler@36510e73, org.apereo.cas.gauth.GoogleAuthenticatorAuthenticationHandler@10e9a5fe, org.apereo.cas.adaptors.u2f.U2FAuthenticationHandler@1cde374]]>

I actually tried the vagrant file b/c I hadn't  used vagrant in a while and I like puppet so I wanted to see how they worked with CAS and I also hadn't done MFA but am thinking about doing, but I am on windows with hyperv so I had to switch the base image to "generic/debian10". The ldap server wasn't starting up with the password of changeme and I didn't really look into why.
Anyway, I think an easier way to re-create it would be with an overlay with the config checked in such that you could run gradlew createKeystore, gradlew copy and gradlew debug on any OS and have a local debuggable CAS server, with maybe a docker container for the ldap server, like this: https://github.com/hdeadman/cas-overlay-template/tree/gitlab-demo/docker/ldap.
To further troubleshoot, you could try turning on "trace" level debugging for the org.apereo.cas.authentication logger (<AsyncLogger name="org.apereo.cas.authentication" level="trace" includeLocation="true"/> in log4j2.xml`) and you should see more logging in MultifactorAuthenticationUtils that might indicate where it is going wrong. If that doesn't help, I know you said you hadn't done Java in awhile but if you got CAS master branch imported in IntelliJ Community Edition you could set breakpoints in the code, attach the debugger and step through MultifactorAuthenticationUtils to see whats going on. https://apereo.github.io/cas/developer/Build-Process-6X.html#intellij-idea. Should work if you have enough memory (e.g. 12-16GB).

@hdeadman  thanks for the feedback ill try to do a bit more digging on this later in the week

@hdeadman thanks for the pointers i tried adding trace put it didn't give much more information.  It looks to me like the mfa attribute resolution fails and then it fails back on to something elses.  I have updated my overlay project to include a docker ldap instance based on you example.

Relevant logs:
<Authentication request is not accompanied by a service given none is specified>
<Locating attribute value for attribute(s): [[preferredLanguage]]>
<Located attribute value [[mfa-gauth]] for [[preferredLanguage]]>
<Selected the provider [AbstractMultifactorAuthenticationProvider(bypassEvaluator=org.apereo.cas.authentication.bypass.DefaultChainingMultifactorAuthenticationBypassProvider@1d23b8ed, failureModeEvaluator=org.apereo.cas.authentication.DefaultMultifactorAuthenticationFailureModeEvaluator@559ca425, failureMode=UNDEFINED, id=mfa-gauth, order=10)] for service [null] out of [2] providers>
Selected a multifactor authentication provider out of
[[AbstractMultifactorAuthenticationProvider(bypassEvaluator=org.apereo.cas.authentication.bypass.DefaultChainingMultifactorAuthenticationBypassProvider@1d23b8ed, failureModeEvaluator=org.apereo.cas.authentication.DefaultMultifactorAuthenticationFailureModeEvaluator@559ca425, failureMode=UNDEFINED, id=mfa-gauth, order=10),
AbstractMultifactorAuthenticationProvider(bypassEvaluator=org.apereo.cas.authentication.bypass.DefaultChainingMultifactorAuthenticationBypassProvider@4b703832, failureModeEvaluator=org.apereo.cas.authentication.DefaultMultifactorAuthenticationFailureModeEvaluator@559ca425, failureMode=UNDEFINED, id=mfa-u2f, order=0)]] for [casuser] and service [null] as [AbstractMultifactorAuthenticationProvider(bypassEvaluator=org.apereo.cas.authentication.bypass.DefaultChainingMultifactorAuthenticationBypassProvider@1d23b8ed,
 failureModeEvaluator=org.apereo.cas.authentication.DefaultMultifactorAuthenticationFailureModeEvaluator@559ca425, failureMode=UNDEFINED, id=mfa-gauth, order=10)]>
<Converting multi-valued element [[mfa-gauth]]>
<Converting multi-valued element [[mfa-gauth]]>
<Attribute value [mfa-gauth] is a single-valued attribute>
<Attribute value predicate [org.apereo.cas.authentication.trigger.PrincipalAttributeMultifactorAuthenticationTrigger$$Lambda$1401/0x0000000800edb040@1a970341]
 has matched the [mfa-gauth]>
<Attempting check for availability of multifactor authentication provider
 [AbstractMultifactorAuthenticationProvider(bypassEvaluator=org.apereo.cas.authentication.bypass.DefaultChainingMultifactorAuthenticationBypassProvider@1d23b8ed,
failureModeEvaluator=org.apereo.cas.authentication.DefaultMultifactorAuthenticationFailureModeEvaluator@559ca425, failureMode=UNDEFINED, id=mfa-gauth, order=10)] for [null]>
<Provider [AbstractMultifactorAuthenticationProvider(bypassEvaluator=org.apereo.cas.authentication.bypass.DefaultChainingMultifactorAuthenticationBypassProvider@1d23b8ed,
failureModeEvaluator=org.apereo.cas.authentication.DefaultMultifactorAuthenticationFailureModeEvaluator@559ca425, failureMode=UNDEFINED, id=mfa-gauth, order=10)] is successfully verified>
<Attempting to find a matching transition for event id [mfa-gauth]>
<Resolved set of events based on the attribute [preferredLanguage] are [[mfa-gauth]]>
<Converting multi-valued element [[mfa-gauth]]>
<Building event based on the authentication provider [AbstractMultifactorAuthenticationProvider(bypassEvaluator=org.apereo.cas.authentication.bypass.DefaultChainingMultifactorAuthenticationBypassProvider@1d23b8ed,
failureModeEvaluator=org.apereo.cas.authentication.DefaultMultifactorAuthenticationFailureModeEvaluator@559ca425, failureMode=UNDEFINED, id=mfa-gauth, order=10)] and service [null]>
<Attempting to find a matching transition for event id [mfa-gauth]>
<Resolved single event [mfa-gauth] via [java.lang.String] for this context>

Same setup as what's reported; two MFA modules gauth and u2f plus an attribute trigger as preferredLanguage.
Closing until further evidence pops up.

@mmoayyed thanks for your work on this and sorry for my lack of update (moving house) .  It appears that the following commit fixed this issue?
https://github.com/apereo/cas/commit/0105b9ca9017e49e3838c6efec230f0ece195449
Is there a way i can update my cas project to test this code? i tried RC6-SNAPSHOT however that has a date stamp of 20190909 so i suspect doesn't have the above commit

I dont think there was ever an issue to begin with, as far as we can tell. But this might help confirm better:
https://github.com/apereo/cas-overlay-template#overview

When i test using RC6-SNAPSHOT i never see the following messages
<Converting multi-valued element [[mfa-gauth]]>
<Converting multi-valued element [[mfa-gauth]]>
<Attribute value [mfa-gauth] is a single-valued attribute>

instead i see <Attribute value [[mfa-gauth]] is not a single-valued attribute> it seems to me that the magic i'm missing is https://github.com/apereo/cas/commit/0105b9ca9017e49e3838c6efec230f0ece195449#diff-c23317b9ce2623086972a6857b216c74R167-R169
full log flow with trace
2019-09-11 13:39:53,839 DEBUG [org.apereo.cas.web.flow.authentication.BaseMultifactorAuthenticationProviderEventResolver] - <Authentication request is not accompanied by a service given none is specified>
2019-09-11 13:39:53,842 DEBUG [org.apereo.cas.authentication.DefaultMultifactorAuthenticationProviderResolver] - <Locating attribute value for attribute(s): [[preferredLanguage]]>
2019-09-11 13:39:53,842 DEBUG [org.apereo.cas.authentication.DefaultMultifactorAuthenticationProviderResolver] - <Selecting a multifactor authentication provider out of [[AbstractMultifactorAuthenticationProvider(bypassEvaluator=org.apereo.cas.authentication.bypass.DefaultChainingMultifactorAuthenticationBypassProvider@783dc0e7, failureModeEvaluator=org.apereo.cas.authentication.DefaultMultifactorAuthenticationFailureModeEvaluator@30133a11, failureMode=UNDEFINED, id=mfa-gauth, order=0), AbstractMultifactorAuthenticationProvider(bypassEvaluator=org.apereo.cas.authentication.bypass.DefaultChainingMultifactorAuthenticationBypassProvider@41bdaa81, failureModeEvaluator=org.apereo.cas.authentication.DefaultMultifactorAuthenticationFailureModeEvaluator@30133a11, failureMode=UNDEFINED, id=mfa-u2f, order=0)]] for [gauth] and service [null]>
2019-09-11 13:39:53,854 DEBUG [org.apereo.cas.web.flow.authentication.RankedMultifactorAuthenticationProviderSelector] - <Selected the provider [AbstractMultifactorAuthenticationProvider(bypassEvaluator=org.apereo.cas.authentication.bypass.DefaultChainingMultifactorAuthenticationBypassProvider@41bdaa81, failureModeEvaluator=org.apereo.cas.authentication.DefaultMultifactorAuthenticationFailureModeEvaluator@30133a11, failureMode=UNDEFINED, id=mfa-u2f, order=0)] for service [null] out of [2] providers>
2019-09-11 13:39:53,855 DEBUG [org.apereo.cas.authentication.DefaultMultifactorAuthenticationProviderResolver] - <Located attribute value [[mfa-gauth]] for [[preferredLanguage]]>
2019-09-11 13:39:53,855 DEBUG [org.apereo.cas.authentication.MultifactorAuthenticationUtils] - <Attribute value [[mfa-gauth]] is not a single-valued attribute>
2019-09-11 13:39:53,855 DEBUG [org.apereo.cas.authentication.MultifactorAuthenticationUtils] - <Attribute value [[mfa-gauth]] is a multi-valued attribute>
2019-09-11 13:39:53,857 DEBUG [org.apereo.cas.authentication.DefaultMultifactorAuthenticationProviderResolver] - <Resolved set of events based on the attribute [preferredLanguage] are [[mfa-u2f]]>
2019-09-11 13:39:53,858 TRACE [org.apereo.cas.web.flow.resolver.impl.mfa.DefaultMultifactorAuthenticationProviderWebflowEventResolver] - <Building event based on the authentication provider [AbstractMultifactorAuthenticationProvider(bypassEvaluator=org.apereo.cas.authentication.bypass.DefaultChainingMultifactorAuthenticationBypassProvider@41bdaa81, failureModeEvaluator=org.apereo.cas.authentication.DefaultMultifactorAuthenticationFailureModeEvaluator@30133a11, failureMode=UNDEFINED, id=mfa-u2f, order=0)] and service [null]>
2019-09-11 13:39:53,858 DEBUG [org.apereo.cas.web.flow.resolver.impl.AbstractCasWebflowEventResolver] - <Resolved single event [mfa-u2f] via [java.lang.String] for this context>

Im probably missing something but I'm not sure how https://github.com/apereo/cas-overlay-template#overview helps however i have updated my overlay and still see the same issue

I don't think the snapshots had successfully been pushed when you tested. They should be in sonatype now. https://travis-ci.org/apereo/cas/jobs/583619242

I'm not sure how https://github.com/apereo/cas-overlay-template#overview helps

My apologies; I should clarified that you should try to run the build with --refresh-dependencies to make sure everything dependencies and modules are force refreshed. (The comment in the referenced section basically says the same briefly)
When CAS pops up, the banner should show you the commit id. Always verify that commit with the commit log here to make sure your build is not too stale.


cmd/dist: replace (*tester).supportedBuildmode with cmd/internal/sys.BuildModeSupported

(*cmd/dist.tester).supportedBuildmode contains a large switch that determines whether a given buildmode is supported for a given GOOS/GOARCH:
https://github.com/golang/go/blob/fa90aaca7d523eaf81c02b48a412cad4ebc57817/src/cmd/dist/test.go#L977-L1038
That switch nearly exactly duplicates the one in cmd/internal/sys.BuildModeSupported, which was itself extracted from a portion of cmd/go/internal/work.buildModeInit.
cmd/dist should be changed to use (instead of duplicating) the function from cmd/internal/sys.

My understanding is that cmd/dist needs to build with Go 1.4, so it can't import cmd/internal/sys.  That's why I wrote two versions initially.

My understanding is that cmd/dist needs to build with Go 1.4, so it can't import cmd/internal/sys.  That's why I wrote two versions initially.

Ah, that could be.
But in that case, we should still have something in place that ensures that the two switches are equivalent, such as a regression-test that compares the two functions.
Or, perhaps (*tester).supportedBuildmode should probe cmd/go to determine support instead of encoding its own parallel assumptions in-process.

Ah, that could be.
But in that case, we should still have something in place that ensures that the two switches are equivalent, such as a regression-test that compares the two functions.
Or, perhaps (*tester).supportedBuildmode should probe cmd/go to determine support instead of encoding its own parallel assumptions in-process.


Detailed manual for GPS base station

Looks good to me. Better squash before/when merging.

I frankly don't have a proper opinion except "simpler is better". We are talking about whole "Find the position of your RTKBase" part where you upload logs to some French service for processing? Yes, it seems to complicated to me if we just need SOME location.

You are right. I will add the simpler Version tonight .
Pavel Veretennikov @.***> schrieb am Mi., 3. Apr. 2024,
17:29:

I frankly don't have a proper opinion except "simpler is better". We are
talking about whole "Find the position of your RTKBase" part where you
upload logs to some French service for processing? Yes, it seems to
complicated to me if we just need SOME location.
—
Reply to this email directly, view it on GitHub
https://github.com/ClemensElflein/openmower.de/pull/29#issuecomment-2034929813,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKJU4W4F2YVJWUDBICTBWLTY3QN7LAVCNFSM6AAAAABC4BUS62VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZUHEZDSOBRGM
.
You are receiving this because you were mentioned.Message ID:
@.***>

I added some content for the easier way. See https://github.com/karlranseierausrom/openmower.de/blob/patch-1/content/en/docs/Knowledge-Base/rtk-base-setup/_index.md . I have no time today to find out how to chenge the PR. Sorry for staying a beginner ;-)


Use srcObject instead of createObjectURL for audio recording example

Was going through https://developers.google.com/web/fundamentals/media/recording-audio/ and got error

(index):9 Uncaught (in promise) TypeError: Failed to execute 'createObjectURL' on 'URL': No function was found that matched the signature provided.

Used suggestion here and works nicely in Chrome stable 72.
https://stackoverflow.com/a/53821674/292408
https://developer.mozilla.org/en-US/docs/Web/API/URL/createObjectURL
What's changed, or what was fixed?

item 1
item 2

Fixes: #issue
Target Live Date: YYYY-MM-DD

[ ] This has been reviewed and approved by (NAME)
[ ] I have run npm test locally and all tests pass.
[ ] I have added the appropriate type-something label.
[ ] I've staged the site and manually verified that my content displays correctly.

CC: @petele

@beaufortfrancois - can you take a look and approve if you're OK with the change?

Change is OK. Thank you @ElijahLynn for this. Do you mind updating all other occurrences of createObjectURL in the doc? If not, I'll do in a follow-up change.

Sure, done. However, I discovered that due to an unescaped `

I'd merge this first and create another PR for the script tag issue.

Thanks, will work on the other one soon, I couldn't get the example working on first try, will have to set aside more time for it.

:tada: This has been pushed live to https://developers.google.com/web/


Idea: Different styles for album covers

I really love how album cover art is displayed currently.
But I wonder, if you could add some styles for customisation.
For example rounded edges for the artwork or round artwork (vinyl style).
Think this could look great :)

A made a poorly drafted sketch of what I meant with rounded corners :)

This looks great! exactly what I was looking for.

Think this would look great with a animation like in the Spotify for iOS app when switching to the next track.

This is how MiniPlay behaves. Quite nice. But animation might be difficult.

yeah I like the animation a lot, makes it even more aesthetically pleasing.

Maybe this could be helpful 🙈
https://github.com/PierrePerrin/PPMusicImageShadow

I like that fork :) https://github.com/andreamazz/SpotMenu

SpotMenu.app.zip
i've added a corner radius to notification widget


Remove kazydek from CODEOWNERS & add NHingerl

Description
As Karolina left the company and is no longer contributing to the project, she must be removed from the CODEOWNERS. As Nina's been with the company for some time now and contributing actively, she, in turn, must be added.
Changes proposed in this pull request:

Remove kazydek from CODEOWNERS
Add @NHingerl to CODEOWNERS

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Error: Failed to create file system: connect: network is unreachable

The problem is, rclone (Go resolver) issues A and AAAA queries at the same time and then tries to connect to the address from whichever response comes first, without falling back to the other one if the first one fails. Luckily A comes first most of the time, but not always.
Disabling IPV6 for all interfaces didn't help, I haven't try disabling the whole IPv6 stack, because I need it from time to time. My current workaround is teredo.
remote type: Google Drive
version: 1.33
OS: linux 4.1.35
command:
rclone copy ...

strace:
[pid 31637] socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[pid 31637] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
[pid 31637] setsockopt(3, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
[pid 31637] bind(3, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = 0
[pid 31637] socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4
[pid 31637] setsockopt(4, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0
[pid 31637] bind(4, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::ffff:<IP_ADDRESS>", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = 0
[pid 31637] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3
[pid 31637] setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid 31637] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("<DNS1>")}, 16) = 0
[pid 31637] getsockname(3, {sa_family=AF_INET, sin_port=htons(49852), sin_addr=inet_addr("<MY_IPV4>")}, [112->16]) = 0
[pid 31641] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP <unfinished ...>
[pid 31637] getpeername(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("<DNS1>")}, [112->16]) = 0
[pid 31641] <... socket resumed> )      = 5
[pid 31641] setsockopt(5, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid 31641] connect(5, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("<DNS1>")}, 16) = 0
[pid 31641] getsockname(5, {sa_family=AF_INET, sin_port=htons(58894), sin_addr=inet_addr("<MY_IPV4>")}, [112->16]) = 0
[pid 31641] getpeername(5, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("<DNS1>")}, [112->16]) = 0
strace: Process 31642 attached
strace: Process 31643 attached
[pid 31641] socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 3
[pid 31641] setsockopt(3, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0
[pid 31641] setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid 31641] connect(3, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "2a00:1450:4014:80d::200a", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable)
2016/11/17 10:54:33 Failed to create file system for "<account>:<dir>/": couldn't read info about Drive: Get https://www.googleapis.com/drive/v2/about?alt=json: dial tcp [2a00:1450:4014:80d::200a]:443: connect: network is unreachable

You might want to check out using one of these
export GODEBUG=netdns=cgo   # force cgo resolver

The default with the precompiled binaries is the go resolver.  I'm not 100% sure trying to use the cgo resolver will work with the precompiled binaries so you might have to compile from source to try that.
This issue looks like essentially what you are reporting: https://github.com/golang/go/issues/17448
If so it is scheduled for go 1.9 fix which should be out in ~7 months.
You can also try configuring your preference in /etc/gai/conf (might not work with the go resolver).

As go1.9 has been released, I'm now compiling rclone with it.
You should find this issue is fixed in the latest beta.
I'm going to close this issue now, please re-open if you are still seeing the issue.

2018/10/28 11:29:42 ERROR : apps/eigen/eigen-eigen-b30b87236a1b/build_dir/doc/snippets/CMakeFiles/compile_Cwise_less.dir/DependInfo.cmake: Failed to copy: upload failed: Post https://content.dropboxapi.com/2/files/upload: dial tcp [2620:100:6016:8::a27d:108]:443: connect: network is unreachable
Arch Linux: community/rclone 1.43.1-1 [installed]

@victoriastuart did you try disabling IPv6 since it doesn't seem to work for you?

No; that's not trivial (or recommended), is it? Recommendations?
https://wiki.archlinux.org/index.php/IPv6#Disable_IPv6 >> "Note: The Arch kernel has IPv6 support built in directly, therefore a module cannot be blacklisted."

No; that's not trivial (or recommended), is it? Recommendations?

I'm just saying rclone is trying to use your IPv6 since your computer supports it and it is configured.  This is the correct thing to do I think.  However your IPv6 isn't working which isn't rclone's fault.
If your IPv6 connection is flakey somehow then disabling IPv6 on your computer might be the best solution.
I normally do this by editing /etc/gai.conf - I'm not 100% sure this will work with rclone but might be worth a try.

Thank you for your response (appreciated)!  You are correct: I thought I had IPv6 access, but in fact it's only IPv4.  :-(
I'm contacting my ISP re: that issue, and will report back here once I hear back from them.

I want to ask, where between getpeername() / getsockname() which is the source and destination IP?
`**getsockname(3, {sa_family=AF_INET, sin_port=htons(49852), sin_addr=inet_addr("<MY_IPV4>")}, [112->16]) = 0
getpeername(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("")}, [112->16]) = 0 [pid 31641] <... socket resumed> ) = 5**`

I want to ask, where between getpeername() / getsockname() which is the source and destination IP?

getpeername gets the remote address, getsockname gets the local one I believe.

So ,
The Source is
getsockname(3, {sa_family=AF_INET, sin_port=htons(49852), sin_addr=inet_addr("<MY_IPV4>")}, [112->16]) = 0
And
Destination is
getpeername(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("")}, [112->16])=0
Is it true?

So ,
The Source is
getsockname(3, {sa_family=AF_INET, sin_port=htons(49852), sin_addr=inet_addr("<MY_IPV4>")}, [112->16]) = 0
And
Destination is
getpeername(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("")}, [112->16])=0
Is it true?

Yes that is right I think.


Cannot install v0.3.0 due to error[E0308]: mismatched types

I have just tried to install mdbook-keeper, but got the following error.
cargo install mdbook-keeper

error[E0308]: mismatched types
   --> /Users/shun/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mdbook-keeper-0.3.0/src/lib.rs:359:42
    |
359 | ...results = self.real_run(preprocessor_config, root, &mut book)?;
    |                   -------- ^^^^^^^^^^^^^^^^^^^ expected `Map<String, Value>`, found a different `Map<String, Value>`
    |                   |
    |                   arguments to this method are incorrect
    |
    = note: `Map<String, Value>` and `Map<String, Value>` have similar names, but are actually distinct types
note: `Map<String, Value>` is defined in crate `toml`
   --> /Users/shun/.cargo/registry/src/index.crates.io-6f17d22bba15001f/toml-0.7.6/src/map.rs:32:1
    |
32  | pub struct Map<K, V> {
    | ^^^^^^^^^^^^^^^^^^^^
note: `Map<String, Value>` is defined in crate `toml`
   --> /Users/shun/.cargo/registry/src/index.crates.io-6f17d22bba15001f/toml-0.5.11/src/map.rs:32:1
    |
32  | pub struct Map<K, V> {
    | ^^^^^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `toml` are being used?
note: method defined here
   --> /Users/shun/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mdbook-keeper-0.3.0/src/lib.rs:330:12
    |
330 |     pub fn real_run(
    |            ^^^^^^^^
331 |         &self,
332 |         preprocessor_config: PreprocessorConfig,
    |         ---------------------------------------

#11 should fix this.


Two Design Docs

-Exception Handeling - try/catch/finally and throwing exceptions
-Native Exceptions - list of planned supported native exceptions

Can you separate these into their own design docs?
Also, don't forget to mention #91.


The log file /tmp/aimlbotkernel.log causes error in Jupyterhub

In a multi-user environment such as Jupyterhub, the first user created a log file in /tmp/aimlbotkernel.log under the privilege of that user.
When there is another user trying to connect to the kernel, the user cannot open the /tmp/aimlbotkernel.log which causes crashes.

Right you are. This was not thought for a multi-user environment. The problem is that the standard Python logger does not care for the logfile permissions, so I guess it depends on the user's umask.
As a quick hack, I've changed the code to add the userid as part of the logfilename, i.e. aimlbotkernel-<uid>.log. That should avoid collisions (and put the logs of different users in a different files, which is probably a bonus).
Should have been solved by https://github.com/paulovn/aiml-chatbot-kernel/pull/15


Is it possible to add Russian language support?

Like a translation for the UI or is there an issue with Cyrillic-encoded books?

There an issue with Cyrillic-encoded books

What's the issue and do you have an example book you could share?

Since the book is in Cyrillic, the program does not find comparisons to English terms. I have attached the log and the book file.
<IP_ADDRESS>.05.2024.txt
Ighry Niemiezidy - Dzhieims Kori.zip

Sorry for the delay -- do you have a source somewhere that lists the terms in the Cyrillic form that show up in the book? My current term source doesn't have any Russian translations available.

Нет, к сожалению я знаю только русскую википедию. Я думал может получится к русскоязычным книгам добавлять аналогичные термины из английских источников. В любом случае спасибо за реакцию.

No, unfortunately I only know Russian Wikipedia. I thought it might be possible to add similar terms from English sources to Russian-language books. In any case, thanks for the reaction.

Yeah the issue is that they have to match exactly with the text of the book
so if the names are in Cyrillic the book I'd need to have a source for
those exact strings. For example, the app supports German because the data
source has German character names available.
On Sat, Jun 8, 2024, 9:47 a.m. Urfee @.***> wrote:

No, unfortunately I only know Russian Wikipedia. I thought it might be
possible to add similar terms from English sources to Russian-language
books. In any case, thanks for the reaction.
—
Reply to this email directly, view it on GitHub
https://github.com/Ephemerality/xray-builder.gui/issues/62#issuecomment-2156024924,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABWQRXFDIM5P522H52WMLALZGL4OFAVCNFSM6AAAAABHSDPYRGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJWGAZDIOJSGQ
.
You are receiving this because you commented.Message ID:
@.***>


Update entra-service-limits-include.md

Adding accuracy from the dynamic group limit from 5,000 to 15,000 as stated in the Dynamic group limits section

@rolyon
Can you review the proposed changes?
Important: When the changes are ready for publication, adding a #sign-off comment is the best way to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

Hi @jorlopama
Thanks for your contribution. Closing this PR because this update was made in another PR.
thanks

#please-close


Move crds directory into templates

Description of your changes
Moves crds folder of helm chart into templates folder for compatibility with helm 3 and adds note to documentation about using helm 3.
This PR does not upgrade our build system or minikube scripts, it only allows people to use helm 3 to install crossplane. We will likely update everything after Helm 3 has got a stable version.
See https://github.com/helm/helm/issues/5892 for details.
Fixes #839
Checklist

I have:

[x] Run make reviewable to ensure this PR is ready for review.
[x] Ensured this PR contains a neat, self documenting set of commits.
[x] Updated any relevant documentation, examples, or release notes.
[x] Updated the RBAC permissions in clusterrole.yaml to include any new types.

This PR adds that actually by removing the crds folder before generation.
On 24 Sep 2019 6:46 PM +0300, Daniel Mangum<EMAIL_ADDRESS>wrote:

@hasheddan approved this pull request.
LGTM! Noticing that when we make manifests it doesn't seem to remove old CRD's unless they are updated (e.g. the Policy CRD's still existed). I wonder if we should add a line to make manifests that removes the CRD's before generating them again? Although that gets a little more difficult with them now being in the general templates/ directory.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

This doesn't make use of the new crd-install mechanism in Helm 3, but, the main problem that they try to solve with that solution is that when you have CRs in templates folder, you have to have an ordered installation that installs CRDs first. So, we don't need that part since we don't install any CR instance.

@muvaf with https://github.com/crossplaneio/crossplane/pull/932 this is no longer the case. Could you share some info on the new crd-install mechanism in Helm 3? Can we keep both helm 2 & helm 3 working also with CR instances?

@turkenh The new mechanism is that you have a folder called crds alongside templates folder in your helm chart, not inside the templates folder. You can read the detailed explanation here https://github.com/helm/helm/issues/5871
In short, moving CRD YAMLs to crds folder does not work with Helm2 because it doesn't scan that folder. If we render it via a template(like before this PR), Helm 3 will first install crds folder and then try installing CRDs again since they are rendered into the templates folder, which results in already exists error. Putting pre-install annotation will make it work with Helm2 but not with Helm3 since these annotations are not used at all. It would've been good if we could switch rendering depending on Helm version but I couldn't find a good way to get Helm's version in the template functions.
So, I'm not sure a good, native solution exists. Maybe we can have a field in values.yaml for Helm version that user can supply, then use that to switch rendering on/off.


why was release bower removed?

before we could do gulp release:bower

nah it's fine ... makes sense.


(portis) delete the full screen

Describe the bug
Opening a claim link in wechat browser, it first takes me to the claim page. And after a few seconds, opens the portis full screen.
That screen needs to go, because it is very confusing.

this should be solved


Is there a need for creating helper for #if inside #each loop where checking is there something inside another collection?

I try something like this:
<select name="electoralUnit" multiple class="form-control" id="exampleFormControlSelect2" size="40">
  {{#each electoralUnits}}
      <option value="{{id}}" {{#if stateResult.electoralUnit}} disabled {{/if}}>{{name}}</option>
  {{/each}}
</select>

where stateResult is a collection outside electoralUnit collection. So, I have a multiple select with electoralUnits and I need to check if one of electoralUnits is in stateResult (already made it ref so ObjectId is in there) then that electoralUnit suppose to be disabled. I try to use ../stateResult in if helper but that's not working. Any helpe with this?

@shone83 you should try asking this on StackOverflow and posting some additional information (like what your data object looks like). The {{each}} and {{if}} helpers are built into handlebars so I'm not sure how we can help in this repository. If you're using helpers from this project and you run into a bug, please feel free to open another issue.

I tried there too, but nothing:
https://stackoverflow.com/questions/49417226/handlebars-disable-attribute-if-id-exist

Hi @shone83, try using the {{log .}} helper to see the context and available properties.

When I use log nothing happens...

Hi @jonschlinkert when I use log nothing happens...

@shone83 I'll post an answer on stackoverflow if one isn't there yet. Thanks for posting the link to it.

@doowb that would be great, still nothing yet ;)


Getting 404 at CF Steps

Hope you can help me find what I'm doing wrong.
I forked the repo, setup my secrets and variables. I'm getting a 404 error. This looks like a CF/Credential error, but I cannot figure it out :(  I run a curl test to verify the token is valid. "This API Token is valid and active"

Pics of my repo action secrets:

Pic of Variables:

Log at Delete old rule and list (If I comment this step out, fails at Create).
 Status: ${response.status});
^
Error: HTTP error! Status: 404
at request (file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/lib/helpers.js:106:11)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async file:///home/runner/work/cloudflare-gateway-pihole-scripts/cloudflare-gateway-pihole-scripts/cf_gateway_rule_delete.js:4:27
Node.js v20.9.0
Error: Process completed with exit code 1.

Try removing the CLOUDFLARE_ACCOUNT_EMAIL secret. It's not needed if you're using token authentication.

I started without that entry and had the same error, so I tried with it.  I just removed it and have the same error.

Resolved.  For others, the ID is NOT the userID, it's the account ID that looks like this:
5c836a76bc95766496cb94756e8675978  (this is fake)
The easiest way to find this is login to cloudflare, then this is the string after the dash URL:
https://dash.cloudflare.com/5c836a76bc95766496cb94756e8675978
Thank you!

closing


Add wise/check-fees refund command

Add refund command to create a compensation between Host and Collective to cover for negative differences in balance.

Note that Added Funds without a matching order may be breaking some expectations.


(DO NOT MERGE) (PUP-4056) Use privatebindir to resolve ruby

This is for review while I finish running acceptance to validate these changes:

Prepends privatebindir to the PATH environment used to execute ruby and gem on all hosts.
For ruby-based ENC scripts executing on the master, resolve ruby relative to privatebindir.
Moves the beaker pin to the version that contains the AIO pathing changes
Removes the AIO path overrides from node config files

CLA signed by all contributors.

$ be rake ci:test:aio SHA=58fbddcf1973262126a3f6699fdb929e4e78a6a5 CONFIG=config/nodes/rhel7.yaml TESTS=tests
...
              - Test Case Summary for suite 'tests' -
       Total Suite Time: 7076.47 seconds
      Average Test Time: 26.70 seconds
              Attempted: 265
                 Passed: 212
                 Failed: 0
                Errored: 0
                Skipped: 52
                Pending: 1
                  Total: 265

This should be merged after PR #3698

:+1:
Not following why the PR ordering matters though. Maybe it will all make sense with coffee :)

Maybe it will all make sense with coffee :)

Hmm I think maybe I need more coffee... I was thinking moving the beaker pin had to be the last thing, but I don't think that's strictly true.


need cursor position to generate suggestions

For multi-word inputs the cursor position is necessary for generating the list of suggestions. This might be accomplished by passing it in the suggestions callback, by passing the ref for the input element, or by exposing the refs on the AutoSuggest class so the caller can inspect the cursor position.

@acthp The input element has ref="input", so you should be able to access the input like that:
<Autosuggest ref="autosuggest" ... />

and then:
this.refs.autosuggest.refs.input


add support for reading only parts of uvfits files

This is addressing the uvfits part of issue #255

I tweaked the tutorial a bit as part of this PR, you can see how it looks here: http://pyuvdata.readthedocs.io/en/uvfits_memmap/tutorial.html

ok, @adampbeardsley I think this is ready for review.

@plaplant I made some changes to try to address your comments. Can you take another look when you get a chance?


AD auth: Error not thrown when Service Account Password is empty, while re-enabling auth after disabling it

Steps to Reproduce:

Enable AD Auth
Disable Access Control
Enable AD again now but with Service Account Password as empty.
No error is thrown

Expected:
Error should be thrown indicating that the Service Account Password is empty.

Tested with  rancher server version - v1.6.8-rc3
Enable AD Auth
Disable Access Control
Enable AD again now but with Service Account Password as empty.
User is presented with error message - "Failed to login, service account password not provided"
No error is thrown


Service state Stopped & Listener State Not Listening

Windows 11 Pro
Everything was working fine until windows 11 pro downloaded updates now after updated RDP stopped working. Also won't let me uninstall.

windows 11 pro downloaded updates

Yor termsrv.dll version have nothing in common with Windows 11 at all.
Repair your system
Not an RDPWrapper issue


Update .openpublishing.publish.config.json to support xref links on API

Per @mairaw
, I’d always use xref links and not regular links that.
For example, to be able to reference APIs from the UWP reference, we added the following line to our repo:
https://github.com/dotnet/docs/blob/master/.openpublishing.publish.config.json#L40

Enable xref links for API
Update a doc with xref API

@scottaddie  can you do this on current sprint?

@Rick-Anderson Yes. I plan to do this after I finish the LibMan docs, which I'm currently working on.


Unable to get up and running from the quick start

Hi, I am unable to scrape the erlang solutions blog as the quickstart guide states here:
https://github.com/oltarasenko/crawly#quickstart
Attempting to run the spider through iex results in:
iex(1)> Crawly.Engine.start_spider(MyCrawler.EslSpider)
[info] Starting the manager for Elixir.MyCrawler.EslSpider
[debug] Running spider init now.
[debug] Scraped ":title,:url"
[debug] Starting requests storage worker for Elixir.MyCrawler.EslSpider...
[debug] Started 2 workers for Elixir.MyCrawler.EslSpider
:ok
iex(2)> [info] Current crawl speed is: 0 items/min
[info] Stopping MyCrawler.EslSpider, itemcount timeout achieved

I'm quite lost as there is no way for me to debug this, if it is a network issue (which is highly unlikely since i can access the esl website through my browser), or if it is an issue with the urls being filtered out.
defmodule MyCrawler.EslSpider do
  @behaviour Crawly.Spider
  alias Crawly.Utils
  require Logger
  @impl Crawly.Spider
  def base_url(), do: "https://www.erlang-solutions.com"

  @impl Crawly.Spider
  def init() do
    Logger.debug("Running spider init now.")
    [start_urls: ["https://www.erlang-solutions.com/blog.html"]]
  end

  @impl Crawly.Spider
  def parse_item(response) do
    IO.inspect(response)
    hrefs = response.body |> Floki.find("a.more") |> Floki.attribute("href")

    requests =
      Utils.build_absolute_urls(hrefs, base_url())
      |> Utils.requests_from_urls()

    # Modified this to make it even more general, to eliminate the possibility of selector problem
    title = response.body |> Floki.find("title") |> Floki.text()

    %{
      :requests => requests,
      :items => [%{title: title, url: response.request_url}]
    }
  end
end

Of note is that the spider does not even call the parse_items callback, as the IO.inspect for the response is not called at all.
Config is as follows:
config :crawly,
  closespider_timeout: 10,
  concurrent_requests_per_domain: 2,
  follow_redirects: true,
  output_format: "csv",
  item: [:title, :url],
  item_id: :title

Could you also try to do Crawly.fetch("https://www.erlang-solutions.com/blog.html") just to check if it's possible to make requests to the blog from your machine?

@Ziinc in any case tell me what you're trying to achieve so I could try to help.

I'm trying to integrate Crawly with an existing phoenix project.
My dependencies are as follows (though I doubt that there would be dependency conflicts:
defp deps do
    [
      {:phoenix, "~> 1.4.0"},
      {:phoenix_pubsub, "~> 1.1"},
      {:phoenix_ecto, "~> 4.0"},
      {:ecto_sql, "~> 3.0"},
      {:postgrex, ">= 0.0.0"},
      {:phoenix_html, "~> 2.11"},
      {:phoenix_live_reload, "~> 1.2", only: :dev},
      {:gettext, "~> 0.11"},
      {:jason, "~> 1.0"},
      {:plug_cowboy, "~> 2.0"},
      {:mix_test_watch, "~> 0.8", only: :dev, runtime: false},
      {:comeonin, "~> 4.1"},
      {:bcrypt_elixir, "~> 1.1"},
      {:distillery, "~> 2.0", runtime: false},
      {:httpoison, "~> 1.4"},
      {:ex_aws, "~> 2.0"},
      {:ex_aws_s3, "~> 2.0"},
      {:sweet_xml, "~> 0.6"},
      {:bureaucrat, "~> 0.2.5"},
      {:hound, "~> 1.0", only: [:dev, :test], runtime: false},
      {:mogrify, "~> 0.7.3"},
      {:honeydew, "~> 1.4.4"},
      {:crawly, "~> 0.5.0"}
    ]
  end

I was actually initially on 1.8.2 when I first encountered the issue, and I had updated to see if it would help.
This is what happens with the Crawly.fetch()
Interactive Elixir (1.9.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Crawly.fetch("https://www.erlang-solutions.com/blog.html")
{:error,
 %HTTPoison.Error{id: nil, reason: {:option, :server_only, :honor_cipher_order}}}

I will try the quickstart on a fresh project and report back

@oltarasenko I've been able to get the quickstart to work in a new project, but it still does not work when i try add it into the existing project. I think it's an issue with dependency conflicts, notably httpoison.

@oltarasenko  Seems like it is an issue with hackney:
https://elixirforum.com/t/hackney-error-option-server-only-honor-cipher-order/25541
I'll try recompiling and updating the deps and try again

Updating the httpoison dependency did the trick, it upgraded hackney to the latest version, where the ssl issue was fixed. Thanks!

I think instead of letting the httpoison errors get swallowed up, it would be good to let them surface as debug logs.

Ok, I am re-opening it, as the thing you have mentioned (https://github.com/oltarasenko/crawly/blob/master/lib/crawly/worker.ex#L43) requires a fix. Will process an error here & log the error message. Good catch!

@Ziinc Could you please have a glance at https://github.com/oltarasenko/crawly/pull/15. It is pretty trivial for now. however, I plan to extend the worker quite soon (as I am currently working on a support of different user agents (aka webdriver support))

looks good. A possible extension (to add to backlog) for error behaviour could be a configurable fallback module to call when the backoff retries fail, thereby allowing the engine to exec some wrap up function, possibly to alert for errors.

A possible extension (to add to backlog) for error behaviour could be a configurable fallback module to call when the backoff retries fail, thereby allowing the engine to exec some wrap up function, possibly to alert for errors.

Possibly specified at either spider level or config level

This is now fixed in 0.6.0


add further navigation bar improvement suggestions

Checklist

[x] I tested all changes and all related features with different users (student, tutor, instructor, admin) on the test server https://artemistest.ase.in.tum.de.
[x] Client: I documented the TypeScript code using JSDoc style.
[x] Client: I added multiple screenshots/screencasts of my UI changes
[x] Client: I translated all the newly inserted strings into German and English

Description

show different exam participation status in the exam exercise navigation bar
the different status:

synced
not synced
submitted

Steps for Testing

check the navigation bar button turns yellow when exercise participation is not synced
-check the icon of the exercise button changes when exercise participation is submitted

Screenshots

The "connection bar" at the bottom of the screen is always green and always says connected.

I don't think the "Offline" network profile is sufficient when you're trying to disconnect from Artemis. Have you tried e.g. blocking your browser using a firewall or disconnecting your internet connection?

the timing of exams was messed up in my PR to remove the browser console error - I will fix it in a separate PR

@florian-glombik thanks for all the findings. Most of them are unrelated to this PR. Can you do me a favor and create a new issue with all these improvement suggestions? (just copy and paste your findings from here)
We will then address them in a follow up PR.


ISPN-7540 Calling getCache(String, String) - template overrides cache

Added version for deprecation
Changed internal method name

https://issues.jboss.org/browse/ISPN-7540

Integrated, thanks Will!


[feature request] Only enable the API keys for one of the two services

Once setting up OpenAI keys but not Perplexity, and toggling Use API keys (__O), will that lead to OpenAI keys being used and Perplexity requests going to the free credits?
https://discord.com/channels/963222352534048818/984966420603482182/1251834399218073641

As far as I understand, this is already implemented: if one of the two API key is left blank, then the 2000 free credits will be used instead of the API key.


Use of a non runtime-blocking sleep/usleep implementation

This PR attempts to add an alternate sleep/ usleep implementation to prevent the issues steming from using the node-sleep package in production.
Thanks for the heads-up about the issue @EMCP.
Deployed , tested & validated my test RaspberryPi

so if I understand this correctly, you basically just wrote your own sleep that utilized a non-blocking promise instead?
Happy to help, thanks for writing this

@EMCP I just wrapped the very useful nanotimer lib (non blocking) in promises for convenience, and voila :)


Cannot search for Microsoft Lists in docs.microsoft.com

Is your feature request related to a problem? Please describe.
It is not actually a feature. It is actually missing a product at this Site. When I go to Office 365 documents site, https://docs.microsoft.com/en-us/microsoft-365/?view=o365-worldwide, I cannot search anything on Microsoft Lists. See the screen captured here https://1drv.ms/u/s!AuZYHSwsevw1ojTgSUqLfctH2cpV?e=4KnKTD.
Describe the solution you'd like
Please add Microsoft Lists into the document.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

@chanmmn Microsoft Lists is not part of Office 365, which is now Microsoft 365.


gh.pennydreadfulmagic.com – 500 error at /.git/config

500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
Reported on github_tools by logged_out
InternalServerError 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
Stack Trace:
Python traceback
File "/home/discord/.pyenv/versions/3.8.2/lib/python3.8/site-packages/flask/app.py", line 2463, in call
return self.wsgi_app(environ, start_response)
File "/home/discord/.pyenv/versions/3.8.2/lib/python3.8/site-packages/flask/app.py", line 2449, in wsgi_app
response = self.handle_exception(e)
Exception_hash: 15c4ac9d81725488a0e577b38f62ab01229a08cf
Request Data
Request Method: GET
Path: /.git/config?
Cookies: ImmutableMultiDict([])
Endpoint: None
View Args: None
Person: logged_out
Referrer: None
Request Data: {}
Host: localhost:5002
Accept-Encoding: gzip
Cf-Ipcountry: RU
X-Forwarded-For: <IP_ADDRESS>, <IP_ADDRESS>
Cf-Ray: 57f4dbb95c763278-SJC
X-Forwarded-Proto: https
Cf-Visitor: {"scheme":"https"}
User-Agent: python-requests/2.23.0
Accept: /
Cf-Connecting-Ip: <IP_ADDRESS>
Cdn-Loop: cloudflare
X-Forwarded-Host: gh.pennydreadfulmagic.com
X-Forwarded-Server: gh.pennydreadfulmagic.com
Connection: Keep-Alive
Labels: github_tools; InternalServerError

This is on the gh.pennydreadfulmagic.com server. Gives a 502 from cloudflare. Not sure I understand it.

odd.

I don't know what this was but at the very least I have stopped receiving notifications about it.


对thrift map类型，支持添加新的key-value对

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Welcome PR


⚠ Bofrost.de has degraded performance

In 95994ea, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1149 ms

Resolved: Bofrost.de performance has improved in 4eb530b.


1385 - Collect statistics by default in ConvertToDelta & Update SQL API

Description
Resolves https://github.com/delta-io/delta/issues/1385
This PR gives the option to collect column statistics per file when doing ConvertToDelta operation. The current behaviour should stay the same (meaning that statistics should not be collected on convert).
How was this patch tested?
Current unit tests + added a couple of unit tests to cover this change
Does this PR introduce any user-facing changes?
Yes. Previously ConvertToDelta command was without the ability to collect statistics. Now, the convert function is overloaded with additional boolean parameter (with false as default) to enable collect statistics.
When this parameter is set to true, after the creation of the table, delta will compute all the files statistics (using legacy functionality and create another COMPUTE STATS commit to the log)

@amirmor1 what if we make the default implementation collect stats, and we have to opt out from doing so. This is a one-time per table operation, and I'd expect that users would prefer the default option of having faster reads henceforth with dataskipping and stats, instead of a slightly faster CONVERT operation.
Thus, the SQL API could look like CONVERT TO DELTA table_name [ NO STATISTICS ] [ PARTITIONED BY clause ]

@scottsand-db Thanks for the review and the comments.
I've fixed the PR with your requests.

@scottsand-db just want to make sure that I understand. You want the collectStats to be true by default but without any mention in the DeltaTable API? If yes -> So on the SQL API the NO STATISTICS has no effect, right? (at least until the second PR).
If yes -> it also means that a multiple unit tests which are not directly related to this PR will need to change since they expect to have a single commit on create while now we'll have two commits (I was able to avoid changing the results with NO STATISTICS during last commit).

just want to make sure that I understand. You want the collectStats to be true by default but without any mention in the DeltaTable API? If yes -> So on the SQL API the NO STATISTICS has no effect, right? (at least until the second PR).

I want collectStats to be true by default.
In the SQL API, when the user includes NO STATISTICS like CONVERT TO DELTA table_name [ NO STATISTICS ] [ PARTITIONED BY clause ] I want it to have an effect. I want statistics to not be collected.

In the second PR, you will add the DeltaTable API.

it also means that a multiple unit tests which are not directly related to this PR will need to change since they expect to have a single commit on convert while now we'll have two commits

You can change these tests to use the NO STATISTICS SQL API.
And then add new tests which do not have NO STATISTICS.
Make sense? Thanks for the great clarifying questions!

Can you update the PR title to be Collect statistics by default in ConvertToDelta & Update SQL API

Hi @scottsand-db.
Thanks for the comments and suggestion to make this on a single commit.
I've added the changes and also added a log message when collectStats is enabled but this option is disabled on config. Figured it requires some "explenation" why no statistics will be in such case.
Let me know if it makes sense.

@scottsand-db Thanks again for the help!
I've pushed the requested changes

@amirmor1 - hope all is well! Just following up on this PR. It's close to being merged and we would LOVE to have this in delta 2.2!

Hi @scottsand-db, I'm all good, thanks. Sorry for the late reply but I'm currently on vacation until next Tuesday.

@amirmor1 - no worries!! Enjoy the vacation! 🍹

Hi @amirmor1 could you please provide me with a public email address? I can't merge this with your private email address<EMAIL_ADDRESS>This is against Linux Foundation guidelines.

@scottsand-db<EMAIL_ADDRESS>


Fixed JS Image issue when using the genes library to compile ES modules

Need to explicitly use:
new window.Image()
otherwise it will create a lime Image object.

Are you sure that this change is necessary? The original code is this:
var image = new JSImage();

And JSImage is explicitly set as an alias of js.html.Image in the imports.
import js.html.Image as JSImage;

It should be impossible for the generated JS to use lime.graphics.Image in this context instead. There would be a conflict between js.html.Image and lime.graphics.Image here only if the constructor call was new Image() instead of new JSImage().
This seems more like a bug in genes.

I think Andres might be right. Looking at the output, new js.html.Image() in Haxe becomes new Image() in JavaScript. Normally, this creates a standard image element.
class Test {
  static function main() {
    trace(new js.html.Image()); //[object HTMLImageElement]
  }
}

But as soon as we define our own Image class, it shadows the standard one.
class Test {
  static function main() {
    trace(new js.html.Image()); //[not HTMLImageElement]
  }
}

@:keep
class Image {
  public inline function new() {
  }

  public inline function toString():String {
		return "[not HTMLImageElement]";
  }
}

There might still be a better workaround. And if there is, we should apply that in the two other locations this code is used.

Oh yes, I see. It slipped by mind that the fully-qualified js.html.Image wouldn't persist in JS. If there's another unqualified Image symbol in scope (which there seems to be, since lime.graphics.Image is imported), it will shadow the native JS API, as you said.
This seems like the correct solution to me.

Is this correct, though? HTML5HTTPRequest always imports lime.graphics.Image, so why do we call new JSImage() under any circumstances? Why not always call new window.JSImage()?

Also, I don't think classes are scoped in the JS output. Pretty sure that if lime.graphics.Image is compiled at all (i.e., not removed by DCE), it's available at the top level.

If I remember correctly, new JSImage() works correctly with the Haxe compiler's default JS output because it adds more qualification than the alternative JS output generated by genes (and probably hxgenjs too). I think the Haxe compiler generates an identifier for lime.graphics.Image, like lime_graphics_Image, so it doesn't conflict.

Oh right, it probably does. So as long as you provide a package (unlike try.haxe.org), you can't shadow the type.

I'd still like to get rid of the conditional compilation.

Done. (Took a couple tries to get it right...)

Looks good to me!

Thank you both! The additional commits work for me.


Convert ModalCurtain to TypeScript

This one needs also needs the base branch to be switched to master. There were a few conflicts when I tried.

Closing this one as part of a tidying up of our issue and PR backlog. Let's re-open once we can switch the base branch to master and rebase.


[BOW-112] Fix shows disappearing after their start but before their end

This is actually more complicated than initally thought due to the fact that determining how long a show is is a non-trivial task, because rundowns/items and continuity items determine it. The best solution I've come up with is to use views.

Codecov Report
All modified lines are covered by tests :white_check_mark:

Comparison is base (fbeca30) 15.88% compared to head (d335722) 15.88%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #124   +/-   ##
=======================================
  Coverage   15.88%   15.88%
=======================================
  Files           5        5
  Lines         957      957
  Branches        2        2
=======================================
  Hits          152      152
  Misses        805      805

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Windows without Titlebars can still be moved

There's a thin area just above the blank window that features the ability to drag the titleless window around. Shouldn't you only be able to drag windows around if they have a titlebar?

I can confirm this behavior will be fixed shortly. Thanks for reporting.

Fixed in 3d6c46f809fee100328965ab206c6cbb3b748cf8

Wow, you guys are on top of it.


Support responsive design layout

The layout of this theme is not responsive and therefore difficult to read on small devices. Would like to see a collapsable list to search through the class names as well as max width for the doc content.

Unlikely to fix since forking template here: https://github.com/pixijs/pixi-jsdoc-template


Unexpected outcome with OR clause

Describe the bug
We have a cluster with 1 config server and 18 content nodes in a group distribution with almost 40 million document
we are trying to run some queries on it but OR clause giving some unexpected outcome.
To Reproduce
First Ran these query
 "select * from sources * where (table contains \"com.gs.pwm.document.store.lex\" AND weakAnd(party_name_s contains \"Al Holbrook\",assignedorg_s contains ([{\"weight\": 1000}]\"FF\"), disclosure_type_s contains \"Quantitative Investment Strategies\", accountname_s contains \"DOUGLAS G. SMITH\", document_access_method_s contains ([{\"weight\":1000}]\"CUSTOM\"), transfereeaccount_s contains \"EA0326627\") AND weakAnd(documentactive_s contains ([{\"weight\":1000}]\"false\"),documentactive_s contains \"true\") AND weakAnd(assetclass_s contains \"PRIVATE_EQUITY\",assetclass_s contains\"EXCHANGE_FUND\") AND weakAnd(documentcode_s contains ([{\"weight\": 1000}]\"APPXA\"),documentcode_s contains \"SUBDOC\") AND weakAnd(signed_s contains ([{\"weight\": 1000}]\"true\"),signed_s contains \"false\") AND weakAnd(ordertype_s contains \"BUY\", saleschannel_s contains \"PWM\"))";

output :
{
"id": "toplevel",
"relevance": 1.0,
"fields": {
"totalCount": 6877
}
then added more clauses on above query.
"select * from sources * where (table contains \"com.gs.pwm.document.store.lex\" AND weakAnd(party_name_s contains \"Al Holbrook\",assignedorg_s contains ([{\"weight\": 1000}]\"FF\"), disclosure_type_s contains \"Quantitative Investment Strategies\", accountname_s contains \"DOUGLAS G. SMITH\", document_access_method_s contains ([{\"weight\":1000}]\"CUSTOM\"), transfereeaccount_s contains \"EA0326627\") AND weakAnd(documentactive_s contains ([{\"weight\":1000}]\"false\"),documentactive_s contains \"true\") AND weakAnd(assetclass_s contains \"PRIVATE_EQUITY\",assetclass_s contains\"EXCHANGE_FUND\") AND weakAnd(documentcode_s contains ([{\"weight\": 1000}]\"APPXA\"),documentcode_s contains \"SUBDOC\") AND weakAnd(signed_s contains ([{\"weight\": 1000}]\"true\"),signed_s contains \"false\") AND weakAnd(ordertype_s contains \"BUY\", saleschannel_s contains \"PWM\"))  AND (collection contains \"com.gs.pwm.document.store.lex.content\") ;"

output:
{
"id": "toplevel",
"relevance": 1.0,
"fields": {
"totalCount": 6400
},
Then added an OR clause on collection fields
 "select * from sources * where (table contains \"com.gs.pwm.document.store.lex\" AND weakAnd(party_name_s contains \"Al Holbrook\",assignedorg_s contains ([{\"weight\": 1000}]\"FF\"), disclosure_type_s contains \"Quantitative Investment Strategies\", accountname_s contains \"DOUGLAS G. SMITH\", document_access_method_s contains ([{\"weight\":1000}]\"CUSTOM\"), transfereeaccount_s contains \"EA0326627\") AND weakAnd(documentactive_s contains ([{\"weight\":1000}]\"false\"),documentactive_s contains \"true\") AND weakAnd(assetclass_s contains \"PRIVATE_EQUITY\",assetclass_s contains\"EXCHANGE_FUND\") AND weakAnd(documentcode_s contains ([{\"weight\": 1000}]\"APPXA\"),documentcode_s contains \"SUBDOC\") AND weakAnd(signed_s contains ([{\"weight\": 1000}]\"true\"),signed_s contains \"false\") AND weakAnd(ordertype_s contains \"BUY\", saleschannel_s contains \"PWM\"))  AND (  collection contains \"com.gs.pwm.document.store.lex.content\" OR collection contains \"com.gs.imd.activities.document.fi_investmentview\") ;"

output
{
"id": "toplevel",
"relevance": 1.0,
"fields": {
"totalCount": 0
}
Don't understand why it returns 0 documents after adding the OR clause.
Expected behavior
It should have returned the same number documents after adding the OR clause even if the new condition does not match with any document.
Vespa version
7.589.25
Additional context
Add any other context about the problem here.

To see how the YQL expression you run is handled, you can add the request parameter trace.level to the request.
You can see an example of the output from the trace using trace.level=2 here: https://docs.vespa.ai/en/query-api.html#query-tracing
If you are not able to get further with this, please include the trace so we can help you debug.
Thanks!

Your queries look fine. I guess the last query probably timed out from matching a log of documents.
You can see this by looking at the coverage information in the response, and you can increase timeout by passing e.g &timeout=5s

@bratseth, I will try this and update you further

traceLevelAfterOR.txt
traceLevelBeforeOR.txt
Attached the tracelevel Output for 2nd and 3rd query in description

Thank you.
tl;dr: Use OR instead of WeakAnd for your structured conditions.
The query here is
AND
table:'com gs pwm document stor lex'
(WEAKAND(100) party_name_s:'al holbrok' assignedorg_s:ff!1000 disclosure_type_s:'quantitativ investment strategies' accountname_s:'douglas g smith' document_access_method_s:custom!1000 transfereeaccount_s:ea0326627)
(WEAKAND(100) documentactive_s:fal!1000 documentactive_s:true)
(WEAKAND(100) assetclass_s:'privat equity' assetclass_s:'exchang fund')
(WEAKAND(100) documentcode_s:appxa!1000 documentcode_s:subdoc)
(WEAKAND(100) signed_s:true!1000 signed_s:fal)
(WEAKAND(100) ordertype_s:buy saleschannel_s:pwm)
(OR collection:'com gs pwm document stor lex content' collection:'com gs imd activities document fi investmentview')

... in the after OR case, in the before OR case the last line is just the first of the two items.
Each WEAKAND(100) operator will return at least 100 hits matching some of the items of it (per node).
Because of this, AND'ing two WEAKAND operators may produce 0 matches, if those two operators happen to produce disjunct sets of hits.
If the other filters in the query are weaker, the probability of that goes up. Adding the OR makes that filter weaker.
Looking at the content of these items, I think these are structured data which should act as filters and you want to OR them to surface all the matches. I guess that's where you started and this is an attempt at optimization? It becomes faster due to discarding matches locally, but unfortunately that also means you can't expect global correctness, although you can of course also increase the probability by using a larger N than 100.
If match details are not accessed in ranking, it would increase performance a lot to mark them as filter terms. If you haven't I would try that first.

@bratseth, in the after OR case, in the before OR case the last line is just the first of the two items.
I don't understand what you mean by above statement

Here tale the example of these simple queries
"select * from sources * where (collection contains "com.gs.pwm.document.store.lex.content") ;"
this returns "totalCount": 1038307
"select * from sources * where (collection contains "com.gs.imd.activities.document.fi_investmentview" OR collection contains "com.gs.pwm.document.store.lex.content") ;"
this returns "totalCount": 3107
But this query should have return all documents that matches either of the condition In OR and should have return large number of documents.
I am unable to understand the behavior here
now the second query: "select * from sources * where (collection contains "com.gs.imd.activities.document.fi_investmentview") ;"
this returns "totalCount": 3107
Now OR the condition in above two queries into single query

By "in the after OR case, in the before OR case the last line is just the first of the two items." I was just summarizing the difference between the two queries you shared.
In your last comment you provide two queries which lack all the WEAKAND clauses of the queries you asked about earlier. Are you saying you observe the same thing running these much simpler queries. If so, please share the traces of those queries instead of the more complex ones discussed above.

tracelevel_2ndQuery.txt
tracelevelWithOR_3rdQuey.txt
Attached the tracelevel output

Okay, both those queries matches the same number of documents (3107), none contains any WeakAnds, and none time out.
Since the second is equal to the first plus an additional OR term, that indicates the second term does not have any other matches, and at most 3107 matches if evaluated standalone. Can you also share the query that shows this is not the case? Also: Check if you have match-phase settings in the rank profile. That will also allow adaptively rejecting matches.

"select * from sources * where (collection contains "com.gs.pwm.document.store.lex.content") ;"
this returns "totalCount": 1038307
This is the result when we use only 2nd term of OR condition so we do have matches for this and it is lot more that 1st term.  I will also attach the tracelevel for this.

@bratseth Any update on this?

@shubh9194 : Can you please also share more detailed backend trace information for the queries used in tracelevel_2ndQuery.txt, tracelevelWithOR_3rdQuey.txt and tracelevel_query1.txt, by also adding "&trace.explainLevel=1" to the queries? That will make us understand how these queries are executed on the content nodes.

tracelevel_query1.txt
tracelevel_query2.txt
tracelevelWithOR_3rdQuery.txt
@geirst please find the attached tracelevel generated with following parameters
"timeout": "120s",
"hits": 0,
"tracelevel":2,
"trace.explainLevel":1

It turns out the information we need are not part of trace information using those levels on the old Vespa 7 version. @shubh9194 can you please run the three queries again using "&tracelevel=6&trace.explainLevel=1" ?

@shubh9194 : After analysing the trace information, we have found that the problem is caused by different stemming in the queries.
In query 1 the language is detected to be ENGLISH and 'store' is stemmed to 'store' (unchanged).
{
    "message": "Stemming with language=ENGLISH"
},
{
    "message": "Stemming: [\nselect * from sources * where collection contains ([{origin: {original: \"com.gs.pwm.document.store.lex.content\", offset: 0, length: 37}}]phrase(\"com\", \"gs\", \"pwm\", \"document\", \"store\", \"lex\", \"content\")) ...
},
...
{
    "message": "sc0.num0 search to dispatch: query=[collection:'com gs pwm document store lex content'] timeout=119998ms offset=0 hits=0 groupingSessionCache=true sessionId=2f62bb90-f462-46eb-8047-9a61a8fc0eb1.1663923489889.343.default grouping=0 :  restrict=[imdamdeloreanlexDocument]"
},

But in query 3 the language is detected to be DUTCH and 'store' is stemmed to 'stor'. The following phrase (collection:'com gs pwm document stor lex content') is sent to the backend and does not match any documents. Only the 3107 hits from the other phrase under the OR are returned.
{
    "message": "Stemming with language=DUTCH"
},
{
    "message": "Stemming: [\nselect * from sources * where (collection contains ([{origin: {original: \"com.gs.imd.activities.document.fi_investmentview\", offset: 0, length: 48}}]phrase(\"com\", \"gs\", \"imd\", \"activities\", \"document\", \"fi\", \"investmentview\")) OR collection contains ([{origin: {original: \"com.gs.pwm.document.store.lex.content\", offset: 0, length: 37}}]phrase(\"com\", \"gs\", \"pwm\", \"document\", \"stor\", \"lex\", \"content\"))) ...
},
...
{
    "message": "sc0.num0 search to dispatch: query=[OR collection:'com gs imd activities document fi investmentview' collection:'com gs pwm document stor lex content'] timeout=119990ms offset=0 hits=0 groupingSessionCache=true sessionId=2f62bb90-f462-46eb-8047-9a61a8fc0eb1.1663923195482.190.default grouping=0 :  restrict=[imdamdealflowlexDocument]"
},

To fix this you need to specify in the query request which language to use, e.g. "&model.language=en". For more details see https://docs.vespa.ai/en/linguistics.html#querying-with-language and https://docs.vespa.ai/en/reference/query-api-reference.html#model.language. FYI @bratseth.

Ok I got it but I have another doubt.
when I add model.language=en to second query, it returns 0 document now instead of 3107
traceLevel_query2_en.txt

In the original query 2 the language was also detected to be DUTCH. The phrase sent to the backend was unchanged: (collection:'com gs imd activities document fi investmentview'). Apparently, this matches with how these documents where stemmed during feeding.
{
    "message": "Stemming with language=DUTCH"
},
{
    "message": "Stemming: [\nselect * from sources * where collection contains ([{origin: {original: \"com.gs.imd.activities.document.fi_investmentview\", offset: 0, length: 48}}]phrase(\"com\", \"gs\", \"imd\", \"activities\", \"document\", \"fi\", \"investmentview\")) ...
},
...
{
    "message": "sc0.num0 search to dispatch: query=[collection:'com gs imd activities document fi investmentview'] ...  restrict=[amCRPlexDocument]"
},

When using "model.language=en" however, we observe that the stemming is different, and the phrase sent to the backend is instead (collection:'com gs imd activity document fi investmentview'), which doesn't match.
{
    "message": "Stemming with language=ENGLISH"
},
{
    "message": "Stemming: [\nselect * from sources * where collection contains ([{origin: {original: \"com.gs.imd.activities.document.fi_investmentview\", offset: 0, length: 48}}]phrase(\"com\", \"gs\", \"imd\", \"activity\", \"document\", \"fi\", \"investmentview\")) ...
},
...
{
    "message": "sc0.num0 search to dispatch: query=[collection:'com gs imd activity document fi investmentview'] ...  restrict=[imdamdealflowlexDocument]"
},

It is important that the stemming used during feeding / indexing is the same as the one used at query time. How is the field collection defined in the schema?

This is how we define the collection fields
field collection type array {
indexing: summary | index
    }

Based on the content of the collection field (some kind of id), we suggest using match: word  (https://docs.vespa.ai/en/reference/schema-reference.html#match). This should fix the stemming asymmetry currently observed between feeding / indexing and querying. Note that such change requires re-feeding of all data.
field collection type array {
    indexing: summary | index
    match: word
}

Which also will significantly improve search performance, avoiding phrase matching.
This is covered here . Using match:word disables stemming and avoids doing phrase search.
If you know the language of your documents, you should consider overriding language detection, see linguistics in vespa

@geirst
but with above solution, it sees we are unable to run queries having wildcard like
"select * from sources * where (collection contains "com.*") ;"
Is there any other alternate way?

Wildcard or prefix search has never been supported for string fields with index, the above will simply search for 'com'

If you need prefix matching you must change the field from index to attribute, and use prefix syntax in the query: https://docs.vespa.ai/en/reference/query-language-reference.html#prefix.
field collection type array<string> {
    indexing: summary | attribute
    match: word
}

@jobergum
I have tried with attribute field as well
select * from sources * where (collection_attr matches "com.gs.pwm.document.store.lex.content") ;
this query return 21992 docs
select * from sources * where (collection_attr matches "com.*.document.store.lex.content")
this query also works and returns 21992 docs
but following query return 0 documents
select * from sources * where (collection_attr matches "com*.document.store.lex.content") ;
select * from sources * where (collection_attr matches "com.gs.pwm.docu*.store.lex.content") ;
select * from sources * where (collection_attr matches "com*.=document.store*content") ;
not sure why wildcard does not work for these queries
Also how many terms can we use with OR? Is there any limit?

First of all when changing from index to attribute you must follow:

https://docs.vespa.ai/en/schemas.html#schema-modifications
https://docs.vespa.ai/en/reference/schema-reference.html#modifying-schemas

Then you use matches which is regular expressions, to see how Vespa parses the regular expressions, use tracelevel to trace the interpretation.


Fix reference to singular test directories renamed to tests

This should fix the remaining issues..  Appveyer passes and travis should with latest commit.


added support for new testnets in blockfrost

Not sure if this is the best approach to deal with the new testnets (preprod and review). Another alternative would be to call them all Network.TESTNET (since they share some "testnet" simmilarities like address), but require (or allow) user to specify the blockfrost URL.

Yeah, I see address derivation is failing with my addition.
Address(payment_vkey.hash(), stake_vkey.hash(), network=Network.PREPROD) is giving a different value from Address(payment_vkey.hash(), stake_vkey.hash(), network=Network.TESTNET) which is not good
Maybe mantaining the same networks, but allowing a custom blockfrost url is actually the best option