New errors (maybe related to ? super) from eisop/checker-framework#746

Using the command given here to test with main-eisop:
/tmp/tmp.OBavpODHqE/guava/guava/src/com/google/common/collect/Multisets.java:1123: error: [argument.type.incompatible] incompatible argument for parameter o of Collections.nCopies.
        entry -> Collections.nCopies(entry.getCount(), entry.getElement()).spliterator(),
                                                                       ^
  found   : E*
  required: E

E* should be a valid substitute for E in lenient mode.
/tmp/tmp.OBavpODHqE/guava/guava/src/com/google/common/collect/TreeMultimap.java:224: error: [assignment.type.incompatible] incompatible types in assignment.
    keyComparator = requireNonNull((Comparator<? super K>) stream.readObject());
                                  ^
  found   : Comparator<capture#767 of ? super K>
  required: Comparator<? super K>

I'm not totally sure what's up with that one, nor even that these two errors are closely related.
Previous discussions (which also touched on the problem that SequencedCollection and friends are unannotated):

https://github.com/jspecify/jspecify-reference-checker/pull/178#issuecomment-2072654491
https://github.com/eisop/checker-framework/pull/746#issuecomment-2239583272
https://github.com/jspecify/jspecify-reference-checker/issues/192

I have promised to try to produce a smaller repro of this problem. For now, I'm filing this issue as a placeholder.

I tried minimizing a test case with ? super TV that doesn't depend on the JDK:
> cat Issue193.java
import org.jspecify.annotations.NullMarked;
import org.jspecify.annotations.Nullable;

class Box<B extends @Nullable Object> {}

interface MyList<T extends @Nullable Object> {
  boolean removeIf(Box<? super T> b);
}

@NullMarked
interface ListSubclass<Z extends @Nullable Object> extends MyList<Z> {
  boolean removeIf(Box<? super Z> b);
}

> ./demo Issue193.java
Issue193.java:12: error: [override.param.invalid] Incompatible parameter type for b.
  boolean removeIf(Box<? super Z> b);
                                  ^
  found   : Box<? super Z>
  required: Box<? super Z*>*
  Consequence: method in ListSubclass<Z>
    boolean removeIf(Box<? super Z>)
  cannot override method in MyList<Z*>
    boolean removeIf(Box<? super Z*>*)
1 error

There is quite some unspecified nullness, but I think they are expected in MyList:

the * on the Box parameter type itself is because the code is in unmarked scope.
the Z* is because type variable uses in unmarked scope are unspecified: https://github.com/jspecify/jspecify-reference-checker/blob/af9dfbc7a018b3b079725d40ebaf87bc4c21e8bd/src/main/java/com/google/jspecify/nullness/NullSpecAnnotatedTypeFactory.java#L162
maybe up for discussion is the cannot override method in MyList<Z*>, which is from the substitution of the nullmarked Z into the unmarked MyList

So the types themselves look correct. What is unclear is why the nullmarked signature is not allowed as override for the unspecified method.
If I use Box<?> as parameter type, the override is accepted - so the top-level subtype test seems okay.
So maybe something with the type hierarchy for ? super and unspecified nullness doesn't work as we want it to.
@cpovirk Do the types themselves look correct to you?

I left some more comments on https://github.com/jspecify/jspecify-reference-checker/pull/178#issuecomment-2252991364 before coming back here....
I might continue to postpone looking at this further until maybe Tuesday (when I'll be in a better position to attach a debugger), but I wonder if we need to expand our existing special case for ? extends to also cover ? super:
https://github.com/jspecify/jspecify-reference-checker/blob/af9dfbc7a018b3b079725d40ebaf87bc4c21e8bd/src/main/java/com/google/jspecify/nullness/NullSpecAnnotatedTypeFactory.java#L662-L669
On the need for the special case, I think I once concluded:

The Checker Framework (to this day) performs capture conversion on the supermethod's arguments.
The JLS just wants you to perform essentially a substitution.
The difference might have something to do with a Checker Framework goal of letting you loosen type-parameter bounds in submethods where that's safe?? (Such loosening would be similar in spirit to contravariant parameter types.)

It looks like our Checker Framework fork had its own checking of parameters for overrides. I don't know if that sort of approach would help us here or not, but even if it did, it would be piling hacks upon hacks :)

Actually, wait, my mistake: That JLS citation is just for method-level type parameters, not the class-level ones we're dealing with here. And there might actually be a place for capture conversion in override checking, though I still suspect it would happen at an earlier step (during computing the value of overridden?).
The JLS section about class-level type parameters is probably "as a member of the supertype of C that names I" in <IP_ADDRESS>. I think that "as a member of" refers to 4.5.2. There we can apply capture conversion but only if we are getting the members of a Foo<? extends ...> or similar. In our example, we're getting the members of a Foo<...> (with no ? extends). (That's probably always the case when checking method declarations but maybe not always when checking member references?) So I don't think we want to apply capture conversion at all when checking method declarations for overriding.
In theory, we could override createOverrideChecker to return our own instance, which could set a global variable that changes the behavior of applyCaptureConversion (to make it only apply substitution) during checking.... Or maybe substitution has already been performed? If so, great, but that doesn't exactly make the solution any less hacky :)
I have no pressing need to import #178 into Google to use there (and eisop/checker-framework#746 alone produces much more limited fallout), so I'm not sure how hard I want to think about this. I still dream that someday we can rewrite override checking and more in a simpler way if we don't need to support the full generality of the Checker Framework....

If we do want to import this, another option might be to arrange for our compilations that currently see unannotated JDK APIs to see annotated JDK APIs instead. (That's actually what I did last time :)) That would probably mean annotating all the major collection interfaces. In particular, as we know from our experience with SequencedCollection, it's not likely to be enough to just annotate, e.g., removeIf without also annotating List itself (to cover its extends Collection<E>).

Removing the capture-conversion step I mentioned above helps more (apparently without creating other problems) but somehow still doesn't solve the problem fully :) I still have problems with replaceAll and computeIfAbsent (and one other method not from the JDK), but I no longer have problems with compute, for example. Maybe now ? super is OK but ? extends is still broken—but only ? extends T, not ? extends @Nullable T / ? extends @NonNull T?
(Certainly we're now at least at the point that I could easily go with suppressions and/or annotating a couple more JDK APIs.)
I sat down intending to run a quick test, and obviously I ended up poking around a little more than that... :) But I'm going to try to stop here for now.

I observe that no tests fail (./gradlew clean assemble test) in either typetools or eisop if I remove that applyCaptureConversion call. Nor do any nullness checks in Google's codebase.
I think I've convinced myself that the applyCaptureConversion call should be unnecessary but mostly(?) harmless: The result of capture conversion should always be a subtype of the original type, and the Checker Framework is checking parameters for subtyping, not equality (though the final JSpecify story may be more complex, and that's another reason to consider removing applyCaptureConversion). The fact that it's apparently not harmless in this case does seem like a sign that we're still getting something wrong in the JSpecify subtyping checking. That's the kind of thing that I'll try to look into on ~Tuesday.

I have somehow still not gotten around to attaching a debugger, but I'll note some things here:
I've been testing with a few combinations but now primarily with:

no wonkyness workaround
no applyCaptureConversion call
various tweaks to treat unspecified nullness as if it one of the possibilities is that it was "intended to be" non-nullable
a @NullMarked type with <E extends @Nullable Object> that extends an unmarked type with just <E>

From what I'm seeing, I think that:

? super is fine alone or in combination with other ? super arguments.
? extends is fine alone or in combination with other ? extends arguments.
A type that contains both ? super and ? extends is a problem, whether it's ? super T+? extends T or ? super A+? extends B.

-AatfDoNotCache doesn't help (nor hurt).
My assumption at this point is that ? super passes the first of the two distinct parameter checks and that ? extends passes the second (or vice versa), so together they fail. Hopefully that assumption will turn out to be correct, and hopefully I won't get confused along the way by the two separate checks and results :)

I observe that no tests fail (./gradlew clean assemble test) in either typetools or eisop if I remove that applyCaptureConversion call.

Oh, I get the impression that I probably need to perform more testing than that to really see if it's working: I can change checkParameters() to always set success to false, and nothing breaks in ./gradlew clean assemble test.
This slightly undercuts my claim above (though I do still have the significant Google testing to fall back on!) and definitely undercuts my attempt to figure out which of the "two distinct parameter checks" actually matters (at least without also testing on Google's codebase).

My assumption at this point is that ? super passes the first of the two distinct parameter checks and that ? extends passes the second (or vice versa), so together they fail. Hopefully that assumption will turn out to be correct

Nope! :) I see the same result if I change checkParameters() to use only...
boolean success = typeHierarchy.isSubtype(overriddenParams.get(i), overriderParams.get(i));

...i.e., without the testTypevarContainment case. (I'm also still omitting the applyCaptureConversion call, as before.)
Incidentally, if I keep only testTypevarContainment, I get errors even for plain ? extends and ? super (though I should probably restore the "wonkyness workaround" if I want to give that a fair shake).
I'm still interested in whether the current Checker Framework override checking could be simplified. (Maybe some of it is a holdover from before the implementation of capture conversion, for example?) To that end, I've put together https://github.com/eisop/checker-framework/pull/829 to test the approach that would work if my dreams came through.
But I will try to actually do the debugger thing shortly... :) After the implosion of my latest theory, I now have no theory about what's happening.

OK, I got the debugger hooked up. I've gotten as far as DefaultTypeHierarchy.visitTypeArgs. There, subtype (like subtypeAsSuper) is Function<? super A*, ? extends B*>*, and supertype is Function<? super A, ? extends B>, both as expected from my example. It doesn't look like we end up in the catch (Exception e) case there, so the question would be which of the two isContainedMany calls should be returning true but isn't. But that's it for today.

(DefaultTypeHierarchy.visitTypeArgs might turn out to be the level at which ? super passes one check and ? extends passes the other.)

DefaultTypeHierarchy.visitTypeArgs might turn out to be the level at which ? super passes one check and ? extends passes the other.

Probably:

? super: The first isContainedMany check succeeds.
? extends: The first isContainedMany check fails, and the second succeeds.

(noting that I'm testing with the changes discussed above)
Here's the code I'm referring to and the JDK 11 spec section it refers to. The specific spec items are:

C<S1,...,Sn>, where Si contains Ti (1 ≤ i ≤ n) (§4.5.1).

Given a generic type declaration C<F1,...,Fn> (n > 0), the direct supertypes of the parameterized type C<R1,...,Rn> where at least one of the Ri (1 ≤ i ≤ n) is a wildcard type argument, are the direct supertypes of the parameterized type C<X1,...,Xn> which is the result of applying capture conversion to C<R1,...,Rn> (§5.1.10).

The former shouldn't apply here at all, since at least one of the type arguments is a wildcard. So the mystery may be why capture-converting the type with ? super produces a type that doesn't act how we'd want in subtyping tests.
In hindsight, I guess I could have seen that coming, given that I previously avoided a problem with plain ? super (in the absence of ? extends in the same parameter) by removing... an applyCaptureConversion call :)

OK, I backed out the applyCaptureConversion removal so that I could test with a type that has only ? super in it.
As before, we end up in DefaultTypeHierarchy.visitDeclared_Declared. In my new test, that's with subtype Predicate<capture#154 of ? extends Object* super E>* and supertype Predicate<? super E>, where Predicate is a locally declared, unannotated type (not the JDK type). That leads us again to the second isContainedMany call in visitTypeArgs with essentially the same values:
isCapturedMany(
   /*subtypeTypeArg=*/ capture#154 of ? extends Object* super E,
   /*supertypeTypeArg=*/ ? super E,
   /*covariantArgIndexes=*/ null)

capture#154 of ? extends Object* super E ought to be contained by ? super E, so let's see what's going wrong....
Actually, regardless of what's going wrong there, I think the supertype should have ? super E* (i.e., unspecified nullness on the E bound). I can't see how fixing that would help our cause here, though, nor do I know if the lack of * is anything new from eisop/checker-framework#746 or other recent changes.
My test code is:
package com.google.foo;

import org.jspecify.annotations.NullMarked;
import org.jspecify.annotations.Nullable;

class Foo {
  interface Predicate<E> {}

  interface Super<E> {
    void g(Predicate<? super E> p);
  }

  @NullMarked
  interface Sub<E extends @Nullable Object> extends Super<E> {
    @Override
    void g(Predicate<? super E> p);
  }
}

Weird: The debuggers shows that we have a previous result for areEqualVisitHistory, which is false. So we... return false immediately?
But areEqualVisitHistory is for checking full-on equality, not containment, right? (It's shared with StructuralEqualityComparer, so probably?) I can understand returning immediately if the answer is true, but if it's false, then we should fall back to the rest of the containment checks, right? For that matter, how can we put into areEqualVisitHistory (which, again, seems to be used for equality checks) based on the results of a containment check? Some of the puts are for "a placeholder in case of recursion, to prevent infinite regress," but then we put a real result at the end. (The part about infinite regress also suggests that we can't "just disable" the history; that is, it's not just a cache.)
@wmdietl, is this something that you understand? I may be totally misunderstanding what the code is doing.

...but the check fails even if I tell it to proceed if the history had a false value, so I guess I can still try to figure out where it's failing.
(The bit with areEqualVisitHistory still seems weird.)

The failing check is:
isSubtype(outsideLower, inside)
i.e.,
isSubtype(E, capture#541 of ? extends Object* super E)
The Checker Framework DefaultTypeHierarchy.isSubtype (which probably delegates back the the JSpecify implementation for at least part of the check) says that it's a subtype, but then the JSpecify isNullnessSubtype check rejects it.
It looks as if isNullnessSubtype should be succeeding by comparing E to E from this call. But the subtype E is has upperBound Object?, and the supertype E has upperBound Object.
The weird upperBound comes from our friend applyCaptureConversion: checkParameters has:

the element of overriddenParams as Predicate<? super E*>*, where typeArgs[0] has a superBound E* whose upperBound is Object
the corresponding capturedParam as Predicate<capture#315 of ? extends Object* super E>*, where typeArgs[0] has a lowerBound E whose upperBound is Object

There's a call to leastUpperBound that passes null and E* (with upperBound Object*) but gets back E (with upperBound Object?).
We end up in lubWithNull, starting with the opening else case. otherAsLub (and thus the value of lub immediately thereafter) is computed fine. We don't enter the following if. Then we compute lowerBounds, which is @NullnessUnspecified as desired. Then isSubtypeShallow is true, so we replace lub with the annotation from null, which is minusNull. That's bad, so the question is what the isSubtypeShallow check is doing and what its result should be.
The call is:
qualHierarchy.isSubtypeShallow(upperBound, otherTM, nullAnno, nullTM)
The parameters are:

@NullnessUnspecified
E (as a TypeMirror)
@MinusNull
null (as a TypeMirror)

Ah, and that calls into our isSubtypeQualifiers, which implements lenient subtyping because we're in lenient mode. So we're, like, sure, @NullnessUnspecified is a subtype of @MinusNull, and thus @MinusNull looks like a least upper bound.
So we don't actually want lenient mode for least-upper-bound computations, or at least we don't want it for this part of the computation. But I'm not sure how easily we can disable that. I guess the thing to try would be to always delegate applyCaptureConversion to the strict-mode factory. That may well break other things, but... it's worth a try?

--- a/src/main/java/com/google/jspecify/nullness/NullSpecAnnotatedTypeFactory.java      2024-08-07 16:00:04.000000000 -0400
+++ b/src/main/java/com/google/jspecify/nullness/NullSpecAnnotatedTypeFactory.java      2024-08-07 17:31:35.000000000 -0400
@@ -532,6 +532,14 @@
   }

   @Override
+  public AnnotatedTypeMirror applyCaptureConversion(
+      AnnotatedTypeMirror type, TypeMirror typeMirror) {
+    return this == withLeastConvenientWorld
+        ? super.applyCaptureConversion(type, typeMirror)
+        : withLeastConvenientWorld.applyCaptureConversion(type, typeMirror);
+  }
+
+  @Override
   protected TypeHierarchy createTypeHierarchy() {
     return new NullSpecTypeHierarchy(
         checker,

That seems to help.
I then tried building the Google code that we use this checker on, but I discovered that I still need the wonkyness check, so I've restored that. (Having already restored the applyCaptureConversion call, I'm left with only some tweaks to unspecified nullness.) I'm kicking off another run, but I need to get going, so I won't have results to report just yet....

Yes, it works in Google, and it fixes 7 unexpected errors in NullSpecTest$Lenient:
SuperTypeVariableUnspec.java:33: error: (argument.type.incompatible)
SuperTypeVariableUnspec.java:65: error: (argument.type.incompatible)
SuperTypeVariableUnspec.java:97: error: (argument.type.incompatible)
SuperTypeVariableUnspec.java:123: error: (argument.type.incompatible)
SuperTypeVariableUnspec.java:126: error: (argument.type.incompatible)
SuperTypeVariableUnspec.java:129: error: (argument.type.incompatible)
SuperVsSuperNullable.java:39: error: (argument.type.incompatible)

I'll create a PR to merge it into main-eisop. I would create a PR for main instead, but there, it leads us to not issue errors in 8 cases in which we should (and currently do):
ToArray.java:25: jspecify_nullness_mismatch
ToArray.java:43: jspecify_nullness_mismatch
ToArray.java:57: jspecify_nullness_mismatch
ToArray.java:74: jspecify_nullness_mismatch
ToArray.java:79: jspecify_nullness_mismatch
ToArray.java:84: jspecify_nullness_mismatch
ToArray.java:89: jspecify_nullness_mismatch
ToArray.java:117: jspecify_nullness_mismatch

Presumably those mixed results are a consequence of how the fix interacts with the eisop changes that revealed the problem.

Ah, this must not have gotten automatically closed because it didn't get merged into main, only into main-eisop. But given that the problems were exposed only by changes in main-eisop, it's fair to consider it to be fixed already.

Note that the Multisets problem from my original post is not fixed. But that is a very rare problem (which I'll work around in https://github.com/google/guava/pull/7330/files) that may well be fixed as part of upstream's reworking of type inference. But the other problems are fixed, and as you can see from https://github.com/google/guava/pull/7330/files, the recent changes to eisop and/or the reference checker have uncovered some missing annotations in Guava.

Correction: Not all the problems I was seeing in Google's repo are fixed by #197. (I keep forgetting that I need to test the extra targets we have for special environments like J2CL....) And I assume that my promised fix to eisop defaulting isn't helpful for those, either.
But it looks like the rest will be fixed once I also land the changes to update our handling of substitution with unspecified types (as discussed in https://github.com/jspecify/jspecify/issues/248). That will require updates to the samples, and it will probably warrant new tests for the handling of @NonNull.
I don't think that even the combination of those things will address the remainder of #200, but one thing at a time :)

Or maybe I'm confused yet again? :)
After kevinb9n's post tonight, I am questioning whether https://github.com/jspecify/jspecify/issues/248 should actually be necessary to fix the problems I'm seeing in some of our implementations of Map.merge and computeIfPresent when building against an unannotated Map class. (I just edited more about this into an earlier mega-post.) It might still be, but I am at least now in quite a bit of doubt.
We'll see when I get back to all this....


Looking for an iOS developer to maintain the iOS version

Hello friends,
We currently don't have anyone maintaining the iOS version. We're looking for a volunteer to maintain the iOS version.

Did you get anyone @caiosba ??

Not yet, @Saad-Bashar :(

oh that's sad. I am actually using this package in one of my projects to share youtube video link from youtube app. It works okay in android. But could not make it work in iOS. I was having one issue with android. getSharedText is null. But saw today that someone fixed it so I also upgraded to latest version. Keep up the good work @caiosba  :)

Thanks @Saad-Bashar !

any update on any development on iOS? I want to use this package in order to share urls when clicking on the share menu. This currently works in android version. Please let me know if there is any alternative applications or even better technology which can be cross-platform to share menus.
But good work on the android, works like a charm.

No updates yet, @saleemadat :(
But thanks for your feedback on the Android version!

https://github.com/alinz/react-native-share-extension
How about contacting them?
Their module doesn’t seem to work successfully and though they have the android version it looks like they are only capable of maintaining the iOS version.

@caiosba @NishidaRyu416
https://github.com/ajith-ab/react-native-file-share-intent
it's Good for Both Platforms

Hey, I'm up for maintaining the iOS portion of this package. I've been looking to contribute to more OSS this year and this seems like an important project for a lot of people.

Hey, I'm up for maintaining the iOS portion of this package. I've been looking to contribute to more OSS this year and this seems like an important project for a lot of people.
I have extensive experience with RN native modules for both iOS and Android, you can check my profile

Happy to hear it, @Gustash !! I can add you as a maintainer when you start hacking on it! Happy to answer any questions you have or if you need anything to start.

@caiosba can you link me to the most pressing issues on iOS? It would probably be wise to start on those

@caiosba can you link me to the most pressing issues on iOS? It would probably be wise to start on those

Sure, I guess the most important one are #26 and #22 - basically bring it back to a working state and document in the README an example that works. I assigned you to both if you don't mind :)

All iOS issues are labelled BTW: https://github.com/meedan/react-native-share-menu/labels/ios

Thanks! I'll get to those first

Btw @caiosba should I request your review for PRs related to iOS? I assume so since there were no iOS maintainers

Btw @caiosba should I request your review for PRs related to iOS? I assume so since there were no iOS maintainers

Just for sanity check, basically :D

We found one! Thanks @Gustash !


Add gradient customization

Currently there was no way to apply gradient colors to the Theme either directly or through remote configuration. With this PR, gradient can be applied to view and button backgrounds. Border, and foreground remain with 1 color.
MOB-1588

Survey does not render correctly

Looks good! But there is one small comment from me

!merge


[8.x] Added support for GCM encryption

This pull request adds support for 128-bit and 256-bit GCM encryption, which will allow Laravel developers to use this stronger encryption mode in their applications.
Galois/Counter Mode (GCM) encryption is a more secure and modern encryption cipher than CBC (which is currently default).
The pull request does not change the default encryption mode and is fully backwards compatible.

Can you explain what a "tag" is in this context?

Sure thing, the tag is generated during the encryption itself and essentially acts as a checksum hash that is used to verify the integrity of the encryption string.
It prevents an attack where the attacker can learn something about the original data by making subtle changes to the encrypted value and then working backwards.
The tag is required for GCM mode as it ensures the integrity of each encrypted block (of 128 or 256 bits), but is not available for older encryption methods.

The $tag variable is passed via reference to the OpenSSL encrypt function, which then assigns it a value. It will add 16 characters to the final mac value that is stored in the db

The tag is another name for the Message Authentication Code (mac), and is used in GCM mode to provide authenticated encryption, meaning that ciphertext tampering can be detected and rejected before decrypting the data. This serves the exact same purpose as the already implemented HMAC in the previous CBC implementation, and using both of these (which this implementation does) is unnecessary and reduces performance.
So, when GCM mode is used, the HMAC part should be skipped. This seems to have been done correctly in the 2017 PR https://github.com/laravel/framework/pull/21963/files.
One of the reasons GCM is considered more secure is that it reduces the risk of implementation error, since the authentication part doesn't have to be implemented by the consumer. However Laravel implements Encrypt-then-mac correctly as far as I can see. https://en.wikipedia.org/wiki/Authenticated_encryption
I might try to make this into a PR in the coming weeks. There is no rush however as I don't see any downside to the current implementation except for performance degradation.
And a minor detail regarding the information above – the AES block size is 128 bits for both AES-128 and AES-256. It's the key length that differs.

There is also a bug in the generateKey method. If AES-128-GCM is used, the key:generate artisan command incorrectly generates a 256 bit key, causing a RuntimeException.
This is not critical, so I'll include that fix in the PR mentioned above, unless that takes longer then expected.

@Krisell thanks for spotting that. I sent in a PR for that: https://github.com/laravel/framework/pull/38474

I might try to make this into a PR in the coming weeks. There is no rush however as I don't see any downside to the current implementation except for performance degradation.

Would very much welcome a PR for this 👍

@Krisell just saw you sent your PR roughly at the same time. I'll close mine.


4.0.1 and 4.0.2 not building on SuSE Enterprise 11

I'm receiving the below build errors on SLES with Ruby 2.1.5 installed via rbenv. Previously we were using 3.5.1 which built successfully.
Building native extensions.  This could take a while...
ERROR:  Error installing byebug:
  ERROR: Failed to build gem native extension.

    /home/digopuser/.rbenv/versions/2.1.5/bin/ruby -r ./siteconf20150318-62725-1803d7a.rb extconf.rb
creating Makefile

make "DESTDIR=" clean

make "DESTDIR="
compiling locker.c
compiling context.c
cc1: warnings being treated as errors
context.c: In function ‘dt_inherited’:
context.c:629: error: unused parameter ‘klass’
make: *** [context.o] Error 1

make failed, exit code 2

Gem files will remain installed in /home/digopuser/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/byebug-4.0.1 for inspection.
Results logged to /home/digopuser/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0-static/byebug-4.0.1/gem_make.out

I'm seeing the same thing on OSX 10.6, running ruby 2.0.0p598

Is 4.0.3 working?

Yep! That works on a gem install. I've updated our Gemfile to include 4.0.3 and am testing our build now. Hopefully this doesn't break again. :four_leaf_clover:

Yep, we're all good. Thanks for the update!

Great!
Out of curiosity, are you installing byebug in your CI environment? It's generally better not to do that unless there's a reason.

No, it's used in our development. Thanks for the heads-up though!

:+1:


Incorrect converter used when multiple field maps exist for the same a and b type.

When setting up a class map I set up two field maps, both of which map from String to String. Each field map uses a converter that trims the string down to a specific length. The problem is that in this scenario the first converter registered to a field map is used by both field maps and the second converter is ignored. Any additional converters registered to other String to String field maps are ignored as well.
https://gist.github.com/brittinfontenot/e70fe3ad92edebb818500c7286226aef

I have the same issue with the version 1.5.0

With @5f6a736 the CustomConverter got a equals and hashcode method.
I think this should be removed from CustomConverter (Maybe the Builtin*Converters should implement them)
In the meanwhile you can try to implement your own equals & hashCode in your Converters.
I also saw that the BuiltinBidirectionalConverter implements the equals method without hashcode which should be fixed.

I would prefer to add apache commons lang as dependency, so the EqulasBuilder could be used in Orika:
https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/builder/EqualsBuilder.html
Maybe for 1.6 ?

You are right. I implemented now my own equals and hashcode method. Now it works as expected.
Thank you.

Thanks @brabenetz. Adding equals and hashcode to my converter works.

If you add a commons lang dependency, can it be optional?

If I add commons lang3, then it will not be optional.
But I'm curious, what could be the problem with apache commons lang3 library?

Some of my projects use only Guava (similar functionality) and exclude a lot of commons stuff. Any time a new dependency pops up it has to be looked at.

Any updates on whether removing equals and hashCode from CustomConverter is planned? I assume this was done in good faith to cache converters, but it's not documented and very confusing. I could raise a PR.


Add count task?

There's a bunch of ways to do this and it may be overblown:
rake cs:get:count[$path]

Return no. of records only i.e. 342.

Here's how I get the count currently: rake cs:get:path[collectionobjects] | jq '.["abstract_common_list"]["totalItems"]'


Unable to provision RKE2 clusters using Amazon cloud provider with k8s 1.22

Summary:

I'm unable to provision any ec2 RKE2 clusters with Amazon cloud provider option.

Environment:
Rancher version: v2.6-head b736007 (also seen on 9740d7a build from earlier today) 11/19/2021
Rancher cluster type: single-node docker install and HA helm install both attempted
Downstream cluster type: RKE2 w/ ec2 node driver and amazon cloud provider
Downstream K8s version: v1.21.6+rke2r1
Steps to Reproduce:

Provision an RKE2 cluster with ec2 node driver. Set up amazon cloud provider.
The cluster is stuck in provisioning state forever. It is stuck on step waiting on probes: calico
Try provisioning an ec2 rke2 cluster but without cloud provider. This works.

Expected Result:
I expected to be able provision an RKE2 cluster using the amazon cloud provider.
Actual Result:
I cannot provision an RKE2 cluster using the amazon cloud provider. This was working recently, just yesterday.
Additional Info:
I attempted to do my due diligence. I didn't spot anything in rancher server logs of use. When I downloaded the SSH key to remove in to the node and investigate the SSH key does not work (and I don't think I made any mistakes using the rancher provided key).

RKE2 PR: https://github.com/rancher/rke2/pull/2163

It was decided that the best way to fix this issue was through RKE2 (the PR is linked).
There is a work-around for this issue: add the environment variable CATTLE_MACHINE_PROVISION_IMAGE to rancher/machine:v0.15.0-rancher72. This will not set the hostname on the EC2 nodes and this will allow provisioning to continue. This can be used until the RKE2 fix is available.
NOTE: this is only for RKE2 clusters provisioned from Rancher with the AWS cloud provider enabled. All other provisioning is unaffected.

@thedadams , thank you. Depending on which RKE2 version the fix is going into and whether it's included in 2.6.3 release's KDM, we may want to add this workaround to release notes hence release-note label.

Gave this a quick check using KDM branch dev-v2.6 on rancher v2.6.3-rc3. I was able to provision a v1.21.7-rc2+rke2r1 cluster using aws cloud provider.
After ssh'ing to a node, I see the following config:
# $ sudo cat /etc/rancher/rke2/config.yaml.d/50-rancher.yaml
{
  "advertise-address": "<redacted>",
  "agent-token": "<redacted>",
  "cloud-provider-name": "aws",
  "cni": "calico",
  "disable-kube-proxy": false,
  "etcd-expose-metrics": false,
  "etcd-snapshot-retention": 5,
  "etcd-snapshot-schedule-cron": "0 */5 * * *",
  "node-ip": [
    "<redacted>"
  ],
  "node-label": [
    "rke.cattle.io/machine=7e6f0cba-7972-437f-b6d1-66a6fb1f94cb"
  ],
  "private-registry": "/etc/rancher/rke2/registries.yaml",
  "protect-kernel-defaults": false,
  "tls-san": [
    "<redacted>"
  ],
  "token": "<redacted>"
}

The cluster is healthy in the rancher UI and running successfully. I'll leave this open so that we can validate with v1.22 when that is functional as well.

(Waiting for https://github.com/rancher/rancher/issues/35683 so we can test 1.22.x)

Moving it out of 2.6.3 since 1.22 is experimental in this release. And 1.21RKE2 version works.

Putting this on pause for https://github.com/rancher/dashboard/issues/5023

My checks Failed
Reproduction Steps:

Not required.

Validation Environment:

Rancher version: v2.6-head da382a8 2/1/2022
Rancher cluster type: single-node docker install
Docker version: 20.10
Downstream cluster type: rke2
Downstream K8s version: v1.22.6+rke2r1 marked as (experimental) in KDM / the dashboard UI
Validation steps:

Create an RKE2 cluster using the EC2 node driver. Pick Amazon as the cloud provider and provide the necessary IAM role.
The cluster still is stuck in a Provisioning state with same issue - waiting for probes: calico

Additional Info:
As normal, I checked web requests/responses and these appear nominal as well. (fleet-default POST includes IAM role and provisioning.cattle.io.clusters contained cloud-provider-name in rkeConfig). I suspect this to be a backend issue still.

@davidnuzik I know what the problem is here. I will reopen the issue.

Thanks @thedadams I just updated my prior comment too, but sounds like you already know the issue. Let me know if there is anything I can do; appreciate it.

This issue is waiting for an RKE2 release. It should be noted that it was me who is doing the work (and did the work that didn't fix it the first time).

Given the timing on issues linked above the expectation is for these fixes to be available in Feb k8s patches releases of RKE2. No additional changes are believed to be needed on Rancher side.

Reproduction Setup

Rancher version: v2.6-head(3df76e4)
Downstream clusters: rke2, k8s version: v1.21.10+rke2r1 & v1.22.7+rke2r1

Results of testing
After following the reproduction steps in the original issue I found that the 1.22.7 k8s version did not successfully provision where as the 1.21.10 did successfully provision.
Moved to reopen after reproducing the issue again.


Update zh_TW translation

@vanessa-chang and I will do more check for the zh_TW :)

I just pulled from Transifex and merged. I guess we can close this one.


Error: Current process got forked after parallelism has been used.

Bug 💥
My program has several APIs (Flask) which use fast_tokenizer of transformers library alongside multi-threading (Joblib) to speed up execution. However, while calling deep_doctection (using torch), I get the error whose screenshot is pasted below. Deep Doctection is not using fast_tokenizer and so analyzer.analyze should work fine.
Important : This issue only occurs when deployed using Docker, local runs always run fine.
Screenshots 🖼

Desktop (please complete the following information, if any other than the one in the install requirements):

OS: Ubuntu 22.04.1 LTS
16 Gb RAM

Additional context 🧬
Using the following:

Python 3.9
Flask for API
Joblib for Multi-threading.
Transformers (4.24.0)
Tokenizers (0.13.2)
Docker (Latest)

Hi,
when you mention to analyzer.analyze do you refer to the built-in analyzer generated from the factory get_dd_analyzer ? Or did you build a custom pipeline ?
The reason I ask is that, while it is true that the analyzer from the factory does not use any components from the transformer/tokenizer library, there are some deepdoctection components who use them (check the notebook Using_LayoutLM_for_sequence_classification.ipynb) and they even require to use fast tokenizers.
I do not know at what stage the tokenizer library starts its multi-threading, but if it is when importing the library then this will cause problems because e.g.
import get_dd_analyzer

requires
import ...pipe

requires
import ..mapper.laylmstruct

requires
if transformers_available():
    from transformers import PreTrainedTokenizerFast

and because you mention, that you use fast tokenizers the condition  transformers_available() is satisfied. There are some other deepdoctection modules that require fast tokenizers as well. (On the other hand, just when writing this I doubt that this might be the reason as all modules are lazy modules and will only be executed when first used).
Did you try to follow the tokenizer suggestion in your suggestion to set the environment variable TOKENIZERS_PARALLELISM to False ?
Also remarkable your remark that everything works fine on local installation but not on Docker.

Thank you for your reply.

I'm using the built-in pipeline (get_dd_analyzer).
I would like to keep the environment variable TOKENIZERS_PARALLELISM to be True otherwise I won't be able to use 'use_fast' in other APIs.
Docker images did have an issue with previous versions of tokenizers and while fast tokenizers would work locally, they'd create an issue inside a docker container (process forks) but the latest build (for transformers and tokenizers) apparently fixed this issue.
However, a quick fix does work (The one you mentioned above) but I was hoping that I'd keep fast tokenizers (the inference time decreases ten fold).

Thank you for the info.
Do you have any GH issue/communication at hand for 3.) ?
I understand that  TOKENIZERS_PARALLELISM=False is not optimal. However, I do not know yet where I could step in from my side in order to get it fixed.
But I understand, once 3.) is fixed it will also resolve this issue, right?

That's the thing.
3). was fixed and runs as intended.
What I was hoping for is an option to not use fast_tokenizer within the analyzer instead of resorting to an environment variable.
Thank you for your time.


Add storybook instructions to default README.md

The README that comes with a generated project should have instructions on running stories:

Run yarn storybook
Change SHOW_STORYBOOK to true in src/app/main.tsx
Reload the app!

Bonus points for info about the VSCode Storybook plugin by "Orta"

Install plugin
Open command palette in VSCode (cmd + shift + P) and run "Reconnect Storybook to VS Code"
Choose story from Storybook pane in left menu (under Outline)

PR: https://github.com/infinitered/ignite-ir-boilerplate-bowser/pull/85


Incorrect positioning when inside a Web Component slot, with a container that has a backdrop-filter

Describe the bug
When a tooltip is rendered within the slot of a web component, within a container that has a backdrop-filter, the tooltip ends up being positioned incorrectly. I've observed quite a variance in just how far off the tooltip is, from just a few pixels to being completely off the page - depending on the rest of the context, but I've provided a simple reproduction below
To Reproduce
Here is a codesandbox that shows the issue (has a dependency on Lit): https://codesandbox.io/p/sandbox/floating-ui-dom-template-forked-xjn3yc?file=%2Fsrc%2Findex.js%3A8%2C16
Screenshots
Incorrectly positioned:

Correctly positioned:

Context:

OS: [e.g. iOS]: MacOS Sonoma 14.5
Browser [e.g. chrome, safari]: Chrome and Firefox. ~Safari does not seem to be affected~. Safari is affected as well, when using -webkit-backdrop-filter
Version [e.g. 22]: Chrome Version 126.0.6478.127 (Official Build) (arm64), Firefox 127.0.2 (64-bit)

I think this may be this Shadow DOM bug
It doesn't seem like the composed-offset-position (offsetParent) polyfill works in this case as it doesn't detect containing blocks correctly
cc: @jcfranco

Thanks, @atomiks - So if I understand correctly, this does not have a workaround currently, right? Since the composed-offset-position trick doesn't work in this case?

It doesn't seem like the composed-offset-position (offsetParent) polyfill works in this case as it doesn't detect containing blocks correctly

@atomiks I'll need to investigate further. A quick local test using isContainingBlock from @floating-ui/utils resulted in test failures.
@josepharhar could you provide any insights or pointers on this?

Not sure if it fixes the issue, but I think the polyfill needs to look at the computed value of backdropFilter, not just filter, since the browser implementation of offsetParent looks at both of those properties: https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/core/layout/layout_object.cc;l=1843-1850;drc=fa1e0cf5238e2600039f60af6909aefbd4cf003c

@atomiks Tests are passing now. I must have messed up the conditional expression previously, so thanks for the catch! I'll work on publishing an update this week.
@josepharhar Thanks for chiming in!

composed-offset-position has been updated and should mitigate this issue (see updated example).
Many thanks, y'all!

Thanks, @jcfranco!  I see it's working in the updated example, but unfortunately this does not fix the issue in my app.
After some digging, it looks like the issue still remains when the container web component has a closed shadow root.
Here's an updated example: https://codesandbox.io/p/sandbox/floating-ui-dom-composed-offset-position-iscontainingblock-repro-case-forked-pklz3x?file=%2Fsrc%2Findex.js%3A17%2C4

I believe it's impossible to traverse through closed shadow roots (feel free to correct) so there's no method that will work to find the correct offset parent.

Ah yes, thanks @atomiks! I'm starting to understand this all better thanks in part to https://developer.mozilla.org/en-US/docs/Web/CSS/Containing_block#identifying_the_containing_block
So if there's an ancestor element with something like backdrop-filter, and that element is inside of a separate web component from where the tooltip is, and that separate web component has a closed shadow DOM, we'll hit this issue.
I was able to get it working by manually passing the backdrop-filter element down into the tooltip element.  It's a bummer we have to do this for these cases, but I'm guessing there's no alternative?
Do you think this is worth a blurb in the documentation?


⚠ Moncai has degraded performance

In 8a44537, Moncai (https://moncai.com.au) experienced degraded performance:

HTTP code: 200
Response time: 5147 ms

Resolved: Moncai performance has improved in 3adde5a.


Formula test failing because build dependency cannot be found

Last week I was having issue where the brew audit step would fail when using core dependencies, with messages like: Error: No available formula with the name "openjdk".
This issue seems to be resolved (I think it was Homebrew/actions#333), and the general test workflow seems to again work on my tap. However, I just tried upgrading one of my formulae, and I'm running into an issue where after the building and installation the brew test step failed because with the message Error: No available formula with the name "cmake"..
I assume that this is still because it parses the formula and does not know about this dependency, since in the test step CMake is not used.
A full build log with the debug enabled can be found here: https://github.com/Dekker1/homebrew-minizinc/actions/runs/4277368900/jobs/7446201245

I've run into the same issue while following this blog post: https://brew.sh/2020/11/18/homebrew-tap-with-bottles-uploaded-to-github-releases/
I used the exact same formula and received the following error: No available formula with the name "go".
The workaround mentioned by @carlocab in https://github.com/Homebrew/brew/issues/15049#issuecomment-1482447051 seems to do the trick though.

I'm running into this as well in Guile Homebrew. See for example:
https://github.com/aconchillo/homebrew-guile/actions/runs/4854739279/jobs/8652466654?pr=178
The error is:
  Error: No available formula with the name "autoconf".

Adding what @carlocab suggested here https://github.com/Homebrew/brew/issues/15049#issuecomment-1482447051 solved the issue.
This also fails in my machine when simply doing brew test guile-cairo for example.

I get forward with the suggested workaround but get "Error: Formula loading disabled by HOMEBREW_DISABLE_LOAD_FORMULA!" from brew test-bot --only-setup even with "HOMEBREW_DISABLE_LOAD_FORMULA: false".

I get forward with the suggested workaround but get "Error: Formula loading disabled by HOMEBREW_DISABLE_LOAD_FORMULA!" from brew test-bot --only-setup even with "HOMEBREW_DISABLE_LOAD_FORMULA: false".

If you did:
export HOMEBREW_DISABLE_LOAD_FORMULA=false

you enabled HOMEBREW_DISABLE_LOAD_FORMULA. The value is irrelevant, only that there is a value set: https://docs.brew.sh/Manpage#environment
unset HOMEBREW_DISABLE_LOAD_FORMULA, then run brew test-bot again.

Also: do brew update

unset HOMEBREW_DISABLE_LOAD_FORMULA, then run brew test-bot again.

I'm running this from a github workflow, which was created by brew new-tap:
name: brew test-bot
on:
  push:
    branches:
      - main
  pull_request:
jobs:
  test-bot:
    strategy:
      matrix:
        os: [ubuntu-22.04, macos-13]
    runs-on: ${{ matrix.os }}
    env:
      HOMEBREW_NO_INSTALL_FROM_API: true # added as suggested in https://github.com/Homebrew/brew/issues/15049
    steps:
      - name: Set up Homebrew
        id: set-up-homebrew
        uses: Homebrew/actions/setup-homebrew@master

      - name: Cache Homebrew Bundler RubyGems
        id: cache
        uses: actions/cache@v3
        with:
          path: ${{ steps.set-up-homebrew.outputs.gems-path }}
          key: ${{ runner.os }}-rubygems-${{ steps.set-up-homebrew.outputs.gems-hash }}
          restore-keys: ${{ runner.os }}-rubygems-

      - name: Install Homebrew Bundler RubyGems
        if: steps.cache.outputs.cache-hit != 'true'
        run: brew install-bundler-gems

      - run: brew test-bot --only-cleanup-before

      - run: brew test-bot --only-setup

      - run: brew test-bot --only-tap-syntax

      - run: brew test-bot --only-formulae
        if: github.event_name == 'pull_request'

      - name: Upload bottles as artifact
        if: always() && github.event_name == 'pull_request'
        uses: actions/upload-artifact@main
        with:
          name: bottles
          path: '*.bottle.*'

I don't see what would have set and exported the HOMEBREW_DISABLE_LOAD_FORMULA in the first place. and how unset would have any effect. I can try with unset HOMEBREW_DISABLE_LOAD_FORMULA && brew test-bot --only-setup but seems to me that it will do nothing.

Also, I believe the HOMEBREW_DISABLE_LOAD_FORMULA is some kind of security measure and disabling/working around it in the workflow will open some kind of vulnerability to the repo.

Looks like --only-setup now passes as I removed the HOMEBREW_DISABLE_LOAD_FORMULA: false but the same problem now occurs in --only-formulae. Adding unset .... && brew ... there changes nothing.

Yes, that was fixed, now the tests go through, but when applying the pr-pull label the publish workflow fails:
==> Auditing Formula k0sctl on os linux and arch intel
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/brew.rb (Formulary::FromPathLoader): loading /home/linuxbrew/.linuxbrew/Homebrew/Library/Taps/k0sproject/homebrew-tap/Formula/k0sctl.rb
k0sproject/tap/k0sctl
  * Can't find dependency '"go"'.
Error: 1 problem in 1 formula detected.
Error: Can't find dependency '"go"'.
Error: Failure while executing; `/home/linuxbrew/.linuxbrew/bin/brew audit --skip-style --debug k0sproject/tap/k0sctl` exited with 1.
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/extend/kernel.rb:241:in `safe_system'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/dev-cmd/pr-upload.rb:149:in `pr_upload'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/brew.rb:94:in `<main>'
Error: Failure while executing; `/home/linuxbrew/.linuxbrew/bin/brew pr-upload --debug` exited with 1.
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/extend/kernel.rb:241:in `safe_system'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/dev-cmd/pr-pull.rb:501:in `block (3 levels) in pr_pull'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/fileutils.rb:128:in `chdir'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/fileutils.rb:128:in `cd'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/dev-cmd/pr-pull.rb:447:in `block (2 levels) in pr_pull'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/tmpdir.rb:93:in `mktmpdir'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/dev-cmd/pr-pull.rb:446:in `block in pr_pull'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/dev-cmd/pr-pull.rb:427:in `each'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/dev-cmd/pr-pull.rb:427:in `pr_pull'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/brew.rb:94:in `<main>'
Error: Process completed with exit code 1.

The workflow has the env: HOMEBREW_NO_INSTALL_FROM_API: 1 but it doesn't seem to help in my case.

https://github.com/k0sproject/homebrew-tap/actions/runs/5143681228/jobs/9258990913?pr=18
https://github.com/k0sproject/homebrew-tap/pull/18

@kke, you will need to publish your .github/workflows/publish.yml changes to the main branch (i.e. commit directly or via another PR) before adding the pr-pull tag.
The reason behind this: the workflow is triggered by the pull_request_target event. It runs in the context of the PR target branch (main in your case).

Fixed by https://github.com/Homebrew/brew/pull/15564


always get set data when harvesting records

Before, for an OAI harvest, set data was collected for each record ONLY in the following circumstances:

A setlist is given
A set blacklist is given
harvestAllSets is set to true

This PR gets set data for records by default.
This has been tested locally with a hub in which all records belong to at least one set (Illinois), and a hub in which sets are not implemented (Kentucky).  This has not been tested for a hub in which only some records belong to sets, because such a hub does not currently exist.
This addresses ticket DT-1472.

This is pretty small PR so I'm going to merge it w/o a second review. @mdellabitta okay?


SSH private key not copying correctly

Hello,
I'm running across this problem which I can't seem to figure out. Any pointers would be appreciated!
While trying to set up SSH, I'm getting the following error:
    amazon-ebs: TASK: [gocd | SSH public key] *************************************************
    amazon-ebs: changed: [<IP_ADDRESS>]
    amazon-ebs:
    amazon-ebs: TASK: [gocd | SSH private key] ************************************************
    amazon-ebs: fatal: [<IP_ADDRESS>] => input file not found at /tmp/packer-provisioner-ansible-local/roles/gocd/files/build_id_rsa, or /tmp/packer-provisioner-ansible-local/build_id_rsa,

What's odd is that 1: the private key is copied correctly, and 2: the public and private key are both present in the source directory:
ls -l /tmp/packer-provisioner-ansible-local/roles/gocd/files/
total 16
-rw-r--r-- 1 ubuntu ubuntu 1675 Feb  6 01:58 build_id_rsa
-rw-r--r-- 1 ubuntu ubuntu  398 Feb  6 01:58 build_id_rsa.pub

For reference, here is the configuration I'm using:
  vars:
    GOCD_SCM_GIT: true,
    GOCD_CONFIGURE_SSH: true,
    GOCD_SSH_PRIVATE_KEY: build_id_rsa,
    GOCD_SSH_PUBLIC_KEY: build_id_rsa.pub

I'm confused why it works for the public key but not the private key. Any thoughts?
Thanks!
David

Hi @gibbster - did you get this sorted out?  I'm not able to duplicate it.
Would it be possible to run Ansible in verbose and share the log snippet around the copying of the key?
BTW: chmod 600 that private key! ;)


[TASK] Update Relayer List (add radio button)

Description
The relayer list should be updated with radio buttons so that this allows users select and deselect a relayer. We should also enable fees % display if possible.
Figma Link
Screenshot

Closed in #1526.


Update DG with use of callbacks & cloudEvents

What type of PR is this?

enhancement/feature

What this PR does / why we need it:
In this PR several notifications update are provided:

enforce use of callbacks in CAMARA API to describe notification event structure
use of CloudEvents format for CAMARA API

Additionally this PR should be accompanied with PR #43.
This PR triggered cancellation of PR #41
Which issue(s) this PR fixes:
Fixes #8 #44
Note: Issue 44 covers several topics. One topic is about general ruling to have separate (or not) swagger for resource-based subscription from the nominal swagger for the API. This point is not tackled in this PR and must be followed in a new issue.
Special notes for reviewers:
Changelog input
- enforce use of `callbacks` in CAMARA API to describe notification event structure
- use of CloudEvents format for CAMARA API

Additional documentation
This section can be blank.
docs

Please provide any additional reviews or comments by 15th September

Despite my updates in suggestions above I wonder if current model of type and subjectis redundant.
Would it be sufficient to have for example:
"type": "org.camara.api.device-status"
"subject": "EventRoamingStatus"

CloudEvents considers versioning in type then we can have:
"type": "org.camara.api.device-status.v1"
"subject": "EventRoamingStatus"

It may be useful, but can complicate the processing and is rather limited to major version number.

@jlurien @rartych Thanks a lot for the help & review.
I hope to have accommodate the guideline following your recommendations.
May I ask to take another look to check if we're aligned or at least check we do not have any blocker on your side.

@bigludo7: Added some comments late yesterday evening.

I am not completely certain about  using CAPITAL_AND_UNDERSCORE format for event type in type attribute.
It should be prefixed with a reverse-DNS name which is typically lower-case.
CloudEvent spec gives the following examples (only lower-case):

com.github.pull_request.opened
com.example.object.deleted.v2

For uniformity I propose to use lower-case (and for  better human readability kebab-case as we already have in API name part:
org.camaraproject.<api-name>.<event-name>
or
org.camaraproject.<type>.<api-name>.<version>.<event-name> if we apply versioning
For developers it may be important to use the most popular formatting as indicated here: https://github.com/camaraproject/Commonalities/pull/43#discussion_r1319631841

I am not completely certain about using CAPITAL_AND_UNDERSCORE format for event type in type attribute. It should be prefixed with a reverse-DNS name which is typically lower-case.
CloudEvent spec gives the following examples (only lower-case):

com.github.pull_request.opened
com.example.object.deleted.v2

For uniformity I propose to use lower-case (and for better human readability kebab-case as we already have in API name part: org.camaraproject.<api-name>.<event-name> or org.camaraproject.<type>.<api-name>.<version>.<event-name> if we apply versioning
For developers it may be important to use the most popular formatting as indicated here: #43 (comment)

@rartych I'm fine with that - fair arguments for the example & uniformity
@jlurien as team Telefonica was pushing for CAPITAL_AND_UNDERSCORE could you be agreed on this?

@murthygorty Thanks for the positive feedback
implicit/explicit: I like your proposal which is less confusing than instance-based or resource-based. If nobody object I can make the change (I tag the 'usual suspects' on the topic to get their view @jlurien @rartych @shilpa-padgaonkar @PedroDiez @patrice-conil ).
I propose also to add you comment on notificationAuthToken in the guideline
For the third point this is more a comment on globally the use of CloudEvents and I do not see a requirement to update the guideline - I'm right?
Again thanks for the review & the proposal.

For the third point this is more a comment on globally the use of CloudEvents and I do not see a requirement to update the guideline - I'm right?

@bigludo7 - you are right in that guideline doesn't need to be updated. I'm taking the opportunity to request @shilpa-padgaonkar @hdamker and the likes to get CNCF improved perhaps via TechSteering so that OUR developers have an overall best experience, for us to succeed holistically :-)

https://github.com/camaraproject/Commonalities/pull/56#issuecomment-1723420492v

Hi @sachinvodafone, so sorry; couldn't fully understand if you are supportive of common approach to both subscription types or not. I really like your differentiation between subscriber and caller, if I understand you correctly. Are you suggesting that the use of the term 'implicit' brings confusion because that implies subscriber/consent involvement?
Personally, I'm not particular about the names, I'm however keen that we view both subscription-types the same in terms of the operations possible. And perhaps, since we are treating them the same largely, we don't even have to name them differently, thus simplifying the concept even further --> in some cases, subscription is 'automatic' as part of an operation; in some cases subscription is the operation itself, that's it. What do you say, @bigludo7 @shilpa-padgaonkar @hdamker

From my perspective all significant comments impacting the introduction of event subscription & notification following CloudEvents format have been taken into account & resolved.
I will provided this feedback during today TSC (@hdamker).
@rartych @shilpa-padgaonkar @jlurien @sachinvodafone @murthygorty please do not hesitate to jump in if there is still blocking point to implement this pattern in our CAMARA API (we can still in future PR improve the wording of the guideline).

@bigludo7  I agree that most important points are covered.
However I would like to point out 2 last points :) :

Even though resource based subscription will be dealt with later I found something in the documentation, For eg. we mention 4 operations under resource based subscriptions and they have paths as "/event-subscriptions" whereas the cloudevent subscription yaml file simply has "/subscriptions". IMHO unless we really need to differ, we can keep the things same?

Version:  How do we expect in resource based subscription the user to know the change in event schema if we dont include version in "type"  for eg. org.camaraproject.device-status..roaming-off?

Thanks @dfischer-tech  @shilpa-padgaonkar  and @akoshunyadi for your review !
@dfischer-tech As I worked on the notification part I have only fixed this part. Probably good to open an issue withe the identified adjustements.

Hi @sachinvodafone, so sorry; couldn't fully understand if you are supportive of common approach to both subscription types or not. I really like your differentiation between subscriber and caller, if I understand you correctly. Are you suggesting that the use of the term 'implicit' brings confusion because that implies subscriber/consent involvement?

HI @murthygorty , I was highlighting that the term "instance-based subscription," as described in our documentation, actually the definition of an "Implicit subscription" rather than an "instance-based subscription."

@bigludo7: Thanks for the comment - issue is on the way.
https://github.com/camaraproject/Commonalities/issues/65

One of the goals of the PR is "enforce use of callbacks in CAMARA API to describe notification event structure". Im not sure if that been covered. If it has, could you kindly point to the line numbers in the commit. Thank you in advance.

[like]  Gorty, Murthy reacted to your message:

From: Shilpa Padgaonkar @.>
Sent: Thursday, September 28, 2023 5:32:09 PM
To: camaraproject/Commonalities @.>
Cc: Gorty, Murthy @.>; Mention @.>
Subject: Re: [camaraproject/Commonalities] Update DG with use of callbacks & cloudEvents (PR #56)
[External]
@shilpa-padgaonkar commented on this pull request.

In documentation/API-design-guidelines.mdhttps://checkpoint.url-protection.com/v1/url?o=https%3A//github.com/camaraproject/Commonalities/pull/56%23discussion_r1340474003&g=ZDc0ZmQyMDgwNjJkZWNjYg==&h=YWEwZGU4MTRlODlmMzk3OWY4OGFjNWRmMzI2ODQ2MDNjOWJlMzdlMGVhMmQ5NzkyNDZiNzgwNzcyZDQ5ZDk4MQ==&p=YzJ1OnRtb2JpbGV1c2E6YzpvOjA5NGU1M2RiYjAxNmE5MjNlMWY2MmIxNTU0OTdhMjEyOnYxOmg6VA==:

-Instance-based subscription
+#### Instance-based (implicit) subscription
@sachinvodafonehttps://checkpoint.url-protection.com/v1/url?o=https%3A//github.com/sachinvodafone&g=MjVhOTI5ZTQwOGZhZjBhMA==&h=YWRiOWNiOWNmYTZlZmY0MDkyNWZlZGViZjk5MjhjZjk3ODQ4OTM2NmExZDhhNTE0MzhjM2FlNzQ2NzUzZjI2MQ==&p=YzJ1OnRtb2JpbGV1c2E6YzpvOjA5NGU1M2RiYjAxNmE5MjNlMWY2MmIxNTU0OTdhMjEyOnYxOmg6VA== : If adding the above said description is not a blocking issue for you, could you please approve this PR as you were active in this review and kindly open a new issue for improving this documentation? Would that work for you?
—
Reply to this email directly, view it on GitHubhttps://checkpoint.url-protection.com/v1/url?o=https%3A//github.com/camaraproject/Commonalities/pull/56%23discussion_r1340474003&g=NDg5ODQyN2I0NjRkZDI3MQ==&h=OTY3OTYzYWVlNGFmMWMzZGRlZTAzZjg4ZDhiNDRjYWIxZDNkNjk0MmI1NGUwNTVlMTM5NGM0YmE4YzFlMTRjMQ==&p=YzJ1OnRtb2JpbGV1c2E6YzpvOjA5NGU1M2RiYjAxNmE5MjNlMWY2MmIxNTU0OTdhMjEyOnYxOmg6VA==, or unsubscribehttps://checkpoint.url-protection.com/v1/url?o=https%3A//github.com/notifications/unsubscribe-auth/AEVULZ6ZQDFZV4TX5BWMBLTX4WYBTANCNFSM6AAAAAA4HIPWCY&g=MTZlZTAyNWZhYWExN2NiZQ==&h=ZjA4M2U2OTAxN2Y4NDRkZGFjODA0YzNiNDJmYmMxOTE4YWU5ODhlNGM2MjFhZGYxNDJhN2YyNDdmN2JmODQ2Nw==&p=YzJ1OnRtb2JpbGV1c2E6YzpvOjA5NGU1M2RiYjAxNmE5MjNlMWY2MmIxNTU0OTdhMjEyOnYxOmg6VA==.
You are receiving this because you were mentioned.Message ID: @.***>

@jlurien : Any blocking issues from your side? IF not, could you kindly approve, as you were an active reviewer?
Once your approval is done, I believe we can merge this PR.

As most active reviewers have approved the PR and confirmed that there are no more blocking issues, will merge this PR. Thank you all for the great collaboration on this PR and a big thanks to @bigludo7 for driving this topic .


[Bug]: CDN FrontdoorOrigin object can't use privatelink-service for loadbalancer with privateLinkTargetIdRef object

Is there an existing issue for this?

[X] I have searched the existing issues

Affected Resource(s)
cdn.azure.upbound.io/FrontdoorOrigin/v1beta1
Resource MRs required to reproduce the bug
---
apiVersion: cdn.azure.upbound.io/v1beta1
kind: FrontdoorEndpoint
metadata:
  name: apim
spec:
  forProvider:
    cdnFrontdoorProfileId: REPLACE-WITH-ID
    enabled: true
---

apiVersion: cdn.azure.upbound.io/v1beta1
kind: FrontdoorEndpoint
metadata:
  name: lb
spec:
  forProvider:
    cdnFrontdoorProfileId: REPLACE-WITH-ID
    enabled: true
---

apiVersion: cdn.azure.upbound.io/v1beta1
kind: FrontdoorOrigin
metadata:
  name: apim
spec:
  forProvider:
    cdnFrontdoorOriginGroupIdRef:
      name: apim
    certificateNameCheckEnabled: false
    enabled: true
    hostName: "APIM.HOSTNAME.COM"
    httpPort: 80
    httpsPort: 443
    originHostHeader: "APIM.HOSTNAME.COM"
    priority: 1
    weight: 50
    healthProbesEnabled: true
---

apiVersion: cdn.azure.upbound.io/v1beta1
kind: FrontdoorOrigin
metadata:
  name: lb
spec:
  forProvider:
    cdnFrontdoorOriginGroupIdRef:
      name: lb
    certificateNameCheckEnabled: false
    enabled: true
    hostName: "<IP_ADDRESS>"
    httpPort: 80
    httpsPort: 443
    originHostHeader: "<IP_ADDRESS>"
    priority: 1
    weight: 50
    healthProbesEnabled: true
    privateLink:
      - location:  "westeurope"
        privateLinkTargetIdRef:
          name: "privatelink"
        requestMessage: "Request from frontdoor origin - lb"
---

apiVersion: cdn.azure.upbound.io/v1beta1
kind: FrontdoorOriginGroup
metadata:
  name: apim
spec:
  forProvider:
    cdnFrontdoorProfileId: REPLACE-WITHID
    loadBalancing:
      - additionalLatencyInMilliseconds: 0
        sampleSize: 16
        successfulSamplesRequired:  3
    restoreTrafficTimeToHealedOrNewEndpointInMinutes:  5
    sessionAffinityEnabled:  true
---

apiVersion: cdn.azure.upbound.io/v1beta1
kind: FrontdoorOriginGroup
metadata:
  name: lb
spec:
  forProvider:
    cdnFrontdoorProfileId: REPLACE-WITH-ID
    loadBalancing:
      - additionalLatencyInMilliseconds: 0
        sampleSize: 16
        successfulSamplesRequired:  3
    restoreTrafficTimeToHealedOrNewEndpointInMinutes:  5
    sessionAffinityEnabled:  true
---

apiVersion: cdn.azure.upbound.io/v1beta1
kind: FrontdoorRoute
metadata:
  name: apim
spec:
  forProvider:
    cdnFrontdoorEndpointIdRef:
      name: apim
    cdnFrontdoorOriginGroupIdRef:
      name: apim
    cdnFrontdoorOriginIdsRefs:
      - name: apim
    patternsToMatch:
        - /api
    forwardingProtocol: HttpsOnly
    httpsRedirectEnabled: true
    linkToDefaultDomain: false
    supportedProtocols:
      - Http
      - Https
---

apiVersion: cdn.azure.upbound.io/v1beta1
kind: FrontdoorRoute
metadata:
  name: lb
spec:
  forProvider:
    cdnFrontdoorEndpointIdRef:
      name: lb
    cdnFrontdoorOriginGroupIdRef:
      name: lb
    cdnFrontdoorOriginIdsRefs:
      - name: lb
    patternsToMatch:
        - /app
    forwardingProtocol: HttpsOnly
    httpsRedirectEnabled: true
    linkToDefaultDomain: false
    supportedProtocols:
      - Http
      - Https

Steps to Reproduce
1: create internal loadbalancer
2: Create private link service for loadbalancer
3: Create frontdoor origin with Privatelink service created above with privateLinkTargetIdRef.name object
What happened?
Creating a frontdoor origin for internal loadbalancer with privatelinkservice fails when privatelink is referred with privateLinkTargetIdRef name object.
Origin creation works fine if the privatelink service id is passed but fails if name reference is used. It seems to support only privatelink service for storage account.
Relevant Error Output Snippet
cannot resolve references: mg.Spec.ForProvider.PrivateLink[i3].PrivateLinkTargetID: cannot get referenced resource: Account.storage.azure.upbound.io "privatelink" not found

### Crossplane Version

v1.16.0

### Provider Version

provider-azure-cdn:v1.1.0

### Kubernetes Version

_No response_

### Kubernetes Distribution

AKS

### Additional Info

_No response_

The private_link_target_id can have multiple types, e.g. Storage account id, private link service id or app services id, depending on what you want to link with. However resource references are limited to only one type https://github.com/crossplane/upjet/issues/414


Can't create index

Describe the bug
Can't create index
To Reproduce

cargo run in meilisearch root folder

curl
    -X POST 'http://localhost:7700/indexes'
    -H 'Content-Type: application/json'
    --data-binary '{
        "uid": "movies",
        "primaryKey": "id"
    }'

Response:
{
	"taskUid": 0,
	"indexUid": "movies",
	"status": "enqueued",
	"type": "indexCreation",
	"enqueuedAt": "2022-11-01T08:44:00.269189619Z"
}

curl \
  -X GET 'http://localhost:7700/tasks/0'

Response:
{
	"uid": 0,
	"indexUid": "movies",
	"status": "failed",
	"type": "indexCreation",
	"details": {
		"primaryKey": "id"
	},
	"error": {
		"message": "Invalid argument (os error 22)",
		"code": "internal",
		"type": "internal",
		"link": "https://docs.meilisearch.com/errors#internal"
	},
	"duration": "PT0.000368400S",
	"enqueuedAt": "2022-11-01T08:44:00.269189619Z",
	"startedAt": "2022-11-01T08:44:00.274817501Z",
	"finishedAt": "2022-11-01T08:44:00.275185901Z"
}

Expected behavior
I expected the "status" field to be "succeeded" rather than "failed" in the response from curl -X GET 'http://localhost:7700/tasks/0'
Screenshots
Attempting to post index:

Attempting to get indexes:

Checking task status:

Meilisearch version: [e.g. v0.20.0]
0.28.1
Additional context
Additional information that may be relevant to the issue.
[e.g. architecture, device, OS, browser]
OS: Archlinux
Architecture: x86_64

It feels like this error message "Invalid argument (os error 22)" is key to solving this issue. So I searched for that error message in this repository and it didn't show any results. I wonder where in the code this error message originates from :thinking:

When I write ulimit -S, I get the output unlimited, so setting it to 10000 seems like a reduction and thus doesn't seem to serve the purpose of increasing the ulimit.
When I write ulimit S (or ulimit -Sn, which gives same output), I get the output 1048576, which is still far greater than 10000. So if 1048576 is too low, I'm not sure why I should reduce it to 10 000?
the 38 operations were just multiple attempts to do the same operation as described in the post. I read in the docs that it displays ma 20 items at a time, so I was curious on what would happen if you exceed 20, thus the large number of identical tasks.
I initially tried to follow the instructions in here to add documents. I did exactly like it said and then in attempt to narrow down the scope of the problem, I looked in the APi documentation and used the info therein to send POST request directly to index, and thus it seems like the problem starts with the index rather than starting with the document.

Meilisearch version: [e.g. v0.20.0] 0.30.o rc1
Additional context Additional information that may be relevant to the issue. [e.g. architecture, device, OS, browser] OS: Windows10 Architecture: x86_64

Are you trying to say that you came across a similar problem, or is this a hint on how to solve it? or is it something else?

Hello everyone, thanks for the report, the core team will work on it asap
For core team: we noticed it's not related to the new index scheduler!

The problem is flaky on macOS and ubuntu but does appear from time to time.
And I can reproduce it 100% of the time on archlinux.
It seems to be linked to a bump of milli from the v0.12.1 to the v0.12.2, but it needs more investigation.

Are there any junior-level contributions I can make to help solve the issue? Or is this something advanced that only people with lots of experience can solve? I've been in Rust for a few months

The problem is flaky on macOS and ubuntu but does appear from time to time. And I can reproduce it 100% of the time on archlinux. It seems to be linked to a bump of milli from the v0.12.1 to the v0.12.2, but it needs more investigation.

What exactly do we need to do to solve the problem? Can you break it down into sub-tasks or specific questions that need to be addressed?

What exactly is a "bump" of milli though? I'm not familiar with the "bump" concept.

It means upgrading milli version in this repo to include the bug fix

Is there an article or something where I can read more about it? I'm trying to understand like the definition of "bump" and how to use the term more generally

A "dependency bump" means "upgrade a dependency", is a common term used, see for example this random tutorial I found when typing "bump dependency" on Google
But no worry with this @amab8901, @irevoire was writing this for traceability and for helping his own team internally to investigate the bug.

I reproduced the issue on an up-to-date KDE Archlinux and think I have a fix:

it looks like it is caused by the Rust lmdb-sys package using the system's version of lmdb if it is installed
On KDE archlinux lmdb (v. 0.9.29-1 for an up-to-date system) is very likely to be installed due to baloo which depends on it and is a KDE program
The system version of lmdb is likely to be incompatible with the version used by meilisearch
Modifying lmdb-sys to always use the vendored version appears to fix the issue. Adding a vendored feature like is done in https://lib.rs/crates/libgit2-sys is a way of adding this behavior back-compatibly
The line of the issue: https://github.com/meilisearch/lmdb-rs/blob/main/lmdb-sys/build.rs#L62
I will open a PR shortly to add such a vendored feature to lmdb-rs and make sure we make use of it in meilisearch

PR is here

Closed by https://github.com/meilisearch/meilisearch/pull/3084

hey why is this closed when there are 2 tasks left?

Tasks have been done. The issue has now been updated, thanks


feature: Add decodeTypeApplication

I was pointed to the ABICoder.decode(), which is exactly what I wanted.
https://forum.fuel.network/t/abi-coders-request-for-the-decodetypeapplication-method/5298


sourceIP missing with ALB?

We've got a fairly generic ALB -> Lambda config and are seeing the following issue.  sourceIp is not defined on event.requestContext.identity (or rather identity is not defined).
{
    "errorType": "Runtime.UnhandledPromiseRejection",
    "errorMessage": "TypeError: Cannot read property 'sourceIp' of undefined",
    "reason": {
        "errorType": "TypeError",
        "errorMessage": "Cannot read property 'sourceIp' of undefined",
        "stack": [
            "TypeError: Cannot read property 'sourceIp' of undefined",
            "    at getRequestValuesFromEvent (/var/task/api/_dist/node_modules/@vendia/serverless-express/src/event-sources/utils.js:52:50)",

Usage is as follows (app is in a Promise so we need to await it and then proxy it?):
module.exports.handler = async (event, context, callback): Promise<any> => {
    console.debug('index.handler: running');
    console.debug(` event: ${util.inspect(event)}`);
    console.debug(` context: ${util.inspect(context)}`);

    // Start a server (if one doesn't exist), then memoize it.
    if (!_server) {
        console.debug('index.handler: creating AWSServerlessExpress');
        const app = await __setup();
        _server = configure({ app, binaryMimeTypes });
        dog_log('api', 'boot', 'complete');
    }

    return _server.proxy({ event, context, callback });
};

Hi Alison, are you running on V3 or V4 (just published an RC for that to the @vendia/serverless-express npm package). If it's not working on V4 I'll take a look (or you can send a PR if you discover a problem). We don't plan on publishing anything but bug fixes to V3  (which doesn't natively support ALB though people have hacked around it)

Hi Alison, are you running on V3 or V4 (just published an RC for that to the @vendia/serverless-express npm package). If it's not working on V4 I'll take a look (or you can send a PR if you discover a problem). We don't plan on publishing anything but bug fixes to V3  (which doesn't natively support ALB though people have hacked around it)

4.0.0-rc2
Alson Kemp
[ m: +1.650.260.8867 ]  [ e<EMAIL_ADDRESS>]
[ Skype: alsonkemp ]  [ gTalk, Jabber<EMAIL_ADDRESS>]
On Fri, Jan 8, 2021 at 1:02 PM Brett Andrews<EMAIL_ADDRESS>wrote:

Hi Alison, are you running on V3 or V4 (just published an RC for that to the @vendia/serverless-express npm package). If it's not working on V4 I'll take a look (or you can send a PR if you discover a problem). We don't plan on publishing anything but bug fixes to V3 (which doesn't natively support ALB though people have hacked around it)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

4.0.0-rc2
Alson Kemp
[ m: +1.650.260.8867 ]  [ e<EMAIL_ADDRESS>]
[ Skype: alsonkemp ]  [ gTalk, Jabber<EMAIL_ADDRESS>]
On Fri, Jan 8, 2021 at 1:02 PM Brett Andrews<EMAIL_ADDRESS>wrote:

Hi Alison, are you running on V3 or V4 (just published an RC for that to the @vendia/serverless-express npm package). If it's not working on V4 I'll take a look (or you can send a PR if you discover a problem). We don't plan on publishing anything but bug fixes to V3 (which doesn't natively support ALB though people have hacked around it)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Cool. I'll check this out as soon as I can and get back to you

Cool. I'll check this out as soon as I can and get back to you

Also (as you probably know), annoyingly the health checker doesn't send in much info to the Lambda so it seems SE fails 100% of health checks.

Also (as you probably know), annoyingly the health checker doesn't send in much info to the Lambda so it seems SE fails 100% of health checks.

Install RC 4 or later. Please let me know any other issues you encounter with ALB as I haven't done thorough testing of this event source.
I'm also curious why you decided to go with ALB over the other event sources

Install RC 4 or later. Please let me know any other issues you encounter with ALB as I haven't done thorough testing of this event source.
I'm also curious why you decided to go with ALB over the other event sources

I'll check RC4 tomorrow.
Other event source?  e.g. APIG?  APIG is seriously complex for our use case and had some weird behavior with deployments via Terraform.  ALB seemed simper (though it's proving not to be so) since we do all routing and documentation via OpenAPI....

I'll check RC4 tomorrow.
Other event source?  e.g. APIG?  APIG is seriously complex for our use case and had some weird behavior with deployments via Terraform.  ALB seemed simper (though it's proving not to be so) since we do all routing and documentation via OpenAPI....

APIGW HTTP API is a simpler alternative to APIGW REST API. V4 has experimental support for that also, as well as Lambda@Edge if you need  Edge computing (and your app can fit within the limits of Edge)

APIGW HTTP API is a simpler alternative to APIGW REST API. V4 has experimental support for that also, as well as Lambda@Edge if you need  Edge computing (and your app can fit within the limits of Edge)

Indeed V2 looks much simpler.  Still, it didn't work (some routes, it did; some routes, it didn't).  Nearly impossible to debug.  I think I'm going to write a quick, custom ALB-Lambda integration...

Indeed V2 looks much simpler.  Still, it didn't work (some routes, it did; some routes, it didn't).  Nearly impossible to debug.  I think I'm going to write a quick, custom ALB-Lambda integration...

What are you having trouble with for debugging? You should be able to run with an offline tool such as Serverless or SAM to work through any initial issues.

What are you having trouble with for debugging? You should be able to run with an offline tool such as Serverless or SAM to work through any initial issues.

I'm can't figure out why I'm getting a LambdaUnhandled error.  More details are at https://serverfault.com/questions/1049284/aws-alb-lambda-502.  Thank you for the quick responses.

I'm can't figure out why I'm getting a LambdaUnhandled error.  More details are at https://serverfault.com/questions/1049284/aws-alb-lambda-502.  Thank you for the quick responses.

Were you able to figure this out?

Were you able to figure this out?

@brettstack Thanks for checking in.  Yes, I just wrote a straightforward adapter from ALB<->Lambda.   ;)  I'll look into open sourcing it and will drop a link to the code in here.    Yes, I just wrote a straightforward adapter from ALB<->Lambda.   ;)  I'll look into open sourcing it and will drop a link to the code in here.    Through an unimaginable amount of painful debugging, I found this out...  See:
// THE ALB SEEMS TO REJECT REQUESTS WITH CERTAIN HEADERS SO ONLY SEND WHITE LISTED HEADERS
const VALID_HEADERS = [ ... ];

@brettstack Gist here: https://gist.github.com/alsonkemp/cd133900bbc664095ee852cd20a5e9f5
AWS ALB has an undocumented bug (or at least I couldn't find docs) where it rejects certain header values in responses.  Through an unimaginable amount of painful debugging, I found this out...  See:
// THE ALB SEEMS TO REJECT REQUESTS WITH CERTAIN HEADERS SO ONLY SEND WHITE LISTED HEADERS
const VALID_HEADERS = [ ... ];

@brettstack To be honest, serverless-express was a bit too abstract/overfactored for my tastes.  While I appreciated abstraction, this was a little too much for me and I felt like I was chasing execution path back/forth through files:
const getResponseToAlb = ({
  statusCode,
  body,
  headers,
  isBase64Encoded
}) => getResponseToService({
  statusCode,
  body,
  headers,
  isBase64Encoded
})

@brettstack To be honest, serverless-express was a bit too abstract/overfactored for my tastes.  While I appreciated abstraction, this was a little too much for me and I felt like I was chasing execution path back/forth through files:
const getResponseToAlb = ({
  statusCode,
  body,
  headers,
  isBase64Encoded
}) => getResponseToService({
  statusCode,
  body,
  headers,
  isBase64Encoded
})

Thanks! I should be able to take that gist and apply some of that header whitelisting logic.

Thanks! I should be able to take that gist and apply some of that header whitelisting logic.

@alsonkemp what headers were causing you issue? I'd rather go with a deny list over an allow list.

@alsonkemp what headers were causing you issue? I'd rather go with a deny list over an allow list.

Deny: agreed... If I knew what to deny.  I didn't feel like binary
searching with 5 minute latencies...
I think it was a partial download header which was sent by express-static.
'accept-ranges: bytes' but I couldn't find any documentation and AWS
Support was unenthusiastic.
I went with a white list not only because I didn't want to fiddle with
identifying the bad header but also because I didn't want to accidentally
discover new ones and I didn't mind controlling which headers are sent
(it's a pretty standard, static list).
I may have mentioned this already but besides copying your code, I did next
to nothing to verify query string functionality.
On Thu, Jan 28, 2021, 3:58 AM Brett Andrews<EMAIL_ADDRESS>wrote:

@alsonkemp https://github.com/alsonkemp what headers were causing you
issue? I'd rather go with a deny list over an allow list.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/vendia/serverless-express/issues/331#issuecomment-769005099,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAIPBO4XB6EY6ZLYVG3GT3S4FGMTANCNFSM4V247DQA
.

Deny: agreed... If I knew what to deny.  I didn't feel like binary
searching with 5 minute latencies...
I think it was a partial download header which was sent by express-static.
'accept-ranges: bytes' but I couldn't find any documentation and AWS
Support was unenthusiastic.
I went with a white list not only because I didn't want to fiddle with
identifying the bad header but also because I didn't want to accidentally
discover new ones and I didn't mind controlling which headers are sent
(it's a pretty standard, static list).
I may have mentioned this already but besides copying your code, I did next
to nothing to verify query string functionality.
On Thu, Jan 28, 2021, 3:58 AM Brett Andrews<EMAIL_ADDRESS>wrote:

@alsonkemp https://github.com/alsonkemp what headers were causing you
issue? I'd rather go with a deny list over an allow list.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/vendia/serverless-express/issues/331#issuecomment-769005099,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAIPBO4XB6EY6ZLYVG3GT3S4FGMTANCNFSM4V247DQA
.


Project admin cannot access preferences

Describe the bug
As a project administrator, I cannot access the preferences screen.
To Reproduce
Steps to reproduce the behavior:

When I go to "Project" tab.
Then I click on "Preferences"
And I am an administrator.
Then I get redirected to my personal Dashboard page.

Screen before clicking on preferences (note that I am an administrator):

Redirect after click:

Expected behavior
Should be able to get to the preferences page.
Environment (please identify where you experience this bug:
*** One of ***

Production
(NOTE: not experienced in Sandbox)
Chrome

@tmcelrath the problem persists if you open the browser in incognito mode?
To enter incognito mode in Chrome these are the shortcuts:
Windows, Linux, or Chrome OS: Press Ctrl + Shift + n.
Mac: Press ⌘ + Shift + n.

And then enter again into sfg.taxonworks.org

Yes - persists in incognito mode.

2.6.5 :001 > User.find_by(name: 'Tommy McElrath').is_superuser?
 => false
2.6.5 :002 > User.find_by(name: 'Tommy McElrath').administers_projects?
 => true

https://github.com/SpeciesFileGroup/taxonworks/blob/a4c558ca25128a24552e373f82ba5c19aea49b3f/app/controllers/tasks/projects/preferences_controller.rb#L4
@mjy is it intentional to now allow super users to alter project preferences? (Haven't checked if the controller is able to tell whether the user is actually a member with admin rights on the project, with those includes alone could be that it is enough to be admin member of at least one project and also be superuser)

Administrators do anything.  Project administrators can modify project preferences, yes.

There is supposed to be one category of user that is either admin or project admin.  That's the group that should be used in this case.  I thought it was super_users, but looks like that is not the case.

Looked at the wrong is_superuser? I think (User vs sessions). When looking sessions code I don't see why access is not granted. Will try on a local copy and debug there. Dump transfer will be finished in around 5 minutes.

This would be the fix
hernan@hernan-pc:~/Projects/taxonworks$ git diff
diff --git a/app/controllers/tasks/projects/preferences_controller.rb b/app/controllers/tasks/projects/preferences_controller.rb
index 88f465e01..b44be8173 100644
--- a/app/controllers/tasks/projects/preferences_controller.rb
+++ b/app/controllers/tasks/projects/preferences_controller.rb
@@ -1,6 +1,5 @@
 class Tasks::Projects::PreferencesController < ApplicationController
   include TaskControllerConfiguration
-  before_action :require_administrator_sign_in
   before_action :require_superuser_sign_in
   before_action :can_administer_projects?

Sounds right?


feat: virtually nilpotent groups

Show that nilpotency is preserved under group isomorphisms and define virtually nilpotent subgroups.
From GrowthInGroups

Thanks :tada:
bors merge


Customize Design

How to customize design ? there is an example ?

or you can use https://github.com/triniwiz/nativescript-popup

missing support customize iOS


Multiple  Retrieval + Generation phases in a single line

Is there an existing issue for the same feature request?

[X] I have checked the existing issues.

Is your feature request related to a problem?
No, but I would really like to chain multiple queries I think this would enhance the overall utility of the Graph function.

Describe the feature you'd like
Based on the context provided by the [[RagFlow Fix]], here's an explanation of "Multiple Retrieval + Generation phases in a single line":
Multiple Retrieval + Generation phases in a single line refers to a process where multiple cycles of retrieving information (retrieval phases) and generating new content or responses (generation phases) are executed sequentially within a single, continuous workflow. This means that the system does not stop after one cycle of retrieval and generation but rather continues through several iterations of this cycle, creating a more refined and comprehensive output.
For example, in a note-taking or knowledge management system like Smart Connect — Notes, this could involve retrieving existing notes, generating summaries or insights based on those notes, retrieving additional related notes or information, and further refining the output, all within a single, uninterrupted sequence.
Steps:

Initial Retrieval: Gather relevant existing notes or data.
Initial Generation: Create a first draft or initial response based on the retrieved data.
Secondary Retrieval: Identify gaps or additional information needed, and retrieve more notes or data.
Secondary Generation: Refine the initial response with the newly retrieved information.
Repeat as needed: Continue this cycle until the desired level of detail and completeness is achieved.

This approach enhances the quality and depth of the final output by leveraging iterative improvements through repeated retrieval and generation cycles.
Describe implementation you've considered
I really have not considered the implementation, other than the way state is maintained might have difficulty with nested recursion.
Documentation, adoption, use case
No response
Additional information
No response

Could you please paste the graph, and elaberate on a real scenario to help us understand that deeply?
Realy important, thanks!


Update namespace-specific policy example to include externally rendered files

Updates the namespace specific policy example to match the updated sample repository given in Configure Kubernetes with Kustomize. This adds files that are loaded outside of the Kustomize root directory.

snippet-bot:force-run


Fixes 1868: Admin Task API

Summary
Adds backend API for admin task interface that allows a user to view all tasks across all organizations/accounts. It is implemented as a feature that can be enabled/disabled and restricted to specific accounts in the config.
The API allows filtering by org_id, account_id, and statuses.
The api struct has a copy of a zest struct in order to properly convert the pulp data to JSON.
The endpoint is not included in the OpenAPI docs.
Testing steps
Add
features:
  admin_tasks:
    enabled: true
    users: ["adminUser"]

to the config.yaml
To see whether the feature is enabled/accessible, you can make a request to the features API:
curl localhost:8000/api/content-sources/v1.0/features/  -H "`./scripts/header.sh 1 adminUser`"

To list tasks:
curl "http://localhost:8000/api/content-sources/v1/admin/tasks/?offset=0&limit=20"  -H "`./scripts/header.sh 1 adminUser`"

To fetch a task:
curl "http://localhost:8000/api/content-sources/v1/admin/tasks/{uuid}"  -H "`./scripts/header.sh 1 adminUser`"

https://issues.redhat.com/browse/HMS-1868

This needs a rebase, but everything looks good and works well!

/retest

Hi @dpang314
Please rebase, because:
I see you have commit for "Fixes 1624" from Date:   Tue Jun 13 11:12:54 , but there are commit from after that date: https://github.com/content-services/content-sources-backend/pull/309/commits
and we have problems with pulp pods not starting when testing this PR.
Thank you

[retest]

/restest

/retest

Hi
testing with:
      parameters:
        FEATURES_ADMIN_TASKS_ENABLED: "True"
FEATURES_SNAPSHOTS_ENABLED: "True"
In [1]: app.content_sources.rest_client.features_api.list_features()
<snip>
Out[1]:
{'admintasks': {'accessible': True, 'enabled': True},
 'snapshots': {'accessible': True, 'enabled': False}}

content-sources-backend]$ curl  -u "$BASIC_AUTH" "https://${URL}/api/content-sources/v1.0/admin/tasks/?limit=6&offset=0" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1916  100  1916    0     0   2162      0 --:--:-- --:--:-- --:--:--  2162
{
  "data": [
    {
      "uuid": "d5c7adeb-66b4-4f72-a25d-d9045767bb80",
      "status": "completed",
      "typename": "introspect",
      "queued_at": "2023-07-14T19:03:39Z",
      "started_at": "2023-07-14T19:03:39Z",
      "finished_at": "2023-07-14T19:03:39Z",
      "error": "",
      "org_id": "3340851",
      "account_id": "0369233",
      "payload": null,
      "pulp": {}
    },
    {
      "uuid": "1ac37a45-51ed-480c-bcaf-fdfc9dffd6f2",
      "status": "completed",
      "typename": "snapshot",
      "queued_at": "2023-07-14T19:03:39Z",
      "started_at": "2023-07-14T19:03:39Z",
      "finished_at": "2023-07-14T19:04:08Z",
      "error": "",
      "org_id": "3340851",
      "account_id": "0369233",
      "payload": null,
      "pulp": {}
    },

content-sources-backend]$ curl  -u "$BASIC_AUTH" "https://${URL}/api/content-sources/v1.0/admin/tasks/{"d5c7adeb-66b4-4f72-a25d-d9045767bb80"}" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   352  100   352    0     0    599      0 --:--:-- --:--:-- --:--:--   600
{
  "uuid": "d5c7adeb-66b4-4f72-a25d-d9045767bb80",
  "status": "completed",
  "typename": "introspect",
  "queued_at": "2023-07-14T19:03:39Z",
  "started_at": "2023-07-14T19:03:39Z",
  "finished_at": "2023-07-14T19:03:39Z",
  "error": "",
  "org_id": "3340851",
  "account_id": "0369233",
  "payload": {
    "Url": "https://stephenw.fedorapeople.org/multirepos/5/repo02/",
    "Force": true
  },
  "pulp": {}
}
 content-sources-backend]$ curl  -u "$BASIC_AUTH" "https://${URL}/api/content-sources/v1.0/admin/tasks/{"1ac37a45-51ed-480c-bcaf-fdfc9dffd6f2"}" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2639    0  2639    0     0    486      0 --:--:--  0:00:05 --:--:--   534
{
  "uuid": "1ac37a45-51ed-480c-bcaf-fdfc9dffd6f2",
  "status": "completed",
  "typename": "snapshot",
  "queued_at": "2023-07-14T19:03:39Z",
  "started_at": "2023-07-14T19:03:39Z",
  "finished_at": "2023-07-14T19:04:08Z",
  "error": "",
  "org_id": "3340851",
  "account_id": "0369233",
  "payload": {
    "SyncTaskHref": "/pulp/api/v3/tasks/018955c9-b6e6-7a7b-8aa1-5a0a39bde143/",
    "SnapshotIdent": null,
    "PublicationTaskHref": "/pulp/api/v3/tasks/018955c9-cf6e-760f-8e42-02dbc6c9df51/",
    "DistributionTaskHref": null
  },
  "pulp": {
    "sync": {
      "pulp_href": "/pulp/api/v3/tasks/018955c9-b6e6-7a7b-8aa1-5a0a39bde143/",
      "pulp_created": "2023-07-14T19:03:50.323528Z",
      "state": "completed",
      "name": "pulp_rpm.app.tasks.synchronizing.synchronize",
      "logging_cid": "ea9504d2e73a4996be344abbc14dd1de",
      "started_at": "2023-07-14T19:03:50.446732Z",
      "finished_at": "2023-07-14T19:03:51.756474Z",
      "worker": "/pulp/api/v3/workers/01895528-d7c0-71d2-9eae-261375b1bf3c/",
      "progress_reports": [
        {
          "message": "Downloading Metadata Files",
          "code": "sync.downloading.metadata",
          "state": "completed",
          "done": 4,
          "suffix": null
        },
        {
          "message": "Downloading Artifacts",
          "code": "sync.downloading.artifacts",
          "state": "completed",
          "done": 0,
          "suffix": null
        },
        {
          "message": "Associating Content",
          "code": "associating.content",
          "state": "completed",
          "done": 1,
          "suffix": null
        },
        {
          "message": "Skipping Packages",
          "code": "sync.skipped.packages",
          "state": "completed",
          "total": 0,
          "done": 0,
          "suffix": null
        },
        {
          "message": "Parsed Packages",
          "code": "sync.parsing.packages",
          "state": "completed",
          "total": 1,
          "done": 1,
          "suffix": null
        },
        {
          "message": "Un-Associating Content",
          "code": "unassociating.content",
          "state": "completed",
          "done": 0,
          "suffix": null
        }
      ],
      "created_resources": [
        "/pulp/api/v3/repositories/rpm/rpm/018955c9-af6f-7dc5-ab6b-d78f2d1b4cda/versions/1/"
      ],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/rpm/rpm/018955c9-af6f-7dc5-ab6b-d78f2d1b4cda/",
        "shared:/pulp/api/v3/remotes/rpm/rpm/018955c9-9b81-7470-bddd-611d275499db/"
      ]
    },
    "publication": {
      "pulp_href": "/pulp/api/v3/tasks/018955c9-cf6e-760f-8e42-02dbc6c9df51/",
      "pulp_created": "2023-07-14T19:03:56.527134Z",
      "state": "completed",
      "name": "pulp_rpm.app.tasks.publishing.publish",
      "logging_cid": "c3247470e6a74e57a884d234c040da28",
      "started_at": "2023-07-14T19:03:56.653339Z",
      "finished_at": "2023-07-14T19:03:58.551824Z",
      "worker": "/pulp/api/v3/workers/01895528-d7c0-71d2-9eae-261375b1bf3c/",
      "progress_reports": [
        {
          "message": "Generating repository metadata",
          "code": "publish.generating_metadata",
          "state": "completed",
          "total": 1,
          "done": 1,
          "suffix": null
        }
      ],
      "created_resources": [
        "/pulp/api/v3/publications/rpm/rpm/018955c9-cff8-7d0f-beeb-f9f4826b49d8/"
      ],
      "reserved_resources_record": [
        "shared:/pulp/api/v3/repositories/rpm/rpm/018955c9-af6f-7dc5-ab6b-d78f2d1b4cda/"
      ]
    }
  }
}

The Custom Repositories page:
**

**

The Task Details (when you click View Details)

LGTM
thank you


Update analyze-cost-management.md - Added missing "

Update analyze-cost-management.md - Added missing "

@JnHs Can you review this PR? When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

@lukemurraynz - thanks!


Write descriptive comments for deletions and the helper function reorder

We need descriptive comments for deletions and reordering, which will properly explain the exact operation.

#3 needs to be fixed first as findable/meldable orders are the basis of reorder


Add Dark Mode button

I want to contribute on this issue. Please assign me this issue under GSSOC'23.