chore: IsIntegral dot notations, noncommutative generalizations, and golfs

Zulip
Initially I just wanted to add more dot notations for IsIntegral and IsAlgebraic; then I noticed near-duplicates
Algebra.isIntegral_of_finite [Field R] [Ring A] and
RingHom.IsIntegral.of_finite [CommRing R] [CommRing A]
so I went on to generalize the latter to cover the former, and generalized everything in the IntegralClosure file to the noncommutative case whenever possible.
In the process I noticed more golfs, which result in this PR. Most notably, isIntegral_of_mem_of_FG is now proven using Cayley-Hamilton and doesn't depend on the Noetherian case isIntegral_of_noetherian; the latter is now proven using the former. In total the golfs reduce mathlib by 226 lines (+680 -454).
The main changes are in the single file RingTheory/IntegralClosure:

Change the definition of Algebra.IsIntegral which makes it unfold to IsIntegral rather than RingHom.IsIntegralElem because the former has much more APIs.

Fix lemma names involving is_integral which are actually about IsIntegralElem.

Move isIntegral_of_(submodule_)noetherian down and golf them.

Remove (Algebra.)isIntegral_of_finite that work only over fields, in favor of the more general (Algebra.)isIntegral.of_finite.

Merge duplicate lemmas isIntegral_of_isScalarTower and isIntegral_tower_top_of_isIntegral into IsIntegral.tower_top.

Golf IsIntegral.of_mem_of_fg by first proving IsIntegral.of_finite using Cayley-Hamilton.

Add a docstring mentioning the Kurosh problem at Algebra.IsIntegral.finite. The negative solution to the problem means the theorem doesn't generalize to noncommutative algebras.

Golf IsIntegral.tmul and isField_of_isIntegral_of_isField(').

Combine isIntegral_trans_aux into isIntegral_trans and golf.

De-capitalize FG in lemma names.

Introduce dot notations for (Algebra.)IsIntegral / RingHom.IsIntegralElem: .map, .map_of_comp_eq, .tower_top, .of_subring, .algebraMap, .fg_adjoin_singleton, .add, .neg, .sub, .mul, .smul, .of_pow, .of_mul_unit, .trans, .tower_bot, .tower_bot_of_field, .of_comp, .quotient

The next changed file is RingTheory/Algebraic:

isAlgebraic_algebraMap/algHom_of_isAlgebraic become IsAlgebraic.algebraMap/algHom

(Algebra.)isAlgebraic_trans/_of_pow/finite/larger_base(_of_injective) become (Algebra.)IsAlgebraic.trans/of_pow/finite/larger_base(_of_injective)
(Note: it's a bit strange that the dot notation for Algebra.IsAlgebraic.of_finite doesn't work.)

The generalization from CommRing to Ring requires an additional lemma scaleRoots_eval₂_mul_of_commute in Polynomial/ScaleRoots.
A lemma Algebra.lmul_injective is added to Algebra/Bilinear (in order to golf the proof of IsIntegral.of_mem_of_fg).
In all other files, I merely fix the changed names, or use newly available dot notations.

cc:
original author of isIntegral_of_mem_of_FG @kckennylau
@erdOne who introduced the generalized Cayley-Hamilton LinearMap.exists_monic_and_aeval_eq_zero allowing to golf the above 97-line proof down to 1+2 lines (I guess it wasn't golfed due to Module.End being noncommutative?)
original author of isField_of_isIntegral_of_isField @dtumad: I'm indeed able to use the Polynomial.reverse and Polynomial.DivX API
original author of isIntegral_trans_aux @jcommelin (the aux lemma doesn't allow noncommutative B, so it's now removed, and the main proof now adjoins elements in A rather than in B)
@eric-wieser @haruhisa-enomoto who may be interested in the noncommutative generalization

Is it possible to split the dot notation changes to their own PR? It would be much better if the mathematical and stylistic content were in separate reviews.

Is it possible to split the dot notation changes to their own PR? It would be much better if the mathematical and stylistic content were in separate reviews.

As I said on Zulip:

The PR golfs some long proofs which collectively trims down mathlib4 by 226 lines, but only adds two new lemmas, removes an auxiliary lemma, and deduplicates some lemmas, so it's hopefully not hard to review.
The PR can't be easily split, as the renamed/reordered lemmas are also used in the golfs.

There is not much "mathematical content" because all I did to the lemma statements is here (replacing [CommRing] by [Ring]). Lemmas may be moved (reordered), golfed, and deduplicated, but the statements are only changed (generalized) in this way, and these are all concentrated in the single file (plus one lemma in Polynomial/ScaleRoots and Algebra/Bilinear each).
On the other hand, the stylistic (dot notation) changes require fixing lots of files, but those fixes are trivial to review. If I first introduce dot notation in a separate PR, I'll need to fix long proofs (e.g. isIntegral_of_mem_of_FG) that will simply be completely removed in the subsequent golfing PR. If I first do the golfs in a separate PR, I'll need to use the old lemma names and switch to new lemma names in the subsequent dot notations PR. So both options don't seem very economic, though I would choose the second option if you insist on a split.

Thanks a lot for this PR. It's a nice clean-up.
I do wonder whether we can make more use of new Lean 4 features now. Since this PR is renaming a lot of things, it might be good to keep the old names around and use the @[deprecated] feature to direct users to the new name.
@semorrison Is there tooling to easily get a list of all decls that are being removed/added in this PR?

@semorrison Is there tooling to easily get a list of all decls that are being removed/added in this PR?

No, but @j-loreaux explained how to implemented this. I'll review! :-)

I think some fuzzy matches may be needed for this PR, as some lemmas changed both name and statement (CommRing -> Ring).
Applying retroactively to #8358 may be easier, as statements are not changed there.

I think some fuzzy matches may be needed for this PR, as some lemmas changed both name and statement (CommRing -> Ring). Mathlib changelog probably can't trace through such changes.

This sounds like an argument for not mixing big renames with other refactors...

@semorrison I'll work on it.

I think Algebra.IsIntegral and Algebra.IsAlgebraic should be classes instead. Should we change it (in the future)? Will such a refactor reverse some changes made in this PR?

Also, does it conflict with #7857 significantly?

I think some fuzzy matches may be needed for this PR, as some lemmas changed both name and statement (CommRing -> Ring). Mathlib changelog probably can't trace through such changes.

On a second thought, in fact the #align commands already provide a way to track declarations renamed in mathlib4; people tend to modify the #aligns (possibly adding ₓ) but rarely remove them. Ideally the mathlib changelog could track all mathlib4 declaration names that have ever been aligned to a mathlib3 declaration and display them in the same page (is it the case now?). However, declarations that don't exist in mathlib3 don't have associated #align commands. Maybe we could introduce a #rename command and ask PR authors to add such a command whenever they rename a declaration (and the changelog could track renaming history transitively, which will be distinct from #aligning to a fixed base mathlib3 declaration), or maybe such commands can be auto-generated, and all the author has to do is to use a UI to pick which new declarations correspond to which old declarations. There may be auto-generated declarations that disappear or get added, but as a first approximation we may ignore them, as dealing with them may require building the whole mathlib rather than building a few modules and parsing some text.
Once we align renamed declarations, we can focus on comparing the declarations. GitHub offers a UI for git diff and we might want a decl diff UI as well. It should check the statement elaborates to the same thing, verify the arguments are the same (either strictly or up to reordering), highlight any the changed parts of the statement (e.g. generalized typeclasses) and definition body with red and green colors, and optionally allow to compare golfed proofs with the original. I imagine this will help reviews significantly, especially for auditing potential AI-assisted big refactors in the future (I wrote down some ideas on Zulip).

@jcommelin The way I generated this isn't ready for prime time, but here is a list of declarations which were added/removed (+/-) in this PR:
+Algebra.IsAlgebraic.of_finite
+Algebra.IsAlgebraic.of_larger_base
+Algebra.IsAlgebraic.of_larger_base_of_injective
+Algebra.IsAlgebraic.trans
+Algebra.IsIntegral.quotient
+Algebra.IsIntegral.trans
-Algebra.isAlgebraic_of_finite
-Algebra.isAlgebraic_of_larger_base
-Algebra.isAlgebraic_of_larger_base_of_injective
-Algebra.isAlgebraic_trans
-Algebra.isIntegral_of_finite
+Algebra.isIntegral_of_surjective
+Algebra.isIntegral_sup
-Algebra.isIntegral_trans
+Algebra.lmul_injective
-FG_adjoin_of_finite
-FG_adjoin_singleton_of_integral
+IsAlgebraic.algHom
+IsAlgebraic.algebraMap
+IsAlgebraic.of_finite
+IsAlgebraic.of_larger_base
+IsAlgebraic.of_larger_base_of_injective
+IsAlgebraic.of_pow
+IsIntegral.add
+IsIntegral.fg_adjoin_singleton
+IsIntegral.map
+IsIntegral.map_of_comp_eq
+IsIntegral.mul
+IsIntegral.neg
+IsIntegral.of_finite
+IsIntegral.of_mem_closure
+IsIntegral.of_mem_closure'
+IsIntegral.of_mem_closure''
+IsIntegral.of_mem_of_fg
+IsIntegral.of_mul_unit
+IsIntegral.of_pow
+IsIntegral.of_subring
+IsIntegral.smul
+IsIntegral.sub
+IsIntegral.tower_bot
+IsIntegral.tower_bot_of_field
+IsIntegral.tower_top
+Polynomial.scaleRoots_eval₂_mul_of_commute
+RingHom.IsIntegral.quotient
+RingHom.IsIntegral.tower_bot
+RingHom.IsIntegral.tower_top
+RingHom.IsIntegral.trans
+RingHom.IsIntegralElem.add
+RingHom.IsIntegralElem.mul
+RingHom.IsIntegralElem.neg
+RingHom.IsIntegralElem.of_mem_closure
+RingHom.IsIntegralElem.of_mul_unit
+RingHom.IsIntegralElem.sub
+RingHom.isIntegralElem.of_comp
-RingHom.isIntegralElem_of_isIntegralElem_comp
-RingHom.isIntegral_of_isIntegral_mul_unit
+RingHom.isIntegralElem_map
+RingHom.isIntegralElem_one
+RingHom.isIntegralElem_zero
-RingHom.isIntegral_quotient_of_isIntegral
-RingHom.isIntegral_tower_bot_of_isIntegral
-RingHom.isIntegral_tower_top_of_isIntegral
-RingHom.isIntegral_trans
-RingHom.is_integral_add
-RingHom.is_integral_map
-RingHom.is_integral_mul
-RingHom.is_integral_neg
-RingHom.is_integral_of_mem_closure
-RingHom.is_integral_one
-RingHom.is_integral_sub
-RingHom.is_integral_zero
+fg_adjoin_of_finite
-isAlgebraic_algHom_of_isAlgebraic
-isAlgebraic_algebraMap_of_isAlgebraic
-isAlgebraic_of_finite
-isAlgebraic_of_larger_base
-isAlgebraic_of_larger_base_of_injective
-isAlgebraic_of_pow
-isIntegral_add
-isIntegral_map_of_comp_eq_of_isIntegral
-isIntegral_mul
-isIntegral_neg
-isIntegral_ofSubring
-isIntegral_of_finite
-isIntegral_of_isIntegral_mul_unit
-isIntegral_of_isScalarTower
-isIntegral_of_mem_closure
-isIntegral_of_mem_closure'
-isIntegral_of_mem_closure''
-isIntegral_of_mem_of_FG
-isIntegral_of_pow
-isIntegral_of_surjective
-isIntegral_quotient_of_isIntegral
-isIntegral_smul
-isIntegral_sub
-isIntegral_sup
-isIntegral_tower_bot_of_isIntegral
-isIntegral_tower_bot_of_isIntegral_field
-isIntegral_tower_top_of_isIntegral
-isIntegral_trans_aux
-map_isIntegral
-mem_integralClosure_iff_mem_FG
+mem_integralClosure_iff_mem_fg

@j-loreaux Thanks! Here's a paired up version by reordering your list:
The one removed lemma:
-isIntegral_trans_aux

Two new lemmas:
+Polynomial.scaleRoots_eval₂_mul_of_commute
+Algebra.lmul_injective

-Algebra.isAlgebraic_of_finite
+Algebra.IsAlgebraic.of_finite

-Algebra.isAlgebraic_of_larger_base
+Algebra.IsAlgebraic.of_larger_base

-Algebra.isAlgebraic_of_larger_base_of_injective
+Algebra.IsAlgebraic.of_larger_base_of_injective

-Algebra.isAlgebraic_trans
+Algebra.IsAlgebraic.trans

-isIntegral_quotient_of_isIntegral
+Algebra.IsIntegral.quotient

-Algebra.isIntegral_trans
+Algebra.IsIntegral.trans

-Algebra.isIntegral_of_finite
+Algebra.isIntegral_of_surjective

-isIntegral_sup
+Algebra.isIntegral_sup

-FG_adjoin_of_finite
+fg_adjoin_of_finite

-FG_adjoin_singleton_of_integral
+IsIntegral.fg_adjoin_singleton

-isAlgebraic_algHom_of_isAlgebraic
+IsAlgebraic.algHom

-isAlgebraic_algebraMap_of_isAlgebraic
+IsAlgebraic.algebraMap

-isAlgebraic_of_finite
+IsAlgebraic.of_finite

-isAlgebraic_of_larger_base
+IsAlgebraic.of_larger_base

-isAlgebraic_of_larger_base_of_injective
+IsAlgebraic.of_larger_base_of_injective

-isAlgebraic_of_pow
+IsAlgebraic.of_pow

-isIntegral_add
+IsIntegral.add

-isIntegral_map_of_comp_eq_of_isIntegral
+IsIntegral.map_of_comp_eq

-isIntegral_mul
+IsIntegral.mul

-isIntegral_neg
+IsIntegral.neg

-isIntegral_ofSubring
+IsIntegral.of_subring

-isIntegral_of_finite
+IsIntegral.of_finite

-isIntegral_of_isIntegral_mul_unit
+IsIntegral.of_mul_unit

-isIntegral_of_mem_closure
+IsIntegral.of_mem_closure

-isIntegral_of_mem_closure'
+IsIntegral.of_mem_closure'

-isIntegral_of_mem_closure''
+IsIntegral.of_mem_closure''

-isIntegral_of_mem_of_FG
+IsIntegral.of_mem_of_fg

-isIntegral_of_pow
+IsIntegral.of_pow

-isIntegral_of_surjective

-isIntegral_smul
+IsIntegral.smul

-isIntegral_sub
+IsIntegral.sub

-isIntegral_tower_bot_of_isIntegral
+IsIntegral.tower_bot

-isIntegral_tower_bot_of_isIntegral_field
+IsIntegral.tower_bot_of_field

-isIntegral_of_isScalarTower
-isIntegral_tower_top_of_isIntegral
+IsIntegral.tower_top

-map_isIntegral
+IsIntegral.map

-mem_integralClosure_iff_mem_FG
+mem_integralClosure_iff_mem_fg

-RingHom.isIntegral_quotient_of_isIntegral
+RingHom.IsIntegral.quotient

-RingHom.isIntegral_tower_bot_of_isIntegral
+RingHom.IsIntegral.tower_bot

-RingHom.isIntegral_tower_top_of_isIntegral
+RingHom.IsIntegral.tower_top

-RingHom.isIntegral_trans
+RingHom.IsIntegral.trans

-RingHom.is_integral_add
+RingHom.IsIntegralElem.add

-RingHom.is_integral_map
+RingHom.isIntegralElem_map

-RingHom.is_integral_mul
+RingHom.IsIntegralElem.mul

-RingHom.is_integral_neg
+RingHom.IsIntegralElem.neg

-RingHom.is_integral_of_mem_closure
+RingHom.IsIntegralElem.of_mem_closure

-RingHom.is_integral_one
+RingHom.isIntegralElem_one

-RingHom.is_integral_sub
+RingHom.IsIntegralElem.sub

-RingHom.is_integral_zero
+RingHom.isIntegralElem_zero

-RingHom.isIntegralElem_of_isIntegralElem_comp
+RingHom.isIntegralElem.of_comp

-RingHom.isIntegral_of_isIntegral_mul_unit
+RingHom.IsIntegralElem.of_mul_unit

If I knew about @[deprecated] (and that it's a good idea to use it here) I would have added them as I rename the lemmas. Since I didn't do that, should there be a utility that takes the list above and automatically add the @[deprecated ...] attributes? (Since most declarations are not in a namespace section, this should be easy for this PR.) Could one declaration have multiple deprecated attributes (I'd like that for IsIntegral.tower_top)?

I do wonder whether we can make more use of new Lean 4 features now. Since this PR is renaming a lot of things, it might be good to keep the old names around and use the @[deprecated] feature to direct users to the new name.

@jcommelin Can you clarify how you intend to apply @[deprecated] attributes to this PR? From what I see in mathlib4, @[deprecated X] is added to a deprecated declaration Y to signify that X should be used instead of Y, so the old lemma Y need to be kept rather than renamed. The lemmas X and Y usually don't have exactly the same statement (for example RingHom.map_list_prod uses RingHom while map_list_prod uses MonoidHomClass), so the attribute doesn't seem to be intended for renamed lemmas.
Of course I can make the new names aliases of the old ones, then this PR would be adding aliases rather than renaming (and I wouldn't even need to fix the 30+ files), but is such proliferation of aliases desirable? Every alias will appear in the docs in its full form, I think. Note that some old names are simply problematic (e.g. isIntegral_sup is lacking namespace) and should be renamed anyway.

I want to preface my comment by saying that this is my personal opinion. As a community, we're still very much trying to figure out what the best strategy/policy is, I guess.
Anyways, I think I would advocate keeping the old names around, and tagging them with @[deprecated fooBar] as a pointer to the new name. In a future PR (can be hours/days/weeks later) we remove all the deprecated decls. The big upshot is that there will be a formal trace in the git history that traces the renaming. This will help:

people bumping mathlib in their personal projects
changelog
future automated refactoring tools that do mathlib bumps for us

@j-loreaux Thanks for generating that list, and @alreadydone thanks for pairing up the old and new names. Would you mind preserving that list in the git history by copying it to the top message?

Since @adomani has gone to the trouble of reducing the diff, I think we should merge #8437 before this PR.

@adomani's #8437 did part of the renames, but the following were not included:
In IntegralClosure:
RingHom.is_integral_map → RingHom.isIntegralElem_map
RingHom.is_integral_of_mem_closure → RingHom.IsIntegralElem.of_mem_closure
RingHom.is_integral_zero/one → RingHom.isIntegralElem_zero/one
RingHom.is_integral_add/neg/sub/mul/of_mul_unit → RingHom.IsIntegralElem.add/neg/sub/mul/of_mul_unit
RingHom.isIntegral_trans → RingHom.IsIntegral.trans
RingHom.isIntegral_quotient/tower_bot/top_of_isIntegral → RingHom.IsIntegral.quotient/tower_bot/top
isIntegral_sup → Algebra.isIntegral_sup
IsIntegral.of_mem_closure' → IsIntegral.of_mem_closure'
isIntegral_of_surjective → Algebra.isIntegral_of_surjective
In RingTheory/Algebraic:
of_larger_base → tower_top (for consistency with IsIntegral)
Algebra.isAlgebraic_of_finite → Algebra.IsAlgebraic.of_finite
Algebra.isAlgebraic_trans → Algebra.IsAlgebraic.trans
I also added a lemma Algebra.IsIntegral.of_injective in IntegralClosure and three in RingTheory/Algebraic: Algebra.IsIntegral.isAlgebraic, isAlgebraic_algHom_iff, and Algebra.IsAlgebraic.of_injective.
(Currently I can't easily use @j-loreaux's #8433 to generate decl name changes, because if I merge this PR directly into #8433's branch the diff generated would include new commits on master. If I first merge master into #8433's branch, then I need to wait for the cache, and then merge this PR into #8433 and wait for the cache again, which is time consuming.)
I also decided to make R implicit in IsIntegral.isAlgebraic, which resulted in more files needing to be fixed.
@adomani's PR means that the current changes in IntegralClosure and RingTheory/Algebraic only cause 24 files to break, so it's possible to split PR into one that changes 24 files and another one that changes 17 files (depending on the previous one, which would just be using the new dot notations to golf proofs), but I'm not sure it's worth doing; most diffs would still be in the first PR. If @adomani would do another renaming PR and include some changes in implicitness, then the number 24 would decrease and a split may be more attractive.

Can you please add the remaining renames to the PR commit message?

I'm inclined to merge this as is. Damiano's tool was a first success. And hopefully we can use it more in the future. But splitting up this PR seems like it will be a lot of work for little remaining gain.

Thanks @jcommelin! PR message edited.

The renames were an experiment to test an automated renaming tool.  Since it was the first "real world" attempt, I aimed to rename only lemmas for which there was no dot-notation involved.
Currently, the tool finds the positions where the old declaration appears and replaces them with the new one.  No fancy namespacing/dot-notation/removal of arguments is implemented.
I would like to add namespacing options, since that is easy to implement as a string-based replacement.
However, doing more context/syntax-aware changes is best done with a tool implemented in Lean.
The tool itself, reads the location of the syntax from the ilean files and then uses bash-commands to do the string replacements.

Thanks :tada:
bors merge

@alreadydone yeah, #8433 is not usable until it's merged because it changes lakefile.lean. Sorry!


Editorial: ensure samples are clear when mixing decoded credentials

Note to self: sometimes we mix decoded forms of credentials and it's unclear to implementors. Ensure we clean up

note to @kimdhamilton does this require action ?

Confirmed relevant samples are confined to the appendix and are appropriately documented; e.g., https://identity.foundation/presentation-exchange/#presentation-submissions


It's not possible to js_name rename an enum

It's not possible to js_name rename an enum.
wasm-bindgen = "0.2.59"
build: wasm-pack build
use wasm_bindgen::prelude::*;

#[wasm_bindgen(js_name = "MyStruct")]
pub struct MyStructJS;

#[wasm_bindgen(js_name = "MyEnum")]
pub enum MyEnumJS {
	Zero,
	One,
	Two,
}

Note that MyEnumJS wasn't renamed to MyEnum below
/* tslint:disable */
/* eslint-disable */
/**
*/
export enum MyEnumJS {
  Zero,
  One,
  Two,
}
export class MyStruct {
  free(): void;
}

Thanks for the report! Currently not all attributes are supported in all places, but that's just a bug and it should be easy enough to plumb this through to get things working!


[wip]: add the new queries to console code

So..... what's in this PR?

[ ] Hooked up the tables of the event triggers to use the latest APIs

[x] Cron
[x] One-Off
[ ] Data (needs fixes on SQL front)

This PR involves a lot of updates on the FilterQuery component and the useFilterQuery method.

TODO and What needs fixing

The API returns all of the entries within the db and in the case of cron triggers the minimum number of elements on the table just goes up to 60 .
Invocation logs need to be hooked up to a new API so that the details can be fetched properly.
The request, response shown in the processed events also need work. (It just crashes now)

@beerose updated the description of the PR for full details on what I'm trying to do in this PR.


Suppress jQuery warning related to settings menu.

If you develop with Firefox, before this commit, you would
get jQuery warnings about empty selectors every time you clicked
in the app, due to a glitch with our old version of bootstrap.js.
This commit makes the error go away by setting
data-toggle to 'nada' for the settings-dropdown
element.
This was the error:
Empty string passed to getElementById().

More details here:
https://github.com/twbs/bootstrap/issues/5566

Merged, thanks @showell!


Sometimes Dio 'cuts off' Rest API responses

Issue Info

Info
Value

Platform Name
Flutter

Platform Version
1.5.4

Dio Version
2.1.0

Android Studio / Xcode Version
both

Repro rate
sometimes 25%

Repro with our demo prj
no

Demo project link
don't have it

Issue Description and Steps
I have some issues with Dio and need some help to resolve it. I'm calling a Rest API while building a flutter page and sometimes I receive some really strange issues because the response is being cut off. I'm 100% sure the webserver is return a full and valid Json response, because I have access to the server backend too.
I receive sometimes, randomly unexpected errors like this:

I/flutter (14410): *** Request ***
I/flutter (14410): uri: https://dev07.changetocomm.net/ac/api/client/C1001005/klantprofiel
I/flutter (14410): method: GET
I/flutter (14410): contentType: application/json; charset=utf-8
I/flutter (14410): responseType: ResponseType.json
I/flutter (14410): followRedirects: true
I/flutter (14410): connectTimeout: 4000
I/flutter (14410): receiveTimeout: 5000
I/flutter (14410): extra: {}
I/flutter (14410): header:
I/flutter (14410):   Authorization:Basic ZmVycnk6bG90dXNub3Rlcw==
I/flutter (14410):   Content-Type:application/json
I/flutter (14410):   Cache-Control:no-cache
I/flutter (14410):   cookie:SessionID=69E21BF718C38B35B8D41692D6F0879BC6DF2696
I/flutter (14410):
I/flutter (14410): *** DioError ***:
I/flutter (14410): DioError [DioErrorType.DEFAULT]: FormatException: Unterminated string (at character 16385)
I/flutter (14410): ...":"Nog te bepalen","id":"2AD67D180C49D744C125824400434336","portal":"N"},{"
I/flutter (14410):                                                                               ^
I/flutter (14410): DIO error DioError [DioErrorType.DEFAULT]: FormatException: Unterminated string (at character 16385)
I/flutter (14410): ...":"Nog te bepalen","id":"2AD67D180C49D744C125824400434336","portal":"N"},{"
I/flutter (14410):                                                                               ^
I/flutter (14410): DioError [DioErrorType.DEFAULT]: FormatException: Unterminated string (at character 16385)
I/flutter (14410): ...":"Nog te bepalen","id":"2AD67D180C49D744C125824400434336","portal":"N"},{"
I/flutter (14410):

And the same request on another moment I receive this:

I/flutter (14410): *** Request ***
I/flutter (14410): uri: https://dev07.changetocomm.net/ac/api/client/C1001005/klantprofiel
I/flutter (14410): method: GET
I/flutter (14410): contentType: application/json; charset=utf-8
I/flutter (14410): responseType: ResponseType.json
I/flutter (14410): followRedirects: true
I/flutter (14410): connectTimeout: 4000
I/flutter (14410): receiveTimeout: 5000
I/flutter (14410): extra: {}
I/flutter (14410): header:
I/flutter (14410):   Authorization:Basic ZmVycnk6bG90dXNub3Rlcw==
I/flutter (14410):   Content-Type:application/json
I/flutter (14410):   Cache-Control:no-cache
I/flutter (14410):   cookie:SessionID=69E21BF718C38B35B8D41692D6F0879BC6DF2696
I/flutter (14410):
I/flutter (14410): *** DioError ***:
I/flutter (14410): DioError [DioErrorType.DEFAULT]: FormatException: Unexpected end of input (at character 8193)
I/flutter (14410): ...xnummer":"FFFFFF","naam":"hap","geslacht":"m","titelvoor":"","partner_naam"
I/flutter (14410):                                                                               ^
I/flutter (14410): DIO error DioError [DioErrorType.DEFAULT]: FormatException: Unexpected end of input (at character 8193)
I/flutter (14410): ...xnummer":"FFFFFF","naam":"hap","geslacht":"m","titelvoor":"","partner_naam"

Please update dio to latest version and try again.

Looks like this works. Thanks

Still facing this issue with v2.1.8. And the strange thing is it occurred in Android Emulator only.


Got to 100% unit test coverage

:tada: This PR is included in version 2.0.4 :tada:
Your semantic-release bot :package::rocket:


Change Operational Store tables names and integrate with EF migrations command

I have an Entity Framework Core 2.2 DbContext as follows:
public class MyContext : IdentityUserContext<User, Int32, UserClaim, UserLogin, UserToken> {
}

And, using Identity Server 4, I would like to use OperationalStore so I added to my Auth project startup:
      services
        .AddIdentityServer()
        .AddSigningCredential(certificate)
        .AddInMemoryIdentityResources(Config.GetIdentityResources())
        .AddInMemoryApiResources(Config.GetApiResources())
        .AddInMemoryClients(Config.GetClients())
        .AddAspNetIdentity<User>()
        .AddOperationalStore(x => {
           x.ConfigureDbContext = y => y.UseSqlServer(connectionString);
           x.EnableTokenCleanup = true;
           x.TokenCleanupInterval = 30;
        });

What entities and configuration should I add to MyContext to support OperationalStore?
And can I change the names of the tables that support Operational Store? How?
Thank You

What entities and configuration should I add to MyContext to support OperationalStore?

This is already encapsulated in the call to AddOperationalStore.

This is already encapsulated in the call to AddOperationalStore.

@brockallen Not sure I got it. I have MyContext in a class library ...
To create the migrations and the database I use the commands on the database:
dotnet ef migrations add "FirstMigration"
dotnet ef database update

How to ensure the SQL tables used by Identity Server are included in the migration and in the database update?
And is it possible to change the name of one of the SQL tables used by Identity Server?

In case I use only OperationalStore, and not configuration, wouldn't dbo.PersistedGrants be the only table / entity needed?


Added code of conduct.

Just a code of conduct file. You can merge it if there's a need of this file in this project.

ok, thanks for your contribution!


Request: include useEffect to reset (with new initial data) form on change of initialValue

Example
const MyComponent = () => {
  let [ item, setItem ] = useState({ foo: 'bar' })
  let [ formState ] = useFormState(item)

  setTimeout(() => {
   // this should trigger a component re-render
   //which will fire a new "item"  value into the useFormState initialValue
   setItem({ fo: 'fum' })
  }, 2000)

  return (
    <pre>
     {
       JSON.stringify(formState, null, 2)
       // this will continue to show the form state as originally
       // set, instead of updating with new form data
     }
   </pre>
  )
}

Expected Behavior
When changing initialValue (e.g. as the product of another hook), I'd expect the form to reset itself (including untouched state) to the new base data.
Current Behavior
When changing initialValue from another hook, the form is left unchanged.  This makes it difficult to dynamically load a form without using component hacks to force a re-instantiation of the useFormState hook
Workaround Hacks

Throwing a useEffect into your component that watches for changes to [what should be] the form initialState, that manually does setFields based on the new object structure.  Risks are that if the form structure changes at all, you may carry old data values along with the new (e.g. it may have old fields that were left in the form body because the new form state didn't have those attributes), as well the fact that they'll all be marked as touched instead of untouched.

Steps Forward

[ ] do not agree that changes to initialValue should be dynamically set form data
[ ] agree and will modify code to match
[ ] agree but would welcome a PR (I'd be happy to take a stab at it)

P.S. - @wsmd Love this library and the elegant interface - keep up the great work!

Any update on this? I, too, am having a hard time updating initial data, for example, after a successful async request.

Our current workaround on this is to do our data fetching one component up the tree, and don't render the component using useFormState until after our data load is complete. Not ideal but it does work. I think what we'd like to see is the ability to just globally set all state in one go, setFields perhaps or the like.

Similar issue #146


[BUG] Cannot return None value from ClosureExpression

Bug description
I'd like to be able to write code like the following:
    let factory = gtk::SignalListItemFactory::new();
    factory.connect_setup(|_, list_item| {
        let image = gtk::Image::new();

        list_item.property_expression("item")
            .chain_closure_with_callback(|args| {
                let file_info: gio::FileInfo = args[1].get().unwrap();

                if matches!(file_info.file_type(), gio::FileType::Directory) {
                    Some(gio::Icon::for_string("folder").unwrap())
                } else {
                    None
                }
            })
            .bind(&image, "gicon", gtk::Widget::NONE);
    });

This code will set the gicon property of the gtk::Image to None if the FileInfo is not a directory. Unfortunately, this code does not compile because of the R: ValueType bound on the closure.
error[E0277]: the trait bound `Icon: ValueTypeOptional` is not satisfied
  --> src/main.rs:11:14
   |
11 |             .chain_closure_with_callback(|args| {
   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `ValueTypeOptional` is not implemented for `Icon`
   |
   = note: required because of the requirements on the impl of `ValueType` for `std::option::Option<Icon>`

For more information about this error, try `rustc --explain E0277`.

I'm new to this API, so it's possible that it isn't intended to be used this way, but perhaps the bound can be relaxed to allow optional types?
Backtrace
N/A
cc @sdroege, who requested I open this issue on Matrix.

See https://github.com/gtk-rs/gtk-rs-core/pull/420


Separate Listening and Discovery Ports in Node Configuration

Description
According to the documentation on P2P Port configuration, the --p2p-port options/BESU_P2P_PORT environment variable/p2p-port configuration file option can be used to configure the listening port for the node, which is used for both the TCP listening port and the UDP discovery port.
Strangely enough, there are sections in the documentation that describe a valid Enode URL that allows setting a different discovery port using a discport query parameter.
This brings the question of if those Enode URLs are supported, how can one configure a dedicated port for discovery, which has a different number than the listening port.
Acceptance Criteria
Allow configuring the listening port and discovery port of the node separatly,
Additional Information
It might seem like a small issue, but in scenarios that require exposing both ports, it creates limitations such as the cloud services that can be used to run nodes (at the time of opening this issue, both Azure ACI and AWS ECS do not support opening the same port number for both TCP and UDP at the same time).

Hi @ItayPodhajcer,
Having two different ports for listening and advertising is not available in the base CLI use case, but if you use Besu with Kubernetes for instance, you can do it by setting services multiple port mappings and using the Besu Kubernetes NAT option.
Could you tell me if your use case includes using Kubernetes?

@NicolasMassart Kubernetes is such an overkill in so many cases, especially when ACI/ECS are enough to get the job done.
Any plans to update the CLI in the near future?

@NicolasMassart How can we tackle this issue and make changes in our code? I have also been facing a lot of trouble getting my ports configured.


Create method to Validator that accepts data argument.

Currently the Validator class requires data be passed into the constructor, though it doesn't do anything important with it then.
In my case, I'm validating in a server and I see no reason to parse the schema over and over for different inputs, especially since it uses two mutexes in the initialize method.
Let me know if I missed something and Validator simply must accept data at construction.
Is #validate_data(data) the way to go, or should I change the signature of #validate to take an optional argument? The former seemed like a clean change with little API change.

I wrote this little script to look for any obvious threading issues.
https://gist.github.com/glennpratt/bf254a896c9b0f15b812

To me this looks fine. :+1:

This needs work, have issues with shared @errors instance var in testing.

OK, pushed a commit that should fix that issue.
Need more tests around that apparently.

Note: This PR is sparse to help decide if this is a good direction to move it.
If it is, I'll update usage, documentation and tests.
So, do people think this is the right direction?
(Thanks @RST-J)

Yes, this is the right direction.  The schema should be initialized once, and then used to validate any number of records.

Yes I agree with the direction completely. I would actually change the signature of the validate method. We already have a 3.x branch for exactly this kind of backwards incompatible change.
Once you're happy with the approach could open a new pull request on the 3.x branch?


Also on a kinda related slime note sometimes when you kill something you can attack the place where it last was. This hitbox doesn't despawn.

@Col-E, found this bug.

Why the wontfix label? Also, a picture of the issue:

It looks like it is fixed now.


final clean ups

What does this PR do?

Motivation

Additional Notes

Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] PR title must be written as a CHANGELOG entry (see why)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have changelog/ and integration/ labels attached
[ ] If the PR doesn't need to be tested during QA, please add a qa/skip-qa label.

Current dependencies on/for this PR:

master

PR #14212

PR #14260   👈
PR #14258
PR #14257
PR #14213

PR #14214

PR #14215

PR #14216

PR #14223

This comment was auto-generated by Graphite.


Define events for TimeSeries to be published for aggregations (part 2 of data contracts work with Joules)

Description:
As per request from the Aggregation domain code-owners, the TimeSeries code-owners need to document the inter-domain events/contracts published by the TimeSeries domain "targeting" the Aggregation domain. Besides being relevant for the actual implementation of the inter-domain integration point, this piece of work is perceived as of high value by the Aggregations code-owners in terms of acquiring TimeSeries-domain knowledge, which is needed to ensure time series data is correctly handled, and used in settlements.
Preconditions:
A draft document has been attached to story #66 showing data properties and types for TimeSeries
Acceptance criteria:

Based on the draft document mentioned under preconditions, TimeSeries events/contracts relevant for the Aggregations domain have been documented.
The events/contracts have been presented to the Aggregations domain code-owners and relevant feedback has been incorporated in the events/contract documentation
The documentation has been made available in the TimeSeries repo under /docs/contracts.md in a similar format to this:
https://github.com/Energinet-DataHub/geh-metering-point/blob/main/docs/contracts.md

Draft PR is up.


Revert "NativeMethodsShared.OSName: Fix the case of OSX"

xbuild has been using $(OS)==Unix on OSX due to an old mono bug
which means that there are lot of targets out there which depend on
this being true on OSX and Linux. "Fixing" this to have the correct
$(OS) == OSX would probably break all such targets which expect
to work on
Revert "NativeMethodsShared.OSName: Fix the case of OSX"
This reverts commit 8fc4ed7713089c5edafa0f3e86f16da268d38dbc.

@dotnet-bot test this please

Should we revert this only for Mono?

No, IMO, we should have uniform behavior w.r.t this. Having different behavior will likely cause unnecessary pain. And it doesn't affect compatibility with existing projects.
If the proposed approach in #539 sounds acceptable, then I'll change this PR accordingly. The proposed approach being:

$(OS) will have values of Windows_NT and Unix only. On Windows - $(OS)==Windows_NT and only any IsUnixLike platform - $(OS)==Unix.

Introduce a new property $(OSName) which would have more specific values like Windows_NT, OSX and Unix. I'm not sure whether Unix is enough here or it should be more specific like Linux, FreeBSD etc.

If the user's requirement is to simply check for "windows or unix-like", then $(OS) can be used. If the requirement is to have more specific behavior, then they can use $(OSName).
@akoeplinger @jonpryor ^
If this is acceptable, then I can update the PR to not revert the patch, hence retain NativeMethodsShared.OSName, set the new property with this and set the $(OS) property explicitly.

@joperezr Does this affect corefx in any way?

@joperezr Does this affect corefx in any way? Please also take a look at #539

In corefx we currently do the 'hack' of setting OSX to Unix because we were seeing issues for this so I don't think that this would cause issues for now, but I'm not sure until I test it.

Based on discussion in #539 this should change to only return Windows_NT when Windows, else Unix. You want to update this PR?

Sure, I'll do that.

LGTM

@dotnet-bot test this please


chore(platform loggging): Change workflow to controller

Change examples for installing deis-logger to reflect recent workflow to controller changes.

We should get this in, then I'll add a PR to update the charts after we cut beta.


[SPARK-36516][SQL] Support File Metadata Cache for ORC

What changes were proposed in this pull request?
The main purpose of this pr is to introduce the File Meta Cache mechanism for Spark SQL and the basic File Meta Cache implementation for Orc is provided at the same time. There was originally a PR that supports file meta cache both Parquet and ORC, but Parquet has non Deprecated API that can be used to pass footer to create new ParquetFileReader and both Apache Spark and Parquet community are reluctant to advertise the deprecated API, so this pr spin off ORC-only part.
The main change of this pr as follows:

Defined a FileMetaCacheManager to cache the mapping FileMetaKey to FileMeta. The FileMetaKey is the cache key, equals is determined by the file path by default. The FileMeta used to represent the cache value and It is generated by the  FileMetaKey#getFileMeta   method.

Currently, the FileMetaCacheManager supports a simple cache expiration elimination mechanism, and the expiration time is determined by the new config  FILE_META_CACHE_TTL_SINCE_LAST_ACCESS

For Orc file format, this pr added OrcFileMetaKey  and OrcFileMeta to cache Orc file Tail and and the Tail cache can be used by Vectorized read scene both in DS API V1 and V2, the feature will be enabled when FILE_META_CACHE_ORC_ENABLED  is true

Currently, the file meta cache mechanism cannot be used by RowBasedReader, and it needs the completion of ORC-746 for further support.
Why are the changes needed?
Support Orc datasource use File Meta Cache mechanism to reduce the times of metadata reads multiple queries are performed on the same dataset.
Does this PR introduce any user-facing change?
Add 2 new config:

FILE_META_CACHE_ORC_ENABLED(spark.sql.fileMetaCache.orc.enabled) to indicate if enable orc file meta cache mechanism
FILE_META_CACHE_TTL_SINCE_LAST_ACCESS(spark.sql.fileMetaCache.ttlSinceLastAccess) to represent Time-to-live for file metadata cache entry after last access, the unit is seconds.
FILE_META_CACHE_MAXIMUM_SIZE(spark.sql.fileMetaCache.maximumSize) to represent maximum number of file meta entries the file meta cache contains.

How was this patch tested?

Pass the Jenkins or GitHub Action
Add new test suites to OrcQuerySuite

Test build #142495 has started for PR 33748 at commit 45f4827.

Test build #142495 has finished for PR 33748 at commit 45f4827.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142495/

Test build #142496 has started for PR 33748 at commit a74c793.

There will be some duplicate codes in the two PR, and this part of the code will be synchronized after one of them is merged

Test build #142497 has started for PR 33748 at commit 1a9bd3b.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46993/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46997/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46998/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46993/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46993/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46997/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46998/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46998/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46997/

Test build #142503 has started for PR 33748 at commit 30df269.

Test build #142503 has finished for PR 33748 at commit 30df269.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class FileMetaKey
case class OrcFileMetaKey(path: Path, configuration: Configuration)
case class OrcFileMeta(tail: OrcTail) extends FileMeta

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142503/

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47004/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47004/

Test build #142496 has finished for PR 33748 at commit a74c793.

This patch fails from timeout after a configured wait of 500m.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142496/

Test build #142513 has started for PR 33748 at commit 42d2bfd.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142497/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47014/

Test build #142515 has started for PR 33748 at commit 0e6c52b.

Test build #142516 has started for PR 33748 at commit 95bae3c.

@dongjoon-hyun  95bae3c add a configurable maximumSize and the default value is 1000. Please help to check whether this configuration is reasonable.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47015/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47015/

FileMetaCacheManager currently holds a cache singleton, which is not easy to test the scenario of cache eviction. Do we need to check these? @dongjoon-hyun

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47014/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47014/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47017/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47017/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47017/

Test build #142513 has finished for PR 33748 at commit 42d2bfd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142513/

Test build #142515 has finished for PR 33748 at commit 0e6c52b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142515/

@dongjoon-hyun 95bae3c add a configurable maximumSize and the default value is 1000. Please help to check whether this configuration is reasonable.

@LuciferYang . It looks good to me because it's per executor.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142516/

Test build #142526 has started for PR 33748 at commit ebb7e0b.

BTW, is it possible to have a benchmark too?

@viirya  I will do this later

Test build #142529 has started for PR 33748 at commit c36a569.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47027/

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47030/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47030/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47027/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47027/

Test build #142526 has finished for PR 33748 at commit ebb7e0b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142526/

Test build #142529 has finished for PR 33748 at commit c36a569.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class V1FileMetaCacheSuite extends FileMetaCacheSuite
class V2FileMetaCacheSuite extends FileMetaCacheSuite

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142529/

Test build #142552 has started for PR 33748 at commit b02de85.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47054/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47054/

Test build #142552 has finished for PR 33748 at commit b02de85.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142552/

Test build #142586 has started for PR 33748 at commit 406f91c.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47086/

Test build #142591 has started for PR 33748 at commit 3b15a82.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47086/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47086/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47091/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47091/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47091/

Test build #142599 has started for PR 33748 at commit 2b99983.

Test build #142602 has started for PR 33748 at commit 1dd174e.

Test build #142604 has started for PR 33748 at commit a339b1b.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47102/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47102/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47099/

Test build #142606 has started for PR 33748 at commit 7153d2a.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47103/

Test build #142586 has finished for PR 33748 at commit 406f91c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142586/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47103/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47103/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47099/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47099/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47107/

Test build #142591 has finished for PR 33748 at commit 3b15a82.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142591/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47107/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47107/

Test build #142602 has finished for PR 33748 at commit 1dd174e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142602/

Test build #142599 has finished for PR 33748 at commit 2b99983.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142599/

Test build #142604 has finished for PR 33748 at commit a339b1b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #142606 has finished for PR 33748 at commit 7153d2a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142606/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142604/

Thank you for updating, @LuciferYang !

Test build #142632 has started for PR 33748 at commit c3838e6.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47132/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47132/

Test build #142634 has started for PR 33748 at commit 4adeb62.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47134/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47136/

Test build #142639 has started for PR 33748 at commit e5f9497.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47136/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47134/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47136/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47134/

Test build #142640 has started for PR 33748 at commit ec8fa1c.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47140/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47140/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47141/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47140/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47141/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47141/

Test build #142634 has finished for PR 33748 at commit 4adeb62.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142634/

Test build #142639 has finished for PR 33748 at commit e5f9497.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142639/

Test build #142632 has finished for PR 33748 at commit c3838e6.

This patch fails from timeout after a configured wait of 500m.
This patch merges cleanly.
This patch adds no public classes.

Test build #142640 has finished for PR 33748 at commit ec8fa1c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142632/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142640/

Ya, it's a little blocked by the on-going discussion because it's related. Thank you for waiting for it, @LuciferYang .

Since the metadata is cached in the executor, does it mean the task reading the same ORC file has to be scheduled on the same executor? How can we guarantee this?
@sunchao please correct me if I'm wrong, if we overwrite the table, the same file names will be reused which potentially could cause inconsistent issue. Shouldn't we have some safeguard such as checking the file sizes?
Thanks,

@sunchao please correct me if I'm wrong, if we overwrite the table, the same file names will be reused which potentially could cause inconsistent issue. Shouldn't we have some safeguard such as checking the file sizes?

Even overwriting the table, will we use same file names? I remember the file names will include unique task id/attempt id.

In Hive it's common that the same file name (e.g., 000000_0) gets used when doing insert overwrite. Even if we check file size and other stuff, it can't completely prevent us from hitting a stale cache.

@dbtsai and @sunchao .
Everything depends on the data lifecycle. For the safety, we can control it by reducing spark.sql.fileMetaCache.ttlSinceLastAccessSec to 10 secs or less which is still effective to reduce the twice footer lookups in the single task.

Since the metadata is cached in the executor, does it mean the task reading the same ORC file has to be scheduled on the same executor? How can we guarantee this?

At present, I think it is try best because there is no guarantee of scheduling, if the there has many NODE_LOCAL scheduling is probably the best scenario.
We may need to collect the information about fileMetaCache distribution to the driver and consider this factor when task scheduling, but this may make the driver use more memory.

In Hive it's common that the same file name (e.g., 000000_0) gets used when doing insert overwrite. Even if we check file size and other stuff, it can't completely prevent us from hitting a stale cache.

Can we add ctime or mtime of the file to the PartitionedFile and use this information for check?
Similarly, how do we ensure that the FileStatus cache(SharedInMemoryCache) is correct when the user overwrites the file and does not send the refreshTable command to the current SparkApp?

Can we add ctime or mtime of the file to the PartitionedFile and use this information for check?

Yea file path + modification time seem like a good way to validate the cache

Similarly, how do we ensure that the FileStatus cache(SharedInMemoryCache) is correct when the user overwrites the file and does not send the refreshTable command to the current SparkApp? There is also the problem of the same name of the file. I feel that solving this consistency problem may become an independent topic :)

I think they are not quite the same: FileStatus cache operates on a per-table basis so you'll only get stale data in the worst case. However, here the cache is on a per-file basis, so one could end up having some files in a partition that are cached while the rest are not. In addition, at least in Parquet, metadata, including row count, stats, bloom filter, etc, are used to filter row groups or data pages. Task failure or correctness issue could happen if we apply stale metadata on a newer file, or if the metadata is used in aggregation pushdown (ongoing work).

Everything depends on the data lifecycle. For the safety, we can control it by reducing spark.sql.fileMetaCache.ttlSinceLastAccessSec to 10 secs or less which is still effective to reduce the twice footer lookups in the single task.

I understand you want to avoid the duplicate footer lookup. In Parquet at least we can just pass the footer from either ParquetFileFormat or ParquetPartitionReaderFactory to SpecificParquetRecordReaderBase for reuse, which I think is much simpler than using a cache.

I understand you want to avoid the duplicate footer lookup. In Parquet at least we can just pass the footer from either ParquetFileFormat or ParquetPartitionReaderFactory to SpecificParquetRecordReaderBase for reuse, which I think is much simpler than using a cache.

If we can add some strategies to Spark in the future to ensure that the task reading the same Orc file has to be scheduled on the same executor, the benefits of fileMetaCache will be more obvious. In fact, in our production environment, about 100000 sql queries are using this feature every day.

Yea, but it adds complexity and more memory consumption like you mentioned earlier, and you'll need to have the driver a long running process like a Presto coordinator, which I'm not sure how many people are using Spark this way.

which I'm not sure how many people are using Spark this way.

There should be many. We can do some survey, haha ~

Yea, but it adds complexity and more memory consumption like you mentioned earlier, and you'll need to have the driver a long running process like a Presto coordinator

Yes, in the production environment, we did change a lot of code for similar optimization

Everything depends on the data lifecycle. For the safety, we can control it by reducing spark.sql.fileMetaCache.ttlSinceLastAccessSec to 10 secs or less which is still effective to reduce the twice footer lookups in the single task.

@dongjoon-hyun Could you elaborate what's the twice footer lookups here in a single task? If it's in a single task, then the files should be the same, so the life of the cache can be just for a single cache, right?
I thought the purpose of this PR is when the same ORC file is used for multiple tasks, the fileMetaCache can be used to avoid reading the footer multiple times with a caveat that those tasks have to be scheduled on the same executor.
As @LuciferYang and @sunchao mentioned above, this requires adding something like Presto coordinator to ensure the footer cache can be reused. I feel it's fairly complicated, and don't know if it worths it. For this use-case, we might just use Iceberg which stores the metadata as a separated manifest.

@dongjoon-hyun db90daf,  7327fdb change to use Guava Cache and  merge with master because https://github.com/apache/spark/pull/33784 revert code aboutCaffeine

Test build #142684 has started for PR 33748 at commit 7327fdb.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47186/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47186/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47186/

Test build #142684 has finished for PR 33748 at commit 7327fdb.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class NoSuchDatabaseException(db: String)
case class NoSuchNamespaceException(
case class NoSuchTableException(
case class NoSuchPartitionException(
case class NoSuchPermanentFunctionException(db: String, func: String)
case class NoSuchFunctionException(
case class NoSuchPartitionsException(override val message: String)
case class NoSuchTempFunctionException(func: String)
class DefaultDateFormatter(
class DefaultTimestampFormatter(
case class DayTimeIntervalType(startField: Byte, endField: Byte) extends AnsiIntervalType
case class YearMonthIntervalType(startField: Byte, endField: Byte) extends AnsiIntervalType

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142684/

Test build #143257 has started for PR 33748 at commit a7eff43.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47760/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47760/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47760/

Test build #143257 has finished for PR 33748 at commit a7eff43.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143257/


Inter-app messaging

Lastuser needs to provide a mechanism for client apps to leave messages for each other. This can be used, for example, for BoxOffice to post an invoice in Billgate (using Billgate's API), and for Billgate to notify BoxOffice of successful payment (via the messaging API).
The messaging API helps communicate across firewalls, since Lastuser is always accessible, and ensures message delivery even if an app is down (though Lastuser is now the single point of failure).

Deprecated. Inter-app messaging should be an app-level concern.


This change enables an checkbox to tick if the streets are used for g…

It is used to deactivate the streets in geocoding and the visualization on the map

@KarlKeu00 check my comment and consider merging after applying the (boolean) where I suggested.


PinInputField Allows Input When isDisabled or isReadOnly  is Set

Which package has an issue?
@yamada-ui/react
Package Version
1.6.2
Description
Bug that allows input even if isDisabled or isReadOnly is set to PinInputField.
It is possible to reproduce the following.
https://yamada-ui.github.io/yamada-ui/?path=/story/components-forms-pininput--is-disabled
https://yamada-ui.github.io/yamada-ui/?path=/story/components-forms-pininput--is-readonly
Link to Reproduction
No response
Steps to Reproduce

Go to '...'
Click on '...'
Scroll down to '...'
See error

Browser
Chrome
Operating System

[X] macOS
[ ] Windows
[ ] Linux
[ ] iPadOS
[ ] iOS
[ ] Android

Do you know how to fix the issue?
None
Are you willing to participate in fixing this issue and create a pull request with the fix?
None
Additional Information
No response

Can I investigate it?


MA issue10 debugMode option

The debugMode option was added which has 3 bits to set:
0x01: debug on RT error
0x02: debug on NVAR SVAR WAVE reference error
0x04: debug on failed assertion
Closes #10
Open Questions:

At the beginning of RunTest(), debugMode is tested to be within 0 and 7. Does it need to be tested for integer values, e.g. by trunk(debugMode) == debugMode ?
The constants DEBUG_ENABLE and DEBUG_ALLOW have the same value, but are used in different contexts. Is it OK to have two constants?
EnableIgorDebugger() gives a 4 bit variable to SetIgorDebugger(), which only uses the 3 least significant bits. Is this a possible bug in the future, so it should only pass 3 bits?

It does not compile as DEBUG_DISABLE is not defined.

Regarding the constants:
The constants should be non-static that a IUTF user is able to use the constants when calling RunTest.
It would be best to define them in unit-testing-basics.ipf and to avoid collisions give them a prefix like IUTF_*.
It is ok to have two different constants with the same value for two different optional parameters.
I think it is ok, if it is documented for SetIgorDebugger that it evaluates only the 3 LSB.

debugMode overwrites allowDebug, as it sets a specific state. This is documented under allowDebugs' information.
Manual also looks good, added an example how to use the IUTF_DEBUG - constants (see RunTest parameter list).

Also "variable" and "string" are written as required by our coding conventions.


[WIP] Generate protos for python client

[x] generate protos for python client
[x] package generated code as an installable

To-Do:

[ ] create PyPI project to host this package
[ ] integrate package generation + release into the CD process

Resolves #6116

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Feature/#26

Description: 概要

Changes: 変更内容

Expected Impact: 影響範囲

Operating Requirements: 動作要件

Additional context: 補足

Codecov Report

:exclamation: No coverage uploaded for pull request base (release/next-v1.3.1@8a88a50). Click here to learn what that means.
The diff coverage is n/a.

@@                   Coverage Diff                   @@
##             release/next-v1.3.1       #27   +/-   ##
=======================================================
  Coverage                       ?   100.00%
=======================================================
  Files                          ?         3
  Lines                          ?        62
  Branches                       ?         7
=======================================================
  Hits                           ?        62
  Misses                         ?         0
  Partials                       ?         0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8a88a50...ea96320. Read the comment docs.


add data archiving capability

User story
As a data curator, I want to archive data so that they can be retrieved in the future.
Acceptance criteria

[ ] archive service ready to receive data from the persistence component in the node

Definition of done

[ ] unit/functional tests added/updated
[x] documentation updated
[ ] live demonstration provided

Initial capability implemented via WIS2BOX_DATA_RETENTION_DAYS (docs: https://wmo-im.github.io/wis2box/running/data-retention.html).  We will need to flush out details for archiving to an archiving service per se.

Update: this was discussed at our 2022-08-02 standup.  The forthcoming demo/work with OpenCDMS for INFCOM will serve to move forward with a clearer picture on data being made available to an archiving service, for future sprint.
cc @david-i-berry @isedwards


bug fix

tftp.py have 2 bug at line 218 & 223 .
"self.block = block + 1"  should be "self.block = block + self.wrap * 65536 + 1"
since block is set by block no from ack message, block will  rotate from 1 to 65536(0), then self.block rotate from 1 to 65536(0) too,
finally ,
"self.fh.seek(self.blksize * (self.block - 1))" would not get right position for file its size > 2**16*512(65536)

tftp.py have 2 bug at line 218 & 223 .
"self.block = block + 1" should be "self.block = block + self.wrap * 65536 + 1"
since block is set by block no from ack message, block will rotate from 1 to 65536(0), then self.block rotate from 1 to 65536(0) too,

Can you submit a PR with a fix?

finally ,
"self.fh.seek(self.blksize * (self.block - 1))" would not get right position for file its size > 2**16*512(65536)

While this may be true, TFTP is unsuitable for a 2 terabyte file.


Do not remove existing encoding declaration

I already added encoding declarations to all files (which is also pep8 compliant). It gets removed by django-baker. That should be retained.

Sorry for the long delay in replying.  It only rewrites a file if there's less than 4 lines.  It seems like the easy solution would be to simply run baker then add the encoding declarations.


Advanced mode

Introduce an "advanced mode" to hide some power user features behind. The advanced mode can be activated by holding down altKey (option key on Mac).
Advanced only features:

Development (switches env to http://localhost:3000/*)
Live (if there is a production host defined)

:tada: This issue has been resolved in version 4.11.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Module '@tsed/logger' does not provide an export named 'colorize'

Informations

Type
Version

Bug
6.7.7

Description
When running my tests locally I received the following error:

I searched about this module and I found out that 6.7.7 version the utils folder was removed from barrelsby.json , that may be causing this error
https://github.com/tsedio/logger/blob/e2efe14c8cfe68d49442fa7e542ddccdbd58ced3/packages/logger/.barrelsby.json#L7-L12
And it is used in @tsed/common in printRoutes.ts
Todos

[x] Example (log, usecase, etc...)
[x] Ts.ED version (bug/question)

I went ahead and created a pull request solving this issue, but I'd like to hear what you think about this topic and open to discuss futhermore

@lucaspagliari sorry I haven’t seen your pr before. But thanks for your reactivity ;)
The fix is released !

thanks for you quick response and also for fixing this 😉


swagger-codegen-cli-2.4.6.jar not found

https://github.com/kubeflow/tf-operator/blob/24798375fead22bb1a78f3039565ccf5a9fae017/hack/python-sdk/gen-sdk.sh#L21
This link not work.
Maybe we could use this link instead.

/cc @jinchihe

/assign
The package is changed, I going to update that. Thanks @pingsutw and @gaocegege


Lambda: not calling .adaptMetric() in some methods in the metric factory

Version
latest
Steps and/or minimal code example to reproduce
https://github.com/cdklabs/cdk-monitoring-constructs/blob/v0.0.42/lib/monitoring/aws-lambda/LambdaFunctionMetricFactory.ts#L156-L161
Expected behavior
We should wrap the metric created in .adaptMetric() call to normalize the metric period.
Actual behavior
The metric is returned as-is, which can result in different period than expected.
Other details
No response

Would be good to check other metric factories as well, and fix this behaviour for good.


Rewrite in Kotlin

Hello, this is just a note to remember that we should rewrite the lib in kotlin when we get the time.

notes from our team meeting of 04/03/18:

Remove the ToolbarHandler and DrawerHandler.
Rename setRouterScreen, getRouterScreen, removeCurrentRouterScreen to setFlowrScreen, getFlowrScreen and removeCurrentFlowrScreen.
Remove SetCustomAnimation.


Initial plugin design

Goal is to support:

UI for provisioning Datasette permissions to specific users
UI for provisioning Datasette permissions to members of groups
UI for managing those groups
Table action menu interface for managing access to a specific table

For the moment I'm going to ignore the execute-sql loophole, where users who can execute SQL can view tables that they do not otherwise have permission to view. We could potentially close that hole later with a mechanism based around SQLite authorizers.

[x] #2
[x] Code for checking permissions
[x] #3
[x] #4

Now on PyPI: https://pypi.org/project/datasette-acl/


The library shouldn't be tied to an async runtime

Currently, the library is tied to tokio, usage here. We chose tokio because it's the async runtime used by chronicle, but ideally the library should be async-runtime agnostic, so users can execute it on their preferred runtime.
One of the options to solve this is waiting for Rust async closures to be stabilized, since that's exactly what we're trying to use in that code snippet.

Closing it for now since we're deeply tied to tokio now (polling system, RwLock/Mutex, mqtt) and can't get back from it. Also iota.rs and reqwest depends on tokio so there's no way to remove that dependency.

Closing it for now since we're deeply tied to tokio now (polling system, RwLock/Mutex, mqtt) and can't get back from it. Also iota.rs and reqwest depends on tokio so there's no way to remove that dependency.


build(deps): update to RN 0.69.10

This PR resolves MOPLAT-720
Description
Most changes come from this guide
The diff might look big due to some breaking changes, now we can opt-in to the new architecture if we want and there is some flag logic about that on the native side.
Some things to note:

added [deprecated-react-native-prop-types](http://npmjs.com/package/deprecated-react-native-prop-types) due to react-native removing some prop types.
had to patch some of our deps due to this (see hacks.md and /patches)
fixed some failing tests (nothing to worry about there)
bonus update -> React is version 18.0.0 🤯
RN -> 0.69.10
hermes comes prebundled from this version onwards 🔥 so it will be easier to integrate/enable

PR Checklist

[x] I have tested my changes on iOS and Android.
[x] I hid my changes behind a feature flag, or they don't need one.
[x] I have included screenshots or videos, or I have not changed the UI.
[x] I have added tests, or my changes don't require any.
[x] I added an app state migration, or my changes do not require one.
[x] I have documented any follow-up work that this PR will require, or it does not require any.
[x] I have added a changelog entry below, or my changes do not require one.

To the reviewers 👀

[ ] I would like at least one of the reviewers to run this PR on the simulator or device.

Changelog updates
Changelog updates

Cross-platform user-facing changes

iOS user-facing changes

Android user-facing changes

Dev changes

update to react-native 0.69.10

Need help with something? Have a look at our docs, or get in touch with us.

@gkartalis - mind updating the PR description with a list of the patches added in this PR?


Testing with various libraries and scenarios.

Add tests to the library to make sure all scenarios for various libraries are handled.

#2

Scenarios:

URL with query params.
GET, PUT, POST, DELETE, PATCH requests.
URL with a port number
URL without a path name. Has only hostname.
http, https, file protocols
Response Types:

JSON
PDF
Image
HTML

Request body:

Text
Encoded Text
JSON
Form URL encoded

Libraries:

Axios
Request
Node-fetch
SuperAgent


Question/Feat - queryEndpoint for Mimir when Ingesting via Distributor

We use this chart to push into a central Mimir instance (and thankful to this team every day for such an amazing helm chart that makes it super easy!)
As we use the Mimir API, our externalServices.prometheus.writeEndpoint is /api/v1/push
I notice there's a queryEndpoint in the prometheus section also. I presume this is for KubeCost (unless the agent needs to query prometheus for other things?).
The natural choice for this would be /api/v1/query. However, I think that in a Mimir environment by default it's expected that those queries go to the Mimir gateway/nginx reverse proxy. We've had challenges with scaling the Mimir Gateway (primarily due to no of gateway replicas needed for the high volume of metrics traffic received so additional nodes required and additional intra-az traffic costs) so instead of using the Mimir Gateway we expose our Mimir Distributors directly via a NLB and set our prometheus host to that value. This has worked really well for prometheus pushes, but of course falls over when it comes to querying.
We still do run the Gateway component for Grafana/querying/ruler etc it's purely metrics ingest we're exposing this way.
So I guess my question is two fold:

What is the queryEndpoint used for in this chart is it purely KubeCost? (which we are keen to enable soon)
In environments like ours where the queryEndpoint is on a different host path, is it feasible to add an additional field to support a query host in the chart? (I may be wrong but I've seen some evidence in the Mimir community of others pushing directly to distributors as well for similar Gateway related reasons).

If this is too niche to be desired in the chart, I wonder if it's possible to use some extraConfig to add it into the agent given extraConfig can't override any existing config just add additional?

Of course if the queryEndpoint is only used for KubeCost and not by the agent (which I suspect by our agents working fine for a few months with an invalid queryEndpoint!) then this can be completely closed and I'll figure out how to override in the sub chart directly :)

Running helm test will also use the query endpoint to check that the metrics the chart expects to publish actually made it to mimir.

So, as of now, you can set different urls for both places where metrics are read:
test:
  envOverrides:
    PROMETHEUS_URL: "https://prom.example.com/api/prom/api/v1/query" <-- The URL that will be queried during helm test
opencost:
  opencost:
    prometheus:
      external:
        enabled: true
        url: "https://prom.example.com/api/prom" <-- The URL that OpenCost will read metrics from

This question has been answered and is several months old, so I'm going to close this.


Additional indexes speed up deleting blocks

OS
Your OS: Linux 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Versions
The db-sync version (eg cardano-db-sync --version): cardano-db-sync 11.0.2 - linux-x86_64 - ghc-8.10
PostgreSQL version: 12+214ubuntu0.1
Build/Install Method
The method you use to build or install cardano-db-sync:
cabal install cardano-db-sync
Run method
The method you used to run cardano-db-sync (eg Nix/Docker/systemd/none):
command line (systemd)
Additional context
Add any other context about the problem here.
Problem Report
Please do not include screenshots or images, but instead cut and paste any relevant log messages
or errors.
db-sync was stuck for many many hours here:
[db-sync-node:Info:772642] [2021-12-01 20:16:54.84 UTC] Rolling back to slot 46396728, hash 0b4e19943029e10894330f9988d25d1104b4503cd41a9f623992cd410fbeb4bc
[db-sync-node:Info:772642] [2021-12-01 20:16:54.90 UTC] Deleting 20768 blocks up to slot 46823508

=> SELECT pid, age(clock_timestamp(), query_start), usename, query FROM pg_stat_activity  WHERE query != '<IDLE>' AND query NOT ILIKE '%pg_stat_activity%' ORDER BY query_start desc;

   pid   |       age       | usename  |                  query
---------+-----------------+----------+-----------------------------------------
 2737934 | 03:21:47.472175 | cardano  | DELETE FROM "block" WHERE "id"=6578171

I tried different things (including canceling the DELETE query) but it would always take several hours again without progress. Eventually I found that adding additional foreign key indexes only took a few minutes and the DELETE query went through immediately
/* From https://www.cybertec-postgresql.com/en/index-your-foreign-key/ */
SELECT c.conrelid::regclass AS "table",
       /* list of key column names in order */
       a.attname as column,
       pg_catalog.pg_size_pretty(
          pg_catalog.pg_relation_size(c.conrelid)
       ) AS size,
       c.conname AS constraint,
       c.confrelid::regclass AS referenced_table,
       'CREATE INDEX ' || c.conrelid::regclass || '_' || a.attname || '_idx ON ' || c.conrelid::regclass || ' (' || a.attname || ');' as fix
FROM pg_catalog.pg_constraint c
   /* enumerated key column numbers per foreign key */
   CROSS JOIN LATERAL
      unnest(c.conkey) WITH ORDINALITY AS x(attnum, n)
   /* name for each key column */
   JOIN pg_catalog.pg_attribute a
      ON a.attnum = x.attnum
         AND a.attrelid = c.conrelid
WHERE NOT EXISTS
        /* is there a matching index for the constraint? */
        (SELECT 1 FROM pg_catalog.pg_index i
         WHERE i.indrelid = c.conrelid
           /* it must not be a partial index */
           AND i.indpred IS NULL
           /* the first index columns must be the same as the
              key columns, but order doesn't matter */
           AND (i.indkey::smallint[])[0:cardinality(c.conkey)-1]
               OPERATOR(pg_catalog.@>) c.conkey)
  AND c.contype = 'f'
GROUP BY c.conrelid, c.conname, c.confrelid, a.attname
ORDER BY pg_catalog.pg_relation_size(c.conrelid) DESC;

          table           |   column    |  size   |              constraint               | referenced_table  |                                           fix
--------------------------+-------------+---------+---------------------------------------+-------------------+------------------------------------------------------------------------------------------
 tx_in                    | redeemer_id | 2770 MB | tx_in_redeemer_id_fkey                | redeemer          | CREATE INDEX tx_in_redeemer_id_idx ON tx_in (redeemer_id);
 pool_offline_fetch_error | pmr_id      | 501 MB  | pool_offline_fetch_error_pmr_id_fkey  | pool_metadata_ref | CREATE INDEX pool_offline_fetch_error_pmr_id_idx ON pool_offline_fetch_error (pmr_id);
 delegation               | redeemer_id | 142 MB  | delegation_redeemer_id_fkey           | redeemer          | CREATE INDEX delegation_redeemer_id_idx ON delegation (redeemer_id);
 withdrawal               | redeemer_id | 105 MB  | withdrawal_redeemer_id_fkey           | redeemer          | CREATE INDEX withdrawal_redeemer_id_idx ON withdrawal (redeemer_id);
 stake_deregistration     | redeemer_id | 11 MB   | stake_deregistration_redeemer_id_fkey | redeemer          | CREATE INDEX stake_deregistration_redeemer_id_idx ON stake_deregistration (redeemer_id);
 pool_offline_data        | pmr_id      | 3320 kB | pool_offline_data_pmr_id_fkey         | pool_metadata_ref | CREATE INDEX pool_offline_data_pmr_id_idx ON pool_offline_data (pmr_id);
 script                   | tx_id       | 2352 kB | script_tx_id_fkey                     | tx                | CREATE INDEX script_tx_id_idx ON script (tx_id);
 collateral_tx_in         | tx_out_id   | 1512 kB | collateral_tx_in_tx_out_id_fkey       | tx                | CREATE INDEX collateral_tx_in_tx_out_id_idx ON collateral_tx_in (tx_out_id);
 reserved_pool_ticker     | pool_id     | 0 bytes | reserved_pool_ticker_pool_id_fkey     | pool_hash         | CREATE INDEX reserved_pool_ticker_pool_id_idx ON reserved_pool_ticker (pool_id);

I've just been able to replicate the reduction in block delete on my system. Although I do wonder, given the index is updated by postgres on every subsequent table INSERT, could this not have a negative performance impact once db-sync gets past the rollback?

I didn't measure it but syncing back the ~20k blocks seemed reasonably fast.
I also don't know which of the indexes had the most impact, but the block deletions taking up to a day is really hurting db-sync users

Because Block table has a lot of foreign keys, indexes and massive size...You can drop db and restore from snapshot instead

but the block deletions taking up to a day is really hurting db-sync users

What? How many blocks are being deleted? This really sounds like something is WAY wrong here.

@erikd I suppose Deleting 20768 blocks up to slot 46823508 as in the issue description. I can confirm a similar case where
[db-sync-node:Info:119] [2021-12-09 10:18:24.02 UTC] Deleting 10453 blocks up to slot 47478285

has been ongoing for 2 hours with no progress.

@xdzurman What is your sync speed like? I have not seen a rollback that small take more than a couple of minutes.

has been ongoing for 2 hours with no progress. (Update: 3,5 hours without progress)

Pretty sure it is making progress, but that its not reporting it (because its all being done in PostgreSQL).

Also 12.0.0 should never roll back that many blocks and longer term we hope to reduce it to basically zero.

Just looking at this again:
> db-sync was stuck for many many hours here:
[db-sync-node:Info:772642] [2021-12-01 20:16:54.84 UTC] Rolling back to slot 46396728, hash 0b4e19943029e10894330f9988d25d1104b4503cd41a9f623992cd410fbeb4bc
[db-sync-node:Info:772642] [2021-12-01 20:16:54.90 UTC] Deleting 20768 blocks up to slot 46823508

by the time you see the second "Deleting X blocks" message, the rollback is complete and the rollback took less than a second.
The "db-sync was stuck for many many hours here" idea is not correct. After the rollback db-sync starts inserting blocks, but each block is not logged. db-sync tried to avoid excessive logging by only logging every block when db-sync is within 100 blocks of the chain tip. When db-sync is more than 100 blocks behind chain tip, only every 1000th block is logged.

I'm using v11 still and I have been facing a rollback of 12k blocks for more than 24 hours on 2 different servers.
Jan 13 17:53:56 dbsync1 cardano-db-sync[883]: [db-sync-node:Info:134] [2022-01-13 22:53:56.18 UTC] Deleting 12143 blocks up to slot 50544807

@frog357 If you are only rolling back 12k blocks, it should take less than 30 minutes. If you are seeing more than that check your postgres logs and configuration.

Nearly all of the suggested indices are already in https://github.com/input-output-hk/cardano-db-sync/blob/master/schema/migration-3-0007-20211022.sql .

Closing.

Just heads up, I created these additional indexes, and my DB sync time after OS restart went down from 4 hours to under 10 second after cardano-node starts. Indeed this fix works.

Keep in mind that if you build db-sync on one machine and then transfer to another, or if you download an already built binary, you must also transfer the schema folder as this is where the migrations reside.


ProjectName.WebView2 folder. Is it required?

I'm currently testing out the project, however I've noticed some odd behavior when it comes to the folder "ProjectName.WebView2" (ProjectName being the name of your Project). This folder is automatically created upon a build, however it is not created if it's published as a Single Executable.
Now. It's about to get weirder.
In the case of the Single Executable, when I start the executable, the folder "ProjectName.WebView2" is created with lots of content inside it. As long as the program is open, you cannot delete this folder.
However after you've done these steps, and then close the program and delete the folder "ProjectName.WebView2", and then re-execute the program, the folder does NOT reappear/recreate- itself, meaning it is not required?
If this really IS required. Is it possible to include it into the Single Executable somehow?

The folder is created by WebView2, obviously, and shouldn't be included if you publish your program (because it may contains your own private cache). If you delete it, it WILL be recreated the next time you run the program.
The folder is necessary in the run time, and you can test that easily: Create a file named "ProjectName.WebView2", and set it to read-only. Then the program cannot run successfully.

Thanks for answering, @Berrysoft!


Use variable in commands.

Its possible to use variable in commands?
Suppose:
var commands = {'hello :name': helloFunction};
var hello = "hello";
var commands = {hello + ' :name': helloFunction};
In this way, wont work, there is some way that i can make it work?
Thanks.

That is a JavaScript issue, not specifically related to annyang.
When constructing an object like that you can't use an expression to define a key.
You could do this instead:
var helloFunction = function() {}
var hello = "hello";
var commands = {};
commands[hello + ' :name'] = helloFunction;

Thanks @TalAter!


The format separator does not match with Carbon when using MS Sql Server

I am using MS SQL Server as my back end server and Larave 5.2 as the framework.
Every time I use toArray() method on eloquent collection I get this error
The format separator does not match

I am NOT overriding the getDateFormat method in any of my models.
But, I also tried to override the using the following trait
trait SqlServerDateTime {

    /**
     * Get the format for database stored dates.
     *
     * @return string
     */

    public function getDateFormat()
    {
        return 'Y-m-d H:i:s.u';
    }

    /**
     * Convert a DateTime to a storable string.
     * SQL Server will not accept 6 digit second fragment (PHP default: see getDateFormat Y-m-d H:i:s.u)
     * trim three digits off the value returned from the parent.
     *
     * @param  \DateTime|int  $value
     * @return string
     */

    public function fromDateTime($value)
    {
        return substr(parent::fromDateTime($value), 0, -3);
    }

}

I figured out how to fix this issue
This is what I have done. I created a BaseModel Class
<?php
namespace Vendor\MyApp;

use Illuminate\Database\Eloquent\Model as EloquentModel;

class BaseModel extends EloquentModel
{

	public function getDateFormat()
	{
		return 'Y-m-d H:i:s.u';
	}

	public function fromDateTime($value)
	{
		return substr(parent::fromDateTime($value), 0, -3);
	}

}

Then for each model I had I extended my BaseModel class something like this
 class User extends BaseModel
 {
 ...
 ...
 }

I hope this helps someone else

Thanks, man, works like a charm

Have you discovered the cause of this problem?
I don't want to create a base model, only to fix a bug. I'll think in a better solution.

This also appears to be an issue in 5.3. Creating a BaseModel class as suggested above works. Thanks!

This shouldn't have been closed as it's a workaround, not a fix? I'm seeing this behaviour with MSSQL and need to maintain compatibility with other DB drivers.

I was able to "fix" the problem by changing the created_at and updated_at column data. The last three digits must be 000. Exemple: 2017-11-21 17:28:54.933 must become 2017-11-21 17:28:54.000.

Encountered this issue in 5.5. As a workaround for writing to storage:
$now = Carbon::now('UTC')->toDateTimeString().'.000';