A few improvements for Flow type definitions

The type definitions for Kefir are a joy to read! I learned some interesting new things - in particular the option to specify a default type for a type parameter (e.g. declare class Observable<+V,+E=*>), and the $NonMaybeType<V> utility type are features that I'm sure will come in handy.
I had a few ideas for improvements. Consider this pull request to be my suggestions.
empty type parameter for vacuous observables
In cases where an observable is guaranteed to not produce either values or errors I changed the relevant type parameter from * to empty. I think that empty is the correct type, because it is effectively an assertion that something will not happen. And empty is an identity under the union operation, so certain operations will work out nicely.
For example I changed the type for constant to:
constant<V>(value: V): Property<V,empty>;

That means that the type checker will allow onError callbacks with any type for the error value argument, which is safe because those callbacks should never be invoked. And the choice of argument type in those callbacks will not affect the type that Flow infers for error values later in the program.
If a constant property is combined with an observable that with an error type parameter of some type E, Flow will infer the error type of the resulting observable to be empty | E, which is equivalent to just E, which is what we want.
more specific types for ESObservable interop
I'm planning on writing a library that consumes reactive streams, and I thought it would be nice to use the ESObservable interface so that I do not have to tie my library to one specific FRP library. I think that having a specific type for toESObservable will be handy.
fix and verify types for overloaded methods and functions
I did some testing and found that overloaded signatures were not behaving as intended. I added a lot of tests, and made a bunch of changes to overloaded functions. One trick that fixed many cases was to put a dummy argument to the end of the argument list in some signatures to help Flow to select the correct overload branch. I used $?: empty for that purpose. For example:
diff($?: empty): Observable<[V, V],E>;
diff<V2>(fn: (prev: V, next: V) => V2, $?: empty): Observable<V2,E>;
diff<V2,V3>(fn: (prev: V | V3, next: V) => V2, seed: V3): Observable<V2,E>;
diff<V3>(fn: null | void, seed: V3): Observable<[V | V3, V],E>;

The problem comes up when Flow encounters an invocation such as obs.diff((x, y) => x - y). If the $?: empty argument were not present in the first signature, then the first two signatures would both be valid interpretations of that invocation. (This is because Flow permits an invocation to provide more arguments than are listed in the function signature.) Flow tends to pick the first signature; so the type of the resulting observable ends up as Observable<[number, number], E> when the type should be Observable<number, E> (assuming that the type of the input observable is Observable<number, E>.
Using the optional argument $?: empty removes the ambiguity. There is no possible argument that matches the type empty. So $?: empty effectively forces a signature to be applicable only in cases where the method is invoked with the exact number of arguments that are listed in the signature (not including the $?: empty argument.) I chose $ for the parameter name is because it reminds me of the "end of input" token in a parser.
I did notice one use of the same trick in the existing signatures:
observe(obs: Observer<V,E>, ...none: empty[]): Subscription;

I used $?: empty instead of ...none: empty[] just because the former is a bit shorter.
It might be possible to remove the $?: empty trick in the future. I saw this in the changelog for Flow v0.46.0, which seems to imply that there might be changes to the way Flow interprets argument list lengths in an upcoming release:

Starting in v0.47.0 we're going to complain when you call a function with more arguments than it expects. You can try it out in v0.46.0 with the .flowconfig option experimental.strict_call_arity=true. For more check out this blog post

I also expanded the definition of combine to use the newish $ObjMap utility type. The new type signature is enormous - but $ObjMap makes it possible to specify type rules very precisely when the inputs are given as objects with observables as property values.

Pinging the folks who wrote and contributed to these definitions previously: @AgentME , @32bitkid , @mAAdhaTTah

I'm still reading through this, and the changes themselves, but I'm not sure I agree with the types here:
diff<V2,V3>(fn: (prev: V | V3, next: V) => V2, seed: V3): Observable<V2,E>;

I'm only picking on this one since it was in your (excellent) writeup. Without having looked at the tests, I'm not sure how this can work. The return value of the fn becomes both the output of the Observable as well as the prev value for the next function call. It can't return a type of V2 and then only accept V | V3 as the prev. Additionally, the seed type gets output by the Observable as well, so fn would have to return V2 | V3 and the method would return Observable<V2 | V3, E>.
I think this can be simplified by using a single type for all of that. seed, prev, the return value of fn and the returned Observable's generic value should all be the same type imo.

This looks promising. I'll be busy next few days, but will try to give this PR a proper look soonish.

The type definitions for Kefir are a joy to read!

Thanks, most of the credit goes to @AgentME , he did a great job!

@mAAdhaTTah I think that type is correct - diff does not use the output from fn as the input to any later fn invocations. The only values that are input to fn are values from the original observable and the seed value. For example:
> Kefir.sequentially(0, [1, 2, 3, 4]).diff((prev, next) => { console.log(`fn(${prev}, ${next})`); return true }, "foo").onValue(v => console.log('value:', v))

fn(foo, 1)
value: true
fn(1, 2)
value: true
fn(2, 3)
value: true
fn(3, 4)
value: true

@hallettj derp I confused diff and scan. nvm!

Looks really good! The use of empty is good; I think Flow didn't have it back when I first made the type definitions so I just sprinkled * around. I later noticed that I hadn't set up any of the overloads for Kefir.* functions correctly, but I had never gotten around to fixing that up.
$ObjMap used for combine seems great. You should also try using $TupleMap for the array-overloads of combine. I think it hasn't been documented yet, but it's basically just $ObjMap for arrays. Flow's internal typing of Promise.all is a good example of its use.

@AgentME Thanks! And thanks for the tip on $TupleMap. It looks like that will work nicely for several cases: combinator with an array of observables and no passive observables or combinator, zip with no combinator, merge, and concat. I would like to find a way to make $TupleMap work in the presence of  combinators and passive observables as well, but there are a couple of details that I don't know how to address at this time.
We want to be able to map values types from an array of input observables to positional arguments to a combinator. For combine that case should look something like this:
// array of observables, combinator
combine: (<E, Obss: Observable<any,E>[]>(
  obss: Obss,
  combinator: Function<$observedValues<Obss>, V>
) => Observable<V,E>)

type $observedValues<Obss> = $TupleMap<Obss, <V,E>(obs: Observable<V,E>) => V>

Unfortunately I do not know of an existing utility type that behaves like I'm trying to make Function behave in that example. I think that expressing the combinator type accurately would require a utility type with two type parameters (e.g. Function<Args, Return>) where the first parameter is a tuple type that expresses types of positional function arguments.
The issue with the passive observable case is that I do not know of any way to express a concatenation of two tuples as a type. We want this:
// array of observables, array of passive observables
combine: (<E, E2, Obss: Observable<any,E>[], PassiveObss: Observable<any,E2>[]>(
  obss: Obss,
  passiveObss: PassiveObss,
  $?: empty
) => Observable<$observedValues<Obss> ++ $observedValues<PassiveObss>, E|E2>)

type $observedValues<Obss> = $TupleMap<Obss, <V,E>(obs: Observable<V,E>) => V>

But again, I don't know of an existing Flow feature that does what I want ++ to do in that example.

That's right, "events" is described in Flow's built-in types.
https://github.com/facebook/flow/blob/master/lib/node.js
On Sat, May 27, 2017, 5:00 AM Roman Pominov<EMAIL_ADDRESS>wrote:

@rpominov approved this pull request.
Overall everything is great! Really good improvements to type definitions!
Just one question regarding the EventEmitter import...
In kefir.js.flow
https://github.com/rpominov/kefir/pull/242#discussion_r118816905:

@@ -1,5 +1,7 @@
/* @flow */

+import type EventEmitter from "events";
I wonder how this import works. We import from Flow build-ins?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rpominov/kefir/pull/242#pullrequestreview-40649202,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAllj5STi-RfopPOZlZ1Ge8RA3ChNZqks5r9-YtgaJpZM4NhsRg
.

Ah, there is a module declaration that makes the import work. OK, this makes sense.

Merged and new version released. Thanks again for the PR!


Nick: Watch vipstealth.com

Nick requests the watch of the watch_keyword vipstealth\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

vipstealth\.com has been seen in 0 true positives, 0 false positives, and 0 NAAs.

Rejected by Jeff Schaller in Charcoal HQ. Reason: 'a good idea, but superseded by a subsequent blacklister in chat: https://chat.stackexchange.com/transcript/11540?m=64645773#64645773'


⚠ Bofrost.lu has degraded performance

In 4160f65, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1198 ms

Resolved: Bofrost.lu performance has improved in ab0518c after 6 minutes.


Kubernetes-worker charm should support binding to communicate with Ceph

Kubernetes-master currently has an interface "ceph-storage" that allows to define a binding and communicate to Ceph cluster via separate network.
Kubernetes-worker requires the same in order to be able to create separate interface and provision the volumes, as well as to attach them to the workloads.

@grebennikov Thanks for filing this. Would you say this is a must-have or nice-to-have? How urgent is this for you?

Any updates on this? This doesn't only affect ceph-based solutions but also nfs-based.
Kubernetes-worker accepts the binding of NFS charm and sets up CSI plugin to use this subnet, however NFS server only receives the default binding of the worker and populates the IP default addresses of the workers as allowed into the exports. Which leads to "access denied" error when trying to mount nfs shares into the containers.


mark-github.svg icon corrupted

After this commit the mark-github.svg icon is corrupted at the top of the circle.
The difference can be seen here.
Please fix this :)

@jeejkang Thanks!

:metal: Thanks for bringing this up @jackmu95

No problem :) I needed the icon for a upcoming project I'am working on and noticed it during the editing process in Illustrator.


How to use Scan?

I am struggling to figure out how to use rueidis to perform the Scan command (https://redis.io/commands/scan/) and iterate over the response. Something that can be done using redis-go as follows:
  var cursor uint64
  var n int
  for {
  	var keys []string
  	var err error
  	keys, cursor, err = rdb.Scan(ctx, cursor, "key*", 10).Result()
  	if err != nil {
  		panic(err)
  	}
  	n += len(keys)
  	if cursor == 0 {
  		break
  	}
  }

  fmt.Printf("found %d keys\n", n)

Hi @sangnnc,
Here you go:
package main

import (
	"context"
	"fmt"
	"github.com/rueian/rueidis"
)

func main() {
	client, err := rueidis.NewClient(rueidis.ClientOption{InitAddress: []string{"<IP_ADDRESS>:6379"}})
	if err != nil {
		panic(err)
	}
	for _, node := range client.Nodes() { // loop all your redis nodes
		var scan rueidis.ScanEntry
		for more := true; more; more = scan.Cursor != 0 {
			if scan, err = node.Do(context.Background(), node.B().Scan().Cursor(scan.Cursor).Build()).AsScanEntry(); err != nil {
				panic(err)
			}
			fmt.Println(scan.Elements)
		}
	}
}

Thank you so much for the quick response. It works like a charm.
P/S: It turns out that the AsScanEntry is released in v0.0.91, which is two weeks ago.  I used the v0.0.90, therefore, I cannot find any ways to iterate a scan :D

Hi @sangnnc,
To inform you that v0.0.92 is out and contains a fix related to this issue: the ScanEntry.Cursor has been corrected to uint64. Please upgrade to the latest version.
If you just used the above code snippet I provided then no code change is required.

Before v0.0.90, I used the following code. It is dumb but worked
scan, err := client.Do(ctx, client.B().Scan().Cursor(0).Match("key*").Build()).ToAny()
if err != nil {
    // handle
}
allKeys, ok := scan.([]interface{})[1].([]string)
if !ok {
    // handle
}


Refresh Recipe, Fix Django Limits, add Maintainer - No Build Number Update - v1.3.6 was never built/published

Checklist

[x] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

https://anaconda.org/conda-forge/djongo/files
version 1.3.6 - was never built or published, not updating build number

@conda-forge-admin, please rerender

@marcelotrevisani - I noticed the current version of djongo - in the feed recipe was not published to conda-forge, and had an incorrect django pin - I have refreshed & fixed the recipe, but left the build number unchanged. Added myself as a co-maintainer too, please review and merge.
Thank You and Regards.


configuration settings for localhost example is not correct

The configuration settings for the localhost example has incorrect value. The example is to only show a tool when the connecting node is not localhost. The settings has the following:
    "conditions": [
    {
        "localhost": true
    }
    ]

Because we want to show when the connecting node is not localhost, the "localhost" value above should be 'false' instead of 'true'.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9473ecb6-7d1d-7b0f-44d3-58690fc21020
Version Independent ID: 1b922f56-101e-e217-0e4e-8bb8c386f7e5
Content: Control your tool's visibility in a solution
Content Source: WindowsServerDocs/manage/windows-admin-center/extend/guides/dynamic-tool-display.md
Product: windows-server-threshold
GitHub Login: @nwashburn-ms
Microsoft Alias: niwashbu

Adding the owner of this content, @nwashburn-ms

#please-close


Add target_dir for 3rd party subcommands

This allows third party subcommands to see the target dir
environment variable as cargo has processed it from the workspace.
Due to moving the processing to ops, this also allows further
variables to be exposed to subcommands in the future.

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @Eh2406 (or someone else) soon.
Please see the contribution instructions for more information.

Thanks for the PR. I left a comment on the issue about how I don't think this fits the subcommand abstraction.


Repo, issue-term, theme Are Not Script Attributes

I see w3c validation warnings on my page, when I use the following HTML
<script src="https://utteranc.es/client.js"
  repo="utterance/utterances"
  issue-term="homepage"
  crossorigin="anonymous"
  theme="github-light"
  async>
</script>

repo, issue, and theme are not standard attributes of the script tag. The checker will not throw a warning if data- is prepended to the attribute. Maybe, it would be better to make the API a function call class method, but still maintain backwards compatibility.

@jdanyow thanks a lot for this fix.
Shouldn't the docs update to the use data- attributes because its better and passes validation tests?

I think the validator you’re using is not up to date. There’s no requirement to prefix attributes with data-.
https://html.spec.whatwg.org/#syntax-attribute-name

I used official W3C validator.
This is the code I tried
<!Doctype html>
<html lang="en">
<head><title>Example</title></head>
<body>
<script src="https://utteranc.es/client.js"
        repo="example"
        issue-term="pathname"
        theme="github-light"
        crossorigin="anonymous"
        async>
</script>
</body>
</html>

And it fails.

Error: Attribute repo not allowed on element script at this point.

It points to this doc https://html.spec.whatwg.org/multipage/scripting.html#the-script-element
Also please have  a look this part of the doc
https://html.spec.whatwg.org/#embedding-custom-non-visible-data-with-the-data-*-attributes
My understanding is that store data or state data- is required.


Todomvc example doesn't compile

Description
I'm submitting a bug report
I have a problem with yew's examples. I've copied todomvc example to another folder, then changed the yew version to 0.9 in cargo.toml and I got errors like these:

method view is not a member of trait Component
the trait bound Model: yew::html::Renderable<Model> is not satisfied
and so on...

It seems that the latest release of yew published in crates.io  doesn't contain features/functionality that used in examples.
Could you publish a new release?
Expected Results
All examples should work with the latest release published in crates.io.
Actual Results
Todomvc example doesn't compile when using latest release 0.9.2
Context (Environment)

Rust: 1.38.0
yew: v0.9.2
cargo-web: v0.6.26

Thanks for the report @Chicker!
This is expected because breaking changes were introduced in version 0.9. The examples in the repo have all been updated to be built correctly with 0.9. Sorry for the inconvenience but until we reach 1.0 you can expect more breaking changes like this 😄


more than 31 bytes in advertsing and scanning

hello, I am using btshell and trying to send more than 31 bytes but not able to send. it shows invalid mfg_data.. but if I put less than 31 bytes in mfg_data it accepts. is there any alternative way to send more than 31 bytes?  is there anything which I am missing?
Thank you in advance

@swapkada Advertising or Scan Response data length is max 31 Octets as per the Bluetooth Core Spec 4.2

Hi,
You can use Advertising Extensions which allows for up to 1650 bytes of data. Check BLE_EXT_ADV and BLE_EXT_ADV_MAX_SIZE syscfg values.
You need scanner that supports extended advertising to be able to see those, NimBLE supports it. Also some latest top Android Phones (eg Sony Xperia XZ2) support those.


sieve doesn't give any type of ore pieces/chunks/etc

no matter which mesh i use or what i put in the sieve it doesn't give any broken ore, ore pieces, etc. the game also doesn't show ore pieces as an item available.

just went further into similar threads, i think unidict might be doing it. i will try removing that first and close this if it fixes it

it doesn't seem to have solved it

Can you run your json file through a json validator (e.g. https://jsonlint.com/) and see if you get any errors. It sounds like the game isn't creating any ore of the items.


Fix typing in Binary Heap

The valtree in the binary heap is not strictly typed. In some codes I have, this causes a 2x performance reduction in the entire code (<.1% is data structure related) due to checking if a binary heap is empty. The simple fix is to make it Vector{T} instead of the non-concerete Array{T}.
Reference for performance problems: https://github.com/JuliaDiffEq/StochasticDiffEq.jl/issues/13

Current coverage is 95.85% (diff: 100%)

Merging #263 into master will not change coverage

@@             master       #263   diff @@
==========================================
  Files            30         30
  Lines          2220       2220
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
  Hits           2128       2128
  Misses           92         92
  Partials          0          0

Powered by Codecov. Last update 52b74de...000fcbc

Thanks!

Could a patch be tagged with this change? This'll have a large effect on some of the main DifferentialEquations.jl methods. It's blocking a release which inlines a few more things which previously needed a function barrier to stop the propagation of this type instability.

Though I understand if you want to wait until after you migrate to JuliaContainers

I'll try to tag this today. The move to JuliaContainers shouldn't affect tagging, as a redirect will be in place.

Thanks!

@ChrisRackauckas, wondering if this is urgent or can wait a few days to get some additional changes in?
Reasoning isn't JuliaContainers (or whatever org that becomes), but more that there are changes in SortedDict constructors that warrant a minor version bump, as well as some upcoming changes in SortedSets and SortedMultiDicts (with pull requests) that warrant a minor version bump, and I would rather only bump the minor version once.
If this is urgent, I can just this change on a branch without the SortedDict changes and tag (cc: @tkelman), and then tag all of the sorted container changes at once (this commit will still be in the history, albeit with a different SHA).
Let me know your thoughts.

It's not urgent. If it takes a week or two I wouldn't mind. I mentioned it because the tag frequency in this repo looks unpredictable, and that is a small enough change that I was hoping to make sure that it wouldn't just sit two months waiting for a larger change to happen. But since it sounds like you have something else on your plate that you're churning through, I can wait.

If it starts to take a while (say longer than a week), bump this and I'll
put it on a branch and tag it. But my hope is that the other changes are in
over the next few days.
On Sun, Jan 29, 2017 at 12:38 PM Christopher Rackauckas <
<EMAIL_ADDRESS>wrote:

It's not urgent. If it takes a week or two I wouldn't mind. I mentioned it
because the tag frequency in this repo looks unpredictable, and that is a
small enough change that I was hoping to make sure that it wouldn't just
sit two months waiting for a larger change to happen. But since it sounds
like you have something else on your plate that you're churning through, I
can wait.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/JuliaLang/DataStructures.jl/pull/263#issuecomment-275944133,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANoElTbL7rrAfXpVffIAPxT8j96lhi9ks5rXPjhgaJpZM4Lwbvf
.

Thanks!


QoL: Copy-Paste overwrite existing arrows

Currently, pasting a selection of notes over an area containing notes will mix both selections' notes together, instead of replacing
A chart containing:
L---
-D--
--U-
---R
In clipboard:
L---
--U-
-D--
---R
After a paste:
L---
-DU-
-DU-
---R
Unsure if this behavior is intended, but having a way to overwrite instead would be nice to save time deleting regions of notes manually.

Added in next update: use Shift+Paste


Dotnet4

@dxwu mind looking over this PR for me?

Coverage decreased (-0.2%) to 59.807% when pulling 8dc5be38e2f27ba786d09ad35245e26836d2f560 on dotnet4 into aecda2636b086cda0aec01bfb8d9726903a34418 on master.

Coverage decreased (-0.2%) to 59.807% when pulling 8dc5be38e2f27ba786d09ad35245e26836d2f560 on dotnet4 into aecda2636b086cda0aec01bfb8d9726903a34418 on master.

looks awesome! I'll be working on using this for a net40 project today

Coverage increased (+10.03%) to 70.032% when pulling 0638efed7f594b297e249f073ae0f8ac28f43916 on dotnet4 into aecda2636b086cda0aec01bfb8d9726903a34418 on master.

Coverage increased (+10.03%) to 70.032% when pulling f5f2ac4d18fcf25eed83d4a44902ed5d3d029c58 on dotnet4 into aecda2636b086cda0aec01bfb8d9726903a34418 on master.


Feature: Shared State with react native and react

for example. App inject the token, userInfo, business id, business info, web get the state; and web can change it, app will auto useEffect the change of any state.
It's a common usecase for every webview hybrid app.

Thank you for your suggestion!!!
I am currently developing the same feature you mentioned.
My goal is to implement two-way binding between the web and React Native.
Of course, for the web part, I am considering a reactive state that does not depend on specific frameworks like React, Vue, or Svelte.
It's likely that libraries will be developed for integrating state across different frameworks as well. Examples could include @webview-bridge/react, @webview-bridge/vue.

I'm working on a design, it's not finished, but this is probably how I'll use it.
React Native Part
// here
export const appStore = storeBridge({
  count: 0,
});

// here
export type AppStore = typeof appStore;

export const { store } = createWebView({
  // register store bridge
  bridge: bind([appMethod, appStore]),

  // ... default options
});

const count = useSharedStore(store, (state) => state.count);
const setStore = useSetSharedStore(store);
setStore("count" , (prev) => prev + 1);

Web
import { linkNativeStore } from "@webview-bridge/react"; // or "@webview-bridge/vue"

const nativeStore = linkSharedStore<AppStore>();

const count = useSharedStore(store, (state) => state.count);
const setStore = useSetSharedStore(store);
setStore("count" , (prev) => prev + 1);

shared state was released in v1.3.1 !!!!
Released a library to integrate with state in vanilla, react, and vue.
https://www.npmjs.com/package/@webview-bridge/react
https://www.npmjs.com/package/@webview-bridge/vue
https://gronxb.github.io/webview-bridge/shared-state/react-native.html


Add the Rust compiler version to the bug report metadata

Motivation
Bugs like #5091 only happen on some Rust compiler versions, so it would be useful to have the compiler version in the startup and bug report metadata.

Hey team! Please add your planning poker estimate with Zenhub @arya2 @conradoplg @oxarbitrage @teor2345 @upbqdn


olivetti 1.5.3

Created with brew bump-formula-pr.

Requeuing these bad boys.

This is new and exciting: https://travis-ci.org/Homebrew/homebrew-emacs/builds/157018403#L4791

Since that error's only on Linux, let's just ignore it

.

:ship:'d


Typo in tf.keras.callbacks.EarlyStopping example

URL(s) with the issue:
Please provide a link to the documentation entry, for example:
https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/MyMethod
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
Description of issue (what needs changing):
The following is from the EarlyStopping page on TensorFlow website.
callback = tf.keras.callbacks.EarlyStopping(monitor='loss', patience=3)
This callback will stop the training when there is no improvement in the validation loss for three consecutive epochs.
In the above line, it should have been "loss" instead of "validation loss".
Clear description
Check above. Please check this gist for an example.
Correct links
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
Submit a pull request?
Yes

I am closing this issue as this was resolved. Thanks!


expose parsed id_token

So that one can actually get at the sub needed to identify a user

Take a look at the uid that the strategy exposes in the auth hash. Seems this comes from the sub claim on the access token. I would hope that these would be the same on both the ID token and Access token...

I no longer have access to the project I was doing this for.  Will check out if this works if I ever have need again.


Colons and spaces in paths are not fun

Description
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Original code results in deadline build task error:
python: can't open file '/sessions/session-7f6d345e82f64e679a7d24ef64113912wj_pyojk/assetroot-24770c319a347a3ab6b1/.deadline/job_history/giadeadlinemonitor-us-west-2/2024-08\n.\nlinux-64/scripts/openjd-vars-start.py': [Errno 2] No such file or directory

Hi there Jason. Thank you for taking the time to fix this up.
Would you mind making two quick modifications, related to the CI failures, to your commit so that we can merge this:

The first is the DCO failure. You can get to instructions on how to resolve that by clicking the 'details' link on the DCO line in the CI checks of this PR.
The second is for the Semantic PR CI failure. Please change the commit title to match semantic PR formatting. In this case, I'd suggest changing the commit title to fix: replace colons and spaces in conda recipe job name


Documentation for class attributes using triple-quotes doesn't show up at all

Describe the bug
No documentation is generated for attributes, even when a triple-quoted string is placed underneath it.
Example code where no attributes are generated:
class MyClass:
    def __init__(self):
        self.a = "b"
        """Example"""

To Reproduce
Steps to reproduce the behavior:

Create a Python class
Define an attribute in the init
Add a triple-quoted string below it
No documentation is generated

Expected behavior
Documentation would be generated in an Attributes table as stated in the README.
Information (please complete the following information):

mkdocstrings version: 0.15.2

Hi, thanks for the report. Indeed, pytkdocs parses the assignments and triple-quoted strings in the __init__ method, but then it has no use for them, since these attributes do not appear as members of the class itself, and pytkdocs does not append an Attributes section to the class' docstring either. Not sure what we should do here.
The workaround is of course to add an Attributes section to the class docstring yourself. Annotations in the __init__ methods will be used in that case.

If I'm already adding an Attributes section to the class docstring, is there any use at all for adding these triple-quoted docstrings after the variables? Will it take them and use them in the sections automatically?

No it will not :confused: It effectively makes attribute docstrings in __init__ methods useless :confused:

No it will not 😕 It effectively makes attribute docstrings in __init__ methods useless 😕

Alright, got it 😔 I'll stick with an attributes section for now - thanks for the help!


Grid column header errors on hover

Short description
When you hover over a grid column header there is a JavaScript error occurring that causes tooltips to not show for the grid column header.
What is current behavior
Grid column header tooltips do not show when you hover over the grid column header.
What is desired behavior
Grid column header tooltips will show when you hover over the grid column header.
Link to jsfidle/codepan with sample code
Simple Grid Demo
Steps to reproduce or sample
Hover over any of the grid column headers in the sample grid demo.  If you open the console you should see an error stating: Uncaught TypeError: Cannot read properties of undefined (reading 'columnTooltipShow') at HTMLTableCellElement.onmouseenter

I cannot reproduce the mentioned error, neither with https://w2ui.com/web/demos/#/grid/1 nor with https://w2ui.com/web/demos/#/grid/31 - I tested with Firefox and Chrome.
When you check your browser's debugger, what w2ui version do you see? Maybe your browser uses a cached version?
The current version is w2ui 2.0.x (nightly) (1/8/2022, 4:08:28 PM)
https://rawgit.com/vitmalina/w2ui/master/dist/w2ui.min.js

I think it was a caching issue as to why the simple grid was giving the error.  However, it still seems to happen for the ES6 grid here: https://w2ui.com/web/demos/#/combo/14.
I found that if I go to the ES6 grid first then the error happens on all the other demo grids.  So I think there is only really an issue with the ES6 grid.

True, probably because now there's 2 versions of w2ui loaded at once.

Fixed. Now es6 version loads in <script type="module"> and no longer ovewrites global grid


semicolon not needed

Best utilized in compound statements, but not Python line endings.

Ups 😳 this was my first project in python; I was still living the JAVA and C/C++ mindset.


Clear alarm register 3 on SMR 5.5 devices

Signed-off-by: Gerben Kroes<EMAIL_ADDRESS>

https://jenkins.fdp.osgp.cloud/view/Nightly Build/job/gxf-nightly-build/558/


Styles.makeStyles cannot infer that the default 'props should be unit

The following doesn't compile
let useStyles = Styles.makeStyles <| fun theme ->
  {|
    root = Styles.create [
      style.height 48
      style.padding(0, 30)
    ]
  |}

unless it is actually used, I was trying to fix the compile error when I was defining the value but the workflow is that I use it somewhere and the compile error would go away. I guess it is OK but this is sub optimal UX. Would there be a way to make add an overload that by default returns a function that takes a unit instead?

Unfortunately that won't work, at least not with the same name.
If I have only the unit overload as a proper type member and the rest as extension members, it will resolve to the unit overload, but the usage later can't change it to non-unit.
If I have both the unit and the generic 'props overload as proper type members, overload resolution fails.
Further tips are welcome!


GPIO now migrates fine (#374), now looking at SPDK migration (#218).

@jlevon I've addressed your comments, please take another look.

@jlevon I've addressed your commetns, please take another look.

@jlevon this PR contains the following diffs that could be individual commits as some of the are not strictly related to each other:

document how to use a vfio-user device with libvirt
document how to use SPDK's nvmf/vfio-user target with libvirt
replace vfio_bitmap with vfio_user_bitmap
replace vfio_iommu_type1_dirty_bitmap_get with vfio_user_bitmap_range
fix bug for calculating number of pages needed for dirty page bitmap
align number of bytes for dirty page bitmap to QWORD because this is how QEMU operates
add debug messages around dirty page tracking
only support flags=0 when doing DMA unmap
set device state to running after reset
allow region read/write even if device is in stopped state
allow transitioning from stopped state to running state
allow transitioning from stop-and-copy state to running state
fix unit tests

This is because they were discovered and fixed incrementally so it wasn't easy to use the workflow where we create a chain of individual PRs where the next one is merge into the previous one. Shall I create a chain of PRs out of this PR, keeping the content the same such that a diff between the last PR of the chain and this PR is empty in order to make sure no additional review is required?

I'd just merge this as one commit, it's not that big. I don't mind if you want to separate it out into at least some parts though.


Add README

This should include a description of the included packages, a list of useful keybindings, and information about nonstandard configuration and/or hacks.

For when this is written, it should include the following useful setup tips:

The setup script should not be affected by environment variables, but it will not clear the environment after it finishes. So if you had a very weird environment before running the setup script, you might have to start a new shell session entirely (the one started by the setup script will not clear your environment) in order for things to work correctly.
The setup script has no effective way of checking to see if you have too-old or too-new versions of your Emacs packages installed. If you experience problems with Emacs, you may want to wipe your ~/.emacs.d/elpa directory so that the correct versions of all packages are downloaded.

Some progress was made in edfbb1e1fc7e287c5f03b07498cd21b811230593 and 82d619f26ae6c1274318d0f059188c2c47c1b7ee.

The Emacs reference is mostly finished as of 40efa664fbd6d573a78ab92777c40efc4de988de.

Since the README is added, and further tasks are mutually orthogonal, this issue has been broken up into #115, #116, #117, #118, and #119.


chore: add typescript 5.1.3 to versions

5.1.3 was released last week https://github.com/microsoft/TypeScript/releases

Type of change

Chore (any other change that doesn't affect source or test files, such as configuration)

Closing in favour of https://github.com/aspect-build/rules_ts/pull/376


chore: import repository https://github.com/Morgan-and-Adam-will-figure-this-out/testing-jx.git

this commit will trigger a pipeline to generate the CI/CD configuration which will create a second commit on this Pull Request before it auto merges

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign
You can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Added an IsDisposed method to LiteDatabase class.

I created this property because currently we do not have a way to check if an existing instance is still valid.

This is an unusual pattern... Why just set a reference to null, as soon as it is disposed (more used pattern)?

Out of curiosity, what is the use case? Why not to set a reference to null, as soon as it is disposed (typical pattern)? And instead of checking for this property check for null.

If you want to use a single point for the connection (inside a service for instance), it might be in a currently disposed state after an action performed inside an using statement, but that instance is still alive/valid after that so it does not have a null value. Having this property available you can perform the necessary checks and only instantiate a new one when is actually needed.

If this PR is accepted then this property should be added to the interface ILiteDatabase as well.
Another issue/question: if this approach is legit and needed then same properties are needed in similar types ILiteEngine, LiteEngine, SharedEngine, and LiteRepository. Otherwise introducing it in just database is going to be somewhat inconsistent. All these types may be used in similar scenarios.

TBH: I don't see a need for this change. You can simply write/use a wrapper class (which might even be derived from LiteDatabase) and this code doesn't really add any functionality to LiteDb itself. Continuing using a disposed object seems like a bad thing to do.


Cache client credentials.

Description of change

Guide to reproduce test results

Checklist

[x] I have self-reviewed this change.
[ ] I have tested this change.
[ ] This change is covered by tests that are already landed, or in this PR.

@iainjwhiteside This is super gross, open to suggestions on how to improve it.  I think the only good way to improve it is to completely restructure the CLI, which I didn't want to do in this PR, so I left a global in.

Approved, but agree the global is unfortunate


dotnetcore/02.echo-with-counter

[x] Is numbered and named correctly and following the spec
--
[x] All files have been spell checked
[x] All style cop/ linting issues have been addressed
[x] Include all DCR changes
[ ] Explicit state save
[ ] Method name changes begin, end etc
[ ] Does not use mainDialog unless that file actually extends Dialog/ ComponentDialog
[ ] Compiles with latest packages
[ ] Runs per instructions in README
[ ] Add commented out code to use blob storage
[ ] Update to look for production/ local (bot configuration)
[ ] onTurnError at adapter
[ ] Include .env or appsettings.json or web.config with botFilePath and botFileSecret
[ ] Deployment scripts folder has been cleaned up and only includes msbot export files
[ ] README to include
[ ] msbot clone steps
[ ] Tools install/ uninstall
[ ] Concise set of steps to run the sample
[ ] C# sample -- Include .bot file in the build output
[ ] All samples include the latest botfile (no "secretKey" field; replaced with "padLock"). Use latest MSBOT off the tools MyGet feed to generate the bot file
[ ] remove and sort usings
[ ] remove azuredeploy.* under deploymentscripts folder (if any)
[ ] Welcoming users is consistent & correct. See 13.basic bot for nodejs & 3. welcome user for C#

Daves PR #348 handles this


Second draft of changes to generate Lexicon regexp in JSON data-format

Fixes #10614
Status
In development
Description
Change code base to provide Lexicon regexp in JSON data-format. The code is independent from external libraries and can be used as simple as following:
python Lexicon.py > lexicon.json
# and it produces the following format, the length -1 means no restrictions
[
{"name": <object-name>}, "regexp": [list of regexps], "length": int},
...
]

Is it backward compatible (if not, which system it affects?)
YES
I adjusted the code such that it is fully backward compatible, and yet can be used outside of WMCore.
Related PRs
#10615
External dependencies / deployment changes
None

@yuyiguo , @amaltaro , @todor-ivanov , @klannon This PR contains second draft proposal to use common Lexicon data-format. After the initial evaluation I think this draft does not have any downsides and fully backward compatible with existing Lexicon. It also generalize better then original PR #10615 , since the format now contains static attribute names and include length of regexp objects. My vote would be to use this proposal.
Please review and provide your feedback.

Valentin, thanks for providing these changes.
I have a slightly different opinion on how it should be implemented though, which is: the current "dumpRegex" changes look fine to me, but I would only use it to generate the Lexicon rules. In other words, once we have the JSON files, the changes provided here are no longer relevant.
We do have to add logic to load the json file in Lexicon though, to be done as soon as any import Lexicon is done.
Regarding the flat JSON format, with static key names, it's indeed more organized and clear, but then it requires iterating over that dictionary every time we want to execute one of the Lexicon functions. Here is where I would be inclined to have a "dynamic" key name such that we can have direct access to whatever rule we want. If we stick to the flat JSON - which I'm pretty sure it's your preference - then we could create a helper function to lookup at the rule we're looking for, which wouldn't be too bad, besides wasting extra CPU cycles.

Alan, please say explicitly what do you want to implement (if any). From your message I think you want:

a function which reads the given json lexicon file and provide dict like look-up of any key

Please note, the static key JSON has nothing to do dynamic keys. The former represent the data, the later is helper function which can be provided. We may have a function which reads the JSON file, then build the dict once and everybody can use. There is no extra CPU cycles involved.
Please reply with concrete action you want to see in this ticket.

a function which reads the given json lexicon file and provide dict like look-up of any key

That's exactly what I had in mind.

Please note, the static key JSON has nothing to do dynamic keys. The former represent the data, the later is helper function which can be provided. We may have a function which reads the JSON file, then build the dict once and everybody can use. There is no extra CPU cycles involved.

Not necessarily, if you simply load your JSON into a python dictionary, you stick to that format (which can or cannot have dynamic keys).
But you just mention something that I had not thought about, indeed we can have a flat and static JSON file, load and parse it into a function (global variable?) and then provide the dictionary key look-up.

@amaltaro, that's exactly my point, the data format should independent from language. They allow to exchange data among applications and different implementations, therefore they should be always generic. If application wants specific structure then the data loaded from generic format can be converted to that structure.
Ok, I'll post an update with load function and a dict you so much desire, but I'll preserve the independent data-format.

Here is second iteration. Now it contains two functions:

dump_regexp to dump Lexicon data into stdout or given file name
load_regexp to load Lexicon data from JSON file and either return it or provide two dictionaries (one which contains key and patterns, and another which provides keys and lengths).

The upstream code can decide how to keep loaded data, it can loaded into an object (within a class), or global variable. Here is an example:
# dump the lexicon data to stdout
python Lexicon.py
[{"name": "logical_file_name", "patterns": [...], "length": 500], ...

# dump lexicon data to a file
python Lexicon.py > /tmp/lexicon.json

# load the Lexicond data from a file
from Lexicon import load_regexp
fname = '/tmp/lexicon.json'
data = load_regexp(fname)
print(data)
[{'name': 'logical_file_name', 'patterns': [...], 'length': 500}, ...]
rdict, ldict = load_regexp(fname, convert_to_dict=True)
print(ldict)
{'logical_file_name': 500, ...}
print(rdict)
{'logical_file_name': [...], ...}

Please review and let me know if it satisfies your view

@amaltaro , @klannon, @yuyiguo  could you please revisit this PR and associated feature request https://github.com/dmwm/WMCore/issues/10614 that I can apply changes and move forward with dbs2go migration to use common lexicon. The PR is already 6 weeks old and I was expecting that at least it would be properly reviewed during this time that we can move forward.

Sorry for the delay, Valentin.
My opinion is that we should not push this code to production as is. Instead, I'm in favor of getting a dump of all the regular expressions that we need, and adapt the Lexicon code to use those patterns from the external file. In other words, I would not push to production this parameter return_regexp

@amaltaro , could you please convert your opinion to practical set of actions. So far I don't know if it is feasible to make backward compatible code with what you're suggesting. For instance, plenty of function from Lexicon.py has hardcoded regexp, e.g. def searchblock. How you can feed into that function? How you can generalize def lfn function from external file, it has quite complex flow of if statement, where majority of regexp are dynamically created by the code? I want to be realistic and provide changes which will allow the transition without major interruption and code-refactoring of multiple places (who knows where exactly in WMCore/DBS these functions are used). The PR I provided can achieve both goals, i.e. preserve current behavior Lexicon.py file and provide ability to dump regexp's into external file.

@vkuznet Valentin, what I have in mind is something like.
First:
Generate a json file like (basically with your current changes in place):
[{"functionName": "blahA", "length": 123, "regexSet": ('^[a-z]', '^[A-Z]', 'blah')},
 {"functionName": "blahB", "length": None, "regexSet": ('^[A-Z]', 'blah')},
...]

Second:
Load rules in memory at global scope and change its structure for easy look-up (inside Lexicon.py)
REGEX_RULES = {}
with open("./Lexicon_rules.json") as fp:
    for item in json.load(fp):
        REGEX_RULES[item.pop("functionName")] = {}
        REGEX_RULES[item[functionName]].update(item)

Third:
Update the Lexicon source code to look like:
def searchblock(candidate):
    rules = REGEX_RULES["searchblock"]
    for item in rules:
        # NOTE: we could iterate through the list of regex values, even if the function is meant to have a single regex
        # at least we would guarantee that whatever changes made in the future to the json file, would be taken into account here
        return check(item["regexSet"][0], candidate, maxLength=item["length"])

def lfnBase(candidate):
    errorMsg = "LFN candidate: %s doesn't match any of the following regular expressions:\n" % candidate
    rules = REGEX_RULES["lfnBase"]
    for item in rules:
        for regex in item["regexSet"]:
            try:
                return check(regex, candidate, maxLength=item["length"])
            except AssertionError:
                errorMsg += "  {}, with length: {}\n".format(regex, item["length"])
        raise AssertionError(errorMsg)

See my comment in the snippet above as well. It's basically a tradeoff between robustness vs simplicity/performance.
What do you think? Do you see possible possible improvements?

@amaltaro I doubt it can work in general way. Here few flaws in your proposal:

where is default location of lexicon.json file? Is it current directory, env variable or should be provided externally?
some regexp contradict each other, e.g. processed dataset is defined as independent at line 27 (see https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Lexicon.py#L27) and as part of lfnParts dict, see https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Lexicon.py#L36. And, those regexp are different. And, neither belong to specific function. Therefore, I can't conclude from your proposal how to combine them in general form since it seems to me it requires different structures, with and without function name, independent and part of another structure
you're mixing function and global variables, e.g. searchblock is a function, while lfnBase is a module (global) variable. The latter, is used in other functions to construction some regexp.
the current codebase uses mixture of global variables and functions, e.g. there are two functions searchblock vs block. The former uses internally defied regexp, while latter relies on global variables. But both represent a block regexp patterns. How to do a separation of patterns in this case?
does the code should be backward compatible, I have no idea where it is used (it can be used in CRAB for instance) and I have no idea which functions are used. Therefore, if I strip off all hard-coded regexp from the code and I can't properly assign them into different parts (see above example with processed dataset regexp) I don't know if code will behave correctly for different clients.
I also don't see any needs to keep function name as part of data-structure since regexp does not represent functions, they represent the data. For instance, there is a SEARCHDATASET_RE, there is searchstr and dataset functions, but all of them represent different dataset patterns. How these variables and functions are used I have no idea.

Bottom line, I don't want to do a third iteration until clear idea is outlined how to represent regexp's. In my view they should represent the data, e.g. list of regexp for all possible dataset values, etc. Since the function defined for a specific use-cases, e.g. searchst vs dataset functions, they use different subset of regexp set.
I think it should be only two APIs: how to load lexicon.json and how to generate it. Then, all function should be cleaned up to use a set of regexp associated with metadata parameters, a block name. Although this will require to change functions and will  break backward compatibility, it will allow to have clear definitions of regexp as a list of allowed expressions. Therefore, I suggest you provide further clarification how to resolve all aforementioned issues.

I think it should be only two APIs: how to load lexicon.json and how to generate it. Then, all function should be cleaned up to use a set of regexp associated with metadata parameters, a block name. Although this will require to change functions and will break backward compatibility, it will allow to have clear definitions of regexp as a list of allowed expressions. Therefore, I suggest you provide further clarification how to resolve all aforementioned issues.

Valentin, your sentence is a summary of what I am proposing.
And yes, I'm planning to have this backwards compatible, thus not affecting anything importing from Lexicon. The only and sole exception would be if someone is actually importing a regex pattern instead of the validate function; then I agree it would not be backwards compatible (however, that would likely be a wrong usage of the Lexicon module!).
The comment I made above should answer some of your concerns. In addition to that, if the same regular expression is used by multiple validate functions, we just need to define that regex for both functions.
I agree that regex represents data (or a standard for the data), however, we need to have a way to map which standard is used by which function, hence my suggestion for that json structure.

@amaltaro please provide provide concrete plan, what you proposed is not complete since I pointed out to its flaws. Therefore, before implementing stuff I need the following:

should we decide on default location of Lexicon.json path, should be take from env, should it be provided to the script
should script load lexicon.json during import lexicon or not? If yes, see above.
if function coverage is required, then we need to define proper data-format for lexicon.json, e.g.

# proposal 1 (data driven, as I define it, but it provides a way to extract from full list of patterns the ones required for function names)
[{"name": "dataset", "patterns": [ full set of regexp], length: int}..., "functions": [{"funcname": string, "funcindexes": [ ids of patterns to use in funcname], ...] ]

# proposal 2 (function driven, as you define it):
[{"name": "dataset", "patterns": [ list of regexp used in func], length: int}..., "funcname": string]

# other proposals ....

The proposal 1 above can be used to generalize the code and json, while in proposal 2 it is not possible.
I suggest that you at least clarify these two main issues before starting another implementation.

@amaltaro please provide provide concrete plan, what you proposed is not complete since I pointed out to its flaws. Therefore, before implementing stuff I need the following:

@vkuznet please do not get stuck waiting for my reply. If effort is low enough and you think you can propose something, please go ahead and bring this up for discussion/review. I count on you and the rest of the team to come up with plans as well ;)

should we decide on default location of Lexicon.json path, should be take from env, should it be provided to the script

Having an environment variable isn't a bad idea (LEXICON_RULES_PATH ?), but I would not enforce it. And if not set, I would just try to load it like with open(".lexicon_rules.json") as ..., and if that fails, then we should raise an ImportError or something like that.

should script load lexicon.json during import lexicon or not? If yes, see above.

Yes, the way I project it is that someone will import Lexicon - just as currently done - and this will bring in the complete validate functions (which will internally access the local module rules cache). Let me know if I'm missing something here though.
For DMWM projects that do not want to have our Lexicon implementation, I assume they would simply pull the lexicon_rules.json file when building/packaging their system. Right?

if function coverage is required, then we need to define proper data-format for lexicon.json, e.g.

Can you please clarify what is your name key/value pair? Your proposal #2 is basically the same as what I proposed a day or two ago, besides the name key/value. I'd like to have a KISS principle here, thus, one rule line/item for each function. If the same regex is used by multiple functions, it's more clear to me to have that defined under each "function's rules".
Thanks for following this up, Valentin.

@amaltaro I provided new PR with new format. Please have a look at #10764 . The PR only contains code changes, but does not yet provide lexicon.json file which I'll generate later if we agree on code structure.

Okay, so the name attribute is meant to refer which information those rules are meant to, e..: "dataset", "request name", "lfn" and so on. I like this idea to characterize the rule being defined!
However, I have to disagree with your structure above, which I paste here as well:
[
{
  "name": "dataset",
  "patterns": [ "^/[0-9a-zA-Z]{1,99}.*", "/[0-9a-zA-Z]{1,99}/[0-9a-zA-Z]{1,99}/[0-9a-zA-Z]{1,99}", ...],
  "length": 200.
  "functions": [
       {"name": "searchdataset", "indexes": [0],
       {"name": "dataset", "indexes": [0,1],

   ]
},
...
}

and the reason is basically because it does not look simple! It's a very nested structure, where indexes matter and it will be more error prone whenever someone needs to update the json content.
I would like to proceed with your proposal #2, which is basically the same structure I suggested in one of my first comments, plus the name key/value pair. We can also rename name to attribute, or attributeType, since name is something extremely generic and vague.
Last but not least, I also agree with your suggestion of -1 for no length limit (contrary to my None suggestion). Thanks

Oh, on top of my reply above, we should make funcName a basic string, instead of a list.

@amaltaro , the structure I proposed is not complex and not nested. It is well defined schema.  The tool will parse it to provide representation of lexicon suitable for specific use-case, e.g. WMCore or Go based applications. Please see load_lexicon in PR #10764. It is not complex at all and can return you any number of representations. With such structure the lexicon code becomes very trivial and very generic.
Anyway, the proposal #2 cannot be generalized. here it is for clarify
[{"name": "dataset", "patterns": [ list of regexp used in func], length: int, "funcname": string}, ...]

And, the reason it can't be generalized across applications because it is function driver (which defeats the purpose of lexicon which you agree should be data driven), i.e. each dict correspond to function used in Lexicon.py. For instance, if we have two functions associated with some attribute name, then we'll have two different entries, e.g.
[
   {"name": "dataset", "patterns: ListOne, "function": "searchdataset"},
   {"name": "dataset", "patterns: ListTwo, "function": "dataset"},

]

But both correspond to the same name dataset. So to get full list of regexp for our dataset we need to merge ListOne and ListTwo, and make a set. This is because the format is function driven and not data driven (as I proposed).
This is NOT the case I use in Go where I don't use python function, and I don't need them. I need full set of regexp rules. Of course, I can resolve this in Go by writing additional code to get all function entries for a given name and then make a union of regexp rules. As such this proposal is Python based, moreover is python based on existing Lexicon.py codebase and has not represent generic list of regexp rules.
As I said please decide on global meaning of lexicon, is it function driven or data driven. Then, the format should reflect this decision. If you worry about adding entries to lexicon.json, then we can provide a tool for that, e.g.
update_lexicon -pattern "regex-pattern" -function "funcname" -length <optional int>

Such tool can update lexicon.json accordingly to avoid manual mistakes.
Since we decide how to make lexicon to be generic I still think it should has data driven format. The functions are legacy artifact of Python code and only required for backward compatibility. Otherwise they have no real meaning since applications should only care about regexp rules.
Anyway, I will still leave final decision to you and will adjust code accordingly. Please evaluate properly the decision (not only from Python point of view but rather how we can use new lexicon format outside of WMCore). Once you decide about it please post final format to avoid confusion and I'll use it and adjust PR #10764 accordingly.

Valentin, if we come back to the original goal and requirements for this new feature, which were:

separate Lexicon data from the Lexicon source code/implementation
having a portable format to any other project in any language
(bonus) have all these changes made in a backwards compatible mode

then all these proposals can meet those requirements.
The same way WMCore/Lexicon can adapt to parse the nested structure you propose, the DBS Go implementation does for the proposal with the validate function name, which I'm advocating to be adopted. The main reason I am against your schema is that I know it will cause issues in the future, that function vs indexes in a list is evil and will definitely hurt us at some point!
If we wanted to make this schema really generic, we should actually ditch the function name from the schema and completely refactor WMCore/Lexicon accordingly. However, I do not think it's worth all that effort.
Anyways, CRAB also uses Lexicon heavily, so perhaps Stefano wants to provide some input on this discussion; and hopefully we can move forward here. @belforte

@belforte Stefano, if you have any opinion on this matter, please do share it with us (sorry for the long thread/discussion as well). Otherwise, let's converge on a final implementation by our next team chat, Valentin.

@amaltaro @vkuznet thanks for keeping me in the loop.
IMHO it would be important to preserve the current inteface, e.g.
 from WMCore import Lexicon
        Lexicon.lfn(testLfn)  # will raise if testLfn is not a valid lfn

that's what I'd call 'backward compatibility' and I'd like to have.
How Lexicon.lfn internally gets the regexp to use in the check, whether it is from reading JSON, or having it hardcoded, or using a cached dictionary... I do not care.
I guess there'd be no problem in changing the above to something like:
 from WMCore import Lexicon2
        Lexicon = Lexicon2()
        Lexicon.lfn(testLfn)  # will raise if testLfn is not a valid lfn

in case having an object instantiation helps for internal caching.
I do not believe that I can give helpful comment on "how to implement".

@todor-ivanov your input is always welcome, please don't wait for an explicit request and feel free to share your thoughts on this as well ;)

Thanks for mentioning @amaltaro . I was going through them both the code changes and the comments, and in all that I can mainly agree on few (high level) remarks.

It is really important to preserve the backwards compatibility (well expressed by @belforte )
The flat dictionary structure does seem to be a good thing in this case

Other than that... I cannot find anything which is not looking good in the code.

@vkuznet Sorry for the delay getting back to this, Valentin.
It looks like we have a consensus on the requirements for this feature, but not yet on the schema for the rules.
So, I would like to proceed according to the comments provided in this comment: https://github.com/dmwm/WMCore/pull/10622#issuecomment-899240578
with the extra data name attribute that you suggested.
So, the final flat JSON rules file would look like (I'm suggesting attrName instead of name, but we can discuss it as well):
[{"attrName": "dataset", "functionName": "blahA", "length": 123, "regexSet": ('^[a-z]', '^[A-Z]', 'blah')},
 {"attrName": "lfn", "functionName": "blahB", "length": None, "regexSet": ('^[A-Z]', 'blah')},
...]

the client load these rules can then:
a) either decide to build the data rule map based on the functionName (our case in WMCore Lexicon)
b) or decide to build the data rule map based on the attrName (likely the case for DBS2Go)
Can you implement these changes (either in this PR, or in one of your other PRs)? Valentin, please also take into consideration the recent changes to Lexicon, provided in this PR:
https://github.com/dmwm/WMCore/pull/10799
Once we have the new set of changes, we will be closer to converging into the final details and getting these changes in for the October release.

should be the proper time to make sure this is also fixed: https://github.com/dmwm/WMCore/issues/10810

should be the proper time to make sure this is also fixed: #10810

This is going to be a big change, and mistakes can be easily done. So, I would rather have the new model in place first (which can be tested by all the unit tests we have - without a single line change), and only then work on the new GH issue. I'm happy to take that issue, once these changes are behind us ;)

@amaltaro before going to implementation (I really would like to avoid writing 3rd time and then we'll decide to rewrite it again), one more detail you should clarify. How in your flat proposal you address multiple functions for the same attribute. For instance, current Lexicon.py has def searchdataset and def dataset, as well as def searchblock and def block. In my view they are redundant since searchdataset (searchblock) is part of dataset (block), respectively. I only worry that it will lead to confusion and inconsistencies. For instance, if someone update searchdataset regexp but does not update dataset one. In your function based approach it will lead to wrong set of regexp since searchdataset function may have some regexp which are not present in dataset one. In attribute approach I can take the union and it is not a big deal, but I do think functions are irrelevant. Don't take me wrong I'll implement it, but at least I want to show you the potential source of errors in the format you propose. Much better approach is to use attribute based format, i.e. a dataset has only one set of regexp and functions should use them all for backward compatibility, i.e. a search dataset is a dataset at the end which is allowed in DBS DB.

could (part of) the confusion be due to the fact that DBS3 has to allow more freedom in format of things to search for than things to write, since we had to allow for stuff written in the more laxed DBS2 times to be still accessible ?

@belforte  the confusion is in function usage. I don't know who uses searchdataset and who dataset, but to me they refer to one and only one entity in DB, a dataset. As such a single entity should have only one set of regexp. I understand the legacy of this codebase, but I would like to move forward with correct data format which represent DB entities. The legacy functions should in my opinion exists but refer to a single, well defined set of regexp based on attributes (entities).

+1

Don't take me wrong I'll implement it, but at least I want to show you the potential source of errors in the format you propose. Much better approach is to use attribute based format, i.e. a dataset has only one set of regexp and functions should use them all for backward compatibility, i.e. a search dataset is a dataset at the end which is allowed in DBS DB.

The way I see it, the attribute based approach would have the same issue if you want to call it a dataset attribute. We cannot allow a dataset in the old (DBS2) construction to be inserted into the database. So, a union of those rules into the same attribute shouldn't be done.
For these quite old backwards compatibility code and rules, I think we will have to keep dedicate attributes to them. Even though those are all dataset in the database, we need to have a way to distinguish them between "before DBS3" and "during/post DBS3".
Perhaps those cases would deserve an attribute/entity like read_dataset and write_dataset instead?

well, read_dataset and write_dataset make sense since they reflect attribute and operations, and I'm fine with them, but how you map them to existing functions, e.g. def searchdataset and def dataset, the same question will go to other attributes too. Since it is unclear to me what function should represent I'm picky about them. I don't mind to implement them, but I don't yet understand the meaning of all functions used in Lexicon. To me the schema should be about DB entities and not about functions. Once, we agree on which attributes should be presented in lexicon.json I'll need to know the mapping of them to existing functions. So far the logic of Lexicon.py is far from clear.
Here are the existing functions:

dataset vs searchdataset vs searchstr
block vs searchblock
procdataset vs userprocdata vs procstring vs procstringT0
lfn vs userLfn

I suggest that you clarify the meaning of those and how we should construct regexp for them and more importantly why do we need to distinguish these regexp patterns. At the end each item above represent a single entity in DBS (even DBS2 vs DBS3 the dataset has identical meaning, even though the conventions for them may have changed). I would understand if you'll say dataset is for write dataset and searchdataset is for read dataset, but look at others, e.g. proxXXX functions and try to clarify their meaning.
I would like to avoid to have data representation based on function which are obsolete and no longer use, especially since other use-cases, e.g. dbs2go, does not use them. I need clarity how we define regexp. So far the suggested format does not provide that.

I will have a look at those and try to clarify them. However, some of them are not used by WMCore services at all, and many have been implemented even before my arrival in the WM system.
Yes, I completely agree that code no longer used should be removed (and inconsistencies should be cleared out). And it's great that you are trying to tidy this up with this new schema. I will get back here once I have some information to share. Thanks!

Actually, giving it a second thought, I'd like to propose the following steps:

do not change any functionality of the Lexicon APIs; besides the separation of rules and code (this should be the 1st commit)
if it passes all unit tests, then it gives us a high level of confidence that no (hidden) mistakes have been made;
then we re-evaluate these confusing functions and their rules and try to refactor Lexicon such that we can make it more clear and remove possibly obsolete code (2nd commit)
updates to unit tests (3rd commit)

My main concern is that a bug can be injected and remain in the system for many years, until we can spot it (like the one Stefano spotted this week!). Lexicon is used across many projects, so I'd rather play as safe as possible here (in addition to the initial backwards compatibility discussed many times in this thread).

I like this because 3. will take some time as we look at code that none of us wrote and try to figure out if that is still needed or anyhow useful. Cleanup is good, but time consuming, the more you want to clean, the more you need to check.
One think I sort of "remember" is that originally there was the idea that Production wanted to make sure they enforce more strict rules for dataset and files formats than what DBS allows, while DBS was more accommodating to let CRAB users adopt more human readable and meaningful names. Things like "for user files, make sure that the username is part of the path, while there is no user name for production", or "make sure users can't create a LFN which Production may want to use one day".
One may surely argue why people did not write two Lexicons, one for DBS and one for WM components...  such arguments usually end with "at the time it seemed the least work".
In a way WMCore/Lexicon is much more than a handy representation of DBS schema constraints.

@amaltaro your last comment does not address the main and primary issue, which is the data format to use and how this data format is related to the code and DB. I gave you clear examples that the format you propose has flaws, i.e. having multiple entries with the same attribute but different functions may lead to inconsistencies and it is unclear how to make union/intersections of regexps across such records/functions. I still believe that only attribute names, lengths, and list of regexps should be presented in a final format. Then different functions which refer to the same attribute, e.g. def searchdataset and def dataset, or def searchblock and def block, or def lfn and def userLfn will use the attribute's regexp list and check that given value satisfies them. I do agree that we may separate read/write entities, such that we'll have read_dataset and write_dataset which will represent operations, e.g. we can use read_dataset in def searchdataset and write_dataset in def dataset. This makes sense to me. But please resolve code ambiguities where multiple functions exists for the same attribute. In particular the following cases should be clarified:

def searchdataset vs def dataset vs def searchstr vs def namestr, all of the refer to dataset pattern and use different regexps, but in DBS/DB we only have one dataset entity
def searchblock vs def block refer to block usage and use different patterns, but in DBS/DB we only have one block entity
def lfn vs def userLfn refer to LFN usage and use different patterns while in DBS/DB we only have one lfn entity
def procdataset vs def userprocdata vs def procstring vs def procstringT0 refer to a processing dataset patterns while in DBS/DB we have only **one** entitiy Since these attributes in DB represented by single entity I need to know which regexp to apply to them in GoLang code. So far reading and using Lexicon.py` functions I have no idea.

So, please clarify your plan with concrete steps, e.g. for your item #1 which data-format we should use. If you still insists on using function name in data format please clarify how should I translate and use it in GoLang code where there is no such functions and there is no way to call them. At the end the code represents access to the DB and regexp applies to the DB entities, and the code itself has no meaning to them, what matters is only set of regexps we need to know to call/use in a code.

Please read my previous comment again. If you still cannot make sense of it, ping me on slack and we can have a quick zoom chat. Sorry to say it, but at this point, this discussion is just becoming a wastage of time!

Valentin and I had a chat and the course of actions discussed is:

Valentin will get a map of DBS REST endpoints, operation (read/write), and the Lexicon API(s) used
Alan will create a new json dump of the rules and separate code from lexicon rules
Then, we will start analysing these ambiguous cases and see what are the stakeholders, where and how it's used, and whether it's still required.

@belforte Stefano, would you be able to check which of the CRAB* services are depending on Lexicon APIs? Are those services using the function/APIs, or is it also using the regular expressions defined in the Lexicon.py module? Once I know which services (and repository) to look at, I can collect a map of APIs used and try to make sense what their use is for.

CRAB does 2 things.

uses the functions to check names, in particular code equivalent to:

from WMCore import Lexicon
Lexicon.lfn(testLfn)
Lexicon.userLfn(testLfn)
Lexicon.userprocdataset(outputDatasetTagToCheck)
Lexicon.primdataset(datasetNameToCheck)

the lines where those are used are:

https://github.com/dmwm/CRABServer/blob/11daa05d7d0866a1dbafbc67960cb70327501e76/src/python/TaskWorker/Actions/DagmanCreator.py#L246
https://github.com/dmwm/CRABServer/blob/11daa05d7d0866a1dbafbc67960cb70327501e76/src/python/TaskWorker/Actions/DagmanCreator.py#L274
https://github.com/dmwm/CRABServer/blob/11daa05d7d0866a1dbafbc67960cb70327501e76/src/python/CRABInterface/RESTUserWorkflow.py#L119
https://github.com/dmwm/CRABServer/blob/11daa05d7d0866a1dbafbc67960cb70327501e76/src/python/CRABInterface/RESTUserWorkflow.py#L199

import the regexp's themselves

in here to print diagnostic information for users when we reject a dataset name

https://github.com/dmwm/CRABServer/blob/11daa05d7d0866a1dbafbc67960cb70327501e76/src/python/CRABInterface/RESTUserWorkflow.py#L131

in here it is more perverse, and as the #TODO in the code says already, it is an historical bad piece of code waiting for being "done right".

https://github.com/dmwm/CRABServer/blob/11daa05d7d0866a1dbafbc67960cb70327501e76/src/python/CRABInterface/Regexps.py#L1-L15
The use case here is the need to provide REGEXPs to the REST input parameter validation in https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/REST/Validation.py

The usage of Lexicon.py and regexp in DBS is a mix bag. The good part is that everything seems to be located in a single python DBS module: https://github.com/dmwm/DBS/blob/master/Server/Python/src/dbs/utils/DBSInputValidation.py, the bad news is that DBS uses a mix of functions from Lexicon.py and its own. For instance, a dataset used for DBS reader relies on searchdataset function from Lexicon.py if given value contains a pattern *, otherwise it uses its internal regexp coming from its function reading_dataset_check (see lines 65-68. The same applies to block, lfns, procds. From what I see in DBS code its reading patterns mostly comes from DBS code functions and sometimes from Lexicon.py, while write patterns comes from Lexicon.py. For instance, in lines 319-331. The first block is for writing and it uses validationFunction (which come from Lexicon.py, and their definitions is at line 240), while for reading patterns it uses its own functions like reading_dataset_check, reading_block_check (which are defined in DBS module itself).
This leads me to conclusion that the following functions from Lexicon.py are used for writing operations:

block, dataset, lfn, primdataset, procversion, acqname, globalTag, validateUrl, DBSUser, dataset

while for reading operations DBS relies on mixture of its own functions and functions from Lexicon.py, e.g. searchstr, namestr (from Lexicon.py) and reading_dataset_check, reading_block_check, etc. from DBS code.
So, it will be fun to deduce from this code a set of regexp lists :) But is is clear that we need two set of regexp rules. One set of rules for DBS reader (DBS GET APIs), and another set of rules for writing operations (DBSWriter, i.e. DBS POST APIs). I'm fine to create two different JSON files which will cover read/write operations and use them in a future. This also suggest that the function I listed above for writing operations is what DMWM/WMCore/CRAB tools should use, while all reading regexp should be correctly identified via DBS codebase and merged with few Lexicon.py functions (like searchstr, namestr and reading_xxx_check DBS functions).

While awaiting your input no how to proceed, I adjusted dbs2go code to use independent lexicon rules based on proposed JSON formats. If you are interested here are the rules:

lexicon_writer.json
lexicon_reader.json
And, I provided two additional json files (for each writer/reader rules) which are used in negative/positive unit tests. In other words the negative json shows wrong examples, while positive contains correct values for tested rules. The files can be found here:
lexicon_reader_negative.json
lexicon_reader_positive.json
lexicon_writer_negative.json
lexicon_writer_negative.json

For the record, the output of dbs2go unit tests looks like this (each test is easy to follow and see what regexp match was applied to each passed value). With such mechanism we can extensively tests the lexicon rules against any DBS entity. With such mechanism you can easily add any lexicon rules and we'll only need to update JSON files. The Python code should load JSON files and use them in functions used by WMCore/CRAB tools.
make test-lexicon
cd test && LD_LIBRARY_PATH=`cat /Users/vk/Work/Languages/Go/gopath/src/github.com/vkuznet/dbs2go/oci8.pc | grep "libdir=" | sed -e "s,libdir=,,"` DYLD_LIBRARY_PATH=`cat /Users/vk/Work/Languages/Go/gopath/src/github.com/vkuznet/dbs2go/oci8.pc | grep "libdir=" | sed -e "s,libdir=,,"` DBS_LEXICON_FILE=../static/lexicon_writer.json DBS_LEXICON_SAMPLE_FILE=../static/lexicon_writer_positive.json go test -v -run LexiconPositive
=== RUN   TestLexiconPositive
2021/10/11 15:09:28 logical_file_name=/store/user/one/two/three/file123.root matched with ^/store/(temp/)*(user|group)/(([a-zA-Z0-9\.]+)|([a-zA-Z0-9\-_]+))/([a-zA-Z][a-zA-Z0-9\-_]*)/(([a-zA-Z0-9\-_]+)/)+([a-zA-Z0-9\-_]+).root$
2021/10/11 15:09:28 logical_file_name=/store/lhe/123/file123.lhe.xz matched with ^/store/lhe/([0-9]+)/([a-zA-Z0-9\-_]+).lhe(.xz){0,1}$
2021/10/11 15:09:28 dataset=/ZMM/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM matched with ^/[a-zA-Z0-9\-_]{1,99}/[a-zA-Z0-9\.\-_]{1,199}/[A-Z\-]{1,50}$
2021/10/11 15:09:28 block_name=/ZMM/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM#8420a3e7-96cf-48b1-a6a8-f9abcd3de8ef matched with ^/[a-zA-Z][a-zA-Z0-9\-_]+/[a-zA-Z0-9\.\-_]+/[A-Z\-_]+#[a-zA-Z0-9\.\-_]+
2021/10/11 15:09:28 processed_ds_name=123-bla-123 matched with ^[a-zA-Z0-9\.\-_]+
2021/10/11 15:09:28 data_tier_name=RAW matched with ^([A-Z]{3,100})+$
2021/10/11 15:09:28 processing_version=123 matched with ^[0-9]+$
2021/10/11 15:09:28 global_tag=123 matched with ^[a-zA-Z0-9\s\.\-_:]{1,100}$
2021/10/11 15:09:28 create_by=123bak matched with ^[a-zA-Z0-9/%*][a-zA-Z0-9/\.\-_%*/#]*$
2021/10/11 15:09:28 create_by=123bak matched with ^[a-zA-Z0-9/][a-zA-Z0-9/\.\-_\']*$
2021/10/11 15:09:28 acquisition_era_name=someEra matched with [a-zA-Z][a-zA-Z0-9_]*$
2021/10/11 15:09:28 physics_group=123 matched with ^([a-zA-Z0-9\-_]+)$
2021/10/11 15:09:28 last_modified_by=123blka matched with ^[a-zA-Z0-9/%*][a-zA-Z0-9/\.\-_%*/#]*$
2021/10/11 15:09:28 last_modified_by=123blka matched with ^[a-zA-Z0-9/][a-zA-Z0-9/\.\-_\']*$
2021/10/11 15:09:28 performed positive lexicon test against ../static/lexicon_writer_positive.json
--- PASS: TestLexiconPositive (0.01s)
PASS
ok  	github.com/vkuznet/dbs2go/test	0.922s
cd test && LD_LIBRARY_PATH=`cat /Users/vk/Work/Languages/Go/gopath/src/github.com/vkuznet/dbs2go/oci8.pc | grep "libdir=" | sed -e "s,libdir=,,"` DYLD_LIBRARY_PATH=`cat /Users/vk/Work/Languages/Go/gopath/src/github.com/vkuznet/dbs2go/oci8.pc | grep "libdir=" | sed -e "s,libdir=,,"` DBS_LEXICON_FILE=../static/lexicon_writer.json DBS_LEXICON_SAMPLE_FILE=../static/lexicon_writer_negative.json go test -v -run LexiconNegative
=== RUN   TestLexiconNegative
2021/10/11 15:09:30 performed negative lexicon test against ../static/lexicon_writer_negative.json
--- PASS: TestLexiconNegative (0.01s)
PASS
ok  	github.com/vkuznet/dbs2go/test	0.428s
cd test && LD_LIBRARY_PATH=`cat /Users/vk/Work/Languages/Go/gopath/src/github.com/vkuznet/dbs2go/oci8.pc | grep "libdir=" | sed -e "s,libdir=,,"` DYLD_LIBRARY_PATH=`cat /Users/vk/Work/Languages/Go/gopath/src/github.com/vkuznet/dbs2go/oci8.pc | grep "libdir=" | sed -e "s,libdir=,,"` DBS_LEXICON_FILE=../static/lexicon_reader.json DBS_LEXICON_SAMPLE_FILE=../static/lexicon_reader_positive.json go test -v -run LexiconPositive
=== RUN   TestLexiconPositive
2021/10/11 15:09:33 dataset=/ZMM/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM matched with ^/([a-zA-Z0-9]+[a-zA-Z0-9\-\._]*)/([a-zA-Z]+[a-zA-Z0-9_\-\.]*)/([A-Z_\-\.]+)$
2021/10/11 15:09:33 dataset=/ZMM/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM matched with ^/(\*|[a-zA-Z\*][a-zA-Z0-9_\*\-]{0,100})(/(\*|[a-zA-Z0-9_\.\-\*]{1,199})){0,1}(/(\*|[A-Z\-\*]{1,50})){0,1}$
2021/10/11 15:09:33 block_name=/ZMM/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM#8420a3e7-96cf-48b1-a6a8-f9abcd3de8ef matched with ^/([a-zA-Z0-9]+[a-zA-Z0-9\-\._]*)/([a-zA-Z]+[a-zA-Z0-9_\-\.]*)/([A-Z_\-\.]+)#([a-fA-F0-9\-\.]+)$
2021/10/11 15:09:33 block_name=/ZMM/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM#8420a3e7-96cf-48b1-a6a8-f9abcd3de8ef matched with ^/(\*|[a-zA-Z\*][a-zA-Z0-9_\*]{0,100})(/(\*|[a-zA-Z0-9_\.\-\*]{1,199})){0,1}(/(\*|[A-Z\-\*]{1,99})(#(\*|[a-zA-Z0-9\.\-_\*]){0,100}){0,1}){0,1}$
2021/10/11 15:09:33 logical_file_name=/store/user/one/two/three/file123.root matched with ^/store/([A-Za-z0-9._\-/])+/([A-Za-z0-9._\-])+(.root|.lhe(.xz){0,1})$
2021/10/11 15:09:33 physics_group=123 matched with ^([a-zA-Z0-9\-_]+)$
2021/10/11 15:09:33 primary_ds_name=123-bla-123 matched with ^[a-zA-Z0-9/%*][a-zA-Z0-9/\.\-_%*/#]*$
2021/10/11 15:09:33 processed_ds_name=123-bla-123 matched with (^[a-zA-Z0-9]+[a-zA-Z0-9_\-]*)$
2021/10/11 15:09:33 processed_ds_name=123-bla-123 matched with ^[a-zA-Z0-9/%*][a-zA-Z0-9/\.\-_%*/#]*$
2021/10/11 15:09:33 data_tier_name=RAW matched with ^([A-Z]{3,100})+$
2021/10/11 15:09:33 acquisition_era_name=someEra matched with [a-zA-Z][a-zA-Z0-9_]*$
2021/10/11 15:09:33 processing_version=123 matched with ^[0-9]+$
2021/10/11 15:09:33 global_tag=123 matched with ^[a-zA-Z0-9\s\.\-_:]{1,100}$
2021/10/11 15:09:33 global_tag=123 matched with ^[a-zA-Z0-9/%*][a-zA-Z0-9/\.\-_%*/#]*$
2021/10/11 15:09:33 create_by=123bak matched with ^[a-zA-Z0-9/%*][a-zA-Z0-9/\.\-_%*/#]*$
2021/10/11 15:09:33 create_by=123bak matched with ^[a-zA-Z0-9/][a-zA-Z0-9/\.\-_\']*$
2021/10/11 15:09:33 last_modified_by=123blka matched with ^[a-zA-Z0-9/%*][a-zA-Z0-9/\.\-_%*/#]*$
2021/10/11 15:09:33 last_modified_by=123blka matched with ^[a-zA-Z0-9/][a-zA-Z0-9/\.\-_\']*$
2021/10/11 15:09:33 performed positive lexicon test against ../static/lexicon_reader_positive.json
--- PASS: TestLexiconPositive (0.01s)
PASS
ok  	github.com/vkuznet/dbs2go/test	0.439s
cd test && LD_LIBRARY_PATH=`cat /Users/vk/Work/Languages/Go/gopath/src/github.com/vkuznet/dbs2go/oci8.pc | grep "libdir=" | sed -e "s,libdir=,,"` DYLD_LIBRARY_PATH=`cat /Users/vk/Work/Languages/Go/gopath/src/github.com/vkuznet/dbs2go/oci8.pc | grep "libdir=" | sed -e "s,libdir=,,"` DBS_LEXICON_FILE=../static/lexicon_reader.json DBS_LEXICON_SAMPLE_FILE=../static/lexicon_reader_negative.json go test -v -run LexiconNegative
=== RUN   TestLexiconNegative
2021/10/11 15:09:36 performed negative lexicon test against ../static/lexicon_reader_negative.json
--- PASS: TestLexiconNegative (0.01s)
PASS
ok  	github.com/vkuznet/dbs2go/test	0.347s

@belforte @vkuznet thanks for providing all these details. I think it will take a couple of weeks before I can resume working on this. Once that happens, I will read carefully your messages and get back to you in case of questions.

@amaltaro please revisit your review before end of the year that we can plan and adjust things.

@vkuznet: Alan is currently on Holiday through middle of next week.  I also know that he has a fairly busy rest of the month coming up for personal reasons.  Is it necessary to finish this in 2021 or can it be deferred until early 2022?

@klannon it is not strictly necessary, but Alan mentioned the review in couple of weeks more than a month ago. Said that, due to lack of response I proceed with my version of lexicon schema in DBS Go implementation and it means that eventually we'll diverse on this subject between different systems which I would like to avoid. Therefore, I don't want to post-pone this issue for too long.

After a quick chat with @vkuznet, added this to the 2022 Q1 to-do list so that we don't lose track of it, since it seems unlikely that we'll manage with the end of year craziness and other Q4 2021 priorities.

Valentin, given that DBS already has its own lexicon logic/format, and you mentioned this no longer to be a critical piece for DBS, I'd suggest to work on this in a low priority this quarter. I'd like to proceed with the remaining implementation myself, but if I do that, it won't happen before march or so.

@amaltaro drop me a line when Lexicon.py changes, in case I need to change the way CRAB uses it.


Fiery Turtle bug

Hi, I'm getting some really strange behaviour from furnace turtles:
it won't accept smelting a single stack in any inventory slot

when you fill all slots but one, it behaves strange and destroys one item:

The 3rd item dissapeared when it was clicked
I looked into the source and could not find any obvious errors, although the whole method works in a strange way.
Also I have another suggestion: as the original inventor of the furnace turtle pointed out, the smelt method should also support just one argument >furnace.smelt(slot)< which smelts the entire stack in that slot without the need for an empty slot in the turtle's inventory.

What version are you running?

MoarPeripherals-1.7.10-1.7-1.53

okay thank you for the report, I will take a look into this ASAP.
as for being able to use furnace.smelt(slot) without the amount, this feature will be available once I have completed theoriginalbit/Peripheral-Framework#3

Pretty sure I already told you about this, and that you already fixed it?
https://github.com/theoriginalbit/MoarPeripherals/issues/27

well apparently it's present again. @BombBloke if you'd like to verify the issue for yourself please? I'm yet to actually verify the issue still exists, busy week.

Ok, tested with 1.53-snapshot-88, all's well. 1.53 "proper" pre-dates the old issue by about a week, so the fix simply never went into a "stable" release.
There's one remaining oddity where, if the turtle's inventory is full, attempting to smelt will throw a "not enough space in inventory" error even if the items that would be removed during the process would leave a slot free for the new items.

Oh really? I thought that made it to stable, hmm. Okay so the changelog for this coming update will be huge!
Yeah I am aware of this issue with it making an open slot, I do need to handle that, it was an oversight. I'll leave this issue open until that one is fixed.


SH rotation matrix

adds SH rotation matrix and operation

Thank you very much for this contribution!
As per our discussion offline we could think about multi rotation extensions. However, I am happy to merge as is, and then extend later.
Any objections?

Perfect! Go ahead!


Add an OnUnbalanced hook for contract rent payments

This closes #2455.

Ok, I've fixed the code to use imbalance. A polkadot PR will have to wait a bit due to polkadot-master being significantly behind master.

Okay at some point before the update of polkadot-master it should be taken care of.
I will leave it to @pepyakin or @jimpo who know contracts better to maybe have a final look?

@thiolliere yes, because
if insufficient_rent || !can_withdraw_rent {
    ...
    return (RentOutcome::Evicted, Some(tombstone_info));
}

happens before T::RentPayment::on_unbalanced(imbalance); is called.

Oh, haha, no changes to polkadot are needed because it doesn't use srml-contracts.


Multiple Translations

If in your application you import a package, which is translated
and in your application create a translation, you get:
Duplicated library name 'messages_all'.
library messages_all;
when using the generate_from_arb functionality.
This applies also to the library in the translated files, e.g. messages_es
The cause is that in generate_localized.dart, the library name(s) are hardcoded.
generate_from_arb.dart allows to specify a file prefix, e.g.
--generated-file-prefix=base-
I suggest to use the file prefix also for the library names.

Right now the intl package only supports a single set of translations
within an app. Are you saying that it actually works to have multiple
translations, and you're just getting warnings about the library name?
On Thu, Jun 4, 2015 at 8:40 PM Jorg Janke<EMAIL_ADDRESS>wrote:

If in your application you import a package, which is translated
and in your application create a translation, you get:
Duplicated library name 'messages_all'.
library messages_all;
when using the generate_from_arb functionality.
This applies also to the library in the translated files, e.g. messages_es
The cause is that in generate_localized.dart, the library name(s) are
hardcoded.
generate_from_arb.dart allows to specify a file prefix, e.g.
--generated-file-prefix=base-
I suggest to use the file prefix also for the library names.
—
Reply to this email directly or view it on GitHub
https://github.com/dart-lang/intl/issues/9.

One translation per library.
When I import a library which has a translation into an application which has a translation, I get the
Warning: Duplicated library name 'messages_all'.
messages

Duplicated library names are only a warning, which can be ignored. But if you actually try to use it I suspect that it will fail for other reasons.

The objective was to create an enhancement request - and if it is accepted, I might attempt to fix it by adding the prefix to avoid the name clash.

The enhancement request is reasonable. Changing the name to reflect that. However, the problem is not the duplicated library name. That should probably be fixed to match the file name. But it's probably more important that the program will not run correctly if it has two different sets of translated messages in it. In particular, only the first one that gets initialized for a particular locale will be used, and message in the other will either fail or will use the translation from the first catalog. I thought there was already an open issue for that, but I don't see one, so we can use this one.

I have this problem as well. My app has some of its functionality implemented as an independent package that should be providing its own localized strings; I had to resort to the following dirty hack on the package side to allow its lookup table to coexist with the application's:
import 'dart:async';
import 'dart:ui';

import 'package:flutter/widgets.dart';
import 'package:my_package/l10n/messages_all.dart';
import 'package:intl/intl.dart' as RealIntl; // HORRIBLE HACK! (see below)
import 'package:intl/src/intl_helpers.dart';

class MyPackageLocalizationsDelegate
    extends LocalizationsDelegate<MyPackageLocalizations> {
  const MyPackageLocalizationsDelegate();

  @override
  bool isSupported(Locale locale) => ... ;

  @override
  Future<MyPackageLocalizations> load(Locale locale) =>
      MyPackageLocalizations.load(locale);

  @override
  bool shouldReload(LocalizationsDelegate<MyPackageLocalizations> old) => false;
}

/// HORRIBLE HACK!
///
/// The intl package only supports loading one global message table, which is
/// held at [messageLookup] in intl_helpers.dart. This prevents us from
/// providing localized strings in this plugin for consumption in a parent
/// application, as only one message table will be used.
///
/// The travesty below works around this. It is so horrible because we are
/// trying to have our cake and eat it too:
///
/// - We want to manage strings in this package with the intl_translation
///   extraction and merging tools, so message lookup calls must (appear) to be
///   made to [Intl]
/// - We don't want to edit generated files (messages_all.dart, etc.) by hand
///
/// How it works:
///
/// 1. We define our own [Intl] class that calls into our own lookup,
///    [myMessageLookup]. The lookup is initially "empty".
/// 2. We import the real intl class under an alias and delegate to it where
///    appropriate.
/// 3. We call the default generated [initializeMessages], then immediately swap
///    out the real [messageLookup] and [myMessageLookup] objects. This
///    returns the real lookup to an uninitialized state for the application's
///    localizations to load. (Yes, all this relies on localization delegate
///    init happening sequentially.)
/// 4. We exhort the consuming application to load our localization delegate
///    before the application's.
class Intl {
  // copied from the real intl package
  static String message(String message_str,
          {String desc: '',
          Map<String, dynamic> examples: const {},
          String locale,
          String name,
          List args,
          String meaning,
          bool skip}) =>
      _message(message_str, locale, name, args, meaning);

  // copied from the real intl package
  static _message(String message_str, String locale, String name, List args,
      String meaning) {
    return myMessageLookup.lookupMessage(
        message_str, locale, name, args, meaning);
  }
}

MessageLookup myMessageLookup =
    UninitializedLocaleData('initializeMessages(<locale>)', null);

class MyPackageLocalizations {

  static Future<MyPackageLocalizations> load(Locale locale) {
    final name =
        locale.countryCode == null ? locale.languageCode : locale.toString();
    final localeName = RealIntl.Intl.canonicalizedLocale(name);
    return initializeMessages(localeName).then((_) {
      // HORRIBLE HACK! (see above)
      // Steal the just-initialized lookup and keep it for ourselves.
      // Then un-initialize the real lookup so the next localizations can load.
      final uninitialized = myMessageLookup;
      myMessageLookup = messageLookup;
      messageLookup = uninitialized;
      RealIntl.Intl.defaultLocale = localeName;
      return MyPackageLocalizations();
    });
  }

⋮

}

This is pretty unexpected and dangerously hidden limitation to allow only one translation set per app. I would not have expected to face this kind of limitation when developing 3 "totally independent" packages each having their own translation set. It took 3 months until I realized this issue... mainly because the default language works normally. You will see the issue only after changing the language.

My workaround no longer functions, it seems due to changes in the timing of when localization delegates are initialized by the framework.
I've managed to hack it again, but I'm not sure if it only works by coincidence since it relies on a specific order of execution:
https://github.com/amake/flutter_l10n_test/commit/6ff8fc632ae728e299e76b38f947f6f2d687ce7e

To have localizations in my packages I'm using a patch that I explain in this article.
https://medium.com/@gerrel/flutter-package-with-intl-localization-394e9da1164a?source=friends_link&sk=48169f60f86aef7eb38bf0fe43041574

Is there a better solution?

Is there a better solution?


more comprehensive tests

better leverage selenium to test interface by clicking buttons and entering text etc.

Closing this. Every component is tested and checked for errors in chrome logs.