Many methods have massive compiler performance drawbacks

I spent much of last week investigating why my work's codebase was taking 2-3 seconds to populate autocomplete or refresh code-intelligence during code changes, and after a lot of tsc tracing and replacing Zod types with any, I found that basically the entire time is being taken up processing Zod types - specifically those using masking (Extend, Omit, Pick) methods.
After removing all uses of masking and dropping back to Zod 3.19 on the suggestion of some comments in the issues, the codebase is now down to 1.5 seconds for autocomplete which is still a lot of time.
I've written this up in a little more detail here and have a small-scale reproduction here which was used in the benchmarks.
Eventually I do plan on a more in-depth benchmarking across a larger variety of methods, as these benchmarks are a somewhat blunt instrument, but it would be great to see type inference performance improved across the board as Zod is lagging behind the competition even in the most basic situations.

I'm experiencing same behaviour. I did finally took time to debug why vscode autocomplete takes 10-15 sec to show in my codebase (which is just terrible while coding frontend) and tracked it down to zod being the culprit even for what I thought are simple schemas. Is there way to somehow blacklist zod from typescript completely? Current DX is not acceptable for me anymore, I would rather have working editor and type everything manualy.

You can notate your Zod types with //@ts-igore (typing //@ts will give you a few options to try.
I've also heard a suggestion to wrap all the Zod types in a function explicitly returning a simple ZodType and ignoring the input.
None of it's ideal of course

I recorded the behavior:

Interesting - I don't know whether it is related to zod, but this terrible behavior which I recorded on simple example is fixed when I set "strict": true instead of "strict": false in tesconfig.json.

Previously in my case it was 10-15 seconds delay, restarting the TS server seemed to fix the problem temporarily. We did a couple of things:

used @typescript/analyze-trace to eliminate some hotspots, found 1 instance of the typescript server being stuck in a recursive type for too long
added //@ts-nocheck to every single file with schema definitions
upgraded typescript to 4.9.4

Somehow the problem seems to have vanished after that...

@Nick-Lucas The performance has been improved dramatically in Zod 3.21, can you try it can report back? I looked at your repo but it wasn't clear to me how to re-run your benchmrak.

Hey @colinhacks sure thing, I've been working on a more granular suite which is a little rough (variances of 5-10ms between runs are normal) but has been quite useful.
I've just updated my project at work to the latest and performance does look much better, the most expensive files in the codebase are no longer traced as related to Zod, which tracks with my experience dog-fooding light-types as a replacement. So definitely on the right lines!
I suspect that Zod's wrapping of types is part of the remaining losses, my light-type development has made of point of always modifying the generic types (ie '.optional' return new LtType<TInput | undefined, TOutput | undefined> instead of ZodOptional<ZodType<etc>>) when chaining to a new class instance, which has some tradeoffs but is proving very efficient for inference. extend/merge aren't possible to do cheaply today, it's actually most efficient to deprecate those methods and encourage users to do object({ ...base.shape, ...extra.shape }) as TypeScript seems to play nicely when duplicate keys are in a spread object, but that can't be hidden inside a generic method because TS can't seem to do native merging when duplicate keys exist 😭
Before
Zod: 3.20.6
TypeScript: 4.8.2

Now
Zod: 3.21.2
TypeScript: 4.8.2

I'm aware typescript has also made some optimisations recently, but just haven't yet updated the suite to 4.9.x
In terms of running this for yourself:
git clone git@github.com:Nick-Lucas/gapstack.git
git checkout e33655ca3139c87264da2edcbdf3d42e62f1697f # just to pin the version

nvm use
yarn install

# Run the benchmarks
yarn light-type-benchmark-2:run

# in theory these alternative scripts should stabilise the CPU cores in use between runs, though at least the Mac one I don't think works as i want yet
yarn light-type-benchmark-2:run:mac
yarn light-type-benchmark-2:run:nux

The suite from the blog post
The blog post suite actually traces the whole repo but I just pulled out the relevant bits for the writeup
# do the setup steps from above ^

# a directory called .trace/ will be created
yarn tsc:trace

I too am concerned about cpu usage of zod. I'm a first time user:

I've imported the lib into two different files (a schema defined in one... a parse function in another)... and my IDE (webstorm) is noticeably showing performance hits...  (im on a 2021 M1 - 32gig)
comment out the zod import... file performance returns.

I'm concerned that if I implement this lib.. ill have to pull it out as other developers here have older/slower machines.

Why are all of these issues being marked as stale? They're still ongoing issues.

This particular issue was actually mostly resolved with a bugfix. I would open/find another one for general DX performance issues, which are definitely still present when used with tools like tRPC


Upgraded the JaCoCo Version

Signed-off-by: Ali Dehghani<EMAIL_ADDRESS>

Codecov Report

Merging #99 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #99   +/-   ##
=========================================
  Coverage     93.44%   93.44%
  Complexity      293      293
=========================================
  Files            33       33
  Lines           687      687
  Branches         84       84
=========================================
  Hits            642      642
  Misses           25       25
  Partials         20       20

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bb68e84...e9a82f0. Read the comment docs.


Marketing slogan

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement
Learn more about why HashiCorp requires a CLA and what the CLA includes

wade seems not to be a GitHub user.
You need a GitHub account to be able to sign the CLA. If you already have a GitHub account, please add the email address used for this commit to your account.
Have you signed the CLA already but the status is still pending? Recheck it.


adds landing-page view

Assignee Tasks

[ ] added unit tests (or none needed)
[ ] written relevant docs (or none needed)
[ ] referenced any relevant issues (or none exist)

closed pull request


Stateful set status

MIrroring the deployment available probe for statefulsets.

Instructions on DCO to add sign off did not work closing this out will resubmit.


minor cleanup of code in WorkerStorage and Users

I changed mostly comments, and a few method/variable names for clarification.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Tachyon-Pull-Request-Builder/2996/
Test PASSed.


New small fixes and features

fix NULL values for BigDecimal and BigInteger
save - if not new - propagate deleteChildren flag and deleteIndexes=true
added expireIndexes support - with an overloaded method "expire"
jedis bump to 2.8.0 because of slow Apache Commons Pool (BaseGenericObjectPool - updateStatsReturn and updateStatsBorrow)

I updated the README to reflect the fact that it does know uses Jedis 2.8.0 since the changes brought by @dpoldrugo


c# - AzureAD.GetClientConfig.Invoke doesn't exist

What happened?
Following the tutorial here for c#: https://www.pulumi.com/registry/packages/azuread/api-docs/getclientconfig/. But var current = AzureAD.GetClientConfig.Invoke(); will never work, since https://github.com/pulumi/pulumi-azuread/blob/8419bec8e7900246a8f23a676c9f8357a45ae43c/sdk/dotnet/GetClientConfig.cs#L12 only exposes InvokeAsync.
Also, on https://www.pulumi.com/registry/packages/azuread/api-docs/getclientconfig/, again the c# sample, it specifies : ["objectId"] = current.Apply(getClientConfigResult => getClientConfigResult.ObjectId), this also doesn't work because it assumes a non-async result, instead, something like this works fine: ["objectId"] = current.Result.ObjectId
Expected Behavior
Should work on MY machine
Steps to reproduce
Try the tutorial!
Output of pulumi about
CLI
Version      3.78.1
Go Version   go1.20.7
Go Compiler  gc
Plugins
NAME     VERSION
azuread  5.40.0
dotnet   unknown
Host
OS       Microsoft Windows 11 Pro
Version  10.0.22621 Build 22621
Arch     x86_64
This project is written in dotnet: executable='C:\Program Files\dotnet\dotnet.exe' version='8.0.100-preview.7.23376.3'
Current Stack: NO/yolo/dev
TYPE                       URN
pulumi:pulumi:Stack        urn:pulumi:dev::yolo::pulumi:pulumi:Stack::yolo-dev
pulumi:providers:azuread   urn:pulumi:dev::yolo::pulumi:providers:azuread::default_5_40_0
azuread:index/group:Group  urn:pulumi:dev::yolo::azuread:index/group:Group::example
Found no pending operations associated with dev
Backend
Name           pulumi.com
URL            https://app.pulumi.com/NO
User           NO
Organizations  NO
Dependencies:
NAME            VERSION
Pulumi          3.56.0
Pulumi.AzureAD  5.40.0
Additional context
No response
Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

Hi @erwinkramer thanks for bringing this to our attention.
I've done a little digging and this looks like a mismatch between how our dotnet support decides to generate the varients of the Invoke vs InvokeAsync and what the example generation expects to be there.
I'm going to transfer this over to the main Pulumi project which is responsible for the code and example generation and reword the issue a little to help draw out the root cause.


Add flashing time separator to Date & Time Patterns

The default Mac menu bar clock has an option for flashing time separators (a colon in my case).
It would be very useful if this functionality could be added to Itsycal.

Actually I did a PR a while back: https://github.com/sfsam/Itsycal/pull/91


Roll engine 309d078b597d..445ab42efc93 (13 commits)

https://github.com/flutter/engine/compare/309d078b597d...445ab42efc93
git log 309d078b597d3a38fe9bcef8d8bae7b56487e8eb..445ab42efc93cbef4b78ec7cb3a7eb97db2ff3d3 --no-merges --oneline
445ab42ef Roll src/third_party/dart 11a3dd7332..600b2831ef (9 commits)
49d0dc6d6 Roll src/third_party/skia 5c5de2146309..b6d04556c34d (1 commits) (flutter/engine#8434)
4ff18fbda Roll src/third_party/dart f6768b6fb3..11a3dd7332 (56 commits)
7b831986f Roll src/third_party/skia 2273c900ea28..5c5de2146309 (5 commits) (flutter/engine#8432)
800ea0a40 Revert "Enable shutting down all root isolates in a VM. (#8402)" (flutter/engine#8431)
b59c4436c Enable shutting down all root isolates in a VM. (flutter/engine#8402)
9acdf1837 Make AccessibilityViewEmbedder final (flutter/engine#8429)
804d54ddf Eliminate unused displayBounds parameter (flutter/engine#8427)
d4c815f57 Roll buildroot (flutter/engine#8425)
20190d707 Roll src/third_party/skia fbe240669394..2273c900ea28 (13 commits) (flutter/engine#8428)
7e38b0aa2 Revert "Revert "Separate the data required to bootstrap the VM into its own class. (#8397)" (#8406)" (flutter/engine#8414)
d8bb9d793 Roll src/third_party/skia 176da0bb8a10..fbe240669394 (9 commits) (flutter/engine#8424)
64fc58329 Remove use of DART_CHECK_VALID. (flutter/engine#8417)
The AutoRoll server is located here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md
If the roll is causing failures, please contact the current sheriff (chinmaygarde@google.com), and stop
the roller if necessary.

Trybots failed. These were the failed builds: https://cirrus-ci.com/task/6694644304838656

Commit queue failed; closing this roll.


Metric Aggregation Rule not working

Hello, I am trying to alert on some key fields like CPU Usage from Elasticsearch.
My config looks like this:
Rule name, must be unique
name: CpuUtil
Type of alert.
type: metric_aggregation
metric_agg_key: node_stats.process.cpu.percent
metric_agg_type: max
doc_type: datapoint
max_threshold: 30
Index to search
index: .monitoring-*
timestamp_field: @timestamp
alert_subject: "CPU Usage above threshold"
alert_text_type: alert_text_only
alert_text: "CPU Usage is above 30 percent :("
(Required)
The alert is use when a match is found
alert:
"email"
(required, email specific)
a list of email addresses to send alerts to
email:
I changed the type to "any" and the emails just kept on coming. But as soon as I change it back to metric_aggregation nothing works.
Any input it greatly appreciated

URL of a site that may be helpful
Elastalert rule for CPU usage in percentage
elastalert_cpu_watch.yaml


Emacs mode: ignore backslash character

Emacs treats the '\' as an escape character by default, which means it sees \(foo : x) -> bar as having unbalanced parentheses and therefore forward-sexp and related functions don't work.

Thanks!


Update transformation types for Cloudinary Video Player #202

Description
Update transformation types for Cloudinary Video Player
Dependency: @cloudinary/url-gen
Issue Ticket Number
Fixes #202

Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Fix or improve the documentation
[ ] This change requires a documentation update

Checklist

[x] I have followed the contributing guidelines of this project as mentioned in CONTRIBUTING.md
[x] I have created an issue ticket for this PR
[x] I have checked to ensure there aren't other open Pull Requests for the same update/change?
[x] I have performed a self-review of my own code
[x] I have run tests locally to ensure they all pass
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes needed to the documentation

Hey @colbyfayock
This PR have some test checks failure
how can I fix it ? Also can you please review the code changes?

looks like a linting error. you can run pnpm run lint from the root of the project


feat: Improve .NET tag command handling

Pinging @busunkim96 for review - I can't assign a reviewer, unfortunately.

cc @amanda-tarafa as this will make more sense of an upcoming PR in google-cloud-dotnet...

I see that the Kokoro failure is due to linting. Will try to clean that up now.

Okay, should be fixed now.

Ah - have now realized I can't merge. Please could you do so when you get a chance? Thanks!


⚠ Spryr has degraded performance

In e66bee5, Spryr (https://spryr.com) experienced degraded performance:

HTTP code: 200
Response time: 4642 ms

Resolved: Spryr performance has improved in bb413d5 after 9 minutes.


[BFCL] Standardize TEST_CATEGORY Among eval_runner.py and openfunctions_evaluation.py

There are inconsistencies between the test_category argument that's used by eval_checker/eval_runner.py and openfunctions_evaluation.py.
This PR partially addresses #501 and #502.

LGTM
Comment: improve warning message in the future, make them more information for user to know immediate action items when aggregating results for data.csv.

Good point. Will address this in a different PR.

Hey @HuanzhiMao and @CharlieJCJ I don't think this is a great idea for the following reason. So we start by showing how to install the dependencies, and then in the middle we go into a long extempore on the different flags and then we come back with a list of commands to run. This ain't helpful back and forth? Maybe we retain the previous structure where after each command we show the options so users have the context on the command. Thoughts?

Hey @HuanzhiMao and @CharlieJCJ I don't think this is a great idea for the following reason. So we start by showing how to install the dependencies, and then in the middle we go into a long extempore on the different flags and then we come back with a list of commands to run. This ain't helpful back and forth? Maybe we retain the previous structure where after each command we show the options so users have the context on the command. Thoughts?

How about we move the flags section to the bottom of the README, just like how all the available model names are displayed at the end?

I have solved this issue here.

Hey @HuanzhiMao and @CharlieJCJ I don't think this is a great idea for the following reason. So we start by showing how to install the dependencies, and then in the middle we go into a long extempore on the different flags and then we come back with a list of commands to run. This ain't helpful back and forth? Maybe we retain the previous structure where after each command we show the options so users have the context on the command. Thoughts?

How about we move the flags section to the bottom of the README, just like how all the available model names are displayed at the end?

I don't think this solves it right? We should present the flags when the user cares about the command, not super down. Folks read and execute code from READMEs linearly, and making them jump around isn't a good idea imo.


Support for Cucumber v8 NDJSON

The JSON formatter in version 8 of Cucumber is in maintenance mode and will at some point be deprecated.
The JSON formatter is to be replaces with the new Message formatter that aims to address a number of shortcomings such as a high memory footprint and a lack of a formal schema.
Given that the JSON formatter is no longer being developed and will be deprecated, will you be looking to support the new NDJSON format that is created by the message formatter?

Hi @humphreyn
Thanks for this heads up. For now I don't have it on my to-do list, but love to look into it. Do you have some more information about the format?

@wswebcreation JSON Formatter is in maintenance mode and no new features will be added. At the moment, it couldn't handle large json files for report generation. Is it possible to use the ndjson file that cucumber generates at the end of test execution to be consumed by this plugin? It would definitely save lot of time for all?

Sure, we can, any help, more docs and so on would be appreciated

@wswebcreation each line is a well-formed json item so other libs basically parse like that. Alternatively there are deps available but from a quick skim it doesn't seem like they are doing anything fancier worth a new dependency.


How long does it take to prepare the waymo dataset using your scripts?

I don't remember exact time. But with a ssd, it should be within 12 hr


Mix cycle includes garbage collection of unused sources.

Each mix cycle:

Make a new empty map[string]*Source, e.g. keepSources
While iterating over the ready & active fires (see issues #11 and #18; implemented as of pull #29)  copy any used *Source to the new keepSources
Replace the mixSources with keepSources

Note that early tests (using an external project) indicate no benefit from this update, in terms of memory usage. Investigating whether go-atomix is the cause at all, and leaving this issue open for now. Some benchmark tests might be helpful.


Fix logic for model type checking

Addresses #29

Ready for review! FWIW these changes fixed my issues on the Ray side.


Add simple notebook to demonstrate configuration of User Interface

Previously the UI has been usable with minimal configuration by developers, and in the container environment has been automatically configured.
#2266 will handle the container environment, but first I will create a simple, somewhat hardcoded, notebook to ensure developers can understand how to get the UI working. Indeed if using the coco pharma environment locally (3 platforms - 8080, 8081, 8082 - and using the documented server names) it will work with no change.

Moving to 1.4 due to #2285

The configuration of the UI platform should eventually be folded into the main server config notebook - so it is possible to configure the coco environment (metadata servers, governance servers and ui servers) all in one place.


Missing subo "Getting Started" page

The subo landing page in the docs refers to a "Getting Started" page that doesn't exist. Should we remove the reference or create the missing page?

My guess is it's referring to the "Get Started" section from Atmo docs.
Happy to fix this if so!


TypeError: cannot make memory view because object does not have the buffer interface

After having installed it via PIP, this is what happens:
fale@x250:~ $ vipaccess
Traceback (most recent call last):
  File "/usr/bin/vipaccess", line 9, in <module>
    load_entry_point('python-vipaccess==0.3.0', 'console_scripts', 'vipaccess')()
  File "/usr/lib/python2.7/site-packages/vipaccess/utils.py", line 273, in main
    response = get_provisioning_response(request)
  File "/usr/lib/python2.7/site-packages/vipaccess/utils.py", line 147, in get_provisioning_response
    return requests.post(PROVISIONING_URL, data=request)
  File "/usr/lib/python2.7/site-packages/requests-2.8.1-py2.7.egg/requests/api.py", line 109, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests-2.8.1-py2.7.egg/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests-2.8.1-py2.7.egg/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests-2.8.1-py2.7.egg/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests-2.8.1-py2.7.egg/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/lib/python2.7/site-packages/requests-2.8.1-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python2.7/site-packages/requests-2.8.1-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python2.7/httplib.py", line 1053, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1093, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1049, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 897, in _send_output
    self.send(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 869, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/site-packages/requests-2.8.1-py2.7.egg/requests/packages/urllib3/contrib/pyopenssl.py", line 216, in sendall
    data = memoryview(data)
TypeError: cannot make memory view because object does not have the buffer interface

Closing this issue since I'm no longer maintaining this project and will be archiving it soon. Please use this fork instead.


Repo sync for protected CLA branch

The pull request is created from main637890447713535104 to main to fix git push error for protected CLA branch

@ghogen : Thanks for your contribution! The author(s) have been notified to review your proposed change.


References not working?

Seems like refs are not get resolved as they were in the previous version.
Environment

What is your OS and version?
What version of qri are you using (qri version)?
0.2.0
What version of Qri Desktop are you using?
What browser(s) did you use? What version?
Issue
What did you do?
package main

import (
    "context"
    "encoding/json"
    "fmt"

    "github.com/qri-io/jsonschema"
)

func main() {
    ctx := context.Background()
    var schemaData = []byte(`{
    "$id": "https://qri.io/schema/",
    "$comment" : "sample comment",
    "title": "Person",
    "type": "object",
    "definitions": {
        "age": {
            "type": "integer",
            "minimum": 0
        }
    },
    "properties": {
        "age": {"$ref": "#/definitions/age"}
    },
    "required": ["age"]
  }`)

    rs := &jsonschema.Schema{}
    if err := json.Unmarshal(schemaData, rs); err != nil {
        panic("unmarshal schema: " + err.Error())
    }
   var valid = []byte(`{
        "age": 10
    }`)
    errs, err := rs.ValidateBytes(ctx, valid)
    if err != nil {
        panic(err)
    }

    if len(errs) > 0 {
        fmt.Println(errs[0].Error())
    }
}

What happened?
/age: 10 failed to resolve schema for ref #/definitions/age
What did you expect to happen?
The same schemas work at https://www.jsonschemavalidator.net/

Please link any related issues.
Do you have a suggested fix?

As of draft2019_09 it should be $defs instead of definitions. Think definitions should be supported for backwards compatibility, but unsure if that works currently.
Let me know if this helps, I'll keep the issue open so I can come back to it sometime.

Wouldn't it be possible to just register definitions as a keyword using the jsonschemas.NewDefs handler?
jsonschema.RegisterKeyword("definitions", jsonschema.NewDefs)
jsonschema.LoadDraft2019_09()

I don't think so, as far as I remember, there was a clash if you included both hence it also being disabled in tests. Happy to be wrong about it though if that's not the case.

I've used this approach and haven't detected any problem so far.
@rbren does it work for you?

I haven't been able to verify yet - still on the old version


Specify the privacy strings to look for in HTTP requests

Here is a list of privacy strings to look for in HTTP requests and what their values should be after opting out:

Restricted Data Processing (RDP): should be rdp=1 per Google's documentation (e.g., calls to Google Ad Manager (GAM))
US Privacy String: should be Y in the third position per the IAB's documentation
...

@stanleymarkman and @kbeliauski, please familiarize yourself with the above. Also, add other candidate strings to the list above. If you are unsure about a particular string (e.g., the GDPR string we discussed today), put it in and we figure it out.

@stanleymarkman and @kbeliauski, let's just put new privacy flags here in this issue. That way we can discuss whether or not to include them. Then, we can put those in the relevant background file or the separate data layer (issue #159).

Here is an overview of privacy flags in HTTP requests:

Google Restricted Data Processing (RDP)
IAB US Privacy String
DAA cookie opt outs
Consent Management Platform (CMP) solutions (e.g., OneTrust)
Individual solutions websites implement themselves
Apple's Centralized Consent API (more general and beyond CCPA)
IAB Transparency Consent Framework (TCF v2.0, TC String encoding/decoding, for GDPR and ePrivacy Directive)
Global Privacy Control (GPC)

We definitely want RDP (1) and US Privacy String (2).
We should be able to identify in HTTP requests that the RDP flag is set (i.e., restricted_data_processing = "true") as well as the US Privacy String (e.g., us_privacy is set to 1NYN).
We may want DAA cookies (3) and CMP solutions (4).
For 3, the DAA does not provide a specific format. So, all of the 100 members or so are doing their own thing. We can come up with a list of their cookie names and values (here is an outdated list we compiled a while back). For 4, we can play around a bit with websites that use certain CMPs to see how their web traffic looks like when opted out vs opted in (via Fiddler or other tools). It may be that they do not have own signals but (at least partially) rely on RDP, US Privacy String, etc.
We probably can't do a whole lot about individual website implementations (5).
5 is somewhat of a scattershot. Sites can do whatever they want. Thus, unless, there is some canonical way developing of how to opt someone out, there is not much we can do principally. Maybe, we can include some keywords to look for, e.g., a cookie named opt_out with a value 1 or something like that. But that is probably about it.
6, 7, and 8 are for Europe or difficult to check for.
6 and 7 are relevant for Europe (which we should consider after having done 1 and 2). But for the time being focusing on the US is OK. We certainly can check for GPC being sent. However, there is no response required beyond the voluntary .well-know/gpc.json. So, no hard evidence that a user has opted out.

I did an analysis to get a better understanding of the privacy strings/flags.
There are a few points to note and to discuss:

As it currently stands, European flags (e.g., TCF v.2.0) and other non-US flags do not help us immediately because those do not have a corresponding browser signal (such as GPC for the CCPA). Certainly, we could opt out manually on the individual sites and check whether the TCF flag is changed now. But that would not be in line with our whole point, which is to research opting out via automated browser signals. The bottom line for me at the moment is that we can leave European flags in the spec, but they are not concretely useful.
Google's rdp flag may not work for us. The reason is that it seems that it only shows up if it is enabled, i.e., rdp=1. Google's documentation does not say it explicitly, but I have yet to see rdp=0, i.e., a situation where restricted data processing is explicitly disabled. In such cases it rather seems that there is simply no rdp parameter at all ("passing in false and 0" does not seem to lead to rdp=0). And that makes it difficult for us to do any kind of before/after comparison. We simply would not know whether a site should have switched rdp to on, i.e., set rdp=1 if we did not observe rdp=0 in the first place. In case of a compliance issue, no rdp parameter would be in the observed traffic either before and after sending the GPC signal.
Essentially, that leaves the IAB's US Privacy String as the only usable privacy string for us at the moment. The question here is how to identify it. The IAB spec requires the URL parameter to be named us_privacy. However, I also have seen us-privacy, usprivacy, or even ccpa. What gave it away in those cases that this is that the string had the format 1***. So, maybe, we can use that format to identify the US Privacy String. E.g., there has to be a 1 in the first place and the other places can have N, Y, or -. Maybe, that format is specific enough to identify the string. This would need to be changed in our spec and implemented in our analysis functionality accordingly.

Let's discuss more tomorrow.

At this point we have converged upon detecting the IAB's US Privacy String. Adding to the previous comment, the rdp flag is not used that much outside of Google. Thus, the focus on the Privacy String is reasonable.
Further points will be discussed in the privacy flag implementation issue (#159).


DOC: optimization tutorial: subscript fixed

This PR fixes a subscript in a formula which incorrectly was not displayed as subscript.

Thanks @gertingold


Is it possible to tethered "upgrade" to a newer but not signed firmware?

I made my iPad Pro 2 (checkm8 compatible) got stuck into recovery mode (I've tried "irecovery -n" but not work) due to failed DelayOTA to 16.6 from 13.7, and I only have two shsh files for this device, which are 13.7 and 16.6. I tried using futurerestore to recover this device but no luck, I am still trying to find a solution to fix it but avoid to upgrade to the latest iOS version via iTunes.

I made my iPad Pro 2 (checkm8 compatible) got stuck into recovery mode (I've tried "irecovery -n" but not work) due to failed DelayOTA to 16.6 from 13.7, and I only have two shsh files for this device, which are 13.7 and 16.6. I tried using futurerestore to recover this device but no luck, I am still trying to find a solution to fix it but avoid to upgrade to the latest iOS version via iTunes.

there is not much solution


Create test notebooks for TrustyAI notebook

The test_notebook.ipynb file should evaluate the Python packages and the versions that are included on.
In this document described in detail the test framework -> https://docs.google.com/document/d/1sctOblkWdpUrv1DzzrzK_nzMMtA44awKmCvXDVAkVXo/edit

This is now completed with #244
Thanks for all the work.


Fix uneccesary latency with suppress/revive logic when rate-limited

Previously, we used Akka streams throttle to rate limit. A disadvantage to this
component was it would buffer a single element in the event of an overflow,
precluding our debounce logic from dropping it in favor of a more fresh
snapshot of the state.
By introducing and using RateLimiterFlow, our debouncing logic always presents
the latest available snapshot as allowed by the rate limiter
JIRA Issues: MARATHON-8664

I also triggered tests a few times to see if something is flaky ;)

...and turns out, that ReviveOffersActorTest has a few broken tests so let's fix these too.


pubspec.yaml error

VSCode Version:
OS Version:

Steps to Reproduce:

Does this issue occur when all extensions are disabled?: Yes/No

/needsMoreInfo


permissions getter for members

Describe the solution you'd like
would be nice to add a method to dpp::guild_member to get the base permissions.
A similar method like dpp::guild::base_permissions.

Aren’t we have this method to do?
guild_find->base_permissions(&event.command.usr).has(dpp::p_kick_members)
I think it still better to use tho.

seems like it would just be a shortcut to the existing methods in channel and guild?


Phoenix: Check calls, spawns, sequence activates, loads, etc against import block library or reusable sequence library

Checked for duplicates
No - I haven't checked
Alternatives considered
No - I haven't considered
Related problems
Many of the sequencing languages we have to support will have the ability to call, activate, spawn, load, etc. other sequences or blocks that live onboard the spacecraft with arguments. Right now, there's no way to check if a VML CALL or an FCPL SEQ_VAR_SEQ_ACTIVATE or @activate matches the definition of an onboard VML block or reusable sequence.
Describe the feature request

Add the ability to import block libraries or reusable sequence libraries that are custom per-project
Add a linting check for a project's activate functions against imported block libraries.

FCPL example:
User could use @activate directive, or call USER_SEQ_VAR_SEQ_LOAD command to load a sequence called "ep_start_mission_thrust.mod" with arguments 125, 0.5, FALSE, -44. When the user types either of these, the UI should check the call against the "ep_start_mission_thrust" definition from the block library and check that the input args are the valid types and are within the allowed value range.

Closing as duplicate of #1501


WIP: Make revad start on a per directory basis

Added a new --dir flag that allows for starting revad on any directory with a set of config files.
Having a directory like so:
~/code/reva
❯ tree ~/automation/reva/separate
/Users/aunger/automation/reva/separate
├── auth.toml
├── frontend-oidc.toml
├── frontend.toml
├── gateway.toml
├── phoenix.oidc.config.json
├── storage-home.toml
├── storage.toml
└── users.demo.json

Would allow for revad to launch every service defined on each config file on the given folder like:
> revad --dir ~/automation/reva/separate
This is a preliminary implementation.
It all lives under the same terminal session so all logs are printed on STDOUT. This is more intended to speed up development as it does not handle graceful shutdown just yet.

Good idea, but i prefer to run each bit in its own container so i can filter logs easy

it is still possible to run each bit on it's own process :D, this would just make it faster to spin something up

Hi @refs, @madsi1m, for testing/dev purposes we can have such funcionality and even with restart logic, we just need to fork one chidren process per configuration file in the directory and the master process will be a dummy process.
Is not the responsibility of Reva to handle the orchestration and observability of the different services, that is delegated to enterprise-grade components like Istio/Envoy and K8s.
Each tool for the right thing

Is not the responsibility of Reva to handle the orchestration and observability of the different services, that is delegated to enterprise-grade components like Istio/Envoy and K8s.

100% agree, although that was not the intention of this PR. As more people join the project we're in the case that we need to explain how to run each individual config "strategy" (strategy as in 1 single file vs n files) when Reva, for some, is a mean to an end as they're working on the oC clients.
I'm more than open to suggestions on how to improve this, as I believe it will streamline development.

@refs I agree for on boarding and developing against make sense.
Can you change the flag name to dev-dir instead of dir so is clear that is only for dev purposes?
Thanks!

Great idea

@refs merged :)

💃 thanks!


add consent history section to profile page

add consent history (to include deleted and expired consents also) section to profile

fix ajax call to set consent only when no valid consent is present (not counting deleted and expired one(s)

minor styling and text formatting fixes in profile creation page, found while testing

Paul - I realized that the changes I made in this pull request are in conflict with the changes I made in pull request #349 (fix frontend procedure date validations) earlier today.  I will wait until #349 is reviewed and merged before I resolve conflicts on this one.  I just want to give you heads up on this. - Amy

@achen2401 #349 is now merged.  You're welcome to rebase this one as you see fit.

Paul - this branch should have the latest from develop and is ready for you to review - I also added some bug fixes - as you will see in the profile_macros.html


JSONSchemaType fails wiht type when a field is declared nullable

What version of Ajv are you using? Does the issue happen if you use the latest version?
8.12.0
Ajv options object
export const ajv = new Ajv({ strict: true })
I'm using Typescript
interface MyType {
  id: string
  value: string | null
}

const MySchema: JSONSchemaType<MyType> = {
  type: "object",
  properties: {
    id: { type: "string" },
    value: { type: "string", nullable: true },
  },
}
export const validateMySchema = ajv.compile<MyType>(MySchema)

I'm getting this issue:
src/user/validation.ts:13:7 - error TS2322: Type '{ type: "object"; properties: { id: { type: "string"; }; value: { type: "string"; nullable: true; }; }; }' is not assignable to type 'JSONSchemaType<MyType>'.
  The types of 'properties.value' are incompatible between these types.
    Type '{ type: "string"; nullable: true; }' is not assignable to type '{ $ref: string; } | ({ anyOf: readonly UncheckedJSONSchemaType<string | null, false>[]; } & { [keyword: string]: any; $id?: string | undefined; $ref?: string | undefined; $defs?: Record<...> | undefined; definitions?: Record<...> | undefined; } & { ...; }) | ({ ...; } & ... 1 more ... & { ...; }) | ({ ...; } & ... 2...'.
      Types of property 'nullable' are incompatible.
        Type 'true' is not assignable to type 'false'.

13 const MySchema: JSONSchemaType<MyType> = {

Also adding null as type:
const MySchema: JSONSchemaType<MyType> = {
  type: "object",
  properties: {
    id: { type: "string" },
    value: { type: ["string", "null"], nullable: true },
  },
}
export const validateMySchema = ajv.compile<MyType>(MySchema)

I have the same error:
src/user/validation.ts:13:7 - error TS2322: Type '{ type: "object"; properties: { id: { type: "string"; }; value: { type: ("string" | "null")[]; nullable: true; }; }; }' is not assignable to type 'JSONSchemaType<MyType>'.
  The types of 'properties.value' are incompatible between these types.
    Type '{ type: ("string" | "null")[]; nullable: true; }' is not assignable to type '{ $ref: string; } | ({ anyOf: readonly UncheckedJSONSchemaType<string | null, false>[]; } & { [keyword: string]: any; $id?: string | undefined; $ref?: string | undefined; $defs?: Record<...> | undefined; definitions?: Record<...> | undefined; } & { ...; }) | ({ ...; } & ... 1 more ... & { ...; }) | ({ ...; } & ... 2...'.
      Types of property 'nullable' are incompatible.
        Type 'true' is not assignable to type 'false'.

13 const MySchema: JSONSchemaType<MyType> = {

also removing the nullable, same issue:
const MySchema: JSONSchemaType<MyType> = {
  type: "object",
  properties: {
    id: { type: "string" },
    value: { type: ["string", "null"] },
  },
}
export const validateMySchema = ajv.compile<MyType>(MySchema)

Type '{ type: "object"; properties: { id: { type: "string"; }; value: { type: ("string" | "null")[]; }; }; }' is not assignable to type 'JSONSchemaType<MyType>'.
  The types of 'properties.value' are incompatible between these types.
    Type '{ type: ("string" | "null")[]; }' is not assignable to type '{ $ref: string; } | ({ anyOf: readonly UncheckedJSONSchemaType<string | null, false>[]; } & { [keyword: string]: any; $id?: string | undefined; $ref?: string | undefined; $defs?: Record<...> | undefined; definitions?: Record<...> | undefined; } & { ...; }) | ({ ...; } & ... 1 more ... & { ...; }) | ({ ...; } & ... 2...'.
      Type '{ type: ("string" | "null")[]; }' is not assignable to type '{ type: "string"; } & StringKeywords & { allOf?: readonly UncheckedPartialSchema<string | null>[] | undefined; anyOf?: readonly UncheckedPartialSchema<string | null>[] | undefined; ... 4 more ...; not?: UncheckedPartialSchema<...> | undefined; } & { ...; } & { ...; }'.
        Type '{ type: ("string" | "null")[]; }' is not assignable to type '{ type: "string"; }'.
          Types of property 'type' are incompatible.
            Type '("string" | "null")[]' is not assignable to type '"string"'.

13 const MySchema: JSONSchemaType<MyType> = {

JSONSchemaType equates "nullable" to optional property definition, string | null is not supported (I think)
cc @erikbrinkman am I right?

Yes, for ajv 8, nullable actually means undefined not null. the v9 branch has a version of the schema capable of recognizing the difference that you can patch in if you want if you really want. Also note, that JSONSchemaType requires that you use the required keyword, which you probably want to specify in this case.

Guys, is it connected to the next error?
JSONSchemaCannotBeCompiledError: JSON schema for response: POST | https://mock.com/ | 200 is found, but cannot be used since AJV cannot compile schema. This is OpenApi spec issue. Got AJV error Error with message: "nullable" cannot be used without "type". Validation cannot be done
Here is a part of my schema, the main problem in Brand property validation

Based on Swagger documentation, null is not allowed as a type value

https://swagger.io/docs/specification/data-models/data-types/?sbsearch=nullable
Do you plan to update your logic according specification?


Test watching and queueing of the events into workers/handlers

Issue : #13

The main goal is to add tests for the asyncio queueing multiplexer ("reactor") of Kopf.
It is incredibly difficult to test async logic without polluting the main codebase with numerous events/pause-points.
Instead, the code is restructured to test the individual methods of this reactor: event streaming, event watching, and event queueing. This goes against the "best practices" of test design, as it now tests the specific implementation, not the behaviour from outside. But this was the only way to do this.
Some terminology hints to make the review easier to understand the reactor:

Streaming means getting the events from the Kubernetes client's Watch.stream() synchronous method, and yielding them into the asynchronous reactor safely (i.e. non-blocking). It is consumed by the watching task.
Watching means getting these events from the infinite never-exiting stream, filtering out some of them silently (e.g. ERROR events cause the stream restart), and passing the normal expected events to the reactor's queueing task.
Queueing (the core of the "reactor") means getting the events to be handled in one stream, and then multiplexing (routing, distributing) them into the per-object (per-uid) queues, starting the workers tasks for each of these queues or reusing the existing ones, and garbage-collecting both the workers & the queues after some idle timeout of no events on the stream (~5s).
Batching of the events means that only the last one of the events should be handled (see #42), while all other events in the small time-window (~0.1s) are ignored.
Handling means getting the events of a specific object in its own dedicated asyncio task (thread), and calling the handler functions. — But this part is out of this PR, and will be done later.

It might be helpful to also read on the Reactor pattern.
As a side-effect of testing, few not-yet-detected issues were fixed:

The watching logic was extracted #62 — no logic changes, just some code restructuring.
The watcher & the workers now exit immediately — by using the EOS (end-of-stream) pseudo-event; previously, they exited after some timeout (5s), making the tests slow. The workers are given some timeout (2s) to exit gracefully before being force-killed, but this was not needed in the tests (might be necessary in the synchronous handlers in real operators).
All hard-coded timing constants are now tweakable if needed.

Rebased on the most fresh master.


[EUWE] Set session[:snap_selected] to nil if Snapshot does not exist

Fixes an error in the logs and the page not loading when the currently selected snapshot is deleted and attempting to view the Snapshot Summary page.
Fixes an Euwe merge conflict with https://github.com/ManageIQ/manageiq-ui-classic/pull/183
https://bugzilla.redhat.com/show_bug.cgi?id=1417763

@miq-bot add_label ui, bug


[Bug]: Non-existing mods

Operating System
Win11
BG3 Mod Manager Version
<IP_ADDRESS>
BG3 Game Version
Patch 5
Bug Summary
Mods that are not real appear in the list. They appear to be part of Patch 5 files. Touching them in any way breaks the game.

(with Show FIlename on)

Someone mentioned this in a Discussion thread and apparently found a solution.

does not change the fact that they're game files and should not be shown in the mod manager, this is not a solution

i did a fresh install of everything for patch 5; the game, BG3MM, and all mods. after the game was updated i opened it and MM before downloading and importing any mods. these two files were automatically showing in MM on the inactive side. judging by their names and file location, they certainly have to do with patch 5 updates for the golden dice and honour mode; both files sit in the folder with the original dice sets from the game. MM won't show their file name, or their path, so i think it's just reading that they exist in the folder, they're like ghost files to MM. they haven't affected my game at all just sitting on the inactive side. but they are clearly game files being shown in MM for some reason. i agree they probably shouldn't show in MM at all. i actually came here looking to see if there was an update to MM to ignore them yet. thought i would add my experience.

does not change the fact that they're game files and should not be shown in the mod manager, this is not a solution

Fair point, I should have said workaround rather than solution.
Anyway it looks like this has already been fixed in a couple recent commits, there just hasn't been a new release since. It's apparently controlled by a loose file:
<manager install dir>\Resources\IgnoredMods.json
so I replaced it with the updated version. But I'm not associated with this project so maybe you shouldn't trust my judgement...

This issue also leads to saves not being able to load due to "missing mods". even when the honor mode "mod" is activated the game still claims that it is missing, and refuses to load the save.

I am also having this issue currently with my Mod manager.


Sanitize/Validate name field

Perform validation on the name field.
Add extra email validation using pydantic in model, for API
General formatting
Closes #4

After this one we can make a release! @dni


[5.1] Terminable Middleware not called during integration tests

Given the following Middleware:
<?php namespace App\Http\Middleware;

use Closure;
use Log;

class Test
{
    /**
     * Handle an incoming request.
     *
     * @param  \Illuminate\Http\Request  $request
     * @param  \Closure  $next
     * @return mixed
     */
    public function handle($request, Closure $next)
    {
        Log::debug('Test#handle() called');
        return $next($request);
    }

    /**
     * Store request/response in log.
     *
     * @param  \Illuminate\Http\Request  $request
     * @param  \Illuminate\Http\Response $response
     */
    public function terminate($request, $response)
    {
        Log::debug('Test#terminate() called');
    }
}

laravel.log after GET / (in browser):
[2015-06-21 12:26:26] local.DEBUG: Test#handle() called
[2015-06-21 12:26:26] local.DEBUG: Test#terminate() called

laravel.log after phpunit (runs tests/ExampleTest.php):
[2015-06-21 12:27:30] testing.DEBUG: Test#handle() called

The test does not use WithoutMiddleware.
Is this intended behavior or a bug?

Most likely $app->terminate() was never called during the test runtime.

I'm having the same issue here. Is there any way to test a terminatable middleware?

@crynobone I tried to include App::terminate() into my test, but $this->terminatingCallbacks are empty ([]) in the test environment.
    /**
     * Terminate the application.
     *
     * @return void
     */
    public function terminate()
    {
        foreach ($this->terminatingCallbacks as $terminating) {
            $this->call($terminating);
        }
    }

@GrahamCampbell Issue is not resolved. Please reopen.

Sorry, but this is not a forum. We do have a forum, so please ask there. I'm sure there will be lots of people who can help you there. :)

So, bug reporting should be handled in the forums?
I consider it a bug, because the existence of WithoutMiddleware implies middleware will be executed if this trait is not included.
Terminable middleware is not executed correctly when doing things like $this->post('foo'). This is either a bug or unexpected behavior (since other middleware is executed just fine).
It can be fixed by contributing to this repository (by code or by clarifying the non-obvious behavior in the docs).
I reported it, @rafaelbeckel has the same issue (so having an issue on a GitHub repo to me seems like the exact right place to discuss this until it is resolved).
Why in your opinion shouldn't it be discussed/resolved here? I am not asking for a tutorial to write integration tests ...
I am confused.

You're going to have to provide more information then. Where are you declaring this middleware?

Middleware inside controllers for example DO NOT terminate. It's simply not supported.

The middleware is App\Http\Middleware\Test, so it is in app/Http/Middleware/Test.php.
I use it in the Kernel (app\Http\Kernel.php) like so:
<?php namespace App\Http;

class Kernel extends HttpKernel
{
    /**
     * The application's global HTTP middleware stack.
     *
     * @var array
     */
    protected $middleware = [
        \Illuminate\Foundation\Http\Middleware\CheckForMaintenanceMode::class,
        \Illuminate\Cookie\Middleware\EncryptCookies::class,
        \Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse::class,
        \Illuminate\Session\Middleware\StartSession::class,
        \Illuminate\View\Middleware\ShareErrorsFromSession::class,

        /**
         * Application specific middleware...
         */
        Middleware\Test::class,
    ];

Pretty standard and right out of the docs.

Hmmm, ok. Thanks for providing that.

Thanks. Let me know if I can help any further to resolve this.

The $response object is an instance of Symfony\Component\HttpFoundation\Response, which current implementation of send() method is:
public function send()
{
    $this->sendHeaders();
    $this->sendContent();

    if (function_exists('fastcgi_finish_request')) {
        fastcgi_finish_request();
    } elseif ('cli' !== PHP_SAPI) {
        static::closeOutputBuffers(0, true);
    }

    return $this;
}

I've not investigated what closeOutputBuffers() do, but that may probably have something to do with that. $response->send() is called just before terminate() in index.

We never call the send method when testing.

Isn't a call to $this->visit('/') a normal http request to the index.php file?

Isn't a call to $this->visit('/') a normal http request to the index.php file?

No?

@rafaelbeckel No. Instead of an actual HTTP request, the instance of Symfony\HttpFoundation\Request gets dispatched to the router directly, because it is substantially faster.

OK. I just read the call() implementation now. Sorry for the dumb question. Should have read it before posting. ;)

This should solve this issue: https://github.com/laravel/framework/compare/5.1...rafaelbeckel:patch-1

Further discussion can continue on the PR if required. Thank you everyone. :)


On-Demand Filter Chain

This Pull Request shows the working progress of "On-Demand Filter Chains".
Design Doc: On-Demand Filter Chains
API PR: https://github.com/envoyproxy/envoy/pull/12599
Related Issues: https://github.com/envoyproxy/envoy/issues/12183, https://github.com/envoyproxy/envoy/issues/12259
Background:
In the current Envoy LDS, filter chains are built on the construction of a listener, requesting dependencies such as SDS to make the listener ready for new connections. However, it’s possible that part of the dependencies will never be ready as envoy is multi-tenancy as an ingress gateway. Additionally, it’s common to see that most filter chains are ready while waiting a long time for a small part of unready filter chains, which is a huge waste of resources to keep these ready filter chains unused. It is also possible that only very few of ready filter chains will actually be used.
Description:
To fix this problem and make better usage of resources, we are proposing to delay the filter chain building process from listener’s construction to the arrival of a new connection that requires one particular filter chain.  The On-demand Filter Chains, instead of requesting dependencies during the construction of listeners, performs on-demand filter chain placeholder rebuilding in a worker-master-worker threading mode. In this way, listeners become available(active) and get into work earlier than before, with no need to wait for some dependencies that will never be truly used.
Implementation Overview:

Build filter chain placeholders when listeners are constructed on the master thread if the filter chain message contains OnDemandConfiguration.
Workers send a rebuilding request to the master thread if a new connection requires a filter chain which is currently a placeholder.
Store the socket for later connection retry.
Send filter_chain_message together with worker dispatcher and callback function to the master thread.
Master thread assigns PerFilterChainRebuilder(with a unique init manager) to request dependencies, rebuild the filter chain, and maintain the lifetime of this filter chain.
Multiple Workers/Listeners request rebuilding the same filter chain -> add to callback list, filter chain will be rebuilt only once.
Temporarily store rebuilt filter chain inside the placeholder. On rebuilt completion, callback with success signal. On timeout: callback with failure signal, rebuilt filter chain will becomes a placeholder again.
Workers receive callbacks from master. If success: retry all pending sockets. Else close all sockets.
Listener update during old listener rebuilding, on draining finished, close all unfinished rebuilding, close all sockets that require filter chain only available to the old listener.

API Changed: api/envoy/config/listener/v3/listener_components.proto

message OnDemandConfiguration: configuration for on-demand filter chain.
google.protobuf.Duration rebuild_timeout: timer for on-demand filter chain rebuilding.

Tests Covered:

filter chain manager

AddOnDemandFilterChain
RebuildOnDemandFilterChain

connection handler

OnDemandFilterChainRebuildingNoCallback
OnDemandFilterChainRebuildingSuccess
OnDemandFilterChainRebuildingFail
OnDemandFilterChainMultipleRebuildingRequests
ListenerUpdateDuringOnDemandFilterChainRebuilding

listener manager

OnDemandFilterChainSingleRebuildingRequest
OnDemandFilterChainRebuildingRequestsAfterRebuilding
OnDemandFilterChainRebuildingRequestWhileRebuilding
ListenerUpdateAfterOnDemandFilterChainRebuiltSuccessfully
ListenerUpdateWhileOnDemandFilterChainRebuilding

listener_lds_integration_test

BasicSuccessWithOnDemandFilterChain (including basic rebuilding and listener update after rebuilding)
RetryRebuildingAfterListenerUpdate (old listener failed rebuilding, updated listener retry rebuilding)
BasicSuccessWithMultipleClients
BasicSuccessOnDemandFilterChainWithMultipleClients (only post request on first connection, store following sockets and wait for callback)

@ASOPVII as I said in other issues, we will not review any code until there is a design document that covers all of the related issues in this area. All of them are driven by Istio so I would like to see a holistic doc on on demand filter chains, warming, etc. cc @lizan @lambdai @yskopets
Thanks!

@ASOPVII as I said in other issues, we will not review any code until there is a design document that covers all of the related issues in this area. All of them are driven by Istio so I would like to see a holistic doc on on demand filter chains, warming, etc. cc @lizan @lambdai @yskopets
Thanks!

Sure, I am working on a design doc for this task.

Design Doc: On Demand Filter Chains

Overall the doc looks good, but we need to drill down into error handling. This is where I think there is a lot of overlap with the work that @kyessenov already did and I want to be consistent. Thank you!

Overall the doc looks good, but we need to drill down into error handling. This is where I think there is a lot of overlap with the work that @kyessenov already did and I want to be consistent. Thank you!

Thanks for that notification! I came up with some ideas based on timeout strategy.

Error occurs during rebuild: Then master just notifies all workers connections failed.
Timeout: I am not sure what is the limit of a timeout on master/worker thread.
a) On Timeout: Master thread gives up rebuilding and notifies all workers that the rebuilding totally failed
b) On Timeout: Master thread notifies workers failed, but the rebuilding continues for another amount of time(additional time).

If rebuilding still failed after the additional time, Master thread does nothing(had notifying workers)

Else if rebuilding succeeded within the last deadline, master thread confirms that rebuilding success. Callbacks workers that the filter chain was built successfully. Since it had already sent a failure signal to workers, this is just to notify workers that this filter chain has been rebuilt successfully and next time a new connection requires this filter chain, they don't need to send rebuilding request to master.

Design Doc: On Demand Filter Chains

Can you share it publicly?

Design Doc: On Demand Filter Chains

Can you share it publicly?

Sure, updated

Thanks for that notification! I came up with some ideas based on timeout strategy.

Let's move the discussion to the doc, but the main complexity is not the timeout itself (that should be configurable as per normal xDS init timeouts), but the different ways of handling it (fail hard, go back to default, etc.). This is where we need to be consistent.

Current Implementation Progress:
PROBLEM required to be solved:
Dependency Cycle between (connectionHandler -> listener -> rebuildInfo -> connectionHandler)

[]x]Build fake filter chain placeholders when listeners are constructed on the master thread. [DONE]
2. Worker thread sends a rebuilding request to the master thread if the filter chain required by a new connection is a placeholder. [DONE]
3. Worker thread pauses the socket(connection) for later connection retrial.
4. Collect necessary rebuild information and pass to the master thread via master thread dispatcher. [TO BE ADDED]
5. Master thread assigns PerFilterChainRebuilder to request dependencies and rebuild the fake filter chain.
6. Multiple Workers/Listeners Same filter chain rebuilding requests -> add to callback list.
7. One dynamic init manager for rebuilding each filter chain.
8. On init ready, point fake filter chain placeholder to the real one.
9. Send retry connection callbacks to each worker.

Followup: multiple listener config update

Recent Update:
Working on writing tests for on-demand filter chains:

ListenerImpl Ctor with on-demand filter chains

New Connection: Rebuild Process:

find filter chain is fake in connection::newConnection
post listener.buildRealFilterChains
listener.buildRealFilterChains  create unique rebuilder (per filter chain)

filter_chain_manager.addRealFilterChain

placeholder->storeRealFilterChains
on_rebuilt_complete-> callback to worker
workerPtr->notifyListener -> handler.retryAllConnections

Tests to be added:

[x] ListenerImpl.Ctor ->buildFilterChains (filter_chain set build_on_demand)
[x] filterChainManagerImpl.addFilterChains (filter_chain set build_on_demand)
[ ] add listener with on-demand filter chains -> new connection -> listener.buildRealFilterChains
[ ] filter_chain_manager.addRealFilterChain -> placeholder update with real -> on rebuilt complete -> worker.notifyListener -> handler.retryAllConnections

API Changed: api/envoy/config/listener/v3/listener_components.proto

bool build_on_demand: option for filter chain to be built on demand.
rebuild_timeout: timer for on-demand filter chain rebuilding.

Tests Covered:

filterChainManager.addFilterChain when build_on_demand is set.
listenerImpl Ctor with build_on_demand filter chain.
Integration Test: listener_lds_integration_test when listenerConfig contains filter_chains with build_on_demand=true.

Making a diagram to show the situation when listener update during old listener filter-chain rebuilding.

Discovery about two cases for duplicate requests (for the same filter chain from the same listener):

Receive requests before rebuild completed, worker name already exists in the list. Do nothing.
Receive requests after rebuild completed, this will happen only when rebuild fails.

It's  possible that the second request has been post from worker to master, however, before the post is executed on master thread, the first rebuilding might be completed and post retry to worker thread. Then the master thread will retry sockets stored including the second one. Then the second rebuilding will continue, even though that the first callback has retried the 2nd socket.
A diagram to show the two cases.

Most features of the on-demand filter chain has been implemented. @lambdai @jplevyak

Recent Update:
PerFilterChainRebuilder should act as the FilterChainFactoryContextCreator for each filter chain.

Please merge main. @lambdai is this ready for my review or do you still need to do more reviews here?
/wait

OK thank I will wait until @lambdai gets back to review.
/wait

Recent Update:

Worker will post worker_dispatcher and callback (connection.retry) to server to replace passing a worker_name and use getWorkerByName.
Put pending_sockets inside each Active_TCP_Listener instead of the connection handler.

Help-Wanted: listener_lds_integration_test
As I set the filterchains of listener to be on-demand, RDSstream can not be created successfully.
Branch link: https://github.com/ASOPVII/envoy/tree/listener_lds_integration_test_debug

Recent Update: Introduce multiple clients making connections in listener_lds_integration_test. Testing that multiple arriving connections will trigger rebuilding request to master thread only once (on the first time). All pending sockets will be retried when rebuilding completes.

@ping-sun I tested this PR for various use-cases such as building a placeholder for a filter chain and then dynamically rebuilding the placeholder when client request comes.
In one of the use-case which can be explained as follows, I got a segmentation fault:
I have an ADS server (gRPC) to push the listener configuration to envoy. Using this ADS server, I pushed the following listener config:
address:
  socket_address: {address: <IP_ADDRESS>, port_value: 443}
filter_chains:
- filter_chain_match:
    server_names: [service1.ocenvoy.com]
  on_demand_configuration:
    rebuild_timeout: 20s
  filters:
  - name: envoy.filters.network.http_connection_manager
    typed_config:
      '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
      codec_type: AUTO
      http_filters:
      - name: envoy.filters.http.router
        typed_config: {}
      route_config:
        name: local_route
        virtual_hosts:
        - domains: [service1.ocenvoy.com]
          name: backend
          routes:
          - match: {prefix: /service/1}
            route:
              cluster: service1
              timeout: 0s
      stat_prefix: ingress_http
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
      common_tls_context:
        tls_certificate_sds_secret_configs:
        - name: service1.ocenvoy.com
          sds_config:
            ads: {}
            resource_api_version: V3
- filter_chain_match:
    server_names: [service3.ocenvoy.com]
  filters:
  - name: envoy.filters.network.http_connection_manager
    typed_config:
      '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
      codec_type: AUTO
      http_filters:
      - name: envoy.filters.http.router
        typed_config: {}
      route_config:
        name: local_route
        virtual_hosts:
        - domains: [service3.ocenvoy.com]
          name: backend
          routes:
          - match: {prefix: /service/1}
            route: {cluster: service1}
      stat_prefix: ingress_http
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
      common_tls_context:
        tls_certificate_sds_secret_configs:
        - name: service3.ocenvoy.com
          sds_config:
            ads: {}
            resource_api_version: V3
listener_filters:
- name: envoy.filters.listener.tls_inspector
  typed_config: {}
name: listener5

In this, the first filter chain is on-demand configuration. The envoy first built a placeholder for that filter chain by invoking addFilterChain(). It means it just created a FilterChainImpl object for this filter chain without building this filter chain completely. This is shown in the following code [link: https://github.com/ASOPVII/envoy/blob/filterchain-dev/source/server/filter_chain_manager_impl.cc#L206]:
// Reuse created filter chain if possible.
    // FilterChainManager maintains the lifetime of FilterChainFactoryContext
    // ListenerImpl maintains the dependencies of FilterChainFactoryContext
    auto filter_chain_impl = findExistingFilterChain(*filter_chain);
    if (filter_chain_impl == nullptr) {
      // If this filter chain include OnDemandConfiguration, we build a placeholder.
      // Otherwise, we build the filter chain directly.
      if (filter_chain->has_on_demand_configuration()) {
        ENVOY_LOG(debug, "filter chain will be built on-demand, first build a placeholder");
        filter_chain_impl = std::make_shared<FilterChainImpl>(filter_chain);
      } else {
        filter_chain_impl =
            filter_chain_factory_builder.buildFilterChain(*filter_chain, context_creator);
      }
      ++new_filter_chain_size;
    }
    addFilterChainForDestinationPorts(
        destination_ports_map_,
        PROTOBUF_GET_WRAPPED_OR_DEFAULT(filter_chain_match, destination_port, 0), destination_ips,
        filter_chain_match.server_names(), filter_chain_match.transport_protocol(),
        filter_chain_match.application_protocols(), filter_chain_match.source_type(), source_ips,
        filter_chain_match.source_ports(), filter_chain_impl);
    fc_contexts_[*filter_chain] = filter_chain_impl;
  }
  convertIPsToTries();
  ENVOY_LOG(debug, "new fc_contexts has {} filter chains, including {} newly built",
            fc_contexts_.size(), new_filter_chain_size);

This means that filter_chain_factory_builder.buildFilterChain(*filter_chain, context_creator) was not called for this filter chain. It implies that the context_creator has not created the filter_chain_factory_context and is not set in the FilterChainImpl object [Code Reference: https://github.com/ASOPVII/envoy/blob/filterchain-dev/source/server/listener_manager_impl.cc#L1029].
After this, I changed a field in the filter-chain config as shown here:
address:
  socket_address: {address: <IP_ADDRESS>, port_value: 443}
filter_chains:
- filter_chain_match:
    server_names: [service1.ocenvoy.com]
  on_demand_configuration:
    rebuild_timeout: 20s
  filters:
  - name: envoy.filters.network.http_connection_manager
    typed_config:
      '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
      codec_type: AUTO
      http_filters:
      - name: envoy.filters.http.router
        typed_config: {}
      route_config:
        name: local_route
        virtual_hosts:
        - domains: [service1.ocenvoy.com]
         # This field is changed
          name: backend-new
          routes:
          - match: {prefix: /service/1}
            route:
              cluster: service1
              timeout: 0s
      stat_prefix: ingress_http
  transport_socket:
    name: envoy.transport_sockets.tls
    typed_config:
      '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
      common_tls_context:
        tls_certificate_sds_secret_configs:
        - name: service1.ocenvoy.com
          sds_config:
            ads: {}
            resource_api_version: V3

and pushed the new listener config again from ADS server.
This triggered draining of the old listener's placeholder filterchain [service1.ocenvoy.com] by calling drainFilterChains() [code ref: https://github.com/ASOPVII/envoy/blob/filterchain-dev/source/server/listener_manager_impl.cc#L785]. diffFilterChain() (invoked by drainFilterChains()) invokes the callback that triggers startDraining() [code ref: https://github.com/ASOPVII/envoy/blob/filterchain-dev/source/server/filter_chain_manager_impl.h#L142 ].
Since factory_context_ (FilterChainFactoryContext) is not set in placeholder case (as explained above), it is nullptr. Thus, invoking factory_context_->startDraining() causing segmentation fault.
This issue can be fixed by adding an if condition in startDraining() function:
void startDraining() override {
    if (is_placeholder_ && !has_rebuilt_filter_chain_ && factory_context_ == nullptr) {
      return;
    }
    factory_context_->startDraining();
  }

Please let me know if the above fix is acceptable.

Any there any plans to implement this feature?


dev api

Hey,
I just wanted to ask if your plugin "PunishmentX" features a dev API, so you can personalize and customize the plugin for your own server, e.g. add some new features or use the plugin as a base to build a more personalized punishment plugin upon.
Kind regards
Flof_Noiflo

It's open sourced under an MIT license.
You are free to use/publish any code featured in PunishmentsX.


react-transform-hmr

The hot reloading feature doesn't work with HOC look component.
I get the following error when a do any update on one of my components.
TypeError: Cannot read property '__reactAutoBindMap' of undefined

Hmm, I actually never used react-transform before. I could check their docs in order to know what prevents the auto binding. Maybe it's because I use a map to store browser state information... Could move that to a good old plain object if it helps

Hmm, I actually never used react-transform before. I could check their docs in order to know what prevents the auto binding. Maybe it's because I use a map to store browser state information... Could move that to a good old plain object if it helps

That some interesting stuff https://github.com/FormidableLabs/radium/pull/255

Woot. Interessting. I will check that after lunch :)

Pushed the fix, will update tomorrow :)

@rofrischmann Thanks.


missing SAR metrics in README in staging

We no longer have SAR metrics in README.

this is something that we are going to fix very soon with the new benchmarks we are doing, this is work in progress, but here https://github.com/Microsoft/Recommenders/blob/benchmark_feb/notebooks/03_evaluate/comparison_movielens.ipynb you can see that we are using the same dataset split to train all the algos. Before we were running the notebooks ,which didn't have the exact same dataset split

duplicate of https://github.com/Microsoft/Recommenders/issues/464, being developed in PR https://github.com/Microsoft/Recommenders/pull/715


Create maintenance_starting.json

This notification will be required in https://github.com/openshift/ops-sop/blob/master/v4/howto/transfer_cluster_ownership.md
Mgmt approval done in https://coreos.slack.com/archives/CCX9DB894/p1670842811176189?thread_ts=1670834364.041809&cid=CCX9DB894

/lgtm

/approve


Move the disabling of samples and operatorhub sources earlier in the installation process

Which issue this PR addresses:
Fixes ADO#15431594
What this PR does / why we need it:
There's a issue where the clusterversion CR waits for samples to go healthy.  If we disable samples before we wait for the clusterversion, then it should not error out in installations.
ICM 331555133 has a similar failure scenario
Test plan for issue:
I can't replicate this, but customers are experiencing this.  The clusterversion CR shouldn't complain about a disabled operator so it's moved earlier in the install process.
Is there any documentation that needs to be updated for this PR?
no

lgtm. Is it possible to also disable the BareMetal operator the same way or do we just rely on it disabling itself?

It's possible in 4.11 now I guess
https://docs.openshift.com/container-platform/4.11/release_notes/ocp-4-11-release-notes.html#ocp-4-11-capabilities
So soon :)


Example Project has many compilation errors in Unity 2019.4 LTS

Looks to be all related to the unit test stuff. I gave fixing it all a quick go but it doesn't appear trivial. Apologies if this is called out elsewhere.
The actual state machine script does compile.

This has now been modernised to Unity 2019.4.19, along with many other changes


fix type 'int' is not a subtype of type 'double' in type cast

Casting using as will sometimes result in the error type 'int' is not a subtype of type 'double' in type cast when the value of data['duration'] is an exact integer on my Android device. Calling toDouble instead of casting resolved the issue.

Added a fix to this issue and raised a PR. Until @sarbagyastha approves the PR #459 you can update your pubspec.yml as follows:
  youtube_player_flutter:
    git:
      url: https://github.com/codeprismtechnologies/youtube_player_flutter.git
      path: packages/youtube_player_flutter


Pattern error in URL regex

The pattern does not compile due to pattern error.
https://github.com/oasis-open/cti-stix2-json-schemas/blob/0f1facfb07d44847e89868e38ed4683fae12d978/schemas/common/url-regex.json#L7
Regex101.com reports ) Unmatched parenthesis as the issue.
Also the regex is very Python specific so to check against the official schema, the checker needs to use Python's implementation of regex. Would it be possible to use a POSIX regex to allow wider support?

I've changed the schema to use the JSON Schema built-in URI format instead. Hopefully this solves the issue for you, but if not please let us know!

Can you pull in the change to the 2.1 branch so the fix can be used via the URLs in the schema files?


#12 Issue

I'm unsure as to what the error means.  Is it because I am treating the array like a list?

Can you send the code rather than a picture of the code so we can test it and help find the problem?

#A
frictionfactor = np.array([0, 0.025, 0.075, 0.125, 0.175, 0.225, 0.275, 0.325, 0.475, 0.5])
Gpoint = 50
ReG = np.array([np.logspace(math.log10(3500),10, Gpoint)])
#B
reLam = np.logspace(math.log10(1000), math.log10(200000), Gpoint)
fLam = np.zeros(Gpoint)
for i in range(0, Gpoint):
fLam[i] = fric_factor(reLam[i], 0)
fG = np.zeros((len(frictionfactor), Gpoint))
for j in range(0, len(frictionfactor)-1):
for i in range(0, Gpoint):
fG[j,i] = fric_factor(ReG[i], frictionfactor[j])
#C
plt.figure('ax',(10,8))
plt.plot(frictionfactor, ReG, '-', linewidth = 3)
mylegend = np.append([('reynolds', 'friction factor')])
leg = plt.legend(mylegend, loc='best')
plt.yscale('log')
plt.xscale('log')
plt.grid(b=True, which='major', color='k', linestyle='-', linewidth=0.5)
plt.grid(b=True, which='minor', color='0.5', linestyle='-', linewidth=0.5)
plt.xlabel('x name', fontsize=30)
plt.ylabel('y name', fontsize=30)
plt.savefig('figure.png')
plt.show()

The error message indicates that you are passing an array to the friction factor function. As it is written the friction factor can only evaluate one number at a time.
So then the question is, which of the two values that you are sending the friction factor function is an array?
I added these two lines of code to see if one of these elements was actually an array
print(roughfactor[0]) print(ReG[0])
The result I got back was

0.0
[3.50000000e+03 4.74047388e+03 6.42059788e+03 8.69619330e+03
1.17783078e+04 1.59527886e+04 2.16067937e+04 2.92646974e+04
3.96367239e+04 5.36848155e+04 7.27118473e+04 9.84824607e+04
1.33386724e+05 1.80661794e+05 2.44692147e+05 3.31416209e+05
4.48877109e+05 6.07968631e+05 8.23445547e+05 1.11529203e+06
1.51057507e+06 2.04595476e+06 2.77108431e+06 3.75321508e+06
5.08343372e+06 6.88510993e+06 9.32533822e+06 1.26304349e+07
1.71069277e+07 2.31699839e+07 3.13819153e+07 4.25043285e+07
5.75687597e+07 7.79723432e+07 1.05607388e+08 1.43036875e+08
1.93732162e+08 2.62394930e+08 3.55393232e+08 4.81352095e+08
6.51953438e+08 8.83019497e+08 1.19598025e+09 1.61986089e+09
2.19397378e+09 2.97156440e+09 4.02474954e+09 5.45120574e+09
7.38322811e+09 1.00000000e+10]

This indicates that the element ReG[0] is an array!
So look at how Reg is defined.
ReG = np.array([np.logspace(math.log10(3500),10, Gpoint)])
You have created an array of an array. Fix this by changing that line of code to
ReG = np.logspace(math.log10(3500), 10, Gpoint)
You are getting close!


Update README.md

Installation steps on UP Squared Board (UP-APL01)
Signed-off-by: Yugansh Mohan<EMAIL_ADDRESS>

Hey @yuganshmohan -- there were some issues identified with your patch by yetus (click on the checks section) https://519-182198941-gh.circle-artifacts.com/0/tmp/yetus-out/diff-patch-markdownlint.txt
Also, it seems that the sign-off on your commit didn't make DCO check happy.

Made a few errors, closing this PR. Will put up a new one with corrected DCO & yetus issues.


injectLabels should not inject on RoleList and RoleBindingList objects

Hi,
We enabled garbagecollection via injectLabels: true on our environments (tk 0.10.0) and we hit a bug with kube-prometheus.
Prometheus generates RoleList and RoleBindingList object kinds and labels are not allowed in its metadata:
error: error validating "STDIN": error validating data: ValidationError(RoleList.metadata): unknown field "labels" in io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta; if you choose to ignore these errors, turn validation off with --validate=false
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleList
metadata:
  labels:
    tanka.dev/environment: someEnv
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    name: prometheus-k8s
    namespace: default
  rules:
  - apiGroups:
    - ""
    resources:
    - services
    - endpoints
    - pods
    verbs:
    - get
    - list
    - watch
...
...

@fculpo that's really annoying. List objects don't support metadata in the same way. And it isn't immediately obvious how to handle this without tanka knowing every type of api-resource.
I suggest you disable injectLabels for now on those environments. We'll hopefully be able to get to this in a couple of weeks time.

@fculpo for a temporary workaround, maybe try unwrapping the List object in Jsonnet. Generally we recommend so far not using List at all, because it prevents Tanka  from analyzing your YAML, like you just noticed.
Due to the high rate of issues caused by this, we will definitely need to revisit this:

maybe unwrap List types by Tanka itself, so they end up in the regular YAML stream
make Tanka have access to the api-resources list, so we can do clever and correct decisions

Without some kind of unwrapping, List objects do not only fail validation, but their children also won't receive our gc labels, so prune will miss them.

Thanks for the feedback. Expanding lists within the tool would indeed help in several ways.
kubecfg does it too IIRC.
As suggested, we worked around that issue by expanding the *Lists.
In that kube-prometheus case, it can be done with:
  prometheus+:: {
    roleSpecificNamespaces: super.roleSpecificNamespaces.items,
    roleBindingSpecificNamespaces: super.roleBindingSpecificNamespaces.items,
  },

Hi, any news on this issue ?
The above workaround actually prevents tanka from detecting changes on servicemonitor objects (for unclear reason)

Hey, list unwrapping will come to Tanka by the end of July.


Fix LB WRR code to work with kernel 4.8.x

Given we can't depend on the 4.9 LTS yet, we really need to fix this so it runs on 4.8. I'm sure we can fix the pseudo c code to llvm produces the instructions in the right order. In particular as we are limiting the llvm release to 3.8.x

@mchalla You can test for HAVE_MAP_VAL_ADJ define.

Thanks. I already sent out a pull request.

https://github.com/cilium/cilium/pull/406


How to replace old prior.py references

Not sure if this is the right place for this, but I recently updated my enterprise code and none of my programs or scripts work as they used the old prior.py code. Is there any reference for how to refactor code to use the new way of defining priors?

It is the right place. Can you share an example that you'd need refactored?

I had to admit that I thought the Prior class was more widely used and panicked a bit when the import failed, but I guess the normal priors are "parameters", so I think that it's only a problem where I am defining my own prior. I think that this code wasn't really that useful in the end, so I can just disable it for now, but if you can suggest the right way to fix it that would also be great.
class VerbiestParallaxPriorRV(rv_continuous):
    def _rvs(self):
        return self.i_invCDF(np.random.uniform(0,1,self._size))
    def _pdf(self,x):
        if x < self.a or x > self.b:
            return 0
        else:
            return self.i_PDF(x)
    def _logpdf(self,x):
        if x < self.a or x > self.b:
            return -np.inf
        else:
            return np.log(self.i_PDF(x))
    def _cdf(self,x):
        if x < self.a:
            return 0
        elif x > self.b:
            return 1
        else:
            return self.i_CDF(x)

def VerbiestParallaxPrior(gl,gb,PXmin=1e-3,PXmax=15,s1400=None,R0=8.5,EE=0.33,size=None):
    GalCo=np.array([gl,gb])
    EE=0.5
    R0=8.5
    PP=np.linspace(PXmin,PXmax,10000)
    RR = np.sqrt( R0 * R0 + np.power( np.cos( GalCo[0] ) / PP, 2.0 )
                           - 2.0 * R0 * np.cos( GalCo[0] ) * np.cos( GalCo[1] ) / PP );
    Pvol = np.power( RR, 1.9 ) \
               * np.exp( - abs( np.sin( GalCo[0] ) / PP ) / EE \
                  - 5.0 * ( RR - R0 ) / R0 ) * np.power( PP, -4.0 )

    Pvol /= np.max(Pvol)
    zz = Pvol

    if s1400 != None:
        Plum = np.power( PP, -1.0 ) *\
            np.exp( -0.5 * np.power( ( np.log10( s1400 ) - 2.0 * np.log10( PP ) + 1.1 )\
                / 0.9, 2.0 ) );
        Plum/=np.max(Plum)
        zz *= Plum

    PDF = zz/np.sum(zz)
    CDF = integrate.cumtrapz(PDF,initial=0)
    CDF -= np.amin(CDF)
    CDF /= np.amax(CDF)

    RV = VerbiestParallaxPriorRV(a=PXmin,b=PXmax)
    RV.i_PDF    = interpolate.interp1d(PP,PDF)
    RV.i_CDF    = interpolate.interp1d(PP,CDF)
    RV.i_invCDF = interpolate.interp1d(CDF, PP)

    class VerbiestParallaxPrior(parameter.Parameter):
        _prior=prior.Prior(RV)
        _size=size
        _gl=gl
        _gb=gb
        def __repr__(self):
            return '"{}":VerbiestParallax({},{})'.format(self.name, gl,gb) \
                + ('' if self._size is None else '[{}]'.format(self._size))
    return VerbiestParallaxPrior

If you look at how the default parameters are defined, that might help you, e.g.Uniform.
You need a way to calculate the prior PDF (which you have) and draw samples from the prior (probably inverse transform samplling since you can calculate the CDF)

To complement Paul's explanation, you can also use UserParameter, which allows giving Enterprise functions as prior or logprior (you can also specify a sampler, but don't have to). The advantage to the new structure is that the prior/logprior can now have its own hyperparameters, which are themselves Enterprise parameters.
For instance (pardon the bitmap, it was the quickest way to get this through):


console fails to connect when app has compute intensive task

Finally got this issue that has been bothering me for ages narrowed to small that I think demonstrates a clear problem.
See program below (which is, in part, an adaptation of the examples/app from console-subscriber).
#![allow(dead_code)]

use std::time::Duration;

static HELP: &str = r#"
Example demo of console failing to connect when you spin up compute intensive task.

By default, the compute-intensive task *does* frequently call yield (but this
does not address the problem). One can also opt out of yielding (which continues
to exhibit failure to connect), or can sleep for 10ms (which resumes the ability
to connect).

USAGE:
    toys [OPTION]

OPTIONS:
    -h, help    prints this message
    no-fact     skips spawning computation (so you can demonstrate successful connect)
    yield-now   computation frequently calls yield_now (this is on by default)
    no-yield    skips calling yield from computation (ideally would also support connect)
    sleep       sleeps for 10ms instead of yielding from computation (enables successful connect)
"#;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    console_subscriber::init();

    let mut run_fact = true;
    let mut yield_type = YieldType::YieldNow;
    for opt in std::env::args().skip(1) {
        match &*opt {
            "no_fact" | "no-fact" => {
                run_fact = false;
                println!("will not run compute intensive loop.");
                break;
            }
            "no_yield" | "no-yield"  => yield_type = YieldType::NoYield,
            "yield_now" | "yield-now" => yield_type = YieldType::YieldNow,
            "sleep" => yield_type = YieldType::SleepMillis(10),
            "-h" | "help" | "-help" | "--help" => {
                println!("{}", HELP);
                return Ok(());
            }
            opt => { panic!("unrecognized command line option: `{}`", opt); }
        }
        println!("Will now yield with yield type {:?}", yield_type);
    }

    let task1 = tokio::task::Builder::new()
        .name("task1")
        .spawn(spawn_tasks(1, 10));
    let task2 = tokio::task::Builder::new()
        .name("task2")
        .spawn(spawn_tasks(10, 30));

    let mut handles = Vec::new();
    if run_fact {
        handles.push(tokio::task::spawn(async move {
            loop { fact_yielding(10, yield_type).await; }
        }));
    }

    println!("handles.len(): {:?}", handles.len());

    println!("joining task1 and task2");
    let result = tokio::try_join! {
        task1,
        task2,
    };

    result?;

    println!("joining handles");
    for h in handles {
        let val = h.await?;
        tracing::trace!("h.await?: {:?}", val);
    }

    Ok(())
}

// Computes factorial of n, printing progress along the way.
async fn fact_stiff(n: u32) -> f64 {
    let mut i = 0_u32;
    let mut accum = 1_f64;
    loop {
        tracing::trace!("i: {I} fact_{N}({I}): {A}", N=n, I=i, A=accum);
        if i == n { break; }
        i += 1;
        accum *= i as f64;
    }
    return accum;
}

#[derive(Copy, Clone, Debug)]
enum YieldType {
    NoYield,
    YieldNow,
    SleepMillis(u64),
}

// Computes factorial of n, printing progress along the way.
async fn fact_yielding(n: u32, yt: YieldType) -> f64 {
    let mut i = 0_u32;
    let mut accum = 1_f64;
    loop {
        tracing::trace!("i: {I} fact_{N}({I}): {A}", N=n, I=i, A=accum);
        if i == n { break; }
        match yt {
            YieldType::NoYield => {}
            YieldType::YieldNow => {
                tokio::task::yield_now().await;
            }
            YieldType::SleepMillis(ms) => {
                tokio::time::sleep(Duration::from_millis(ms)).await;
            }
        }
        i += 1;
        accum *= i as f64;
    }
    return accum;
}

#[tracing::instrument]
async fn spawn_tasks(min: u64, max: u64) {
    loop {
        for i in min..max {
            tracing::trace!(i, "spawning wait task");
            tokio::task::Builder::new().name("wait").spawn(wait(i));

            let sleep = Duration::from_secs(max) - Duration::from_secs(i);
            tracing::trace!(?sleep, "sleeping...");
            tokio::time::sleep(sleep).await;
        }
    }
}

#[tracing::instrument]
async fn wait(seconds: u64) {
    tracing::debug!("waiting...");
    tokio::time::sleep(Duration::from_secs(seconds)).await;
    tracing::trace!("done!");
}

Interesting, thanks for the report!
Can you also post the Cargo.toml for the reproduction? In particular, it would be nice to know if you're using the multi-threaded or single-threaded Tokio runtime.

Okay, I played around with the repro and I think I've figured this one out.

@pnkfelix do you have a minute to try out #178 and let me know if it fixes your issues?


Read multiple remotes for deletion from file

The branch command can only delete one branch at a time.
(Except when --all is given)
Make it delete many remotes by reading them from a file.
I have to delete around 2000 remote branches.
Calling branch --delete 2000 times would take ages.

Why are the remotes not delete automatically, when they are deleted from the server btw?
The last changeset almost always is "Delete branch".

Why are the remotes not delete automatically, when they are deleted from the server btw? The last changeset almost always is "Delete branch".

That is a good question. I assume that it simply wasn't implemented, but I am not familiar with the code, as it predates my involvement and I personally haven't used the --delete feature at all here.


Don't suggest <?php on > character in HTML not fixed by 2.0.0 (on 2.0.1)

I had the extension disabled whilst waiting for the fix to arrive, having re enabled it (having realised I had 2.0.1 now) I am still encountering the behaviour

I don't know if having the extension disabled meant it didn't really update but I have tried uninstalling and reinstalling the extension to no avail.

Looks like VS Code doesn't send the new parameter yet.
@dbaeumer?
{
    "id": 4,
    "method": "textDocument/completion",
    "params": {
        "textDocument": {
            "uri": "file:///Users/felix/git/felixfbecker/php-language-server/Performance.php"
        },
        "position": {
            "line": 1,
            "character": 4
        }
    },
    "jsonrpc": "2.0"
}

This requires a new version of the LSP npm modules which I will publish with the next stable release of VS Code.

Sill no support (Either out of the box or via extensions) for html inside php file :(
Come on VS !!!

looks like the intellisense is strictly based on file types, e.g. if it is a pure html, php, javascript, css it worked well, any mix of them will not(e.g. php+html mix), but these file sometimes do mix up. Can intellisense detect php/js/css/html tags and do intellisense/auto-complete based on those blocks instead of just the file name type?
really need to make sure intellisense, emmet, auto-complete work  for code that mixes html/js/css/php etc together.

I can confirm that with VS Code 1.19 stable, and PHP IntelliSense 2.1.0, <?php is still being suggested on almost all keystrokes.
For example, if typing

 the <?php suggestion is present after typing d, i, v, >, <, d, i, v, and >.
It's only not present on the very first < or the / character.


I need Assembly References List ! (Screenshots are preferable)

LLMR's chat-flow itself works, but always fails in compiler stage.
(Error: Object reference not set to an instance of an object)
The cause must be "Assembly References Assets" setup.
Some elements (Assembly References at Inspector tab) remains "None"

Although guids are listed on GitHub...
Although GUIDs for Assembly References are listed in the GitHub repository, I’m unsure how to correctly create the "Assembly References Assets" and which specific assets to assign to each element (Element 0, 1, ..., 22) in the Inspector tab　.

Additional Question
When I clicked the Select Loaded Assembly button, assemblies with PublicKeyToken = null appeared. As an alternative, I used Select Assembly File and chose .dll files from LLMR-main/Assembly/xxxxxx.dll, but I’m uncertain if this setup is correct.

Request for Screenshots
Could someone provide screenshots of the correct Assembly References setup for each of the following scenes?

DalleCLIP_Modular.unity
Playground_DallE-CLIP_Refinement.unity
Playground_new.unity
I’d like to see the exact lineup of Assembly References (Compiler object > Inspector > Assembly References).

Unity Version: 2022.3.11f1

I have identified the cause of the issue and will close this myself.
The problem was due to mistakenly adding unnecessary (self-created) Assembly Reference .asset files to RoslynC#Settings.
I reverted RoslynCSharp/Resources/RoslynCSharpSettings.asset back to its original list of 14 referenced .dll files
(removing extra .dll references by clicking the "–" button to clean up).
After setting up following these steps,
https://github.com/microsoft/LLMR/issues/10#issuecomment-2470759771
I successfully managed to make a cube appear, confirming that the runtime compiler is now working.


Community standards page

Good afternoon. I made following changes:

adding a link in Community's submenu to Standards page,
changing directory structure in README.md,
renaming community.html to standards.html,
replacing asking for contribution with conduction rules in a new site,
uploading happy children' photo as header image.

Have a nice day!
Fixes #18

I signed your CLA.
It's the source of the photo: https://pixabay.com/photos/children-s-children-asian-man-1256840/

Thanks!
LGTM

You're welcome.

@netprise why did you close this PR?

Because I thought 'Add a community standards page' issue is resolved.

No, you 're closed this PR before merge. Don't worry, we'll do it as soon as we fully approve all changes

I'm sorry.

No worries! And thanks for you contribution!

@fox1t Is it okay to make changes? In my opinion, yes. perhaps texts should be updated in the future

@fox1t ping

@netprise thanks for adding this!! 💛 ❕


Update Cocoapods version in CI after DT_TOOLCHAIN_DIR fix is in an official release

An error was introduced to Cocoapods with Xcode 15, which causes macOS builds to fail.
error: DT_TOOLCHAIN_DIR cannot be used to evaluate LD_RUNPATH_SEARCH_PATHS, use TOOLCHAIN_DIR instead (in target 'Pods-Runner' from project 'Pods')

The issue was fixed upstream in Cocoapods (https://github.com/CocoaPods/CocoaPods/pull/12009), but the fix is not in an official release yet.
We added a workaround to Flutter in https://github.com/flutter/flutter/issues/132755, but we had to ignore some errors in tests due to this.
This is the tracking issue to upgrade Cocoapods in CI once https://github.com/CocoaPods/CocoaPods/pull/12009 is in an official release.

This looks like a likely dupe of https://github.com/flutter/flutter/issues/131527.

Hi @vashworth :)
According to the CocoaPods releases the mentioned PR (https://github.com/CocoaPods/CocoaPods/pull/12009) is now official in 1.13.0: https://github.com/CocoaPods/CocoaPods/releases/tag/1.13.0

I've sent a patch to bump the recommended version in https://github.com/flutter/flutter/pull/135447

Umbrella bug for updating the ruby self contained package: https://github.com/flutter/flutter/issues/136377

@godofredoc I can see there's active work happening on blocking work; do you have an ETA for Cocoapods itself being updated? I'm trying to evaluate whether to invest time in a flutter/packages workaround for the old version (https://github.com/flutter/flutter/issues/136737) or if it's become moot shortly due to having 1.13 on the bots.

@cbracken can you please validate this has been fixed with the new ruby dep?

Removing myself from this bug, everything has landed and only the confirmation is missing. @vashworth FYI

Verified fixed in https://github.com/flutter/flutter/pull/140395.


creating epoches without event

Describe the bug
trying to create an epoch with a continuous dataset which doesn't have events so creating events manually but unsuccessful.
using np array yet it says needs numpy array of int and has numpy.ndarray
Steps to reproduce
events=np.linspace(0,15*60,600))
epochs = mne.Epochs(edf_h01_r_hf.,events,  tmin=-0.2, tmax=0.5, preload=True)
Expected results
epoch formation
Actual results
TypeError: events should be a NumPy array of integers, got <class 'numpy.ndarray'>
Additional information
Platform:      Linux-5.4.0-74-generic-x86_64-with-glibc2.29
Python:        3.8.5 (default, May 27 2021, 13:30:53)  [GCC 9.3.0]
Executable:    /usr/bin/python3
CPU:           x86_64: 4 cores
Memory:        3.7 GB
mne:           0.23.0
numpy:         1.19.5 {blas=openblas, lapack=openblas}
scipy:         1.6.3
matplotlib:    3.4.2 {backend=module://ipykernel.pylab.backend_inline}
sklearn:       0.24.2
numba:         Not found
nibabel:       Not found
nilearn:       Not found
dipy:          Not found
cupy:          Not found
pandas:        1.2.4
mayavi:        Not found
pyvista:       Not found
vtk:           Not found

can you see if this helps?
https://mne.tools/dev/auto_tutorials/epochs/60_make_fixed_length_epochs.html
https://mne.tools/dev/generated/mne.make_fixed_length_epochs.html#mne.make_fixed_length_epochs
https://mne.tools/dev/generated/mne.make_fixed_length_events.html
please use the forum https://mne.discourse.group/ to ask usage questions.
close the issue if your problem is solved. thanks

Thanks! that was  a huge help.
I will remember to ask in the forum next time.