make it compatible with ftrobopy/fttxpy/ftifpy

this may be a dumb question, but what is not compatible about this?

As far as I can see the whole API of this lib seems to have at least a
different naming scheme for the peripherals and their functionality. Just
adding some aliases should solve the problem.
Christian Bergschneider @.***> schrieb am Sa., 6. Jan.
2024, 00:46:

this may be a dumb question, but what is not compatible about this?
—
Reply to this email directly, view it on GitHub
https://github.com/Bloeckchengrafik/ftswarm.py/issues/4#issuecomment-1879387178,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEDLVELIX6SOJO6UD2UQLILYNCGDXAVCNFSM6AAAAABBOOHT5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZZGM4DOMJXHA
.
You are receiving this because you authored the thread.Message ID:
@.***>

In this API, I try to improve the following parts over the ftrobopy api:

Synchronous Behavior:
The current synchronous behavior in the ftrobopy API poses a challenge, particularly considering the slower nature of communication with external hardware. The time spent waiting for IO could be better utilized by other parts of the program, especially given the text-based protocol. Addressing this could significantly enhance the overall efficiency of programs interacting with external hardware.

Naming Convention:
The ftrobopy-style mixedCase naming convention diverges from the Python PEP 8 Style Guide, which I prefer to adhere to in all my Python software projects. For reference, you can review the PEP 8 guidelines on function and variable names here. Aligning with these conventions contributes to better code readability and maintainability.

Lack of Typehints Support:
Another notable aspect is the absence of support for Python type hints in the ftrobopy API. With everything structured as one large class with subclasses, incorporating type hints could enhance code clarity and facilitate better collaboration among developers.

Another part is the naming of things: (Why is there a digital input and a switch?)
In my development practices, I strive to align function and class naming with the official ftSwarm C++ API. This ensures a smoother transition when building swarm applications in both Python and C++.
It's worth mentioning that the last part of my feedback explains why I name my components the way I do, and it's not a critique of the ftrobopy API itself. While a synchronous wrapper API with ftrobopy-style naming could be a workaround, maintaining it may not be feasible due to time constraints.
I apologize for the length of this feedback, and I appreciate your understanding. I'm open to any suggestions or alternative solutions you might have.


Large buttons

input.btn.large gets width:210px; from input.large, so it is not sized according to its content.
Maybe there should be another class like .big for buttons with bigger font size, if you want to keep backwards compatibility with .large.

We no longer use .large and other classes for inputs. All form size classes are prefixed with .input-.


Add note about cron job suspend behavior

Call out the fact that suspended cron jobs count as missed.

/check-cla


[Bugfix] Preserving scrollable on subsequent rerenders - reported by @cprussin

Also added and additional _unbindEvent call in _bindEvent, since @cprussin also reported overly lavishly bound events.

@cprussin:
Can you check with my fork if these changes solve your issues?

Had to put this on the backburner for a few days.  Will get to this sometime this week, but not today.

backburner!  #ember-puns

My most recent commit solves @cprussin's problem as well as my previous problem with a simple solution: Naming the scrollable property holding the jquery Element a bit different(_scrollable), so it's not reset when route refreshing sets properties passed to the component to initial values.

This looks a lot cleaner @mkorfmann.  Still interested to know if it fixes @cprussin's case before moving forward

Sure, I'm eager to know if it solves his issues as well.

Sorry for the late reply, but yep, this seems to do the trick!
Thanks a bunch!

Cool, thanks for the feedback @cprussin. I will squash these commits today, so this PR can be merged.

The build failed due to Ember 2.0 being released, I could fix these minor issues, but that would probably not be within the scope of this PR.

:+1: glad to see this getting addressed, I temporarily have a fork in place to get around this issue

Is there still something to be done for this to get fixed?

tests need to be passing

Have looked at the details of the travis build? The tests are failing because this addon is not yet upgraded to Ember 2.0. So, naturally the 2.0 build is failing, like in any other current PRs.

Should I upgrade this addon to 2.0 within this PR?

Ideally the upgrade would be a separate PR, doing the bare minimum work needed to get it to support both 1.x & 2.x
I won't have time for this for a week or two at least

Ok, I will look into this as soon as possible and create a seperate PR.

@mkorfmann - this is ready to go yeah?

Yes, looks good to go.

Added the additional error message and prefixed the existing twos with "Ember Infinity:".

thankyou @mkorfmann !


Strange behavior with CollapsibleIfStatements

if (BuildConfig.DEBUG) {
    if (Constants.isTestApp) {
        doSomething()
    } else {
        throw UnsupportedOperationException("This method only exists for tests")
    }
}

I'm not sure how it expects me to collapse this.  I don't want to do anything if it's a RELEASE build.
If there weren't an else case, It would make sense, but I don't think this should be triggering CollapsibleIfStatements.

Thanks for reporting. I'll take a look.


tracking projections that work and do not work

I did the following subset of projections (almost all available from Proj4) to test how many are done correctly and show them in BeautifulMakie(output). Is better than expected, still some corner cases [that I don't use], but those are really hard to get right. Either way, I leave the code here as reference. This is just for lines, I can imagine that for surfaces (heatmaps) less will be successful.
using GLMakie, GeoMakie
GLMakie.activate!()
let
    projections = ["+proj=adams_hemi", "+proj=adams_ws1", "+proj=adams_ws2",
        "+proj=aea +lat_1=29.5 +lat_2=42.5", "+proj=aeqd", "+proj=airy", "+proj=aitoff",
        "+proj=apian", "+proj=august", "+proj=bacon", "+proj=bertin1953", "+proj=bipc +ns",
        "+proj=boggs", "+proj=bonne +lat_1=10", "+proj=cass", "+proj=cea",
        "+proj=chamb +lat_1=10 +lon_1=30 +lon_2=40", "+proj=collg", "+proj=comill",
        "+proj=crast", "+proj=denoy", "+proj=eck1", "+proj=eck2", "+proj=eck3",
        "+proj=eck4", "+proj=eck5", "+proj=eck6", "+proj=eqc", "+proj=eqdc +lat_1=55 +lat_2=60",
        "+proj=eqearth", "+proj=euler +lat_1=67 +lat_2=75", "+proj=fahey", "+proj=fouc", "+proj=fouc_s",
        "+proj=gall", "+proj=geos +h=35785831.0 +lon_0=-60 +sweep=y", "+proj=gins8", "+proj=gn_sinu +m=2 +n=3",
        "+proj=goode", "+proj=guyou", "+proj=hammer", "+proj=hatano",
        "+proj=igh", "+proj=igh_o +lon_0=-160", "+proj=imw_p +lat_1=30 +lat_2=-40", "+proj=isea",
        "+proj=kav5", "+proj=kav7", "+proj=laea", "+proj=lagrng", "+proj=larr", "+proj=lask",
        "+proj=lcca +lat_0=35", "+proj=leac", "+proj=loxim",
        "+proj=lsat +ellps=GRS80 +lat_1=-60 +lat_2=60 +lsat=2 +path=2", "+proj=mbt_s", "+proj=mbt_fps",
        "+proj=mbtfpp", "+proj=mbtfpq", "+proj=mbtfps", "+proj=merc", "+proj=mill", "+proj=misrsom +path=1",
        "+proj=moll", "+proj=murd1 +lat_1=30 +lat_2=50",
        "+proj=murd3 +lat_1=30 +lat_2=50", "+proj=natearth", "+proj=natearth2",
        "+proj=nell", "+proj=nell_h", "+proj=nicol",
        "+proj=ob_tran +o_proj=mill +o_lon_p=40 +o_lat_p=50 +lon_0=60", "+proj=ocea", "+proj=oea +m=1 +n=2",
        "+proj=omerc +lat_1=45 +lat_2=55", "+proj=ortel", "+proj=ortho", "+proj=patterson", "+proj=poly",
        "+proj=putp1", "+proj=putp2", "+proj=putp3", "+proj=putp3p", "+proj=putp4p", "+proj=putp5",
        "+proj=putp5p", "+proj=putp6", "+proj=putp6p", "+proj=qua_aut", "+proj=robin", "+proj=rouss",
        "+proj=rpoly", "+proj=sinu", "+proj=times", "+proj=tissot +lat_1=60 +lat_2=65", "+proj=tmerc",
        "+proj=tobmerc", "+proj=tpeqd +lat_1=60 +lat_2=65", "+proj=urm5 +n=0.9 +alpha=2 +q=4",
        "+proj=urmfps +n=0.5", "+proj=vandg", "+proj=vandg2", "+proj=vandg3", "+proj=vandg4",
        "+proj=vitk1 +lat_1=45 +lat_2=55", "+proj=wag1", "+proj=wag2", "+proj=wag3", "+proj=wag4",
        "+proj=wag5", "+proj=wag6", "+proj=wag7", "+proj=webmerc +datum=WGS84", "+proj=weren",
        "+proj=wink1", "+proj=wink2", "+proj=wintri"]

    fig = Figure(resolution=(1200, 9000))
    k = 2
    for i in 1:39, j in 1:3
        try
            ga = GeoAxis(fig[i, j]; dest=projections[k],
                title=projections[k], coastlines=true)
            hidedecorations!(ga)
        catch ex
        end
        k += 1
    end
    fig
end;

Neat!  This will definitely be useful, I'd love to add it as a test if you're OK with that!
Many of the projections which fail here have singularities or infinities at the traditional lon in (-180,180) and lat in (-90,90) limits.  E.g., web mercator cannot show the poles, airy also has singularity conditions. You could try to set lonlims=automatic, latlims=automatic in GeoAxis, which I'm pretty sure should help with at least the Mercator-type and Airy projections.
I'm currently changing the projection system which breaks a lot of this locally (basically all of it, actually) but the only way we could do this in the way which Cartopy does would be if we had knowledge of the domain of any arbitrary projection, which I don't believe Proj gives.
In general, I'd have to say that if lines work, so should surface (for the same domain).  Note that all the axes are either well-formed or nonexistent (which happens when the bounds of the axis project to infinity).

I'd love to add it as a test if you're OK with that!
Sure, that's why I'm sharing :D

Many of the projections which fail here have singularities or infinities.
Indeed, in this regard maybe some proper initial default values for certain projections and domains will be helpful.

I'm currently changing the projection system which breaks a lot of this locally...
Great!, I hope the user syntax is still similar, which as of now feels natural (in the Makie sense :D ).

The syntax shouldn't change, only the internals :D (right now there are issues with the inverse transformation which are pretty fundamental, for example).

Note for this issue: at this point, with the PR Block geoaxis, there's something strange with the size.  This is what I get:

The errors I get are:

Errors
```julia
Invalid point for transformation: Float32[-100.0, -5.31714f7]
+proj=cass
Proj.PROJError("cass: Invalid latitude")
┌ Warning: No strict ticks found
└ @ PlotUtils ~/.julia/packages/PlotUtils/bZEEj/src/ticks.jl:191
+proj=cea
InexactError(:trunc, Int64, NaN)
Invalid point for transformation: Float32[-100.0, -165.28813]
+proj=eqdc +lat_1=55 +lat_2=60
Proj.PROJError("eqdc: Invalid latitude")
Invalid point for transformation: Float32[-100.0, -164.70897]
+proj=euler +lat_1=67 +lat_2=75
Proj.PROJError("euler: Invalid latitude")
+proj=geos +h=35785831.0 +lon_0=-60 +sweep=y
InexactError(:trunc, Int64, NaN32)
+proj=guyou
ErrorException("Invalid text boundingbox")
+proj=igh
ArgumentError("range step cannot be zero")
+proj=imw_p +lat_1=30 +lat_2=-40
InexactError(:trunc, Int64, NaN32)
Invalid point for transformation: Float32[-100.0, -121.2566]
+proj=lcca +lat_0=35
Proj.PROJError("lcca: Invalid latitude")
Invalid point for transformation: Float32[-100.0, -171.52461]
+proj=murd1 +lat_1=30 +lat_2=50
Proj.PROJError("murd1: Invalid latitude")
Invalid point for transformation: Float32[-100.0, -171.27933]
+proj=murd3 +lat_1=30 +lat_2=50
Proj.PROJError("murd3: Invalid latitude")
+proj=ortho
ErrorException("Invalid text boundingbox")
Invalid point for transformation: Float32[-100.0, -526.6071]
+proj=poly
Proj.PROJError("poly: Invalid latitude")
Invalid point for transformation: Float32[-100.0, -51990.34]
+proj=rouss
Proj.PROJError("rouss: Invalid latitude")
Invalid point for transformation: Float32[-100.0, -174.89333]
+proj=tissot +lat_1=60 +lat_2=65
Proj.PROJError("tissot: Invalid latitude")
Invalid point for transformation: Float32[-100.0, -167.77003]
+proj=vitk1 +lat_1=45 +lat_2=55
Proj.PROJError("vitk1: Invalid latitude")
```

which leads me to believe that we should:
(a) just return NaN or Inf when Proj errors with invalid lat/long.
(b) create a custom "text boundingbox finder" which ignores all infinite textboxes, i.e., textboxes with origin at infinity.

You may find this PROJ issue informative on why some projections have not worked at all.

Yeah, I encountered that a while ago - that's why the transformation code in utils.jl is so verbose, it's actually a try-catch on each transformation in case Proj reports an error :(.
Maybe we could speed that up in Proj.jl itself, perhaps a fast path when applying a transform (or fast_apply(transform, ...) which returns NaN when the point is invalid for whatever reason?
With the latest changes to geoaxis that I've made, pretty much every projection just works (although we still have quite a bit of work to do to figure out valid limits etc).

Sorry,  can't comment on that. I know nothing about GeoMakie code. Just thought the info could be useful as I found the same problem when calling PROJ trough GDAL.


[regression - 8.0.7-9.1.0] Fix native fullscreen on macOS

Problem statement
macOS supports two types of fullscreen: Zoom and Native Full Screen. Zoom will maximize the window similar to Windows/Ubuntu. Native Full Screen maximizes the window to its own separate workspace. This is used by default by all macOS apps (all 3rd party incl.)
It would be nice to have Native full screen on pressing maximize(green) button.

[X ] I looked at nativefier --help and https://github.com/jiahaog/nativefier/blob/master/docs/api.md , no existing option fits my needs.
[ X] I searched existing open & closed issues. Yes, my issue is new.

Suggested solution
Electron definitely supports native full screen as electron apps are able to full screen natively. Thus it must be possible to include it as an option in nativefier.

This worked in version 8.0.7 but I just upgraded to 9.1.0 and it is broken.

I am facing the same issue. Please look into it :)

Issue persists in 10.1.0 (on macOS Catalina 10.15.6 (19G2021)).
But, interestingly, when using nativefier --full-screen '<url>' to force the app to open in native full-screen: that works!
And, I can exit full-screen and re-enter it with no issues using the native window controls:

I'm also running into this issue of missing native full screen on macOS. In addition, when creating an app for YouTube, the video doesn't full screen to the entire monitor- it only fills the app window.

I'm having the same problem on Catalina with the latest version of Nativefier (via Homebrew).
After fiddling around a bit and reading some documentation of Nativefier and Electron I tried adding the following parameter:
--browserwindow-options '{ "fullscreenable": "true", "simpleFullscreen": "false" }'
That fixes my problem and windows behave as they should regarding fullscreen support. Maybe this helps in finding a general solution. If not, it is (at least for me) a possible workaround.

Don't think this works on Big Sur... I ran brew update just to be sure , as well as copy-pasting your parameter (but those already the defaults, no? Or would this imply there's a bug with it). It still only zooms (not the native macOS fullscreen) for me.
Macbook Pro (2019), macOS Big Sur 11.2.1

@firefinchdev @axelav @navinkumarg @rvanmazijk @AlJohri @ZepDonald can anyone confirm if this is still an issue with the latest version? When clicking maximize for me on the latest version on Catalina, it seems to be maximizing to Native full screen (on its own workspace).

For now native fullscreen is working like expected. Thanks for fixing this bug.


[WIP] shuffle array

hierarchy of pulls:
pull/208 "yubrew/132-implement-rand"
-pull/221 "jhernandezb/132-implement-rand-list"
-- pull/291 "yubrew/working-token-index"
--- pull/294 "yubrew/pub-shuffle"
---- pull/299 "yubrew/shuffle-array"
if pull 299 works then we're gucci
random minter tests:

try minter instantiate look at tx for gas
try random minting and look at tx's for gas
try shuffle and look at tx's for gas
try random minting and look at tx's for gas

array is cheaper instantiation, but because of item load -> save, incurs too much gas for mint


Yan MB-6296 Incorporate Eula into the Admin App

Description
Add the EULA Modal into the admin app so that it displays when the user clicks the sign in buttons.
Reviewer Notes
I had to add the React USWDS styling to the EulaModal so that it would look consistent across all the pages.
Setup
Add any steps or code to run in this section to help others prepare to run your code:
make admin_client_run
make server_run

Code Review Verification Steps

[ ] There are no aXe warnings for UI.
[ ] This works in Supported Browsers and their phone views (Chrome, Firefox, IE, Edge).
[ ] User facing changes have been reviewed by design.
[ ] Request review from a member of a different team.
[ ] Have the Jira acceptance criteria been met for this change?

References

Jira story for this change

Screenshots
If this PR makes visible UI changes, an image of the finished UI can help reviewers and casual
observers understand the context of the changes. A before image is optional and
can be included at the submitter's discretion.
Consider using an animated image to show an entire workflow instead of using multiple images. You may want to use GIPHY CAPTURE for this! 📸
Please frame screenshots to show enough useful context but also highlight the affected regions.

Warnings

:warning:
This PR does not include changes to unit tests, even though it affects app code.

:warning:

Files located in legacy directories (src/shared or src/scenes) have
been edited. Are you sure you don’t want to also relocate them to the new file structure?
View the frontend file org ADR for more information

Messages

:book:
:link: MB-6296

  Generated by :no_entry_sign: dangerJS against 106ea46e16d58a792ad70854859af7c439405844


push_notifications: Remove redundant APNs retry loop

aioapns already has a retry loop. By default it retries forever on ConnectionError and ConnectionClosed, so our own retry loop would never be reached. Remove our retry loop, and configure aioapns to retry APNS_MAX_RETRIES times on ConnectionError like the previous version did. It still retries forever on ConnectionClosed; that’s not configurable but probably fine.
Testing plan: Not tested live.

Looks great -- thanks @andersk, merged!


NetworkError during sync

Expected Behavior
Sync should work correctly because my google drive credentials have not changed and the network is not having any problems.
Actual Behavior
Sync shows Error: NetworkError when attempting to fetch resource. No, there is no network error. You have some other error.
Specifications

OS: Fedora 34
Browser: Firefox 88.0.1
Other extensions that might interfere:

@byron-hawkins
I cannot reproduce the problem with Firefox 89 on Windows 10.
Does it occur with a brand-new profile?

Sorry, I can't really create a fresh profile without completely destroying my entire browser configuration, which represents many hours of effort, much of which is not integrated into any export/import feature.

Could you create a new profile in about:profiles, install this extension and enable sync? It doesn't destroy your existing profiles.

Sure, well I'm trying to do this now, but when I go to sync, it asks where to sync. There is only my one google drive, which is already synced to uBlacklist in my main firefox profile. So if I also sync the test to the same google drive, what happens? And if I sync the test somewhere else, then it's not a test of the target scenario anymore, it's some other scenario that has nothing to do with this bug. So what can I do here?

So if I also sync the test to the same google drive, what happens?

The blocklist used in the original profile will be downloaded to the new profile via your Google Drive. (NOTE: If you update the blocklist in the new profile BEFORE enabling sync, it will overwrite your existing blocklist.)

So what can I do here?

clarify the condition under which the problem is reproduced; does it happen with a new profile of Firefox on Fedora?
tell me the detail of the error, which may be displayed in  about:devtools-toolbox?id=%40ublacklist&type=extension > Console or Network.

It does indeed function correctly in the test profile. My blocklist is downloaded, though it is not current because of course the main profile has not been able to sync in a long time. I tried restarting the browser in each profile and nothing changed: it works in the test profile, but not in my main profile.

Did you try re-installing the extension in your main profile?
(You may need to export your blocklist before it)

tell me the detail of the error, which may be displayed in about:devtools-toolbox?id=%40ublacklist&type=extension > Console or Network.

What about this?

Well, the error doesn't occur today, so I can't investigate. This is good though, the sync works! If the error happens again later, I'll try to check the log.

If you run into the problem again, please feel free to comment on this issue.


fix(bch-js): Updating to v4.15.2 to test rate limit idea

:tada: This PR is included in version 1.27.2 :tada:
Your semantic-release bot :package::rocket:


add possibility to use custom bin-folder for composer

This PR resolves issue #14.
I still haven't tested it fully, as I get some errors when testing locally or with my fork as git repo.
If I run it on my TYPO3 Crawler repository, which has a custom bin-folder, I get the following error.
$ .Build/bin/phpstan-bodyscan

In JsonLoader.php line 20:

  Could not decode JSON from phpstan: "At least one path must be specified to analyse.
  "

In JsonLoader.php line 18:

  Syntax error

The repository have a valid phpstan.neon file, so I don't know what I'm overlooking.
I went for the simple implementation and used file_get_content of the composer.json to avoid pulling in an additional dependency.

Thanks for the PR :+1:
I'll merge it, as it solves the custom vendor dir issue :muscle:

The follow up issue is probably this one: https://github.com/TomasVotruba/phpstan-bodyscan/issues/11
We'll need a way to pick up paths from phpstan.neon instead of the hardcoded ones

You're right, looks like that issue. I can confirm it by adding my Classes directly to https://github.com/TomasVotruba/phpstan-bodyscan/blob/main/src/Process/AnalyseProcessFactory.php#L20.
Thanks for merging it.

Thanks for taking this up.


Survival Mode Waves/Difficulty

I need to scale zombie count & spawnrate for rounds. also lower the amount of zombies added each round.


Support fine-grained configuration per channels for the same bot (e.g. Slack)

Overview
This issue is an epic and aggregates all the subtasks to finalize the complete feature.
Reason
There is a demand in community—5 open issues with a lot of reactions—to make the configuration more flexible in regards which notification should be collected and where to send.
Acceptance Criteria

[x] #643
[x] User is able to route notifications:

[x] to individual channels without needing to run multiple deployments.
[x] to a given channel based on the Kubernetes Namespace.

[x] User is able to define kubectl permissions per channel—in particular, configuring what commands can be executed and in which Namespaces.
[x] User-facing documentation is up-to-date.
[x] Unit-test/e2e test coverage is added to ensure no regression in the future.

To do

[x] Prepare design draft / proposal - @mszostok, see:

https://github.com/infracloudio/botkube/pull/623,
https://github.com/infracloudio/botkube/pull/626

[x] Gather community feedback about new design.
[x] Prepare a list of subtasks. Split them into:

[x] required—those which need to be finished for MVP
[x] follow-up—those which can be addressed later

[x] Start the asynchronous work to finish all required subtasks.

Related issues
This issue aggregates the feature requests from:

https://github.com/infracloudio/botkube/issues/250
https://github.com/infracloudio/botkube/issues/486
https://github.com/infracloudio/botkube/issues/444
https://github.com/infracloudio/botkube/issues/508
https://github.com/infracloudio/botkube/issues/542

All the tasks above will be eventually closed and the tracking of the feature will be done here.

Findings
The https://github.com/infracloudio/botkube/issues/250 is not strictly related to this issue, see: https://github.com/infracloudio/botkube/issues/250#issuecomment-1155030581
New related issues:

https://github.com/infracloudio/botkube/issues/508
https://github.com/infracloudio/botkube/issues/542

Closing as all subtasks have been resolved.


ENH: stats: more analytical formulas for fitting distributions to data

In the CZI Proposal, we wrote:

The continuous distributions in SciPy all have a method for fitting the distribution to data using
the method of maximum likelihood estimation (MLE). The proposed enhancements are:

Where possible, use an analytical formula rather than numerical optimization for greater
speed and accuracy
...

This is part of our effort to address the roadmap item "Improve the options for fitting a probability distribution to data".
One of the best sources I've found (via a NIST Engineering Statistics Handbook) is the Parameter Estimation sections from:
Evans, Hastings, and Peacock (2000), Statistical Distributions, 3rd. Ed., John Wiley and Sons.
Any other sources that are easily accessible?
Also, for method of moments (gh-11695), I could write a program that tries to solve for fitting formulas (from the formulas for the moments) using SymPy.  I don't think that would work as well for MLE, though. Might not even get anything new for MM.
Given the formulas, this should be pretty straightforward work: just override the fit methods of the distributions defined in  /scipy/stats/_continuous_distns.py, following existing examples (e.g. beta_gen) for extending documentation and any other conventions. For tests, I suppose we could mainly compare against the generic implementation. In cases where the generic implementation isn't working very well, we could check the partial derivatives of the likelihood function?
If you have any thoughts / words of wisdom before we get started, please let us know here.
@fletcheaston @swallan I think this is next.
Update:
@swallan @WarrenWeckesser A simple but really useful script for generating first-order conditions is here. Now it can (sometimes) solve for a variable in terms of the rest.

Continuous distributions
Current status of fit override
Page Number of MLE (Evans)

laplace
merged: gh-11988, good
124

rayleigh
merged: gh-12097, gh-17090, good
175

logistic
merged: gh-12738, gh-17117, gh-18177, almost good, but should use root_scalar
130

pareto
merged gh-15567, good
149

invgauss, wald
merged: gh-12514, further investigation needed
121

gumbel_r, gumbel_l
merged: gh-12737, good
101

powerlaw
merged: gh-13053, good
159

nakagami
see gh-10908
-

vonmises
merged: gh-17585, gh-18013, gh-18128. Almost good; see gh-18128.
192

weibull_min, weibull_max
see gh-11806
196

asymmetric laplace
awaiting PR
notes

genextreme
see gh-10446
-

Other distributions that have overridden fit method:
norm - looks good, nothing to do here
beta - investigate fitting loc and scale? Adjust inequality conditions for FitDataError.
expon - works perfectly, AFAICT
weibull_min - should implement analytical solution when loc is fixed
gamma - look into fitting when loc is not fixed. Adjust inequality conditions for FitDataError.
lognorm - merged: gh-16839; works quite well. I'd rather not work on it more, but others are welcome to.
uniform - works perfectly, AFAICT

@mdhaber Some of the Parameter Estimation sections from Evans, Hastings, and Peacock (2000), Statistical Distributions  say that the parameters are the solutions to a system of equations.
For example, in the Logistics Distributions (Section 28.3):

Should I only pay attention to sections that provide explicit formulas to derive the shape/location parameters, or is there a way to solve this type of system?

Take note of those. It can be easier to solve a system than the original optimization problem.

There are several distributions already in SciPy that could have MLE fit methods added based on the documentation from the Evans, Hastings, and Peacock textbook:
scipy stats.xlsx
Sorry if a excel file is not ideal to upload for this, it didn't look easy to format it as a table for markdown.
I didn't see a ton of use of the term "method of moments" so I haven't fully documented other methods of estimation yet.

Hi Sam,
Thanks for putting this together.
Fwiw MLE tends to be favored by statistics community for various reasons so you’ll see them more commonly mentioned.
For MLE sometimes you can get an estimator via EM as well. You can find one if your distribution can be represented as a mixture, either continuous or discrete.
Here is explanation I wrote recently about this:
https://stats.stackexchange.com/a/466873
In the case you can find such a representation, you can also use as a means of efficient r.v. Generation, simulate mixture rv then simulate next rv in your graph with first rv places as the value, repeat for each draw.
So if your documenting and find a mixture relationship, you’ve probably found two uses.
Sincerely,
-Lucas Roberts

On May 18, 2020, at 5:11 PM, Sam Wallan<EMAIL_ADDRESS>wrote:

There are several distributions already in SciPy that could have MLE fit methods added based on the documentation from the Evans, Hastings, and Peacock textbook:
scipy stats.xlsx
Sorry if a excel file is not ideal to upload for this, it didn't look easy to format it as a table for markdown.
I didn't see a ton of use of the term "method of moments" so I haven't fully documented other methods of estimation yet.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Sorry if a excel file is not ideal to upload for this, it didn't look easy to format it as a table for markdown.

That seems fine, but for potential future use recent releases of Pandas can export tables in Markdown

for potential future use recent releases of Pandas can export tables in Markdown

Thanks for the tip, @grlee77

We may keep track of the progress of this issue with this list of pull requests:

Continuous distributions
Current status of fit override
Page Number of MLE (Evans)
other notes

laplace
#11988
124

rayleigh
#12097
175

genlogistic

130

pareto

149

powerlaw

159
has matching moments

vonmises

192

Will Require Further Investigation to Cnsure Compatability

gumbel_l

101
related to genextreme and gumbel_r

genextreme

101

gumbel_r

101

invgauss

121
related to wald

wald

121

weibull_max

196
related to weibull min

weibull_min

196

I still need to look into how these related but distinct distributions will work with the definitions found in the text.

Thanks, Sam!

Don't know if this is in scope, but L-moments parameters estimates are often more robust to outliers, especially for extreme value distributions. There is an existing implementation in the lmoments3 package, but it does not seem to be maintained anymore. Having this functionality in scipy would be great and a lot of people in the hydrological sciences would be grateful.
Note that fortran code is available and googling turns up a couple of wrappers.

@huard, L-moments would be a great addition.  We considered including this in our CZI proposal, but we already had enough to do without it.

Here is a module for L-moments

There are several distributions in SciPy whose PDFs match that of the Evans textbook, but the given maximum likelihood estimates do not provide a good guess based on my observation and comparison with log-likelihood calculations.
scipy.stats.powerlaw

This is the MLE given for shape c in the textbook (a in scipy.)

import numpy as np
from scipy.stats import powerlaw

data = powerlaw.rvs(a=1.5, scale=3, size=10000)

a = len(data) / (np.sum(np.log(data)))

args = [data, (powerlaw._fitstart(data), )]
func = powerlaw._reduce_func(args, {})[1]

mle = powerlaw.fit(data, fa=a)
opt = powerlaw.fit(data)

print(a)
print(opt)
print(func(opt, data) - func(mle, data))

results in
=> 2.2951735969907996
=> (1.4981329611989254, 0.007238375787595926, 2.9926593757680857)
=> -433.6743531238026
Negative difference of the log likelihood indicates that it is not a good fit, which makes sense since the shape a that was created with the MLE is not close to the expected a, 1.5.
scipy.stats.vonmises

And we are given the MLE for the location, a:

However this estimate is much worse than the default optimizer:
import numpy as np
from scipy.stats import vonmises

data = vonmises.rvs(kappa=2, loc=2, scale=1, size=100)

a = np.arctan(
    np.sum(np.sin(data)) /
    np.sum(np.cos(data))
    )

mle = vonmises.fit(data, floc=a)
opt = vonmises.fit(data)

args = [data, (vonmises._fitstart(data), )]
func = vonmises._reduce_func(args, {})[1]

print(mle)
print(opt)
print(func(opt,data) - (func(mle, data)))

=>
(0.7097792398327256, -1.21691026300755, 0.47757754771381733)
(1.517470770406523, 2.145220714568419, 1.376472083641273e-35)
-7883.697993409623

The data was created with a loc of 2, but a is -1.2169....

@swallan I wrote some code to help derive the first-order conditions for MLE.
I plugged in the pdf of the powerlaw distribution using the notation in the book and get this output:

Those derivatives equal zero at critical points. Solving the second one for c, I see two omissions in the formula given by the book.

there should be a negative sign
x_i needs to be divided by b

Here's an update to your code:
import numpy as np
from scipy.stats import powerlaw

np.random.seed(12)
scale  =3
data = powerlaw.rvs(a=1.5, scale=3, size=10000)

a = -len(data) / (np.sum(np.log(data/scale)))

args = [data, (powerlaw._fitstart(data), )]
func = powerlaw._reduce_func(args, {})[1]

mle = powerlaw.fit(data, fa=a)
opt = powerlaw.fit(data)

print(a)
print(opt)
print(func(opt, data) - func(mle, data))

prints
1.505896207715652
(1.7277164581361997, -0.1642901635441375, 3.1642133311355245)
94.20666782387707

Usually the default fit method gets a better objective, actually, but at least the value of a is always close to what it should be.

For von Mises, there's nothing wrong with the formula. You just needed to use arctan2.
import numpy as np
from scipy.stats import vonmises

np.random.seed(9)
data = vonmises.rvs(kappa=2, loc=2, scale=1, size=100)

a = np.arctan2(
    np.sum(np.sin(data)),
    np.sum(np.cos(data))
    )

mle = vonmises.fit(data, floc=a)
opt = vonmises.fit(data)

args = [data, (vonmises._fitstart(data), )]
func = vonmises._reduce_func(args, {})[1]

print(mle)
print(opt)
print(func(opt,data) - (func(mle, data)))

prints:
(1.4209176118306086, 2.0857610265575404, 3.976067874882936e-31)
(1.733333333170029, 1.835804809404621, 3.779079569992055e-30)
215.89894443765752

Again, the fit is not always better, but the parameters are always reasonably close to what they should be.

@swallan note that in the code snippets above, I fixed the parameters at the values used to generate the random variates. e.g.
scale = 3
loc = 2
data = powerlaw.rvs(a=1.5, scale=scale, loc=loc, size=10000)

# I didn't change loc or scale after generating the data
# I'm using them as my fixed values to calculate the shape
a = -len(data) / (np.sum(np.log((data-loc)/scale)))

The formulae are still valid even if loc and scale are fixed at different values. E.g. if I do
...
scale = 4
loc = 1.3
a = -len(data) / (np.sum(np.log((data-loc)/scale)))
...

then I get the output
[1.8648278776216605, 1.3, 4]
(1.8648437500000021, 1.3, 4)
3.622226358857006e-07

You might notice then that for some values:
scale = 3.1
loc = 1.6
a = -len(data) / (np.sum(np.log((data-loc)/scale)))

you get output that suggests that the analytical solution is not as good as the numerically optimized one:
[2.30702272670952, 1.6, 3.1]
(1.9406250000000023, 1.6, 3.1)
-135.47177179157734  # objective function for analytical solution is worse?

This happens because the first-order conditions on the MLE are derived without considering the domain of the distribution's PDF, and they can return MLE parameters that would cause the data to be out of the domain. Strictly speaking, the analytical MLE parameters do maximize the log-likelihood function of the distribution, but SciPy's _reduce_func is only equal to the log-likelihood function when the data is inside the domain. _reduce_func introduces a penalty when the parameters are such that data is outside the domain.
We already ran into this sort of thing with one distribution, right? Do you remember which it was?

@swallan for powerlaw:
np.random.seed(1)
a = 5
s = 15
u = 3.5
data = powerlaw.rvs(a=a, scale=s, loc=u, size=10000)

a, u2, s2 = powerlaw.fit(data, fa=a)
print("Default fit errors: ", u2-u, s2-s)

def f(u):
    N = len(data)
    s = np.max(data)-u
    res = (a-1)*np.sum(1/(u-data))+N*a/s
    return res

u_min = -1000  # I'm still trying to figure out a theoretical lower bound
u_max = np.min(data) - 0.0000001  # epsilon less than np.min(data)
u3 = root_scalar(f, bracket=[u_min, u_max]).root
s3 = np.max(data) - u3

print("Custom fit errors: ", u3-u, s3-s)

I added some notes about MLE for the power law distribution to mlenotes.html.  Some of the notes reiterate @mdhaber's comments from above.  Earlier, @mdhaber said

I think it means you really need to fix the scale to get a maximum likelihood estimate

I think something like this is necessary, but I think it is the location that should be required to be fixed.  This set of requirements should always have a well-defined result:

The location must be fixed, and the given location must be strictly less than min(x).

If the scale is not fixed, the MLE for scale is
scale = max(x) - loc

@mdhaber said this above, and I also make the argument for this in my notes.
(If the scale is fixed, we require scale >= max(x) - loc.  Otherwise there is a data point that is not in the support of the distribution.)

If the shape a is not fixed, it is given by formula shown above in @mdhaber's sympy calculations (and in the notes that I put online).

@WarrenWeckesser some of this is in my code above. I no longer think that the location or scale must be fixed; the optima seem well defined based on the support of the distribution. I haven't written up the notes formally, but the code above efficiently calculates both the optimal scale and location given the shape. Since I already worked it out, I think we should talk before you spend too much time on it.

Thanks @WarrenWeckesser @mdhaber, I've had a busy week but will be adding this soon.

I've gone through the power law equations again.  Below are some notes about the issues that come up, but first I should note two things:

The shape parameter a is only required to be positive; it can be less than 1.  There is a plot in Forbes, et al, that shows the PDF for the shape being 0.5.  I only point this out because in one of our in-person conversations, I recall saying (or at least thinking) that the parameter was restricted to being greater than 1; that was not correct.  This introduces a complication, because in the three cases 0 < a < 1, a == 1 and a > 1, the PDFs are qualitatively very different.

@mdhaber's comments about the domain of the MLE optimization problem are important.  The support of the distribution is loc < x <= loc + scale, and we require that the provided data be in the support, which imposes boundary constraints on the parameters.
(Just like many other distributions, if the user provides fixed parameters that constrain the support, and also provides data that is outside the support, fit must raise a FitDataError, because the user is asking for an impossible calculation.)
So we know the following:

loc cannot exceed min(data).
loc + scale cannot be less than max(data).
Those conditions impy that the absolute smallest that scale can be is max(data) - min(data), which  can only occur if loc = min(data).

These conditions are boundaries for the optimization problem.  We've seen earlier that the first order conditions lead to some strange conditions, but in general, using the first order conditions for an optimization problem is only valid in the interior of the domain.  One has to consider the boundary separately.

I mentioned these notes before; I've updated them recently.  Only look at those notes for the nicely rendered expressions for the likelihood and log-likelihood functions at the top of the document; ignore the rest of the what I wrote there.  (In those notes,  I use a for the shape, μ for the location, and σ for the scale.)
Suppose all three parameters are free.  In the likelihood function, the scale only shows up in the denominator.  If all the other terms in the likelihood function are finite and nonzero, we can make the likelihood larger by making the scale smaller.  For the MLE, we make the scale as small as possible, which is max(data) - min(data).  That implies that the location must equal the minimum of the data, loc = min(data).  This is a problem, because this is exactly the case where the terms in the big product in the likelihood function are not finite and nonzero.
Suppose loc = min(data).  If a > 1, the PDF at x = min(data) is then 0.  In other words, trying to set loc = min(data) removes the x value min(data) from the support, and forces the likelihood function to be identically zero.  This breaks an assumption of MLE that the PDF of each data point is nonzero.  If 0 < a < 1, the term (min(x) - loc)**(a - 1) in the likelihood function results in division by zero, so the PDF is not defined there.  (If we try to pretend that infinity is simply a very big number and we allow loc = min(data) as the estimate, then the shape parameter a is left undetermined, since any value of a > 0 still gives an infinite likelihood function.)
To ensure that the MLE problem is well-defined, we could impose the constraint that loc must be fixed, and the fixed value must be strictly less than min(data).  Then the problem is well-defined.
Does fixing a different parameter result in a well-defined MLE problem?  Fixing just the shape does not, because the argument above still applies.
What if just the scale is fixed, with scale > max(data) - min(data)?  Then the MLE for the location depends on whether 0 < a < 1 or a > 1 (and we don't know which case holds a priori):

If 0 < a < 1, we maximize the likelihood by making the location as large as possible, so loc = min(data)--but this results in the same problem described above.
If a > 1, we maximize the likelihood by making the location as small as possible, so loc = max(data) - scale.   This gives a solution, but we had to know a > 1 to get it.

I guess the code could accept the case where just the scale is fixed, and then set loc = max(data) - scale.  Then it would use the formula for a derived from the first order condition.  But if this results in a < 1, the solution is not correct, and the code would have to generate an error stating that the data appears to be in a regime where the MLE is not well-defined (and perhaps suggest fixing the location instead of or in addition to the scale).
So to ensure that we're providing a solution to a well-defined MLE problem, at minimum it looks like we need to require that either the location is fixed (and the value is less than min(data)), or the scale is fixed (and the value is greater than max(data) - min(data)).  In the latter case, if the shape a is fixed, we require that a > 1. if the shape is not fixed, then we have to check that when the shape calculated (using the formula derived from the first order condition), it is greater than 1.

@swallan for Weibull the conditions are:
(whenever shape is free)

(whenever scale is free)

(whenever location is free)

Re: powerlaw

So to ensure that we're providing a solution to a well-defined MLE problem, at minimum it looks like we need to require that either the location is fixed (and the value is less than min(data)), or the scale is fixed (and the value is greater than max(data) - min(data)).

At least for a > 1, I'm certain there can be a well-defined solution, as I wrote above. Here's a (terrible) plot of the log-likelihood function for data generated with shape = 5, loc = 3.5, scale = 15.

The blue line is loc = np.min(data) and the orange line is sigma = np.max(data) - loc. All points outside the feasible region were assigned a value 0. You can't easily tell from the plot where the maximum is, but you can tell it occurs on the line sigma = np.max(data) - loc. You can find it using the method of Lagrange multipliers for constrained optimization. The condition is that the gradient of the objective is aligned with the constraint normal, and that's what the code above solves.  There were 1000 data points here, and for that the MLE of the location is 3.26 and the scale is 15.23. For 10000 data points, I get 3.46 and 15.04, so very close.
I haven't studied the shape < 1 case analytically, but it looks like for shape = 0.5, loc = 3.5, scale = 15, it is also well-defined, and the solution is at the intersection of those two lines - so even easier to compute.

This is true, but the argument that leads to it is only valid for the shape < 1 case.  But the closer we get to that line, the higher the LL function, so it seems that a supremum exists, and that's good enough for me. I think it's better to return a point that's very close to that intersection than to require that the user provide one of the parameters.
Please let's talk in real-time to discuss this further. (It's not efficient to type everything or to duplicate work, and we don't have time to spare!)

Thanks @mdhaber, I found I was mistaken about the power-law MLE for the case a > 1.  I was decoupling the behavior of the location and scale parameters too much, and reached an incorrect conclusion. The only conclusion that I should have drawn is the one that you stated earlier: if we know a > 1, then the location and scale must satisfy the constraint loc + scale - max(data) = 0.  That's the orange line in your plots.
The case 0 < a < 1 is still problematical; I edited my notes with a few comments about this.

Hi all,
As I started #13396, I discovered this issue and potential improvements for MLE. I'd be happy to help improve this part of scipy.stats.
Can I freely pick remaining distributions and submit analytical ML estimators implementations? Or do you have specific guidelines or tasks attributions? 🙂

Hi all,
As I started #13396, I discovered this issue and potential improvements for MLE. I'd be happy to help improve this part of scipy.stats.
Can I freely pick remaining distributions and submit analytical ML estimators implementations? Or do you have specific guidelines or tasks attributions? 🙂

The table  above lists the ones we had planned based on what we had found in this reference.
That said, as we went through we found that since SciPy's parameterization always has scale and loc, we frequently have to derive the conditions ourselves, and the reference just gave us hints at which distributions it's worth trying for. I wrote a little script that helps a bit with that, if you're interested.
@swallan had you started on von Mises? If not, that would be a good one for @jjerphan.
@lispsil said he was interested in Weibull. (gh-12787). exponweibull is another one I've seen an issue about (gh-11806).
Otherwise, any distribution is fair game., I think.

The table  above lists the ones we had planned based on what we had found in this reference.
That said, as we went through we found that since SciPy's parameterization always has scale and loc, we frequently have to derive the conditions ourselves, and the reference just gave us hints at which distributions it's worth trying for. I wrote a little script that helps a bit with that, if you're interested.
@swallan had you started on von Mises? If not, that would be a good one for @jjerphan.
@lispsil said he was interested in Weibull. (gh-12787). exponweibull is another one I've seen an issue about (gh-11806).
Otherwise, any distribution is fair game., I think.

@swallan had you started on a PR for von Mises? If not, that would be a good one for @jjerphan. There are some notes about it above.

@mdhaber I haven't formally started it yet / written anything so it's fair game

@swallan had you started on a PR for von Mises? If not, that would be a good one for @jjerphan. There are some notes about it above.

@mdhaber I haven't formally started it yet / written anything so it's fair game

Oh, @jjerphan I remembered that I wrote out notes for the Asymmetric Laplace distribution:
https://www.dropbox.com/s/4tyrqjr6708yhh1/MLE - Asymmetric Laplace.pdf?dl=0
I think that would be another good one as everything to be done is laid out explicitly there in the "Computation" section. It's possible there is an error in a formula there, but I imagine they will be identified and fixed once they are implemented.

Oh, @jjerphan I remembered that I wrote out notes for the Asymmetric Laplace distribution:
https://www.dropbox.com/s/4tyrqjr6708yhh1/MLE - Asymmetric Laplace.pdf?dl=0
I think that would be another good one as everything to be done is laid out explicitly there in the "Computation" section. It's possible there is an error in a formula there, but I imagine they will be identified and fixed once they are implemented.

I think that would be another good one as everything to be done is laid out explicitly there in the "Computation" section. It's possible there is an error in a formula there, but I imagine they will be identified and fixed once they are implemented.

I guess we can cook something similar to the references notes for Nakagami. Would this layout (i.e a subsection for the analytical derivation of the MLE in the distribution's page in the tutorials) be suitable to you?

I think that would be another good one as everything to be done is laid out explicitly there in the "Computation" section. It's possible there is an error in a formula there, but I imagine they will be identified and fixed once they are implemented.

I guess we can cook something similar to the references notes for Nakagami. Would this layout (i.e a subsection for the analytical derivation of the MLE in the distribution's page in the tutorials) be suitable to you?

@swallan had you started on a PR for von Mises? If not, that would be a good one for @jjerphan. There are some notes about it above.

@mdhaber I haven't formally started it yet / written anything so it's fair game

OK, thanks. I'll start with that.

@swallan had you started on a PR for von Mises? If not, that would be a good one for @jjerphan. There are some notes about it above.

@mdhaber I haven't formally started it yet / written anything so it's fair game

OK, thanks. I'll start with that.

If you are interested, I am looking for a second reviewer for #13396 (which relates to this issue). 🙂

Adding truncweibull_min as a candidate for possible fit override. See gh-9974.

Improved fit has been requested for genextreme, too (gh-10446). Since it generalizes gumbel, weibull, and invweibull, perhaps we can use the progress toward fitting those to fit genextreme.

Improved fit has been requested for truncnorm (gh-16382).

vonmises can be removed from the list since #17585. The fit is not analytic per se (requires root finding) but is very fast and stable.

After the last iterations vonmises should be good now. I will try to look into gamma/beta soon.


Prisma error message incorrect when function trigger fails to find column

Bug description
When a linked Postgres function trigger throws an error for a missing column -> record "new" has no field "missing_column" Prisma returns this as The column new does not exist in the current database. when a prisma.create or any ops linked to the trigger are attempted.
How to reproduce
Create a basic Postgres table and set up a trigger on insert. The trigger should try to reference a column that does not exist in the table.
BEGIN
    NEW.full_name = CONCAT(NEW.first_name, '  ', NEW.last_name);
    RETURN NEW;
END;

NEW.full_name does not exist
Now attempt to run a prisma.table_name.create({ data: {  first_name: "Test", last_name: "Name" } }); on this table, and Prisma returns The column "new" does not exist in the current database
More details here - https://github.com/prisma/prisma/discussions/13423
Expected behavior
Prisma should return either the same error as Postgres, or a message that better indicates the error originated from a function attempting to read a NEW record and not a new column.
Prisma information
generator client {
  provider      = "prisma-client-js"
  binaryTargets = ["native", "rhel-openssl-1.0.x"]
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

// any model

Environment & setup
OS: MacOS Big Sur 11.6.1
Database: Postgresql
Node.js version: v14.18.1
Using prisma inside a serverless express backend

Prisma Version
prisma                  : 3.12.0
@prisma/client          : 3.12.0
Current platform        : darwin
Query Engine (Node-API) : libquery-engine 22b822189f46ef0dc5c5b503368d1bee01213980 (at<EMAIL_ADDRESS>Migration Engine        : migration-engine-cli 22b822189f46ef0dc5c5b503368d1bee01213980 (at node_modules/@prisma/engines/migration-engine-darwin)
Introspection Engine    : introspection-core 22b822189f46ef0dc5c5b503368d1bee01213980 (at node_modules/@prisma/engines/introspection-engine-darwin)
Format Binary           : prisma-fmt 22b822189f46ef0dc5c5b503368d1bee01213980 (at node_modules/@prisma/engines/prisma-fmt-darwin)
Default Engines Hash    : 22b822189f46ef0dc5c5b503368d1bee01213980
Studio                  : 0.459.0

I'm running into this now... What do you need to confirm this bug? All you have to do is create an erroneous trigger in your test environment and this error will appear.


Support for ETW event processing?

I have a general question regarding this WPA SDK project that I can't find answered elsewhere.  Based on the samples so far, it seems the emphasis is on being able to ingest non-.etl files and transform them in a way such that they can be visualized via WPA Tables.
The primary use case I'm interested in though is this: we have an ETL trace consisting of "NT Kernel Logger" data (call stacks, dispatch, context switch, all that sort of stuff), plus our own internal events (which typically get combined into a single final trace file via xperf -stop "NT Kernel Logger" -stop "OurProduct Logger" -d combined.etl).
WPA is great for allowing us to visualize the native kernel logging events it knows about, but it can't show us much about our custom events (other than that really generic view that I've forgotten the name of).
Now, WPA is already processing all the ETW events in the combined.etl trace file, so I guess I was wondering if there was a way (or plans for such a way) to hook into that existing ETW/ETL pipeline and get callbacks for events (perhaps by specifying the GUIDs we're interested in up-front).  I can't see anything like that currently, but I wanted to double check.
In lieu of an integrated pipeline... what would be the current option?  Custom data source that ingests .etl files, and use a C# .etl processing library (like TraceProcessor or whatever it's called) to extract events and make them play nice with the WPA plugin environment?  (Would this prevent WPA from displaying all the existing graphs/tables for events it knows about in the combined.etl trace?)
Hopefully the question makes sense, let me know if you need any more information (or if there's a better forum to ask questions, as this isn't really an issue per-se).
Thanks in advance!

FYI, for MsQuic, we've worked with the WPA folks to put together a custom, open source plugin (code) for our custom ETW events (manifest). Now, I'm not sure if that's exactly what you want here or not. It sounds like there might already be a plugin for some of your events, and that might just need to be updated. I'll let the WPA folks comment to that.

Actually, here's a concrete example of exactly the type of xperf invocation I'm talking about, using an open source project of mine (PerfectHash) as a reference.  The PerfectHash project emits .man-based ETW events based on various keywords (https://github.com/tpn/perfecthash/blob/main/src/PerfectHash/PerfectHashEvents.man).
A typical trace session, as adminstrator:
xperf -start "NT Kernel Logger" -on LATENCY+DISPATCHER+CPU_CONFIG+CSWITCH+DISK_IO+DPC+HARD_FAULTS+INTERRUPT+LOADER+MEMINFO+MEMINFO_WS+PERF_COUNTER+POWER+PROC_THREAD+PROFILE+PRIORITY+WDF_DPC+WDF_INTERRUPT -stackWalk Profile+CSwitch+ReadyThread+ThreadCreate+ThreadDelete+ProcessCreate+DiskFlushInit+DiskReadInit+DiskWriteInit -f kernel.etl -BufferSize 4096 -MinBuffers 4096 -MaxBuffers 8192 -MaxFile 8192 -FileMode Circular

xperf -start "PerfectHash Logger" -on PerfectHash:0x3e

I run my PerfectHashCreate.exe program, then issue:
xperf -stop "NT Kernel Logger" -stop "PerfectHash Logger" -d combined.etl

I then want to take that combined.etl, which contains all the kernel event goodies, plus events specific to my PerfectHash library, and load it into a single WPA instance, leveraging this SDK to visualize my PerfectHash events as first-class WPA tables (so they play nicely with things like zooming, filtering, etc).
Based on my review of the SDK so far, it seems like I'd need to achieve this by writing a custom .etl processor -- if so, would that still allow the ability to depict the kernel events and my custom events in the same WPA session?

FYI, for MsQuic, we've worked with the WPA folks to put together a custom, open source plugin (code) for our custom ETW events (manifest). Now, I'm not sure if that's exactly what you want here or not. It sounds like there might already be a plugin for some of your events, and that might just need to be updated. I'll let the WPA folks comment to that.

Ah!  Very interesting!  So, it looks like:
a) Yes, you'd have to write your own .etl data processor for now (but that's pretty easy thanks to Microsoft.Diagnostic.Tracing).
b) Yes, you'd end up with a WPA session that displayed all the normal stuff like call stacks etc, in addition to your new tables/views provided by your plugin.

Yes, our plugin only adds to existing WPA functionality. It doesn't overwrite or remove anything.

Great, thanks!  This answers all my questions, so I'll close out the issue.

I successfully got my plugin to depict my .etl-backed tables side-by-side with all the normal xperf-provided tables, which is awesome!  However, it doesn't work via the command line approach, i.e. when I did the following:
wpa -addsearchdir c:\foo -i c:\foo\trace.etl

It comes back with the following error:
No -processor was specified, and multiple plugins can handle the file `c:\foo\trace.etl'. Please specify the `processor' option to disambiguate.

If I added my plugin's bin\Release directory as a junction in WPA\CustomDataSources, then launch WPA with no arguments, then do File->Open and select the .etl file... it successfully loads both xperf-derived tables and those from my plugin.  So, it'd be nice if there was a way to replicate this behavior on the command line.

Hi @tpn,
When opening a file in WPA through File->Open, WPA will pass the file to all data processors available to process it (unless overridden in the Window->Select Tables setting). When loading through the command line this does not happen, and the user must specify which processor (singular) to use.
A future WPA release will fix this issue, and allow opening your file through the command line.
In the future, if you have issues specific to WPA and not the SDK, please submit them through Feedback Hub.

That makes sense, thanks for the clarification; much appreciated!

Just to add some easy links here, if you want to process ETW/.etl files programmatically it's recommended to use DataLayer / TraceProcessing. Here are the links:
https://www.nuget.org/packages/Microsoft.Windows.EventTracing.Processing.All
https://aka.ms/TraceProcessing


Integrate REVA appprovider

oCIS WOPI server duplicates code which is already in REVA. This can be used with the OpenInApp request like done in https://github.com/owncloud/ocis-wopiserver/pull/16.
In order to use the OpeninApp request we need to integrate / start the AppProvider in oCIS like done in https://github.com/owncloud/ocis/pull/2204.
Also a rework of the AppProvider and the CS3 WOPI server are in progress:

https://github.com/cs3org/wopiserver/blob/master/CHANGELOG.md#tue-jun-29-2021---v700
https://github.com/cs3org/reva/pull/1785, https://github.com/cs3org/reva/issues/1779

further TODOs:

see requirements for integration into ownCloud Web
https://github.com/owncloud/ocis/issues/59

futher procedure:

merge https://github.com/cs3org/reva/pull/1968
update REVA in oCIS
merge https://github.com/owncloud/ocis/pull/2204

-> /app/list/ and /app/open/ endpoints are available in oCIS

wait for mimetype handler / extension bootstrap overhaul https://github.com/owncloud/web/issues/5135

-> write small app provider app for ownCloud Web, that uses /app/list/ endpoint for registering file handlers and does the file handling action (do /app/open request and then open in iframe)
cc @fschade

Integration work in oC Web is also in progress:

https://github.com/owncloud/web/pull/5752
https://github.com/owncloud/web/pull/5712

Current state:

Fix needed https://github.com/cs3org/reva/pull/2118
Add file actions in oC Web https://github.com/owncloud/web/pull/5805

Missing in general:

documentation
archive oCIS WOPI server
update deployment example https://github.com/owncloud/ocis/pull/2478

Missing in oC Web:

default action
new file dialog
Fix https://github.com/owncloud/web/issues/5845


[POC] Add third batch of charts

Description

Rename chart-gallery to visual-vocabulary (I'll properly rename the folder after review, so the actual changes are better visible)
Minor updates to layout to optimise spacing
Removed the column_factory and created a separate example for time/column to properly display a time dimension (Had to introduce a custom chart here to properly align the ticks with the columns 😬 https://github.com/mckinsey/vizro/pull/634/commits/9379fb6ac817be189bbf0b1ec95af4bf8fda7b2c. From a data viz perspective worth doing, not entirely sure whether we should add this complexity though..)
Added third batch of charts

[x]  Scatter matrix
[x]  Funnel
[x]  Column + Line
[x]  Bubble
[x] Stepped line
[x] Parallel coordinates
[x] Heatmap

Screenshot
Notice

[x] I acknowledge and agree that, by checking this box and clicking "Submit Pull Request":

I submit this contribution under the Apache 2.0 license and represent that I am entitled to do so on behalf of myself, my employer, or relevant third parties, as applicable.
I certify that (a) this contribution is my original creation and / or (b) to the extent it is not my original creation, I am authorized to submit this contribution on behalf of the original creator(s) or their licensees.
I certify that the use of this contribution as authorized by the Apache 2.0 license does not violate the intellectual property rights of anyone else.
I have not referenced individuals, products or companies in any commits, directly or indirectly.
I have not added data or restricted code in any commits, directly or indirectly.

The only thing I still need to look at are the column charts.

Thanks! I've decided to remove the column_factory there because the same column example didn't make sense on the time category 😬 So I've replaced it with an example where one of the axis actually displays time. However, re-using the time category example for all column examples also didn't make sense for me, because normally you would have categories on one of the axis. Curious what you think 🤔


Module 1:static is duplicated after entering delta value

What happened?
When changing the delta value of existing module "1:static", an additional module appears as an duplicate.
The value of the duplicate can not be changed.
Deleting the duplicated module is possible.
Adding of further modules is possible.
Deleting the module 1:static and 2:passthrough is not possible.

How to reproduce
Can be reproduced by changing value.
Relevant log output
No log output

Which version are you running?
Beta 14
Diagnostics file

[X] I have attached a diagnostics file

Additional information
config_entry-smart_irrigation-d309574885fe55cb4f63702993289a78-6.txt

this is caused by some or your modules still having string type IDs, not numeric. I switched to numeric IDs (int) for zones/modules/mappings. There are two ways of getting out of this state:

easiest: uninstall the component, reboot HA, install the component.
harder: go into hidden storage folder and edit storage file for smart irrigation. Make sure all IDs for zones/modules/mappings are ints, so something like: id: 0, and not id: "0",

Just upgrading to the next beta is not going to help you here.

The easy way helped me out. Now it works with beta 16 after new installation


--hot flag should not add HMR plugin when it's already there, causes wrong moduleId in hot.accept

I am struggling getting HMR to work.
I see in the generated code that the moduleId in the hot.accept call is generated wrong.
In my case it reads 1010 whereas it should be 10.
It looks like the moduleId is inserted twice?
I found an existing issue in which the same problem is described:
https://github.com/webpack/webpack/issues/406#issuecomment-62756894
However I don't know what the solution is. It's suggested that the issue is with the output.path but I have no idea what to do. My output path is set to an absolute directory and the docs suggest that this is correct?

Ok... So after re-reading the issue it suddenly dawned upon me that I had both --hot and the HotModuleReplacementPlugin in my config... @sokra explicitly mentions that as the cause but it never clicked when I read it the first time... When I removed the plugin it all started working!
However, I have to say I find this behavior very unexpected. I would expect command line arguments to override the config, not just add the plugin again... Especially when that ends up breaking everything.
If it's possible to detect that the plugin is already there then maybe that would be much better behavior for the --hot switch. To add the plugin, only when it's not already configured. I'm going to change the title of this issue to reflect that, and remove the question label. My question is answered, but I feel this is actually a bug.

Ah I can't remove labels. Ok.  :)

Consider closing the issue? This makes it a bit easier to navigate open issues (there are so many). Thanks.


OpenZiti Streaming Metrics

We're going to have to revisit our metrics infrastructure for both account limits (#96) and also to implement sparkline metrics for private sharing scenarios.
Investigate the streaming metrics available and see what might be possible (even if it requires a few tweaks).

https://docs.openziti.io/docs/reference/configuration/controller/#events
https://github.com/openziti/fabric/blob/main/events/file_logger.go


Create workflow for collecting and sending diagnostic information

Is your feature request related to a problem? Please describe.
I'm always frustrated when I don't have enough information to help resolve an issue.
Describe the solution you'd like
The ability to collect and send diagnostic information similar to https://docs.docker.com/docker-for-mac/troubleshoot/#diagnosing-from-the-terminal
Describe alternatives you've considered
Spending lots of time going back and forth to collect additional information.
Additional context
We should collect things like:

k8s status (esp failed items)
logs (k3s, containers, & system)
check-config output
network info (iptables, routes, netstat, etc)
system info (processes, memory, cpu, etc)
other...?

Other considerations:

encrypt by default?
cloud upload?

Per Craig had moved this over to test but QA wants to wait until collection PR is in. For now QA is going to start testing it out and playing with it for local collection.

Per Craig had moved this over to test but QA wants to wait until collection PR is in. For now QA is going to start testing it out and playing with it for local collection.


Fix shim socket permissions on Darwin

When creating a socket for a shim on Darwin OS, it fails with permission error:
create new shim socket: listen unix /var/run/containerd/s/61a930be46013380e0376220342308b188da82e446ed263125ab52ccf5f96ce6: bind: permission denied
On OS X socket path needs +x in order to be created without elevating privileges.
Signed-off-by: Maksym Pavlenko<EMAIL_ADDRESS>

/retest

/test all


Sampler rework (gradual refactor)

Maybe we close this issue in favour of #365?

[x] MH (by Babak already) #365
[x] ULA #369
[x] MALA #369
[x] PCN #365
[ ] Gibbs #367

[x] Test if Gibbs work with new sampler #398 #397
[ ] Base it also on the new sampler
[ ] Not create new instance of sampler at each Gibbs step

[ ] NUTS #366
[x] LinearRTO #368
[ ] RegLinearRTO #368

Some notes from the meeting on sampler rework
# user interface for doing sampling
# 1) current way
sam = sampler.sample(1000, warm_up=1000)
# potentialy reset sampler after sampling call (don't have to)
sam2 = sampler.sample(1000) # sample with the same state as the previous call
# 2) object-oriented way
sampler.warmup(100)
sampler.sample(1000)
samples = sampler.get_samples()
# 3)
sam = sample("MH", x, x0=x0, scale=0.1, n_samples=1000, warm_up=1000)
# sampler is created and deleted dynamically
# 4)
# use option 2 for sampling under BayesianProblem's interface

It's agreed to use the 2nd option

as from a user's perspective, it's just natural to do some warmup/tuning, have a check at the chain, and then decide to have further tuning or go to full sampling.
The 3rd method doesn't seem to fit with other programming conventions/styles of CUQIpy.

# user interface for BayesianProblem
# 1)
BP.MAP()
# 2)
BP.MAP
# what happens if we call it multiple times?
# 3)
BP.compute_MAP()
BP.get_MAP()

The apis for BayesianProblem, like map and uq, will be decided later after further meetings.

@chaozg @nabriis @jakobsj
fyi, I am closing this issue as we discussed. Github issue numbers for specific tasks were added


Add CDI support to peer pods

How can we enable Dynamic Resource Allocation (DRA) based on Container Device Interface (CDI) for peer pods?
K8s v1.26 introduced DRA and Kata agent is recently enabling CDI. In my understanding, when we want to use GPUs in a peer pod, we need to manually specify an instance profile with GPUs. The webhook simply removes nvidia.com/gpu device requests to an annotation kata.peerpods.io.gpus, but it seems to be not used to select an instance profile.
Any suggestions?

@bpradipt Thank you for your responses. Can we extend the current webhook-based approach to support DRA and generate a CDI spec in a peer pod VM?

@zvonkok - you might also be interested and extremely helpful here?

According to this comment, it looks like CDI needs to be enabled in both runtime and kata-agent.
https://github.com/kata-containers/kata-containers/issues/9543#issuecomment-2366800482
The CDI support for runtime has been only enabled in runtime-rs, but not in Go version of kata-shim runtime.
https://github.com/kata-containers/kata-containers/issues/10145
I don't think runtime-rs supports the remote hypervisor for peer pods. Do we need to enable CDI in the Go version of kata-shim runtime?

In the Go version of kata-shim runtime, the remote hypervisor just ignore devices for now. I think we also need to fix this when CDI support is enabled in the kata-shim runtime.
https://github.com/kata-containers/kata-containers/blob/ca416d883729c7888287a89de836d67bc0975528/src/runtime/virtcontainers/remote.go#L203-L215
func (rh *remoteHypervisor) AddDevice(ctx context.Context, devInfo interface{}, devType DeviceType) error {
	// TODO should we return notImplemented("AddDevice"), rather than nil and ignoring it?
	logrus.Printf("addDevice: deviceType=%v devInfo=%#v", devType, devInfo)
	return nil
}

func (rh *remoteHypervisor) HotplugAddDevice(ctx context.Context, devInfo interface{}, devType DeviceType) (interface{}, error) {
	return nil, notImplemented("HotplugAddDevice")
}

func (rh *remoteHypervisor) HotplugRemoveDevice(ctx context.Context, devInfo interface{}, devType DeviceType) (interface{}, error) {
	return nil, notImplemented("HotplugRemoveDevice")
}

I think another possible workaround to support CDI in peer pods is to manipulate Devices in CreateContainerRequest by cloud-api-adaptor.
https://github.com/confidential-containers/cloud-api-adaptor/blob/aab207c82de836587bfa62e192bfd18a4af6d19a/src/cloud-api-adaptor/pkg/adaptor/proxy/service.go#L77-L82

@bpradipt Thank you for your responses. Can we extend the current webhook-based approach to support DRA and generate a CDI spec in a peer pod VM?

I think @yoheiueda proposal to do it in the CreateContainerRequest may be easier.  We can just keep the webhook to handle resource removals from the spec which doesn't apply to peer-pods.
Also I'm unclear how DRA will impact the peer-pods resource management? Can we do away with the webhook completely and rely on DRA for peer-pods resource management ?

There are several parts to the story. I am ramping up on peer-pods so excuse my ignorance on some parts. There are several aspects here.
Enable CDI in the kata-agent which is completely independent if peer-pods, or local VMM. This is enabled here: https://github.com/kata-containers/kata-containers/pull/9584.  @bpradipt This will eliminate the prestart-hook.

We've build DRA to request special features of a GPU, like give me a GPU with 40G, MIG slice, vGPU or a specific architecture. I am still unsure how we're going to map this exactly with peer-pods since we do not know what the CSP pool is capable of.
We need some advertisement system (NFD) for CSP like infrastructure?
The peer pods add a new layer of complexity. I need to think of how to enable DRA and CDI.

@bpradipt We need to think how to enable DRA properly. The logic you have is a good start but ignores MIG, or vGPU.

According to this comment, it looks like CDI needs to be enabled in both runtime and kata-agent.
kata-containers/kata-containers#9543 (comment)
The CDI support for runtime has been only enabled in runtime-rs, but not in Go version of kata-shim runtime.
kata-containers/kata-containers#10145
I don't think runtime-rs supports the remote hypervisor for peer pods. Do we need to enable CDI in the Go version of kata-shim runtime?

Yes, both runtime and kata-agent need integrate with CDI. Currently AFAIK, kata runtime and runtime-rs have both support CDI for GPU scenarios.
And another thing remote hypervisor in runtime-rs, is also under reviewing, which is a Project of Summer of Code

Hmm, since the mapping is Pod per CSP VM we need to make sure that DRA in the case of peer-pods only allows creation of GPUs that map to CSP instance types. Interesting ....

All the managment and configuration of devices is now pushed into DRA, whereas with device-plugins you consume what the infrastructure offers. We have conflict here with peer-pods.
In the bare-metal use-case we can request a full-passthrough GPU (vGPU) where DRA would bind a proper GPU to VFIO or MDEV and create the CDI spec with the vfio device and send this to Kata which then passes-through the GPU and in the VM we use CDI to create the proper device nodes in the OCI spec to be mounted into the container.
In the case of peer-pods DRA would just act as a proxy to pass-through the wanted typed to peer-pods which then in the end would choose the proper instance-type and to the CSP magic.

@zvonkok Thank you very much for the explanation of how CDI works with DRA.

And another thing https://github.com/kata-containers/kata-containers/pull/10225, is also under reviewing, which is a Project of Summer of Code

@Apokleos That sound great! I have a basic question regarding runtime-rs. At some point in the future, will the Go version of kata-shim runtime  be deprecated and replaced with runtime-rs?

@zvonkok Thank you very much for the explanation of how CDI works with DRA.

And another thing kata-containers/kata-containers#10225, is also under reviewing, which is a Project of Summer of Code

@Apokleos That sound great! I have a basic question regarding runtime-rs. At some point in the future, will the Go version of kata-shim runtime be deprecated and replaced with runtime-rs?

Hah, Yeah, good point. I think I should invite AC members @stevenhorsman @fupanli @zvonkok .etc. to help answer this question.

@Apokleos That sound great! I have a basic question regarding runtime-rs. At some point in the future, will the Go version of kata-shim runtime be deprecated and replaced with runtime-rs?

The short answer here is yes. The more nuanced version is yes, but we are not sure on the timeframe. The current plan is for Kata Containers 4.0 to ship with runtime-rs as the default shim, but the go runtime won't be removed here, however it might have security fixes only, or best-effort feature support with all new features targeted primarily at the rust runtime first.
4.0 is planned for so time in 2025, but there is still quite a bit of work required to close the gap as listed in https://github.com/kata-containers/kata-containers/issues/8702 including the remote hypervisor support that @Apokleos mentioned.

@bpradipt @stevenhorsman @yoheiueda @zvonkok @Apokleos Thank you very much for your helpful comments. Let me summarize the discussions and suggestions (and my understanding😃). Feel free to correct or add anything:

A user can run a Pod which refers a ResourceClaim as a peer pod. The user can pass a structured parameter to define the allocated resource.
The requested resource will be actually allocated when a peer pod VM is created.
The resource request is reflected to a pod VM instance profile and a CDI spec used by the kata agent inside of the pod VM.
The worker node must advertise the available resources in advance (similarly to the existing kata.peerpods.io/vm extended resources).
The container runtime in the worker node also must enable CDI.
We need some custom component (Kubelet plugin?) to pass through the resource allocation request to the cloud provider and pod VM.

@inatatsu thanks for summarising it.
Few inline questions for my understanding:

@bpradipt @stevenhorsman @yoheiueda @zvonkok @Apokleos Thank you very much for your helpful comments. Let me summarize the discussions and suggestions (and my understanding😃). Feel free to correct or add anything:

A user can run a Pod which refers a ResourceClaim as a peer pod. The user can pass a structured parameter to define the allocated resource.
The requested resource will be actually allocated when a peer pod VM is created.
The resource request is reflected to a pod VM instance profile and a CDI spec used by the kata agent inside of the pod VM.
The worker node must advertise the available ResourceSlice in advance (This is somewhat similar to what is currently done by the peerpod webhook using the kata.peerpods.io/vm extended resources).

Is this about advertising external VMs as resources instead of the current per node extended resources?

The container runtime in the worker node also must enable CDI.
We need some custom component (Kubelet plugin?) to pass through the resource allocation request to the cloud provider and pod VM while creating a dummy CDI spec for the container runtime on the worker node.

How CDI is useful for peer-pods case? The availability of the GPU resource is taken care by the cloud infra provider and all GPUs available in the VM gets allocated to the pod as there is 1-1 mapping between VM and pod.

Is this about advertising external VMs as resources instead of the current per node extended resources?

While I did not imagine such a use case😅, it is interesting and may simplify the VM management. FYI, ResourceSlice is also per node.

How CDI is useful for peer-pods case? The availability of the GPU resource is taken care by the cloud infra provider and all GPUs available in the VM gets allocated to the pod as there is 1-1 mapping between VM and pod.

In my understanding, CDI allows flexible device mapping and is runtime-agnostic. But as you point out, peer pods primarily rely on selecting an appropriate instance profile (or flavor) to allocate resources, and CDI just provides a mapping between the resources and containers.

@bpradipt Thank you for your questions.

Is this about advertising external VMs as resources instead of the current per node extended resources?

While I did not imagine such a use case😅, it is interesting and may simplify the VM management. ResourceSlice can be also per node.

How CDI is useful for peer-pods case? The availability of the GPU resource is taken care by the cloud infra provider and all GPUs available in the VM gets allocated to the pod as there is 1-1 mapping between VM and pod.

In my understanding, CDI allows flexible device mapping and is runtime-agnostic. But as you point out, peer pods primarily rely on selecting an appropriate instance profile (or flavor) to allocate resources, and CDI just provides a mapping between the resources and containers.

So, CDI will be helpful on the kata-agent side to assign the GPU (or other devices) to the container and additionally using the same building blocks (CDI). Is my understanding correct?

So, CDI will be helpful on the kata-agent side to assign the GPU (or other devices) to the container and additionally using the same building blocks (CDI). Is my understanding correct?

Yes. That's my current understanding.

* The container runtime in the worker node also must enable CDI.

The go runtime merged PRs to enable CDI:

https://github.com/kata-containers/kata-containers/pull/7325
https://github.com/kata-containers/kata-containers/pull/8861
https://github.com/kata-containers/kata-containers/pull/10276

@zvonkok Does this mean the go runtime (except for the remote hypervisor) already supports CDI?


deps: update gax.version to v1.60.1

com.google.api:gax-grpc 1.60.0 -> 1.60.1 patch
com.google.api:gax-bom 1.60.0 -> 1.60.1 patch
See gax release PR here: https://github.com/googleapis/gax-java/pull/1239

Close as an auto PR has been created.


Reimplement 'Reboot VM' with Stop/Start combined action

The current implementation of 'rebootVirtualMachine' is broken in CloudStack when SGs are in use. Can we implement 'reboot' action with 'Stop & Start' combined together?

The problem is fixed in 4.11.2, discard the task.


$breadcrumb-font-family is not !default

This is especially annoying when customizing the icon font, even while following the orange icons' project's README (which tells to use project's name as icon-font name).
Although I perfectly understand the need to lock many other variables, this specific one shouldn't be locked.

Hello,
This is fixed in commit 07a0d157ec25d
Everything now is !default, yay !


[5.3] Single point for making Eloquent model instance using IoC Container

Resolving Eloquent objects out of the container doesn't make any sense. They should not have any dependencies.


Multi-select retains first item when clear_button plugin is enabled

When you've enabled the clear_button plugin and have a multi-select dropdown that allows you to "activate" the items (i.e. not using the no_active_items plugin), pressing the keyboard shortcut for selecting everything (Ctrl+A, Cmd+A, etc) followed by backspace results in the first item in the list being retained. The list appears empty visually but the clear button is still visible while it should only appear when there are selected items, plus the first item is not selectable anymore (due to using the default setting for hideSelected).
The problem does not occur if you actually use the clear button, or if you remove all the items one by one (by simply holding backspace for example).
This seems to be due these pieces of code (added some comments for clarification):
removeItem(item = null, silent) {
    const self = this;
    item = self.getItem(item);
    if (!item) return;
    var i, idx;
    const value = item.dataset.value;

    // Returns an index that is 1 too high, so the last item in the list is
    // not removed until the next call, which was actually intended for
    // removing the first to last item
    i = nodeIndex(item);

    // All items are visually removed...
    item.remove();
    if( item.classList.contains('active') ){
        idx = self.activeItems.indexOf(item);

        // ... but this never removes the first item from the internal list
        // due to everything being already visually cleared before it gets
        // to the appropriate index, so the `getItem()` call at the top will
        // cause it to return early
        self.activeItems.splice(idx, 1);
        removeClasses(item,'active');
    }
    // ....
}

const nodeIndex = (el, amongst) => {
  if (!el) return -1;

  // `amongst` is not set by `removeItem()` so it will simply use the
  // `nodeName`, which is 'DIV' (which the clear button also is)
  amongst = amongst || el.nodeName;
  var i = 0;
  while (el = el.previousElementSibling) {
    if (el.matches(amongst)) {
      i++;
    }
  }
  return i;
};

The clear button simply calls a clear() function instead of removing a range of items one by one, like how it works when you "activate" all the items and press backspace. Removing items one by one targets every item specifically so it's no problem either.
Since you're already in the context of removeItem(), I don't think there will be any side effects by simply having nodeIndex() target the .item class.

Nothing stale about it.


Credential Definition and Trusted Issuers restrictions are disabled in presentation request

When creating a presentation request under the Verification tab the Credential Definition and Trusted Issuers form fields are disabled.
When looking at the code they are disabled with a hardcoded boolean. Removing this for credential definition works fine and allowed me to set restrictions to the presentation request. Is there a specific reason these are disabled?
https://github.com/hyperledger/aries-toolbox/blob/848890ef085705eaf7e1ae80774a02f6eee98681/src/renderer/components/Verifications/Verification.vue#L85-L97

About a year ago when these additions were made, we didn't have a clean way of selecting one of potentially many credentials that could fulfill the request. At the time we decided to disable the UI and punt on coming up with a good credential selection interface and associated messages.
So there's the historical background. I think we have the same general problem today but I've been away from this code for awhile and would have to review how ACA-Py is handling this scenario now.

Any update on this?
When using the Lissi wallet, it will prompt you for a self-asserted value (input field). This means you can't use any credential to do the proving. In my testing it works if you have a single credential, which is enough for most demo's.

I haven't had the opportunity to look at this issue in more detail recently. I'm happy to accept a PR re-enabling these features if it appears to be working as expected.


verifyChecksum semantics

With recent commits to the PSA spec, I think this issue has been resolved with the following answers: There is no separate verifyChecksum control block in PSA.   All (or at least most) checking of header checksums is expected to be performed in the parsers, setting some kind of result for use in the following Ingress or Egress control block, such as the parser error.  The Ingress and Egress control blocks are expected to check for parser errors, if desired, and then also perform the desired actions on packets that experienced errors during parsing.
This issue can be closed then, I think?

@cc10512 I think this issue can be closed now, with recent commits to the draft PSA spec?

Agreed.


Wrong Chrome extensions (again)

Let me say first that I'm really sorry.
azuretls-client:
"extensions": [
  {
    "name": "TLS_GREASE (0xaaaa)"
  },
  {
    "name": "ec_point_formats (11)",
    "elliptic_curves_point_formats": [
      "0x00"
    ]
  },
  {
    "name": "server_name (0)",
    "server_name": "tls.peet.ws"
  },
  {
    "name": "signed_certificate_timestamp (18)"
  },
  {
    "name": "application_layer_protocol_negotiation (16)",
    "protocols": [
      "h2",
      "http/1.1"
    ]
  },
  {
    "name": "psk_key_exchange_modes (45)",
    "PSK_Key_Exchange_Mode": "PSK with (EC)DHE key establishment (psk_dhe_ke) (1)"
  },
  {
    "name": "extensionRenegotiationInfo (boringssl) (65281)",
    "data": "00"
  },
  {
    "name": "application_settings (17513)",
    "protocols": [
      "h2"
    ]
  },
  {
    "name": "supported_groups (10)",
    "supported_groups": [
      "TLS_GREASE (0xbaba)",
      "X25519 (29)",
      "P-256 (23)",
      "P-384 (24)"
    ]
  },
  {
    "name": "extended_master_secret (23)",
    "master_secret_data": "",
    "extended_master_secret_data": ""
  },
  {
    "name": "key_share (51)",
    "shared_keys": [
      {
        "TLS_GREASE (0xbaba)": "00"
      },
      {
        "X25519 (29)": "sensitive"
      }
    ]
  },
  {
    "name": "compress_certificate (27)",
    "algorithms": [
      "brotli (2)"
    ]
  },
  {
    "name": "signature_algorithms (13)",
    "signature_algorithms": [
      "ecdsa_secp256r1_sha256",
      "rsa_pss_rsae_sha256",
      "rsa_pkcs1_sha256",
      "ecdsa_secp384r1_sha384",
      "rsa_pss_rsae_sha384",
      "rsa_pkcs1_sha384",
      "rsa_pss_rsae_sha512",
      "rsa_pkcs1_sha512"
    ]
  },
  {
    "name": "session_ticket (35)",
    "data": ""
  },
  {
    "name": "extensionEncryptedClientHello (boringssl) (65037)",
    "data": "sensitive"
  },
  {
    "name": "supported_versions (43)",
    "versions": [
      "TLS 1.3",
      "TLS 1.2"
    ]
  },
  {
    "name": "status_request (5)",
    "status_request": {
      "certificate_status_type": "OSCP (1)",
      "responder_id_list_length": 0,
      "request_extensions_length": 0
    }
  }
]

real chrome:
"extensions": [
  {
    "name": "TLS_GREASE (0x0a0a)"
  },
  {
    "name": "ec_point_formats (11)",
    "elliptic_curves_point_formats": [
      "0x00"
    ]
  },
  {
    "name": "server_name (0)",
    "server_name": "tls.peet.ws"
  },
  {
    "name": "signed_certificate_timestamp (18)"
  },
  {
    "name": "application_layer_protocol_negotiation (16)",
    "protocols": [
      "h2",
      "http/1.1"
    ]
  },
  {
    "name": "psk_key_exchange_modes (45)",
    "PSK_Key_Exchange_Mode": "PSK with (EC)DHE key establishment (psk_dhe_ke) (1)"
  },
  {
    "name": "extensionRenegotiationInfo (boringssl) (65281)",
    "data": "00"
  },
  {
    "name": "application_settings (17513)",
    "protocols": [
      "h2"
    ]
  },
  {
    "name": "supported_groups (10)",
    "supported_groups": [
      "TLS_GREASE (0x2a2a)",
      "X25519 (29)",
      "P-256 (23)",
      "P-384 (24)"
    ]
  },
  {
    "name": "extended_master_secret (23)",
    "master_secret_data": "",
    "extended_master_secret_data": ""
  },
  {
    "name": "key_share (51)",
    "shared_keys": [
      {
        "TLS_GREASE (0x2a2a)": "00"
      },
      {
        "X25519 (29)": "sensitive"
      }
    ]
  },
  {
    "name": "compress_certificate (27)",
    "algorithms": [
      "brotli (2)"
    ]
  },
  {
    "name": "signature_algorithms (13)",
    "signature_algorithms": [
      "ecdsa_secp256r1_sha256",
      "rsa_pss_rsae_sha256",
      "rsa_pkcs1_sha256",
      "ecdsa_secp384r1_sha384",
      "rsa_pss_rsae_sha384",
      "rsa_pkcs1_sha384",
      "rsa_pss_rsae_sha512",
      "rsa_pkcs1_sha512"
    ]
  },
  {
    "name": "session_ticket (35)",
    "data": ""
  },
  {
    "name": "extensionEncryptedClientHello (boringssl) (65037)",
    "data": "sensitive"
  },
  {
    "name": "supported_versions (43)",
    "versions": [
      "TLS_GREASE (0xbaba)",
      "TLS 1.3",
      "TLS 1.2"
    ]
  },
  {
    "name": "status_request (5)",
    "status_request": {
      "certificate_status_type": "OSCP (1)",
      "responder_id_list_length": 0,
      "request_extensions_length": 0
    }
  },
  {
    "name": "TLS_GREASE (0xaaaa)"
  }
]

for the tests:
{
  "ja3": "771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,11-0-18-16-45-65281-17513-10-23-51-27-13-35-65037-43-5,29-23-24,0",
  "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"
}

So "supported_versions" lost its grease since last pr.
Also real chrome adds grease at the end and there is no padding. Last time you said grease added before padding because of the hello client payload length. But when there is no padding azuretls-client doesn't add grease alltogether. So if I'm just don't understand the subject then nevermind.
P.S. Without using ApplyJa3 the last grease and padding are placed correctly.

It should be like this (code is not tested):
if navigator == Chrome {
    lastIndex := len(extensions) - 1
    last := extensions[lastIndex]

    switch last.(type) {
        case tls.UtlsPaddingExtension: // also "pre_shared_key (41)" but there is none in the library as I see
            extensions[lastIndex] = &tls.UtlsGREASEExtension{}
            extensions = append(extensions, last)
        default:
            extensions = append(extensions, &tls.UtlsGREASEExtension{})
    }
}

Chrome adds TLS_GREASE at the end except when there is "pre_shared_key (41)" or "padding (21)".

You should not include pre_shared_key (41) in the ja3 fingerprint since it's a new TLS connection

Yep I don't use it. I just wrote about it for the info.

I'm really sorry again. But I don't know if you understood correctly. It is really should be like in the code I'm mentioned.
Chrome always adds TLS_GREASE at the end except when there is UtlsPaddingExtension then TLS_GREASE is second to last.


Support custom jobs

This PR adds custom job specifications to the cluster API using the Cap'n'p AnyPointer. A user can then convert their payload to a pointer and have their worker handle their custom cases. The kind field was meant to allow distinguishing between multiple custom job specifications (I don't know if there's a better way to do that).
The simple_custom test is meant to pretend the OBuilder job specification doesn't exist and creates a custom job where the payload is an OBuilder struct with a spec field.
The PR also exposes the default_build in the cluster-worker library. The idea there is that if #151 is merged and released then people can extend the worker to custom jobs whilst still supporting docker and obuilder ones? It might be nice to then expose some of the logic currently in bin/worker too. See this branch of solver-service for an example of using the cluster_worker libraries.

Thanks for the review. I also hit that snag and opted for the more wasteful approach of copying (mainly because I hadn't even considered open types so my use of conversion functions didn't work!).

This PR now also adds support for submitting the custom jobs using the ocurrent ocluster plugin. In order to build a "solver worker" it would be great to make the cluster worker library publicly available. Here I've just added it to the ocluster package with ocluster.worker but that's probably not desirable (it was just to unblock solver-worker) in which case it will need https://github.com/ocurrent/ocluster/pull/151 probably.

Thanks for the review @talex5 (and sorry for the delay)!
The PR is rebased, the worker is no longer public, custom jobs require a cache_hint and the Custom API for Current_ocluster now also has a test.

Thanks @talex5 for the pointers, I didn't know about the Connection API! I think all of the custom parts have been removed from Current_ocluster and the tests now show how to use the underlying Connection.t to submit custom jobs (I think...)


Anurag cycle sort  worksheet

Adding cycle sort in new branch

The provided solution still doesn't solve the problem.
Please go through the problem statement again and try replicating the sample input and output with your code.
If you have any queries, feel free to comment on the problem's respective issue for clarification.
This PR also contains changes to another file, the solution to which has already been merged.
Make sure to verify your commits before creating a PR.
Happy coding!


Add IsInherited property to RBAC entries

This will support

visually differentiating role assignments that are inherited by the container
support modifying only non-inherited RBAC assignments

Converting PR back to draft; another new property is required


Generate static JSON of example values

We ultimately want to have a public database of tracking requests and dynamically list examples of observed values for each data path.
Until we get there, I want to generate a static JSON of such examples values based on the data from my master's thesis.

Updated code that takes this into account:
import { db, pg } from './db';
import type { AnnotatedResult, Request } from 'trackhar';
import { processRequest } from 'trackhar';
import { writeFile } from 'fs/promises';

type DbRequest = {
    appId: string;
    platform: 'android' | 'ios';
    appVersion: string;

    content?: Buffer;
} & Omit<Request, 'content'>;

(async () => {
    const requests = await db
        .manyOrNone<DbRequest>(
            `select requests.id as "requestId", apps.name as "appId", apps.platform as platform, apps.version as "appVersion", requests.start_time as "startTime", requests.method as method, requests.host as host, requests.path as path, content_raw as content, requests.port as port, requests.scheme as scheme, requests.http_version as "httpVersion", regexp_replace(concat(requests.scheme, '://', requests.host, requests.path), '\\?.+$', '') as "endpointUrl"
        from apps
          join runs r on apps.id = r.app
          join requests on r.id = requests.run;`
        )
        .then((reqs) => reqs.map((r) => ({ ...r, content: r.content?.toString('binary') })));

    const processedRequests = requests
        .map((r) => {
            try {
                return processRequest(r);
            } catch (err) {
                console.error(err, r);
                throw err;
            }
        })
        .filter((r): r is AnnotatedResult => !!r);

    const values = processedRequests.reduce<Record<string, Set<any>>>((acc, cur) => {
        for (const r of cur) {
            // https://github.com/tweaselORG/tracker-wiki/issues/3#issuecomment-1505548347
            const key = `${r.adapter}/${r.property.toLowerCase()}/${r.context}/${r.path}`;
            if (!acc[key]) acc[key] = new Set();

            acc[key].add(['string', 'number'].includes(typeof r.value) ? r.value : JSON.stringify(r.value));
        }

        return acc;
    }, {});

    const json = JSON.stringify(values, (k, v) => (v instanceof Set ? [...v] : v), 4);
    await writeFile('adapterExamples.json', json);

    pg.end();
})();

process.on('unhandledRejection', (err) => {
    console.error('An unhandled promise rejection occurred:', err);
    pg.end();
    process.exit(1);
});

Fixed with #1.