Update fromMessage API in data-prepper-common Sources

Description

Changes access modifier of fromMessage API to public to leverage in Sources
Updates usage in RandomStringSource and StdInSource

Issues Resolved
Resolves #646
Check List

[ ] New functionality includes testing.
[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #1074 (405881d) into main (cc0a6fc) will decrease coverage by 0.03%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #1074      +/-   ##
============================================
- Coverage     91.32%   91.28%   -0.04%
  Complexity      683      683
============================================
  Files            84       84
  Lines          1994     1986       -8
  Branches        168      168
============================================
- Hits           1821     1813       -8
  Misses          132      132
  Partials         41       41

Impacted Files
Coverage Δ

...m/amazon/dataprepper/model/event/JacksonEvent.java
98.41% <ø> (ø)

...dataprepper/plugins/source/RandomStringSource.java
73.33% <100.00%> (-3.14%)
:arrow_down:

...amazon/dataprepper/plugins/source/StdInSource.java
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cc0a6fc...405881d. Read the comment docs.


Visualization wtih Grad-CAM

Hi! Thank you for your wonderful project! I've tried to visualization with gradcam, and the results are as follows.

There seems to be something wrong with my results.  I was going to input one frame [3,224,224] to the method https://github.com/jacobgil/pytorch-grad-cam. But it failed to the problem "shape [-1,15,224,224] is invalid for input of size 150528", because the input tensor should be [-1, 3*5, 224, 224] in your model.
So I sampled the frames according to the strategy in your code for visualization in each segment. 5 consecutive frames [X1,X2,X3,X4,X5] was concated in each segment , and got the tensor [3x5,224,224]. Then, I repeated it 8 times to obtain the input tensor [3x5x8, 224, 224]. Finally, I got the CAM and multipled it onto X3.
Why is there an obvious CAM in each result frame? It seems to be different from the picture in your paper. Could you tell me what's wrong with my visualization?

You can try this repo for visualization.


Full parameter options and licensing

Summary:

Includes the full options to specify frequency and time arrays as used in pyuvsim.
Adds BSD 3-clause license and a changelog in preparation for first release.
Style change --- Ordering import statements
Some changes that enable simulations to run with the minimum required parameters.
New scripts

Frequency/time param explanation.
How best to define the time and frequency arrays is a matter of some debate among the datasim team. For instance -- if each frequency channel has a center frequency and a channel width, then the full bandwidth (that is, the amount of frequency space covered by the channels) should be (Nfreqs +1) * channel_width, where the +1 accounts for the extra half channels above the last and below the first frequency centers.
The debate is whether this is sensible in a simulation context. Certainly if we want to compare with data, the frequency channel centers should match the channel centers of a data file. But how do we define the bandwidth in either case? Is bandwidth even meaningful for a simulator?
Overall -- This PR sets the following convention, using the parse_..._params() functions borrowed from pyuvsim:
Bandwidth = channel_width  * (Nfreqs + 1)
freq_array = (f_0, f_1, ..., f_Nfreqs) where f_i = f_0 + i * channel_width

and similarly for the time array.
I've included functions in utils that will tell you what the parameters are for a given frequency or time array, and a unit test that checks that those parameters return the same array when run through the parser.
Skymodel frequencies
I also made the change that the simulator will error if a loaded sky model doesn't have frequencies that match the obsparam frequencies. I think this is the preferable behavior, since the user may not notice that the simulation is going forward with settings different than what was specified. This can probably be downgraded to a warning instead, but it shouldn't happen silently.
New scripts
Still a little undercooked... these will be better made in a future PR. I've added a script that generates a random antenna layout of a given size and maximum baseline, and another that makes a HEALPix map of evenly spaced points whose brightnesses correspond with their declinations. I want to make an image of a recognizable pattern on the sky and confirm that sky rotation and positioning look reasonable.
Sorry for the long explanation, and I hope this doesn't disrupt anything you're working on!

@aelanman I disagree that bandwidth = chan_width * (Nfreqs + 1). Furthermore, the frequencies that come from the correlator and that of a UVData object are, to my knowledge, the channel start. Therefore, bandwidth = Nfreqs * channel_width and freq_array should be defined as
np.linspace(start_freq, start_freq + bandwidth, Nfreqs, endpoint=False)
It seems like we are perhaps thinking too hard about this? It seems like the question is that a frequency is defined by its bin center or bin start, in both cases freq_array and bandwidth are defined as I have above, its just that in the latter case we would give start_time += chan_width / 2 relative to the former.
Long story short, I don't think we should change our definition of bandwidth or Nfreqs in order to conform to a bin start or bin center convention, which should only change the values of freq_array, not their total extent or number of elements. Let me know what your thoughts are on this.

@aelanman I've got some suggestions for the parse_ functions in simulator.py. In general, I don't like the inplace editing of the dictionaries which will cause trouble when passing dictionaries whose values will be overwritten silently. I also think its easier to enforce a policy of parsing order on the multiple keys that can be specified, rather than enforce that all keys are compliant with each other.
Also added extensive testing for the parsing functions.
If you disagree with this change we can revert the commit: Let me know what your thoughts are on this in the meantime.

@nkern I think the change makes sense and makes it a little clearer what's happening. I've put back in the assertion tests on the final freq/time arrays, because I don't think it hurts to be extra sure that the array and parameters are consistent.
I've also added a test on the redundancy selection. The actual redundancy selection methods used by pyuvdata are well-tested there, but this is to make sure the selection is understood properly.

@aelanman looks like an error in your redundancy test. once that is fixed should be good to go. Also, do you want to make a commit that bumps the VERSION file to 1.0.0, that way once we merge this in we can tag a a v1.0.0 release?


upgrade karmada

Signed-off-by: hejianpeng<EMAIL_ADDRESS>What type of PR is this?

/kind bug
What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
upgrade karmada to avoid https://github.com/karmada-io/karmada/issues/1836
Does this PR introduce a user-facing change?:

NONE

/lgtm
/approve


Maccy starts when double clicking or copying

Before Submitting Your Bug Report

[X] I have verified that there isn't already an issue reporting the same bug to prevent duplication.
[X] I have seen the FAQ.

Maccy Version (see 'About' window)
0.31.0
macOS Version
14.5 (23F79)
Maccy Settings
{
    "KeyboardShortcuts_delete" = "{\\"carbonModifiers\\":2048,\\"carbonKeyCode\\":51}";
    "KeyboardShortcuts_pin" = "{\\"carbonKeyCode\\":35,\\"carbonModifiers\\":2048}";
    "KeyboardShortcuts_popup" = "{\\"carbonKeyCode\\":8,\\"carbonModifiers\\":256}";
    "LaunchAtLogin__hasMigrated" = 1;
    "NSStatusItem Visible Item-0" = 0;
    "NSWindow Frame com.sindresorhus.Settings.FrameAutosaveName" = "295 1761 442 321 -698 900 2560 1415 ";
    SUAutomaticallyUpdate = 0;
    SUEnableAutomaticChecks = 0;
    SUHasLaunchedBefore = 1;
    SULastCheckTime = "2024-07-26 03:02:29 +0000";
    SUSendProfileInfo = 0;
    enabledPasteboardTypes =     (
        "public.rtf",
        "public.html",
        "public.utf8-plain-text",
        "public.tiff",
        "public.file-url",
        "public.png"
    );
    ignoredPasteboardTypes =     (
        "Pasteboard generator type",
        "net.antelle.keeweb",
        "com.agilebits.onepassword",
        "com.typeit4me.clipping",
        "de.petermaurer.TransientPasteboardType"
    );
    migrations =     {
        "2020-04-25-allow-custom-ignored-types" = 1;
        "2020-06-19-use-keyboardshortcuts" = 1;
        "2020-09-01-ignore-keeweb" = 1;
        "2021-02-20-allow-to-customize-supported-types" = 1;
        "2021-06-28-add-title-to-history-item" = 1;
        "2021-10-16-remove-dynamic-pasteboard-types" = 1;
        "2022-08-01-rename-suppress-clear-alert" = 1;
        "2022-11-14-add-html-rtf-to-supported-types" = 1;
        "2023-01-22-add-regexp-search-mode" = 1;
    };
    searchMode = mixed;
    showInStatusBar = 0;
    suppressClearAlert = 1;
}

Description
Firstly, thanks for maintaining such an awesome and useful tool for us.
However, as I'm new to maccy, I have some confusion about default actions.
Now, when I copy something or do double clicking, maccy will automatically pop up, and may block my work.
So I wonder if there is a way to disable these features so that I can only call maccy by command+shift+c?
Steps to Reproduce
Click twick in a row or copy things

That should never happen, can you please record a video how double-clicking something opens Maccy window?

Sure! Here is the video and I changed the hotkey to command+C
https://github.com/user-attachments/assets/357e5029-afcd-4b3e-9853-354c0ce0058f

It's strange that when I change the hot key not to be command+C, I notice that maccy works well. And it won't open when clicking twice. So maybe command+C will trigget something in macOS?

Nevertheless, thanks for your help and contribution to maccy!

command+C

This is the default shortcut for copying in macOS - you should not be able to use it normally.


Correction for raw affiliation Faculté de Pharmacie Université de Montréal C.P. 2940 Chemin de Polytechnique  Montreal Quebec H3C 3J7 Canada

Correction needed for raw affiliation Faculté de Pharmacie Université de Montréal C.P. 2940 Chemin de Polytechnique  Montreal Quebec H3C 3J7 Canada
raw_affiliation_name: Faculté de Pharmacie Université de Montréal C.P. 2940 Chemin de Polytechnique  Montreal Quebec H3C 3J7 Canada
new_rors: 0161xgx34
previous_rors: 05f8d4e86;0161xgx34
works_examples: W4372348743
contact: f3ab2c157a1e648a1c6bdf87bc618323:c4f3dbf47d1209970e3819 @ polymtl.ca

This issue was accepted and ingested by the OpenAlex team on 2024-11-01. The new affiliations should be visible within the next 7 days.


getExperimentCrossCorrelation: colData variables

We were discussing that it might be good to have an option to calculate correlation between assay and sample metadata. After considering different options, I ended up to these new arguments that are used to specify columns from colData.
Any ideas/feedback?

For me this is quite intuitive (although I'm very biased...); coldata_variable1 (should it have a better name?) works just like assay_name1. Instead of fetching a matrix from assay, it fetches a variable from colData
Because of the amount of different arguments and options it might be little bit hard to grasp the idea, but I think we can improve examples (And we can improve this after feedback from users)


AST: Fix GenericSignatureBuilder bug with protocol typealiases

If a nested type of a generic parameter was a typealias, and the
generic parameter was not at depth 0, index 0, we would resolve
the type down to the wrong PotentialArchetype, causing bogus
diagnostics and crashes.
Make sure we apply a substitution to the typealias's underlying
type before resolving it, to map it to the correct PotentialArchetype.
Note that the original test case in the radar works now, but the more
comprehensive test I added exposes an existing problem where generic
signature canonicalization is not idempotent. When this is fixed,
the RUN: line in the test can be changed from -typecheck to -emit-ir.
Fixes rdar://problem/30248571.

@swift-ci Please smoke test

@swift-ci Please smoke test

@DougGregor and @huonw this might amuse you.

@huonw Wait, is this the same bug as https://github.com/apple/swift/pull/7389/?

This does the same stuff as the first commit as #7389, but I had to/am doing more to fix more things with them (e.g. explicit same-type requirements mentioning protocol typealiases will sometimes get dropped).

Yeah, this is the same as #7389, although that PR is growing beyond what fits in 3.1. This narrow fix (as noted, the same as the first commit in #7389) does look safe enough for 3.1.

@swift-ci Please smoke test

@swift-ci Please smoke test


Evaluate the exposure of trusted RPC port in production env

In order for privileged users to query the storage via RPC, integritee has built-in support for RPC requests of trusted state. We should evaluate:

is it safe to keep it open in production env?
if so, which TrustedGetter should we have?

The pros and cons are obvious IMO:
++ you get the synchronous responses for your trustedGetter queries - very F/E friendly
--  the worker IP/port gets exposed, attackers might try ddos to bring the service down => maybe we'll have some non-validateer nodes

the worker IP/port gets exposed, attackers might try ddos to bring the service down

We can put an AWS LoadBalance or even an Nginx reverse proxy in front of those workers to protect them.

Does TrustedGetter have any auth mechanism? For example, does the user still need to sign a transaction to call the rpc?

Get it. I think making the trusted RPC port be exposed to F/E is acceptable.


Filter logs

adding default filtering to "notice" level to calm down the demonstration log output

@benbp @dalebremner @brianparry adding filtering to the demo setup - will make a corresponding PR to take the low-level notices about blocking the event loop from notice to debug in on-core

info level perhaps better in light of https://github.com/RackHD/RackHD/issues/34 ?

yeah, thinking so - when we merge in any log level changes, I'll swing back by and update this

:+1:


Document SkipNonexistentProjects="Build" parameter of MSBuild task

The documentation of the MSBuild task claims that SkipNonexistentProjects is a Boolean parameter. In the source code however, it is a String parameter that recognizes the values "True", "False", and "Build". Please document the meaning of Build.
MSBuild emits <SkipNonexistentProjects>Build</SkipNonexistentProjects> as metadata of ProjectReference items in *.sln.metaproj when MSBUILDEMITSOLUTION=1.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 24ae888a-0777-7e09-cd0c-c864df8376db
Version Independent ID: 29dd0025-ff74-ec07-ad71-4a75a276d8d8
Content: MSBuild Task - Visual Studio
Content Source: docs/msbuild/msbuild-task.md
Product: visual-studio-windows
Technology: msbuild
GitHub Login: @ghogen
Microsoft Alias: ghogen

@rainersigwald

I'm not sure this option is meaningful for users outside of MSBuild itself; it means "build this project even though it doesn't exist on disk, trusting that something else has already created an object that will match it in the engine", but it's not easy to meet that guarantee outside of MSBuild (which generates .metaproj projects in memory that don't exist on disk).

Given that, we'll close this PR.


Added convex hull treshold

Could you please add some tests to ensure this is working as intended?

Test added


Fixing Errors and Improving Clarity in CONTRIBUTING.md File.

Issue Description:
The current CONTRIBUTING.md file in the project repository contains several errors, typos, and unclear wording. This issue aims to address these issues and improve the overall quality and clarity of the CONTRIBUTING.md file. By doing so, we can enhance the onboarding experience for new contributors and ensure that the contributing guidelines are accurate and easy to understand.
Proposed Changes:
https://github.com/kubevirt/kubevirt/blob/22ad8cca72e54dc760398f6556d3a7d03b0cb4ab/CONTRIBUTING.md?plain=1#L5
→ * **Kubernetes** is a container orchestration system, and is used for running.
https://github.com/kubevirt/kubevirt/blob/22ad8cca72e54dc760398f6556d3a7d03b0cb4ab/CONTRIBUTING.md?plain=1#L10
→"they have" should be "it has" to maintain consistency with the singular noun "KubeVirt."
https://github.com/kubevirt/kubevirt/blob/22ad8cca72e54dc760398f6556d3a7d03b0cb4ab/CONTRIBUTING.md?plain=1#L17
→ "shall" should be replaced with "will" for better readability.
https://github.com/kubevirt/kubevirt/blob/22ad8cca72e54dc760398f6556d3a7d03b0cb4ab/CONTRIBUTING.md?plain=1#L48
→"issues" should not be capitalized
https://github.com/kubevirt/kubevirt/blob/22ad8cca72e54dc760398f6556d3a7d03b0cb4ab/CONTRIBUTING.md?plain=1#L81
→by this act, expressing that they comply to improve the sentence structure.
https://github.com/kubevirt/kubevirt/blob/22ad8cca72e54dc760398f6556d3a7d03b0cb4ab/CONTRIBUTING.md?plain=1#L116
→ "Getting started" should be "Getting Started" for consistency with the title casing.
https://github.com/kubevirt/kubevirt/blob/22ad8cca72e54dc760398f6556d3a7d03b0cb4ab/CONTRIBUTING.md?plain=1#L141
→"A Golang BDD Testing Framework" should be "a Golang BDD Testing Framework" as "a" should be lowercase.
Expected Outcome:
By addressing these issues, we can ensure that the CONTRIBUTING.md file provides clear and accurate guidelines for contributors, fostering a more streamlined and efficient contribution process. This will ultimately contribute to the overall success and growth of the project.

@aroramrinaal it is hard to review and apply the changes in an issue. Would you mind opening a PR with your proposed changes?


python3Packages.mdformat: 0.7.9 -> 0.7.10

Motivation for this change
Update to latest upstream release 0.7.10
Change log: https://mdformat.readthedocs.io/en/stable/users/changelog.html#id1
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

The tomli update causes a mass rebuild. We probably should target staging.

The tomli update causes a mass rebuild. We probably should target staging.

We must target staging here.


fix failed render during page navigation

Description
This PR fixes the issue #279 where some players fail to load when rendering multiple CldVideoPlayer on one page.

Also fixes the issue where the CldVideoPlayer component fails to render properly after navigating between pages
Also adds a cleanup function that removes the widget instance when a CldUploadWidget component unmount

Issue Ticket Number
#279
Fixes #<ISSUE_NUMBER>

Type of change

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Fix or improve the documentation
[ ] This change requires a documentation update

Checklist

[x] I have followed the contributing guidelines of this project as mentioned in CONTRIBUTING.md
[x] I have created an issue ticket for this PR
[x] I have checked to ensure there aren't other open Pull Requests for the same update/change?
[x] I have performed a self-review of my own code
[x] I have run tests locally to ensure they all pass
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes needed to the documentation

nice work @JoshuaRotimi it seems to clear the issues we were seeing!! just a few small tweaks needed

@colbyfayock  I'm currently having some issues calling the dispose function when unmounting the component. I think the types of the CloudinaryVideoPlayer might need to be updated. It currently has only one type of on which is a function.

@colbyfayock I'm currently having some issues calling the dispose function when unmounting the component. I think the types of the CloudinaryVideoPlayer might need to be updated. It currently has only one type of on which is a function.

can you clarify if it's a Type issue or if it's an issue with the instance not being available at the time of unmounting? just in case there's a race condition there
if it's a Type issue, let's at a @ts-ignore for now and i can open a ticket to get this addressed on the widget. let me know!

@colbyfayock I'm currently having some issues calling the dispose function when unmounting the component. I think the types of the CloudinaryVideoPlayer might need to be updated. It currently has only one type of on which is a function.

can you clarify if it's a Type issue or if it's an issue with the instance not being available at the time of unmounting? just in case there's a race condition there
if it's a Type issue, let's at a @ts-ignore for now and i can open a ticket to get this addressed on the widget. let me know!

Oh yes it is a type issue, @ts-ignore is a valid temporary fix. Also, are you still experiencing the issue of one or more components failing to load? Not sure if you saw my previous comment about using the id prop for each component.

i need to test it again
i just found out however that there's another onReady callback for the Script tag, it may be worth considering playing around with that to see if its a bit more reliable

onLoad: Execute code after the script has finished loading.
onReady: Execute code after the script has finished loading and every time the component is mounted.

https://nextjs.org/docs/pages/building-your-application/optimizing/scripts#executing-additional-code
i would imagine that's actually what we would want, correct? it may not make the initial load more reliable, but it should make the page navigation more reliable and prevent us from having to include the additional initialization inside useEffect
h/t: https://twitter.com/stordahldotdev/status/1719098207651905820?s=61&t=ya091PlFoeyCWxWs2FVkIQ

Alright. You can pull the latest changes and test it again then let me know if it works. I will check out the onReady callback to see if it provides a better solution.

It's better than it was before but it's still having issues loading completely reliably between page loads
https://d.pr/v/W1Oahw
Notice on the 3rd and 4th navigation as well as some others we're missing a video.
Let's see how onReady works here, it might resolve the issue

@JoshuaRotimi any luck?

the onReady callback only runs once, so only the first component is loaded properly so I'm still unable to figure out the best solution so far. any ideas? @colbyfayock

i see what you mean. i dont have any quick ideas ill need to spend more time thinking about this
i think there's enough value to move forward with this now though without waiting to help address some issues for people
can you remove the examples at the bottom of: docs/pages/cldvideoplayer/basic-usage.mdx
then i can merge it in and perhaps if you'er still interested in looking into this we can do it in another ticket, otherwise i can take over
thanks @JoshuaRotimi

Alright done.

thanks @JoshuaRotimi - awesome job with this!!


Fixes #179 -- generate a 2018 edition fuzzer on init

Err, I'm baffled by the error here, 2018 edition crates should be able to use macros like that... right?

bors r+

@nagisa Thanks for reviewing all of these!
Is there any chance of a new cargo-fuzz release, to support the libfuzzer-sys update?


create a bio file @ychen0515

:smile:

Looks good. I'm going to go ahead and delete your old branch.


EventLog target to support SuccessAudit and FailureAudit events

Currently EventLog target translates logEvent.Level into EventLogEntryType.Error,.Warning or .Information.
It will be good to have new entryType configuration parameter that will allow to specify SuccessAudit and FailureAudit events
(reference http://msdn.microsoft.com/en-us/library/system.diagnostics.eventlogentrytype(v=vs.110).aspx)

@MNF I would like to look at this. Any ideas how to configure this feature?

https://github.com/304NotModified/NLog/commit/d1bab0a9dca2cfafd7ac437f8940c64dd0d19cc1


SRTM endpoint return 404

It seems that the SRTM endpoint have been removed. All of them return code 404 at the moment.

Yes, they changed the URLs and the way the data is downloaded. It now requires a basic html authentication.
Once I fix https://github.com/tkrajina/go-elevations/issues/3 I'll move to srtm.py to make it work again. Unfortunately my free time for this is limited, so this (probably) won't be ready in a couple of days :( But I'll try my best to do it next week.

Seconding @jpenninkhof on this issue, and issuing a thanks in advance!

It would be very helpful if this could be fixed. Many thanks in advance @tkrajina !

@jgliss @farawayhillsaregreener @jpenninkhof I'm still working on the earthdata branch, but as a temporary solution I changed master to use a SRTM mirror provided by https://srtm.kurviger.de
So, master works now. I also released a new version on pypi.
I'm leaving this issue open until I also fix and merge the earthdata branch.

@tkrajina great thanks!


Page Change Shortcuts loose alignment of the new page

[x]  When we navigate to the next/previous page using page change icon shortcuts or icon, if there is no opened tab next then it is opening the next page from the directory but the alignment of the new loaded page is lost.

Solved the bug, now you can use the shortcuts and icon to shift pages easily. Please refer to 06860dca759a78cbcdf7ed2ce1a28b1716f01f41


"Cannot read property 'globalStore' of undefined" - error presented when trying to log into git hub enabled setup.

**Rancher versions: v2.0.5-rc4
Steps to reproduce the problem:
github enabled set in 2.0.4 with access site set to - "Allow members of Clusters, Projects, plus Authorized Users" with few users added.
Upgrade to 2.0.5-rc4.
After rancher server gets upgraded , I see "Cannot read property 'globalStore' of undefined" presented every time I log in as git hub admin user

This issue happens even on fresh install of  v2.0.5-rc4

Tested with latest build from master.
No error seen when logging into github auth enabled setup as admin user.


[CAS-1037] Add membase support to MemCacheTicketRegistry

Membase (http://www.couchbase.org/membase) adds disk persistence and replication to the Memcached server. There are three ways to deploy the Membase server: http://www.couchbase.org/wiki/display/membase/Deployment+Strategies
The two strategies using the membase proxy work with the existing MemCacheTicketRegistry. The 'vBucket aware' deployment strategy requires the latest version of the spymemcached client (2.7) and some extra config options.
Reported by: Eric Pierce, id: epierce
Created: Fri, 19 Aug 2011 16:15:06 -0700
Updated: Sun, 9 Oct 2011 13:27:23 -0700
JIRA: https://issues.jasig.org/browse/CAS-1037
Punting for now because we need to get the dependency in central repo

There is now support for couchbase. Closing this for now until the need resurfaces.


Docs sprint

Per chat with @GretaCB we should work on a quick docs sprint.
Main tasks:

Make sure data API is documented well: mapnik.Datasource/Featureset/Features
In particularly integrate toJSON improvements in docs: https://github.com/mapnik/node-mapnik/issues/308
Port markdown docs (https://github.com/mapnik/node-mapnik/tree/master/docs) to new format @brianreavis sketched out in https://github.com/mapnik/node-mapnik/tree/documentation

Documentation TODO

[ ] mapnik.Layer
[ ] Finish mapnik.Map documentation
[ ] Add at least one example per class
[x] Polish up documentation style
[ ] output throws tags https://github.com/documentationjs/documentation/issues/86
[ ] output undefined returns https://github.com/documentationjs/documentation/issues/62

@tmcw - thanks for pushing this ahead. Want to re-iterate that I would love to pull in @GretaCB to hone and add some critical docs. Her experience with the performance aspects of Mapnik in production will help highlight key places to cover - e.g. datasource plugin options like cache_features.

@springmeyer @GretaCB think we could close this? Docs have been updated heavily since this issue was created.

Closing, docs have been updated quite a bit since this sprint plan!


[WIP][Bug] Issue 934

Digging into #934

I have filed a bug on the HF Hub model:
https://huggingface.co/bartowski/Meta-Llama-3-8B-Instruct-GGUF/discussions/9
But I think this is really a problem in the LlamaCpp layer


On Windows: Division by zero error

Dear Inigoflores,
on windows i started gettin a division by zeror erro. Do you have any idea why? It was working some weeks ago.
Using logs in folder processlogs\logs
Warning: Division by zero in processlogs\processlogs.php on line 168

General Witnesses Overview
----------------------------------
Total witnesses                   =     1   (INF/hour)
Succesfully delivered             =     0     (0%)
Failed                            =     1   (100%)
  ├── Max retry    =    0     (0%)
  └── Crash/reboot =    1   (100%)

Max Retry Failure Reasons
----------------------------------
Timeout                           =     0     (0%)
Not Found                         =     0     (0%)
Other challenger issues           =     0     (0%)

Challengers
----------------------------------
Not Relayed                       =     0     (0%)
Relayed                           =     0     (0%)
Unknown (Probably Not Relayed)    =     1   (100%)

console.log

I am all of a sudden getting the same error as well:
Fatal error: Uncaught DivisionByZeroError: Division by zero in C:\MinerTools\processlogs\processlogs.php:168
Stack trace:
#0 C:\MinerTools\processlogs\processlogs.php(78): generateStats(Array)
#1 {main}
thrown in C:\MinerTools\processlogs\processlogs.php on line 168

@Secarius, that's weird. It works for me with your logs
General Witnesses Overview
----------------------------------
Total witnesses                   =    29  (2.06/hour)
Succesfully delivered             =    18 (62.07%)
Failed                            =    11 (37.93%)
  ├── Max retry    =   11 (37.93%)
  └── Crash/reboot =    0     (0%)

Max Retry Failure Reasons
----------------------------------
Timeout                           =     7 (24.14%)
Not Found                         =     2   (6.9%)
Other challenger issues           =     1  (3.45%)

Challengers
----------------------------------
Not Relayed                       =     9 (31.03%)
Relayed                           =    10 (34.48%)
Unknown (Probably Not Relayed)    =    10 (34.48%)

What version of the tool are you using? Have you downloaded it recently?
@DCYeahThatsMe please provide your log files and let me know when you downloaded the tool.

I configured my tool to download the newest version of your tool on startup.
So its the newest version :) (same for DCYeahThatsMe, he used also the newest version, as we both communicate before.

@inigoflores yes, what @Secarius said is correct.  This has been happening for a few days at least now.  It happens on all the UG65 Milesight console.log files from various miners.
console.log
I'll attach one as an example.

But only on my windows machine. If i run it with ubuntu it works

Yes, I am also running Windows.  I haven't installed any software specifically.... maybe there was a windows update, but nothing that I'm aware of.

Same here running on Windows using Sensecap logs

One of the latest commits introduced a bug on Windows. It should now be fixed. Please confirm.

One of the latest commits introduced a bug on Windows. It should now be fixed. Please confirm.

Can confirm its working now on Windows 👍

Yes it works now. Thank you very much for the fast support. 👍


Regarding Keymapping

Is your feature request related to a problem? Please describe
Hello Developers kindly add shooting games and add keymapping in wsa for shooting games like (PUBG,FF) with custom key option . Now you developing well. All the best for your future growth.
Describe the solution you'd like
My suggestion is use bluestacks keymapping beacuse the keymapping is good
Describe alternatives you've considered
No response
Please specify the version of Windows Subsystem for Android
Build 2209.40000.26.0

Closing as duplicate of #82


[New Model]: PackingList 1.0.0

Model Description
This new model is part of various aspect models linked to the logistic twin (others e.g. GTL). The PackingList Aspect Model will be a simple representation of a packing list commonly used in logistics. Find an example attached and further information here: https://confluence.catena-x.net/display/BDPQ/DT+transport+UNIT+and+matching+aspect+model+GTL

Example Data
{
"orderNumber": "123456",
"shippingDate": "2023-02-11",
"orderDate": "2023-01-12"
"receiverAddress": {
"bpnaProperty": "BPNA0123456789ZZ"
},
"senderAddress": {
"bpnaProperty": "BPNA0123456789ZZ"
},
"listItems": [
{
"itemName": "Valve",
"quantity": {
"quantityNumber": 2.5,
"measurementUnit": "unit:litre"
},
"position": 1,
"itemDescription": "eOMtThyhVNLWUZNRcBaQKxI"
}
],
}
MS1 Criteria

[ ] The proposed aspect model does not exist already
[ ] The proposed aspect model does not extend an existing aspect but introduces completely new functionality
[ ] The proposal references a Catena-X use case
[ ] Relevant standards are mentioned/linked
[ ] A first draft of the data structure is provided in an example
[ ] A modeling steward has been assigned

modeling_team

The proposed aspect model does not exist already -> OK
The proposed aspect model does not extend an existing aspect but introduces completely new functionality -> OK
The proposal references a Catena-X use case -> OK
Relevant standards are mentioned/linked -> OK
A first draft of the data structure is provided in an example -> OK
A modeling steward has been assigned -> @agg3fe


⚠ Samsung has degraded performance

In 67a0c06, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2395 ms

Resolved: Samsung performance has improved in 8488871.


Add filled character class

Add way to style filled characters

Hey @andreaswilli! I would like to add different styles to the characters that have been filled, so placeholders could be gray for example and filled chars black. What do you think of this approach?

Thanks for the PR @priidikvaikla! I like your proposal, since this could also improve the cursor recipe (which currently does not work if we use a placeholder).
Implementation looks good, please add/update unit tests. Afterwards, I can update the cursor docs.

Added tests

Published v4.1.0.


srb init fails out of the box with ruby 2.4.6 / rails

While trying to reproduce another issue, I ran:
docker run -it --rm ruby:2.4.6 bash
gem install rails
apt-get update && apt-get install -y nodejs
cd /tmp
rails new sorbet
cd sorbet/
echo "gem 'sorbet', :group => :development" >> Gemfile
echo "gem 'sorbet-runtime'" >> Gemfile
bundle install
bundle exec srb init

This fails with:
[...]
Would you like to continue? [Y/n] y
Generating: sorbet/config
Generating: sorbet/rbi/sorbet-typed/
Cloning into '/root/.cache/sorbet/sorbet-typed'...
remote: Enumerating objects: 73, done.
remote: Counting objects: 100% (73/73), done.
remote: Compressing objects: 100% (44/44), done.
remote: Total 306 (delta 28), reused 49 (delta 15), pack-reused 233
Receiving objects: 100% (306/306), 92.95 KiB | 0 bytes/s, done.
Resolving deltas: 100% (109/109), done.
Generating: sorbet/rbi/gems/
/tmp/sorbet/config/application.rb:9:in `<top (required)>': Sorbet is not a module (TypeError)
	from /usr/local/bundle/gems/sorbet-0.4.4426/lib/require_everything.rb:25:in `require'
	from /usr/local/bundle/gems/sorbet-0.4.4426/lib/require_everything.rb:25:in `load_rails'
	from /usr/local/bundle/gems/sorbet-0.4.4426/lib/require_everything.rb:18:in `require_everything'
	from /usr/local/bundle/gems/sorbet-0.4.4426/lib/gem-generator-tracepoint.rb:39:in `block in main'
	from /usr/local/bundle/gems/sorbet-0.4.4426/lib/gem-generator-tracepoint/tracer.rb:70:in `trace'
	from /usr/local/bundle/gems/sorbet-0.4.4426/lib/gem-generator-tracepoint.rb:38:in `main'
	from /usr/local/bundle/gems/sorbet-0.4.4426/bin/srb-rbi:229:in `block in make_step'
	from /usr/local/bundle/gems/sorbet-0.4.4426/bin/srb-rbi:118:in `init'
	from /usr/local/bundle/gems/sorbet-0.4.4426/bin/srb-rbi:196:in `main'
	from /usr/local/bundle/gems/sorbet-0.4.4426/bin/srb-rbi:234:in `<main>'

Try rails new sorbet with another name. You appear to be reusing sorbet.

Wow ... so obvious in hindsight. Thanks for the quick feedback.


Error message suggests wrong command "ontospy --bootstrap"

Running ontospy gendocs --lib fails with the following error message:
Your local library is empty. Use 'ontospy --bootstrap' to add some ontologies to it.

The command ontospy --bootstrap does not exist, however.
Correct command seems to be ontospy lib --bootstrap

Thanks, was fixed in https://github.com/lambdamusic/Ontospy/commit/fb778ee031157e376cd8140a16a7d03874f59ebe#diff-d3371ef67d5f601acefec8569cdd5354


Listview generator broken

What's going on?
Cannot use the generator as intended, the name is set always View.js and that leads to the error:
View.js: Duplicate declaration of "View"
Steps to reproduce
$ ignite generate listview EventList
? What coding style do you want for your list? Flatlist (new)
? What kind of List would you like to generate? Row
? How will your data be presented on this list? Single
✔ App/Containers/View.js
✔ App/Containers/Styles/ViewStyle.js
At first I thought it was because of this new Flatlist, but the classic gives the same results:
$ ignite generate listview EventList
? What coding style do you want for your list? Listview (classic)
? What kind of List would you like to generate? Row
? How will your data be presented on this list? Sectioned
✔ App/Containers/View.js
✔ App/Containers/Styles/ViewStyle.js
I tried with a quoted name too with no luck
ignite doctor results:
System
platform           darwin
arch               x64
cpu                4 cores                                       Intel(R) Core(TM) i5-4260U CPU @ 1.40GHz
directory          /Users/jumper/Developer/ReactNative/Balance
JavaScript
node               8.7.0        /usr/local/bin/node
npm                5.5.1        /usr/local/bin/npm
yarn               1.3.2        /usr/local/bin/yarn
React Native
react-native-cli   2.0.1
app rn version     0.49.3
Ignite
ignite             2.0.0        /usr/local/bin/ignite
Android
java               1.8.0_144    /usr/bin/java
android home       -            /Users/jumper/Library/Android/sdk
iOS
xcode              8.3.3

Hi @JonathanJumper  listview  Deprecated - Use list

This video has the same error
https://www.youtube.com/watch?v=lv00UOXxy0g

@joelbarbosa still deprecated but not clear in the cli? Is that intentional?

Unfortunately, this codebase has gotten dusty.  We'll be rolling out a new boilerplate soon, but if you want to kick over a PR to fix this issue, that'd be awesome.

Oops... unintentional close.  Damn those 2 buttons are close together.

No longer relevant, so I'm closing this. 😎


Trying to access Ticker objects for multiple tickers results in AttributeError

According to the documentation we can initialize multiple Ticker objects like this

import yfinance as yf
tickers = yf.Tickers('msft aapl goog')
tickers.tickers.MSFT.info
tickers.tickers.AAPL.history(period="1mo")
tickers.tickers.GOOG.actions

However, I get the following error:
print(tickers.tickers.MSFT.info)
AttributeError: 'dict' object has no attribute 'MSFT'

I'm using v 0.1.59

In case anyone else has this issue I realized that I had to iterate over the key-value pairs of the dictionary, so I did this:
tickers = yf.Tickers('msft aapl goog')
for key, value in tickers.tickers.items():

Then, I can access each ticker (key) in the dictionary. For example:
	pb = value.info['priceToBook']


SCSS variable highlighting stopped working

Using v2.0.7
Highlighting just disappeared for SCSS variables. Pretty sure this was working prior to v2.0.7.

That might just be what variables look like in your color scheme. What do variables look like in say JavaScript?

@braver same, here, again:

where:

@media should be red
all variables orange and italic
backdrop-filter should not be red

@braver here is how js code is highlighted

Parameters in a function definition should be orange italic, and they are.
Variables should be white. When they’re defined and whenever they’re used. Variables passed as parameters to a function call should also be white.

Your JS example clearly shows that as well:

In the definition of sampleFunc the args are orange
The obj variable is always white
The obj variable is also white when passed as a paramter to set.add

The old version of Sass got it wrong, it’s probably the only syntax out there that scopes things like it does.

@braver same, here, again:

where:

@media should be red
all variables orange and italic
backdrop-filter should not be red

You may be right about the @media and the backdrop-filter though.

Not sure if this one deserves separate issue:
.sidebar
  background: none
  @media(max-width: $break-parent)
    will-change: transform, scroll-position

.sidebar
  background: none
  will-change: transform, scroll-position

Where transform and scroll-position has two different colors

I'm closing this. The original issue as far as there was any has been resolved. backdrop-filter isn't a standardized property and therefore not recognized. In any case that's something completely seperate from the original issue. @dr-dimitru this is something else, unrelated to the other issues raised here. I don't have a fix for that yet, but have a test in place and will look at a solution soon.


Configuration File Manager

Add ConfigurationManager class ("ivadomed/config_manager.py") to help with retrocompatibility when the configuration file is modified. The default config file is defined in "config/config_default.json". From now on, it will only be necessary to update "config/config_default.json" and not all config files.

For now I only did the minimal code to make it work. I did not integrate the config documentation in the class, since I wasn't sure where to include it.

Pull Request Test Coverage Report for Build 322682184

35 of 36 (97.22%)  changed or added relevant lines in 6 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.4%) to 91.26%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

ivadomed/config_manager.py
19
20
95.0%

Totals

Change from base Build 322441987:
0.4%

Covered Lines:
5597

Relevant Lines:
6133

💛  - Coveralls


[1.22.0] [BUG] Can't set waypoint for construction sub in the trench

Factorio & Maraxsis version number
Factorio 2.0.23
Maraxsis 1.22.0
Describe the bug
Can't set waypoint for construction sub in the trench
To Reproduce
Steps to reproduce the behavior:

Select the waypoint tool
Right click in the trench

Expected behavior
A waypoint for the construction sub is set (I assume the sub will move from topside to the trench and back if there aew waypoints for it)

So I am not sure if this is a bug or not.
It seems to me that construction subs should only be used for construction tasks (construct, deconstruct and upgrade) and the diesel sub is the one that you should use for automated schedules as you can set a waypoint in the trench by riding in it and manually submerging.
It is a little confusing though
@notnotmelon what are your thoughts here? is this intended behavior, a lack of communication or just an incomplete feature?

Okay I spoke too soon,
With patch 1.22.0 you can successfully use submerge option in the schedule (you previously couldnt manually submerge while riding in a construction sub so you couldnt create the trench way points)
Okay so as it stands right now, the only way to make waypoints in the trench is to manually ride in the sub you want to automate, manually submerge, then create those waypoints in the trench
I agree this is cumbersome and it would be more intuitive if you could set waypoints while in remote view

Maybe there should only be one type of sub for both types of tasks instead of two separate classes?

When a sub submerges, all spidertron/patrol remotes are disconnected, so you have to reconnected it by click-dragging, or get a new connected one from the sub's GUI (or Control+Shift+Q). Once you've done that, you can happily carry on placing waypoints in remote view. You can't place waypoints on a surface that the submarine is not currently occupying.
There was a bit of jank in the patrol renders not updating correctly during a submerge, that has now been fixed.


Vsblty vision captor and data captor app

Request you to please update the README.md file as per the guideline shared. The license is missing in all the yaml files. Request you to fix that too.

@architag21 updated the Readme.md file

@architag21 updated the Readme.md file

Could you please mention the flavor of Smart Edge-Open used..


Can't connect PS Move Controller. Driver reinstallation.

I connected my controllers to pc, everything worked fine, but i reinstalled motion controller drivers in zadig and now i can't connect my controllers to psmoveservice. Please, help me.

Problem solved with full reinstalling of windows

Glad to hear you got it sorted.


[Statement checker] Always wire up case vars with bindSwitchCasePatternVars

bindSwitchCasePatternVars() was introduced as a simpler way to wire up
the "parent" links for case variables with same-named case variables
from the previous case item, and is used in the function builders code
to handle switch statements. It duplicated some logic from the
statement checker that did the same thing using a more complicated
algorithm.
Switch (ha ha) the logic in the statement checker over to using
bindSwitchCasePatternVars(), fixing a bug involving unresolved
patterns along the way, and remove the old code that incrementally
wired up the parent links. The resulting code is simpler and is
unified across the various code paths.

@swift-ci please smoke test

@swift-ci please test source compatibility

@gottesmm you might be interested in the refactor here, because I think you wrote the code I'm calling "complicated" ;)


Add static-safe mapping to client/galaxy/images

This fixes various fugue/etc icons not displaying correctly (https://github.com/galaxyproject/galaxy/issues/9080#issuecomment-564590700)

Thanks @dannon


from qiskit.pulse import SamplePulse for type hints

Summary
from qiskit.pulse import SamplePulse for type hints
Details and comments
Help readers (and mypy) know where SamplePulse is defined.

Closing as this was addressed in #3902 as part of sweeping documentation updates. Thank you for your effort to clean up our code 😄

Wow.  What was the logic behind removing all the type hints and moving them into comments?

The sphinx type hints extension was unable to resolve the type hints correctly without actually importing it as you do in this PR. But, when you do import it as @taalexander reported (and CI here showed) it causes both test and linter failures around circular imports. To fix the documentation build and properly resolve a link in the compiled documentation it was necessary to stop using type hints

Got it.  The return values as strings were a better solution than no type checking at all.


disable/enable extension for specific language

Hello
This can be a useful option - disable or enable a specific extension for a specific language.
It could be identify by extension id
Maybe something like:

What do you mean by disabling or enabling an extension per language?

For example, I'm using a plugin like Highlight Matching Tag. I want to use it for XML files, but I do not want to use it in HTML files at the same time
The next example is that I use the Prettier plugin. I would like it to function only in JS files.
These are two examples that immediately came to my mind, but I think there are more cases where it could be useful

A little similar ticket: #40239
Not the same but similar

I do not think it is possible by disabling the extension. Because we can disable an extension completely but not only for some files.


fix: Model name replaced by drive letter in JSON schema on Windows #7860

Change Summary

This fixes an issue where the JSON schema can end up using the drive letter (like C, D etc) instead of the model name on Windows if the module with the model is imported from an absolute path on the filesystem (for example using importlib).  This was caused by splitting at the first colon : character to remove it and the ID following it, thus on an absolute path on Windows of the form C:\... ended up becoming just the drive letter. This fix changes it to remove the LAST colon character and anything following it.
Related issue number

fix #7860
Checklist

[x] The pull request title is a good summary of the changes - it will be used in the changelog
[ ] Unit tests for the changes exist
[x] Tests pass on CI
[ ] Documentation reflects the changes where applicable
[ ] My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

Please update 👍
As discussed in the issue, with a unit test 😄

I think I should have put this here, rather than in the issue (well, it's now in both):
Hi,
Sorry for the delay with the unittest and the PR in general. I ended up checking a bit deeper into the issue still while adding the unittest and realized a mistake in the way importlib was used and then got interrupted with some more urgent work for couple of days.
This was the original proof of concept code (based on our project that was using it the same way):
    poc_dir = Path(__file__).parent.relative_to(Path.cwd())
    p = (poc_dir / "models.py").absolute()
    spec = importlib.util.spec_from_file_location(name=str(p), location=str(p))

    if not spec.loader:
        raise RuntimeError(f"Failed to import {p} module")

    module = spec.loader.load_module(str(p))

    return getattr(module, "MyModel")

An import of a module file should really be made like this instead: https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
But if one uses the way shown above it will cause the name of the module to also be something like C:\path\to\module.py (on Windows). I was originally trying to change the name parameter to something without a colon, but each try to do that resulted in an error. That way of doing the import will only work if the name and location match (based on empirical tests at least), whereas when using the correct way you can set the name to something that looks like a python module name.
Somehow I didn't figure out that this was the wrong way to do the import and what the right way was.
I think the fix itself still makes sense; it makes the code more robust, but on the other hand I'm not sure if I really think the unit test for it makes sense; one should not really name a python module with a colon in the name. Showing that being done in the unittest and cluttering one of the fixtures with an extra parameter is not something I really feel happy about. On the other hand I can't think of any better way to add the test without unnecessary duplication either.

I think this PR should be renamed to Make the model name generation more robust in JSON schema and the description  something like this:

This makes the model name used in the JSON Schema be correct also if the model is loaded from a module that has a colon in the name. This should not really happen in any correct use case (python modules should not typically have colons in them), but it's possible to achieve by accident if you use the absolute file path as the module name on Windows when working with importlib. In that case you would have ended up getting just the drive letter as a model name in the JSON Schema, this fix ensures it will still work correctly in that case.


This commits adds a migration binary to moves clients on a postgresql…

… db to a smart contract db

This is a good way to implement everything here. I appreciate it, now we just have to migrate a test exist instance and see if that breaks.


Use RH friendly images

/ok-to-test

/hold awaiting for the CI job @andfasano

/test ?

/test e2e-tests
It was just optional, so it could be manually triggered meanwhile I put it back as mandatory

/retest

/test e2e-tests

/test e2e-tests

/retest

/retest

/hold cancel

/test e2e-tests

/test e2e-tests

/test e2e-tests

/test e2e-tests

/test e2e-tests

/test e2e-tests

/test e2e-tests

thanks, lgtm,  but would you mind squashing both commits into 1

/test e2e-tests

The change looks fine also to me, I'd like to see the e2e-test running first

/approve

/lgtm
thanks


fix(DatePicker): normalize min/max dates before comparison

SPT-2117
Description
In staff-portal form many DatePicker's we're using minDate in the following way:
const minDate = useMemo(() => new Date(), [])
<DatePicker minDate={minDate} />

But currently, this causes a bug when the user manually entered the current date inside DatePicker's input.
This is because internally in isDateBefore (and isDateAfter) we're comparing date and dateToCompare as is, so because of the minDate={new Date()},  dateToCompare contains current time and helper returns false:
date: Thu Nov 18 2021 00:00:00 GMT+0300
dateToCompare: Thu Nov 18 2021 15:23:46 GMT+0300

In this PR i added normalization of min and max dates, so just the resetting of the time to the 00:00:00:000 values
How to test
To reproduce the bug:

Open https://picasso.toptal.net/?path=/story/picasso-lab-datepicker--datepicker
For any example add minDate={new Date()}
Enter the current date to the date input
The date won't be selected

To test the bug:

Open https://picasso.toptal.net/SPT-2117-normalize-min-max-dates/?path=/story/picasso-lab-datepicker--datepicker
For any example add minDate={new Date()}
Enter the current date to the date input
The date should be selected

Screenshots

Before.
After.

Review

[x] Read CONTRIBUTING.md and Component API principles
[n/a] Annotate all props in component with documentation
[n/a] Create examples for component
[x] Ensure that deployed demo has expected results and good examples
[x] Ensure that tests pass by running yarn test
[x] Ensure that visuals tests pass by running yarn test:visual. If not - check the documentation how to fix visual tests
[x] Ensure the changed/created components have not caused accessibility issues. How to use accessibility plugin in storybook.

PR commands

List of available commands:

@toptal-bot run all - Run whole pipeline
@toptal-bot run build - Check build
@toptal-bot run visual - Run visual tests
@toptal-bot run deploy:documentation - Deploy documentation
@toptal-bot run package:alpha-release - Release alpha version

PR Review Guidelines

When to approve? ✅
You are OK with merging this PR and

You have no extra requests.
You have optional requests.

Add nit:  to your comment. (ex. nit: I'd rename this variable from makeCircle to getCircle)

When to request changes? ❌
You are not OK with merging this PR because

Something is broken after the changes.
Acceptance criteria is not reached.
Code is dirty.

When to comment (neither ✅ nor ❌)
You want your comments to be addressed before merging this PR in cases like:

There are leftovers like unnecessary logs, comments, etc.
You have an opinionated comment regarding the code that requires a discussion.
You have questions.

How to handle the comments?

An owner of a comment is the only one who can resolve it.
An owner of a comment must resolve it when it's addressed.


Parameterized Postman test case environment files

Currently hardcoded to localhost.env.

Now on the testing page MDWHub has a new option for entering the postman env.  Also, the CLI test command has a new option --postman-env.  Location is relative to .test file, or can use pkg/asset syntax.


Make 2fa required error readable

Patches in a check for the 200 2fa verification response, and raises a readable exception for it
Decided to patch the main request method in rest.py as this can be returned in (all?) requests requiring auth
Let me know if I should make any changes

@ariesclark Untested but LGTM


Video image is broken after app is restored

Hi,
I'm using card showing the video stream from my camera (TPLink Tapo C200).

I put the mobile application on the background
-- battery saver is disabled for the app
-- automatic start is enabled
Left the app there for some period of time (looks like it is necessary to turn off the screen and put phone to "standby").
After the application is opened again (restored from background), the video is screwed:

I have to close the app and start it again completely to fix the stream.

HA is running on 64bit Pi4 4GB.

After some more testing it seems to be random. Sometimes it happens even after fresh app start. Sometimes it is fixed after few seconds of playback. And sometimes it not happen even application is restored from background after some longer mobile standby.
I would give this very low prio as the workaround exists and I'm not able to test it with different camera.

Same here, did you solve this? I just slide to different page in dashboard and come back to restart the card, is not a big problem but would be great to avoid this annoying issue, also with Tapo C200.

Check my new project. Maybe it handle this situation
https://github.com/AlexxIT/go2rtc

Should be fixed in v3


Support paginating over nested doc elements

This is now enabled when the page-size parameter is used even without a where clause.
Note: this is a slight change in API behaviour as previously
the page-size parameter was ignored in this case. However, there
was no reason for specifying the page size in this case before,
so hopefully end-user use cases are not broken by this change.
Fixes #1040

Example usage:
~ $ curl -L -H "X-Cassandra-Token: $AUTH_TOKEN" -g '<IP_ADDRESS>:8082/v2/namespaces/demo1/collections/test2/1f7af925-3dd2-4dd3-8cda-74a1814f4155'|jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   102  100   102    0     0   7285      0 --:--:-- --:--:-- --:--:--  7285
{
  "documentId": "1f7af925-3dd2-4dd3-8cda-74a1814f4155",
  "data": {
    "b": 1,
    "o": {
      "1": {
        "x": "1"
      },
      "2": {
        "x": "2"
      }
    }
  }
}

Paginate over top-level elements:
~ $ curl -L -H "X-Cassandra-Token: $AUTH_TOKEN" -g '<IP_ADDRESS>:8082/v2/namespaces/demo1/collections/test2/1f7af925-3dd2-4dd3-8cda-74a1814f4155/*?page-size=1'|jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   177  100   177    0     0  11062      0 --:--:-- --:--:-- --:--:-- 11062
{
  "documentId": "1f7af925-3dd2-4dd3-8cda-74a1814f4155",
  "pageState": "JDFmN2FmOTI1LTNkZDItNGRkMy04Y2RhLTc0YTE4MTRmNDE1NRT_VVVVVVVVVVQBYv9VVVVVVVVVVfB_____8H____8=",
  "data": [
    {
      "b": 1
    }
  ]
}

~ $ curl -L -H "X-Cassandra-Token: $AUTH_TOKEN" -g '<IP_ADDRESS>:8082/v2/namespaces/demo1/collections/test2/1f7af925-3dd2-4dd3-8cda-74a1814f4155/*?page-size=1&page-state=JDFmN2FmOTI1LTNkZDItNGRkMy04Y2RhLTc0YTE4MTRmNDE1NRT_VVVVVVVVVVQBYv9VVVVVVVVVVfB_____8H____8='|jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    98  100    98    0     0   8909      0 --:--:-- --:--:-- --:--:--  9800
{
  "documentId": "1f7af925-3dd2-4dd3-8cda-74a1814f4155",
  "data": [
    {
      "o": {
        "1": {
          "x": "1"
        },
        "2": {
          "x": "2"
        }
      }
    }
  ]
}

Paginate over second-level elements:
~ $ curl -L -H "X-Cassandra-Token: $AUTH_TOKEN" -g '<IP_ADDRESS>:8082/v2/namespaces/demo1/collections/test2/1f7af925-3dd2-4dd3-8cda-74a1814f4155/*/*?page-size=1'|jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   195  100   195    0     0  16250      0 --:--:-- --:--:-- --:--:-- 16250
{
  "documentId": "1f7af925-3dd2-4dd3-8cda-74a1814f4155",
  "pageState": "JDFmN2FmOTI1LTNkZDItNGRkMy04Y2RhLTc0YTE4MTRmNDE1NRj_VVVVVVVVVUABbwExAXj_VVVVVVVVVVXwf_____B_____",
  "data": [
    {
      "o": {
        "1": {
          "x": "1"
        }
      }
    }
  ]
}

~ $ curl -L -H "X-Cassandra-Token: $AUTH_TOKEN" -g '<IP_ADDRESS>:8082/v2/namespaces/demo1/collections/test2/1f7af925-3dd2-4dd3-8cda-74a1814f4155/*/*?page-size=1&page-state=JDFmN2FmOTI1LTNkZDItNGRkMy04Y2RhLTc0YTE4MTRmNDE1NRj_VVVVVVVVVUABbwExAXj_VVVVVVVVVVXwf_____B_____'|jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    84  100    84    0     0   7000      0 --:--:-- --:--:-- --:--:--  7636
{
  "documentId": "1f7af925-3dd2-4dd3-8cda-74a1814f4155",
  "data": [
    {
      "o": {
        "2": {
          "x": "2"
        }
      }
    }
  ]
}

Get all nested elements (same as before this change):
~ $ curl -L -H "X-Cassandra-Token: $AUTH_TOKEN" -g '<IP_ADDRESS>:8082/v2/namespaces/demo1/collections/test2/1f7af925-3dd2-4dd3-8cda-74a1814f4155/o'|jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    90  100    90    0     0  15000      0 --:--:-- --:--:-- --:--:-- 15000
{
  "documentId": "1f7af925-3dd2-4dd3-8cda-74a1814f4155",
  "data": {
    "1": {
      "x": "1"
    },
    "2": {
      "x": "2"
    }
  }
}

@semichkin: Please review the example output above. Does it work for your use cases in #1040? Please note the presence of the full chain of parent object all the way to the root when paginating over nested elements. This is necessary to disambiguate the identity of the element since the path may include *.

The intermediate build failure was indeed obscure - the test got not results from SG, but no errors were logged. I reviewed the related code (searchPathWildcardsInMemoryMatching) but could not find any obvious problems with how that request is executed. I did notice a two second pause in C* logs at the time the test method was running... perhaps it was a stop-world GC and something timed out.
Re-run succeeded, so I'm proceeding to merge this PR.


Possible memory leak in WinUI 3 Preview 3 when using ContentDialogs and UserControls

Describe the bug
I'm developing a Win32 Net5 app using the WinUI 3 Preview 3 and I'm getting some problems with possible memory leaks. I have a complex app which uses a lot of UserControl and ContentDialogs but, after they are out of scope, their memory is never released.
I'm trying to do some memory analyses using DotMemory and in the screenshot below it is possible to see one of the UserControl (named SongControl) that is never released. In particular, it is possible to see that the only thing that is retaining the UserControl is the so called "Regular Handle", which I think is a GCHandle from the internal WinUI code.
This SongControl is used in a ListView and is generated using an ObservableCollection as ItemSource. When I navigate away from the page with the ListView, I remove all the items present in the ObservableCollection (removing also all the event handlers linked to each item and setting all of them to null) and then set the collection to null.
As there is nothing in particular retaining the control, I believe that in a standard situation the internal WinUI code would have to remove that Regular Handle and allow the object to be collected, but it doesn't happen. Because of this, the memory footprint of my app keeps growing forever and so, soon or later, the user will get an OutOfMemoryException.
The same behaviour happens for every ContentDialogs I use. When they are closed, they are never released from memory, even setting them to null and removing all event handlers to which they are linked.
I already tried to set each UserControl and each ContentDialogs I use to null, set each control they contain to null, remove all event handlers, but nothing changed.
Any ideas on how to solve the problem?
Steps to reproduce the bug

Create a UserControl
Load it in a ListView using an ObservableCollection
Navigate away from the page
Destroy all the contents of the ObservableCollection
See that the memory is never released

Expected behavior
Memory has to be released
Screenshots

Version Info
WinUI 3 Preview 3. Windows 10 20H2. Net 5.
NuGet package version:
[Microsoft.WinUI 3.0.0-preview3.201113.0]

Windows app type:

UWP
Win32

No
Yes

Windows 10 version
Saw the problem?

Insider Build (xxxxx)

May 2020 Update (19041)
Yes

November 2019 Update (18363)
Yes

May 2019 Update (18362)

October 2018 Update (17763)

April 2018 Update (17134)

Fall Creators Update (16299)

Creators Update (15063)

Device form factor
Saw the problem?

Desktop
Yes

Xbox

Surface Hub

IoT

@Scottj1s and @stevenbrix is this the same memory as this? https://github.com/microsoft/CsWinRT/issues/413

@Scottj1s and @stevenbrix is this the same memory as this? https://github.com/microsoft/CsWinRT/issues/413

@StephenLPeters, yeah this does look like the same C#/WinRT issue.

@StephenLPeters, yeah this does look like the same C#/WinRT issue.

Yes, this looks like a duplicate of microsoft/CsWinRT#413

Yes, this looks like a duplicate of microsoft/CsWinRT#413

Closing this, as seems to be a dupe of https://github.com/microsoft/CsWinRT/issues/413, which was fixed. If you still see this problem in the latest WindowsAppSDK release, comment here and I'll reopen the issue, or open a new issue.

I see the same problem
wappsdk 1.2.220930.4-preview2
winui 3
windows sdk build tools 10.0.25227-preview


[FRONTEND] fix same var used in single op

Fix if same var in used in single op #367

see CI error https://github.com/dmlc/nnvm/pull/383

len was added 3 month ago, revert to len(list_outputs()) or update mxnet in CI?

I see, let us make things backward compatible first. As the CI env is a bit fragile atm


View merge graph error message

click on branch

click on three dots

click on merge graph

click on pop up full screen

click X to close full screen

click X to close the versioning graph

Error message:
Not Found
We're sorry, but the server returned a 'Not Found' error
Details
{
  "headers": {
    "normalizedNames": {},
    "lazyUpdate": null
  },
  "status": 400,
  "statusText": "OK",
  "url": "https://mauro.dev.dataproducts.nhs.uk/api/tree/folders/VersionedFolder/b8b85e7a-8f33-47c1-8cf9-941575486aa9",
  "ok": false,
  "name": "HttpErrorResponse",
  "message": "Http failure response for https://mauro.dev.dataproducts.nhs.uk/api/tree/folders/VersionedFolder/b8b85e7a-8f33-47c1-8cf9-941575486aa9: 400 OK",

  "error": {
    "status": 400,
    "reason": "Bad Request",
    "errorCode": "TIS01",
    "message": "Catalogue Item retrieval for catalogue item [VersionedFolder] with no supporting service",
    "path": "/api/tree/folders/VersionedFolder/b8b85e7a-8f33-47c1-8cf9-941575486aa9",
    "version": "5.4.0-SNAPSHOT"
  }
}

NOTE: the error message only appears if the pop up full screen option is used.

System C did not build this feature of the Mauro UI, it pre-dates our start of the project and is not directly related to our project scope. Please could you change the label to "OXFORD" so it can be looked at by someone else?

22/10/24 - this may be because of the way the branches were ingested. James will have a look.
This isn't a high priority.


Bad template string refactor w/ existing template strings

Doing a Convert to template string refactors
const a = `x` + `y` + text + "z";

to
const a = `${`x` + `y` + text}z`;

which isn't invalid but would be better to convert it to
const a = `xy${text}z`;

(There is also a point to not do the refactor at all when the author meant to split the template strings, but in that case they just won't use the refactor.)

const a = `${`x` + `y` + text}z`;

I'm kind of surprised that works. I'd expect the backticks inside the ${} to need to be escaped.


belindas-closet-nextjs_6_348_adding-security-research-file

Resolves Issue #348
This PR adds a .pdf file that contains the best app security practices from A01-A05 on OWASP for Belinda's closet.
View file here:
Security-Research-1_A01-A05.pdf
Any feedback or changes is appreciated. Is the format ok? or should I use markdown instead?

You can just link to the pdf (or google doc, notion, etc) from the issue. No PR/code review is necessary for this type of documentation. @brinkbrink Do devs have access to creating notion docs or do they need some type of access from you? Is there a specific folder they can use?

They shouldn't need special permissions but if anybody has trouble, just send me a DM on Slack. A few folks have been able to do so without any permission changes. I have created a "Developer Wiki" where docs should live. I'll go over that on Thursday but I can always move the doc if people don't see the dev wiki.

You can just link to the pdf (or google doc, notion, etc) from the issue. No PR/code review is necessary for this type of documentation. @brinkbrink Do devs have access to creating notion docs or do they need some type of access from you? Is there a specific folder they can use?

@taylorpapke, I'm not sure what you mean by link to the pdf from the issue, do you mean just have a link to the pdf on the issue page #348 and approve it from there, no PR? would you guys be notified if I added the link there?

@Asfand00 Yeah, it looks like you have the pdf included in issue #348 and I personally think that should be enough to complete your user story. I think we should discuss this more on Thursday but I don't think the research documents need to be reviewed and approved. The documents will just be used to help aid implementation in the next sprint so having it documented in an issue would be good (and a centralized place like notion would be ideal).

@Asfand00 Yeah, it looks like you have the pdf included in issue #348 and I personally think that should be enough to complete your user story. I think we should discuss this more on Thursday but I don't think the research documents need to be reviewed and approved. The documents will just be used to help aid implementation in the next sprint so having it documented in an issue would be good (and a centralized place like notion would be ideal).

@taylorpapke I see so should I close this PR then? I checked on notion and it seemed my file was published on the developer wiki for people to view. I'm also doing the second half still of my user story so it's not done yet for A06-A10 issue #349

@Asfand00 Yeah, it looks like you have the pdf included in issue #348 and I personally think that should be enough to complete your user story. I think we should discuss this more on Thursday but I don't think the research documents need to be reviewed and approved. The documents will just be used to help aid implementation in the next sprint so having it documented in an issue would be good (and a centralized place like notion would be ideal).

@taylorpapke I see so should I close this PR then? I checked on notion and it seemed my file was published on the developer wiki for people to view. I'm also doing the second half still of my user story so it's not done yet for A06-A10 issue #349

@Asfand00 Yes, you can close this PR. Thanks!


Initial version

Add first version of the middleware package.

Out-of-the-box support for Express and Next.js.
Supports Node.js 12, 14, 16, and 17.
100% test coverage.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@53c598e). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             master        #1   +/-   ##
==========================================
  Coverage          ?   100.00%
==========================================
  Files             ?         3
  Lines             ?        46
  Branches          ?         5
==========================================
  Hits              ?        46
  Misses            ?         0
  Partials          ?         0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 53c598e...f092ab8. Read the comment docs.


ArrayIndexOutOfBoundsException

database: Oracle
audit4j-db version: 2.4.1
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at org.audit4j.handler.db.AuditLogDaoImpl.createTableIfNotExists(AuditLogDaoImpl.java:112)
String values[] = tableName.split("."); // tableName = "audit"
query.append("select count(*) from all_tables where table_name = upper('")
.append(values[1]).append("') and tablespace_name = upper('")
.append(values[0]).append("')");
audit4j.conf.yml
!Configuration
handlers:

!org.audit4j.handler.db.DatabaseAuditHandler
embedded: false
db_connection_type: jndi
db_jndi_datasource: myJndiDS
!org.audit4j.core.handler.ConsoleAuditHandler {}
layout: !org.audit4j.core.layout.SimpleLayout {}

The Issue seems majorly due the implementation where its expects that the table name will be prepended with the Tablespace but its not. Ideally the framewrok should pass the table name in that manner if thats what is expected. Plus i can see that the code is using ALL_TABLES which is incorrect and should use USER_TABLES. The logic to create New audit table seems faulty too.

This will be fixed in #12


Could your mod have be incompatible with FabricSeasons?

Its snowing in sparse_jungle in summer, probably it is your mod. Before i added him all was summer, now its winter :(
Im using 1.4.3 of your mod.

Report it to Fabric Seasons, I don't know anything about that mod. I doubt anything we do would cause that.

https://www.curseforge.com/minecraft/mc-mods/fabric-seasons


Late bind external data

Instead of bundling external data into the docker image, download it at model load time using b10cp. b10cp download is very fast and we avoid the decompression step that comes with bundling into docker image. We also avoid the longer build times by bundling weights into docker image.
With https://github.com/basetenlabs/flan-t5-xl-truss/tree/ext-data Downloading the weights, which total around 11 GB, takes 40 seconds compared to 2 minutes when downloading from HuggingFace directly. This is significant improvement in startup time.

One thing that's not clear to me here (maybe I missed it in the last PR),
Where do we ignore that paths in external_data when zipping up the PR? I worry that users will end up with a copy in their image anyways if they don't clean up the data_dir before deploying their model (and I know what the client doesn't check for external_data at this point)

You're right, at this time this will need to be taken care of manually. In future, we should provide a truss cli feature similar to git lfs track, where we do this automatically, i.e. upload file to provided storage, remove local file and update external_data.


Add elf_relocations table

Add the new elf_relocations table.
Not 100% sure how I want to leverage it but the markings of the table are there.
I have also included a new example directory for object creation.

@markrwilliams lets think of interesting queries.
max relocation distance?


Update 01-prerequisites.md

The gcloud compute zones list command will return following error if it's not authenticated with the Google Cloud SDK.
ERROR: (gcloud.compute.zones.list) Some requests did not succeed:
 - Failed to find project [Project ID]

Thanks for the suggestion. I've reworded this section of the guide on master.


CNF-9796: Add functions to cleanup backups and remove OADP

Some code refactors and add functions for backups and OADP operator removal which is needed during abort, rollback or finalize.

/cc @nishant-parekh @jc-rh @browsell

/lgtm

/approve

Rebased

/lgtm


Update charts/kubeslice-worker pod spec to add resources to container spec

📜 Description
Review and update charts/kubeslice-worker pod spec to add resources to container spec
👟 Reproduction steps
Install kubeslice-worker chart
👍 Expected behavior
kubeslice-worker pod with a container level security context (with min required permissions)
👎 Actual Behavior
Missing security context for the manager container - please review and add
🐚 Relevant log output
No response
Version
No response
🖥 What operating system are you seeing the problem on?
Linux
✅ Proposed Solution
No response
👀 Have you spent some time to check if this issue has been raised before?

[X] I checked and didn't find any similar issue

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hey there I came to know about this project on KCDPUNE2024, would really like to contribute to this by taking up this issue.

Sure  - please sync up with @narmidm for details.

sure @Sheikh-Abubaker. I have assigned this issue to you. Let us know @Sheikh-Abubaker, if you need any help with this issue or you can connect directly with @richiesebastian for any discussion on slack channel.

Thank you @narmidm.

Join the #kubeslice channel on the Kubernetes Slack
https://kubernetes.slack.com/archives/C03Q64HNEJF


new idea

could someone make a vhd (hyperV) with streama installed?

Why don't you use the docker image?

You can test docker image docker pull gkiko/streama

We are closing this issue as the virtualization technology you mentioned is not suitable for this kind of task


More coherent slot support

Currently:

Via Configure you can specify a token by label or serial, but you can't get the slot number where that token resides back out.
You can generate a key on the configured token, but you can't specify the new key's label or ID.
You can generate a key on a numbered slot, but you have to find the slot yourself.
You can generate a key on a specific session, but you have to open and close the session yourself.

None of this is insurmountable but it does leave a lot of work for the callers of the key generation APIs.
(The situation is better for finding keys, but see #16).
I propose the following changes:

GenerateDSAKeyPair, GenerateECDSAKeyPair, GenerateRSAKeyPair and GenerateSecretKey all get id and label arguments, consistent with the other Generate... functions. This is an incompatible change, but since the functions are almost useless I don't foresee much downstream fallout.
withSession will become WithSession, allowing applications to play nicely with the library's own session management.
WithSession will take a PIN argument rather than using the configured PIN. The present callers of withSession will use the configured PIN.
The logic in Configure to find a slot given a token label/id will be exposed as a public function.

I'm down with exposing a key id. In general I wonder whether we need more of an interface around keys if we want to set labels as they're mutable. In addition I wonder if we should have a "factory" interface around tokens for generating keys and managing said token.

It occurred to me recently that we can add a variadic parameter to capture any additional PKCS#11 attributes the caller wishes to add to the generated key. This largely removes my concerns about only offering partial support for key generation. It would be a breaking change, however, because it changes the function signature.
We've just released v0.1.0, so we can consider a few well-placed breaking changes that we introduce in v1.0.0.
I plan to work on:

Adding variadic parameters as mentioned. I'll keep the existing behaviour, wherein a nil id or label parameter generates a random value. To explicitly request an empty id or label, the caller can pass pkcs11.NewAttribute(pkcs11.CKA_LABEL, nil).
I'll address [1] above, i.e. adding the id and label attributes to the Generate... functions.

I'll then look at [2], [3] and [4].

I'm closing this. Comments on the new design are welcome at #36.


Change app name

As suggested by a user, the app, when viewed in Finder, is "Standing Desk App.app", which is redundant. We should change the name so it's not.

Due to the surprising of emails/inquiries we have received about this project over the last year or so (i.e. more than 0), changing the name seems foolish since its current moniker is why people have found it in search engines! Thus, closing.


Explaining the how is missing in this documentation

[Enter feedback here]
Many incoming links to this page is highlighted on how does competing consumer pattern works. However, the explanation is void of it rather generic in telling the why it is good. How does smart consumer work here. Who manages which consumer for which message and what if more consumers needed to be spun up, which service assesses the need? Is there a service discovery node and is there a zookeeper.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 9425297d-8ddb-c092-5c2a-79015f4e8954
Version Independent ID: 95fcd443-8a9f-0838-e73a-c6578f2d57f8
Content: Competing Consumers pattern - Azure Architecture Center
Content Source: docs/patterns/competing-consumers.yml
Service: architecture-center
Sub-service: azure-guide
GitHub Login: @martinekuan
Microsoft Alias: architectures

@khalilCodeX
Thank you for bringing this to our attention.
I've delegated this to content author @martinekuan, who will review it and offer their insightful opinions.

@martinekuan
Could you please review add comments on this, update as appropriate.

Hi @khalilCodeX
Sorry for the delay.
Competing Consumers pattern enables multiple concurrent consumers to process messages received on the same messaging channel (it is usually implemented using a queue system).
It is a generic idea, and it depends on implementation details.
Each consumer asks for a message to the queue, processes it and asks for the next message. The queue is FIFO, so when a consumer asks for a message, it is going to get the oldest at that moment. There is not a service discovery, each consumer has the connection string to the queue and starts pulling-processing data.
In this approach all the consumers are able to handle all messages.  If you like to create some consumer for same messages and some for others, you could use something like Priority Queue pattern and some technology like Service Bus Topic as messaging manager.
How many consumers. In Azure, you could manage the way to scale, for example, Event-Diven scaling Azure Functions. An interesting measure for scaling in this context is the size of the queue (exampleFor Azure Event Hubs, Azure Functions scales based on the number of unprocessed events distributed across all the partitions in the event hub).
The solution section has a good set of benefits. You could provide some suggestions if you find something interesting to add.
I will close the ticket, but feel free to open a new one if you still have concerns.
#please-close


How can I handle this in a modified model?

 This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:787: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:787: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:752: UserWarning: Using non-full backward hooks on a Module that does not return a single Tensor or a tuple of Tensors is deprecated and will be removed in future versions. This hook will be missing some of the grad_output. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using non-full backward hooks on a Module that does not return a "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:752: UserWarning: Using non-full backward hooks on a Module that does not return a single Tensor or a tuple of Tensors is deprecated and will be removed in future versions. This hook will be missing some of the grad_output. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using non-full backward hooks on a Module that does not return a "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:787: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:752: UserWarning: Using non-full backward hooks on a Module that does not return a single Tensor or a tuple of Tensors is deprecated and will be removed in future versions. This hook will be missing some of the grad_output. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using non-full backward hooks on a Module that does not return a "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:752: UserWarning: Using non-full backward hooks on a Module that does not return a single Tensor or a tuple of Tensors is deprecated and will be removed in future versions. This hook will be missing some of the grad_output. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using non-full backward hooks on a Module that does not return a "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:762: UserWarning: Using non-full backward hooks on a Module that does not take as input a single Tensor or a tuple of Tensors is deprecated and will be removed in future versions. This hook will be missing some of the grad_input. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using non-full backward hooks on a Module that does not take as input a "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:762: UserWarning: Using non-full backward hooks on a Module that does not take as input a single Tensor or a tuple of Tensors is deprecated and will be removed in future versions. This hook will be missing some of the grad_input. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using non-full backward hooks on a Module that does not take as input a "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:762: UserWarning: Using non-full backward hooks on a Module that does not take as input a single Tensor or a tuple of Tensors is deprecated and will be removed in future versions. This hook will be missing some of the grad_input. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using non-full backward hooks on a Module that does not take as input a "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:762: UserWarning: Using non-full backward hooks on a Module that does not take as input a single Tensor or a tuple of Tensors is deprecated and will be removed in future versions. This hook will be missing some of the grad_input. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using non-full backward hooks on a Module that does not take as input a "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:777: UserWarning: Using a non-full backward hook when outputs are generated by different autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_output. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using a non-full backward hook when outputs are generated by different autograd Nodes "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:777: UserWarning: Using a non-full backward hook when outputs are generated by different autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_output. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using a non-full backward hook when outputs are generated by different autograd Nodes "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:777: UserWarning: Using a non-full backward hook when outputs are generated by different autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_output. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using a non-full backward hook when outputs are generated by different autograd Nodes "
/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:777: UserWarning: Using a non-full backward hook when outputs are generated by different autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_output. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using a non-full backward hook when outputs are generated by different autograd Nodes "
2022-03-21 23:17:09 - nan_detector.py[line:89] - WARNING: NaN detected in output of encoder.layers.2.moe.moe_layer, shape: torch.Size([60, 1, 768]), forward input max: 3.67578125, input min: -7.75
Traceback (most recent call last):
  File "/workspace/OFA/trainer.py", line 871, in train_step
    grad_norm = self.clip_grad_norm(self.cfg.optimization.clip_norm)
  File "/workspace/OFA/trainer.py", line 1208, in clip_grad_norm
    return self.optimizer.clip_grad_norm(
  File "/workspace/OFA/fairseq/fairseq/optim/fp16_optimizer.py", line 200, in clip_grad_norm
    self.scaler.check_overflow(grad_norm)
  File "/workspace/OFA/fairseq/fairseq/optim/dynamic_loss_scaler.py", line 61, in check_overflow
    raise FloatingPointError(
FloatingPointError: Minimum loss scale reached (0.0001). Your loss is probably exploding. Try lowering the learning rate, using gradient clipping or increasing the batch size.

Then the training broke down. So how can I fix this problem? Hyperparameter Tuning? Or something else I need to pay attention to?
I will really really really appreciate it if you can help me!

It seems that you are working with moe and facing training instabilities. Any more information about your implementation of the moe layer and modified ddp?

Yes, I am adding an MoE layer with 4 experts ( dimension same as FFN ), using the tutel library. And I skip the pre-training stage, directly use it in the fine-tuning. At the start, I will load the checkpoint from a base model, copy the weights from FFN and then add some random noise to give weights to MoE experts ( gate with random initialization ).

You can just check my Fork to see the source code inside models/ofa/unified_transformer_layer.py

Have you tried it on a single GPU? If you run it on multiple GPUs, you should consider about your implementation of DDP (mainly in concern of all reduce) and also gradient clipping (in concern of the norm). Also, use fewer MoE layers and fewer experts (like 2) first, to maintain a relatively large batch size (as smaller batch size may cause instabilities due to the ResNet).

thx! I will have a try.

So here comes a confusing problem. I tried as you mentioned ( n_experts=2, single GPU ), but during the training, the GPU memory is keep growing until I ran out of it. Do you have any clue why this happened and how can I fix it?

Seems you got rid of training instabilities but got stuck in memory consumption. I have never met this when I was training MoE models, so I am not sure whether this is caused by MoE layers. Are you sure it is a continuous growing instead of a spike? I guess there might be some other reasons. Try with a smaller model and record the memory consumption to better examine your implementation.

Ok, thx. BTW, were you using the MoE module implemented by yourself or open-source libraries?

We did it by ourselves on some of our previous works in Tensorflow. Later we'll look into implementing MoEs for OFA and maybe tutel is a choice :)

Cool! I am doing research about moe in multimodal pre-trained models and really admire your guys' work. You can drop me an email if you have plans and interests for some collaboration and intern opportunities!