Mute bug resulting in early unmuting when 2 or more are active for a member.

Related to #98
Occasionally, moderators may apply a mute to a member who already has an existing mute. A good example is the bot's automute when spam is detected, and then a proper mute being given for a longer duration after moderator evaluation. In the current implementation, the mute that was first applied by the bot automatically already has a task in the bot to unmute the member after 10 minutes. The member will then end up with an active mute in infractions, but no mute role allowing them to talk in the server again.
In the related issue, it's recommended to give an error output saying that a mute can't be given because they already have one, but I feel requiring an unmute to an actively spamming account is potentially going to cause issues for that small period of time until the proper mute is applied. If it's an automated sending thing for example, maybe the spamming will immediately resume.
Instead, perhaps we should implement something in the backend to ensure the following:

Only a single unmute task per user is pending of which is always the longest active mute.
Don't allow weirdness such as new mutes that are meant to end earlier than existing ones. Perhaps one of the two approaches for this might suit:

Accumulate durations on top of existing active infractions. So if they have a 1 day mute but they complain via dm rudely and a mod wants to extend it by another 2 days, they could just immediately make a 2 day mute instead and it would be observed after the existing one is over.
Error output to say that it's shorter than the existing duration, including saying what the currently longest active duration is.

When a mute is due for removal, perhaps a double check of active infractions via the api just in case.

Only a single unmute task per user is pending of which is always the longest active mute.

I agree but it's somewhat tricky unless the same scheduler instance is used. Well, the database can be queried for active infractions but technically there's nothing enforcing that a scheduled infraction is also present and active in the database. It's a bit of a mess I suppose.

Don't allow weirdness such as new mutes that are meant to end earlier than existing ones.

On one hand, infractions durations can already be edited. On the other hand, it may be more convenient to have durations accumulate in the sense that if it did error out, then it'd be an extra step to run merely a different command with similar parameters. However, it would not be convenient (quite the opposite) if the actor did not wish for the times to accumulate and would have rather had their second infraction attempt invalidated. Therefore, I am leaning towards this option:

Error output to say that it's shorter than the existing duration, including saying what the currently longest active duration is.

When a mute is due for removal, perhaps a double check of active infractions via the api just in case.

Ideally, such a scenario should be made impossible in the first place (both client side and API-side constraints). Problem with API constraints for the active field is that different infractions have different behaviours. For example, the active field doesn't really make sense for notes and warnings (yet I believe such infractions are considered always active) but it makes sense to have multiple notes and warnings for a user. So this would be a bit convoluted.

Another idea is to have custom behaviour for mutes that were authored by the bot because it's easy to check who the actor of the active mute is. I'm not sure what custom behaviour would be implemented but I wanted to point out that it is a possibility.

On one hand, infractions durations can already be edited with a separate command. On the other hand, it may be more convenient to have durations accumulate in the sense that if it did error out, then it'd be an extra step to run merely a different command with similar parameters. However, it would not be convenient (quite the opposite) if the actor did not wish for the times to accumulate and would have rather had their second infraction attempt invalidated. Therefore, I am leaning towards this option:

Error output to say that it's shorter than the existing duration, including saying what the currently longest active duration is.

I agree. I'm also not always sure if we really mean to accumulate infractions when we hand out a secondary infraction. It's easier to reason about "I'm going to add a more severe infraction and it should have this duration" than "I want to add a more severe infraction that should end about here; how long should the duration be so it accumulates nicely with an existing infraction?".
I could also see the later causing issues if the actor makes a mistake, accidentally instating a much longer infraction than that they intended by accumulating to an existing one they were unaware of.
A safer option would be to err out when the new duration is shorter and to replace when the new duration is longer, although we can obviously still provide feedback in some way to the actor that this happened.

For example, the active field doesn't really make sense for notes and warnings (yet I believe such infractions are considered always active)

They currently are, but I think we've (no idea who were involved) discussed it once that it doesn't really make sense for these infractions to be active in the first place. The database also isn't consistent at this point, as I think they were migrated as "Inactive" by default (or whatever caused it, but some notes are marked as active and some as inactive.)
I'd say it makes sense to add a restriction to the back-end that only one infraction of a given type can be active for a given user at any time. To do so we'd have to change:

Restrict notes/warnings to active=False in the back-end;
Write a data migration that makes sure all the old notes/warnings are set to inactive;
Add validation to make sure only one active infraction per type per user can exist.

One downside is that this means we can't reverse the order of API/carrying out the punishment, as was suggested in the discussion surrounding the back-up moderation features for if the API goes down. That said, it wouldn't be difficult to work around that; we'll just have to handle a negative API response in a proper way and we'll need to anyway.

Another idea is to have custom behaviour for mutes that were authored by the bot because it's easy to check who the actor of the active mute is. I'm not sure what custom behaviour would be implemented but I wanted to point out that it is a possibility.

There's already another unanticipated consequence¹ of recent changes that may require this, but I don't think it's a solution to the double mute/ban infraction problem. The issue with double mutes/bans happens most frequently when a human actor escalates an earlier infraction, with the first possibly being an automated infraction.
[1] An automated infraction triggered by an antispam rule now shows up in the original Context (channel) to give context to the situation, since it often means that one of the participants in a conversation suddenly falls silent. We've recently removed the reason from the infraction confirmation message, though, meaning that part of the context is now missing.

Thanks for the well thought out response. I like your proposed API changes. I believe it'll be a lot simpler to add those restrictions in the API then have convoluted checks client-side. And, in general, not needing any logic to account for multiple active infractions (relevant for unmute/unban currently) also reduces client-side code complexity.
One concern I have about making notes and warnings inactive always is that it makes infraction queries (once that is implemented, it's already an issue) more complicated. It'd be more convenient to just list all active infractions rather than having special cases in the query for notes and warnings. I don't know how much that inconvenience would come up in practice though.

@scragly @SebastiaanZ I believe this was addressed by https://github.com/python-discord/site/pull/278, which disallows multiple active infractions. Can you confirm?

Finally, I've introduced a database-level UniqueConstraint to prevent a user from having multiple active infractions of the same type.


[bug] HTML5 document encoding differs from HTML4

Please describe the bug
The encoding of an HTML5 document differs from the encoding of an HTML4 document:
Nokogiri::HTML4::Document.parse(File.open(SHIFT_JIS_HTML)).encoding == "Shift_JIS"
Nokogiri::HTML5::Document.parse(File.open(SHIFT_JIS_HTML)).encoding == "UTF-8"

I haven't had time to dig into why this is (and whether it's intended behavior), so I'm opening this issue to look into it later. cc @stevecheckoway
Help us reproduce what you're seeing
#! /usr/bin/env ruby

$: << "lib"
require 'nokogiri'
require_relative 'test/helper'

class Test < Nokogiri::TestCase
  describe "document encoding" do
    describe "HTML4" do
      describe "given a File" do
        it "should detect shift_jis" do
          assert_equal(
            "Shift_JIS",
            Nokogiri::HTML4::Document.parse(File.open(SHIFT_JIS_HTML)).encoding,
          )
        end
      end

      describe "given a File and an encoding" do
        it "should detect shift_jis" do
          assert_equal(
            "Shift_JIS",
            Nokogiri::HTML4::Document.parse(File.open(SHIFT_JIS_HTML), nil, "Shift_JIS").encoding,
          )
        end
      end

      describe "given a String" do
        it "should detect shift_jis" do
          # fails
          assert_equal(
            "Shift_JIS",
            Nokogiri::HTML4::Document.parse(File.read(SHIFT_JIS_HTML)).encoding,
          )
        end
      end

      describe "given a String and an encoding" do
        it "should detect shift_jis" do
          assert_equal(
            "Shift_JIS",
            Nokogiri::HTML4::Document.parse(File.read(SHIFT_JIS_HTML), nil, "Shift_JIS").encoding,
          )
        end
      end
    end

    describe "HTML5" do
      describe "given a File" do
        it "should detect shift_jis" do
          # fails
          assert_equal(
            "Shift_JIS",
            Nokogiri::HTML5::Document.parse(File.open(SHIFT_JIS_HTML)).encoding,
          )
        end
      end

      describe "given a File and an encoding" do
        it "should detect shift_jis" do
          # errors
          assert_equal(
            "Shift_JIS",
            Nokogiri::HTML5::Document.parse(File.open(SHIFT_JIS_HTML), nil, "Shift_JIS").encoding,
          )
        end
      end

      describe "given a String" do
        it "should detect shift_jis" do
          # fails
          assert_equal(
            "Shift_JIS",
            Nokogiri::HTML5::Document.parse(File.read(SHIFT_JIS_HTML)).encoding,
          )
        end
      end

      describe "given a String and an encoding" do
        it "should detect shift_jis" do
          # fails
          assert_equal(
            "Shift_JIS",
            Nokogiri::HTML5::Document.parse(File.read(SHIFT_JIS_HTML), nil, "Shift_JIS").encoding,
          )
        end
      end
    end
  end
end

yields
Failure:
document encoding::HTML5::given a File#test_0001_should detect shift_jis [./html5-document-encoding.rb:52]
Minitest::Assertion: Expected: "Shift_JIS"
  Actual: "UTF-8"

Failure:
document encoding::HTML5::given a String#test_0001_should detect shift_jis [./html5-document-encoding.rb:71]
Minitest::Assertion: Expected: "Shift_JIS"
  Actual: "UTF-8"

Error:
document encoding::HTML5::given a File and an encoding#test_0001_should detect shift_jis:
TypeError: no implicit conversion of Hash into Integer
    /home/flavorjones/code/oss/nokogiri/lib/nokogiri/html5.rb:266:in `read'
    /home/flavorjones/code/oss/nokogiri/lib/nokogiri/html5.rb:266:in `read_and_encode'
    /home/flavorjones/code/oss/nokogiri/lib/nokogiri/html5/document.rb:119:in `do_parse'
    /home/flavorjones/code/oss/nokogiri/lib/nokogiri/html5/document.rb:95:in `parse'
    ./html5-document-encoding.rb:63:in `block (4 levels) in <class:Test>'

Failure:
document encoding::HTML4::given a String#test_0001_should detect shift_jis [./html5-document-encoding.rb:31]
Minitest::Assertion: Expected: "Shift_JIS"
  Actual: "UTF-8"

Failure:
document encoding::HTML5::given a String and an encoding#test_0001_should detect shift_jis [./html5-document-encoding.rb:81]
Minitest::Assertion: Expected: "Shift_JIS"
  Actual: "UTF-8"

Expected behavior
I think these should both be the same?

There are a couple of things worth digging into here:

why isn't the HTML4 EncodingReader discovering the encoding in the "given a string" case?
looks like passing a File and an encoding isn't supported by HTML5 (it raises an exception)
can we make the HTML5 encoding detection match the HTML5 behavior? I think we might be able to use the EncodingReader if it's helpful.

Since Gumbo doesn't support anything other than UTF-8, it performs the standard encoding detection pre-scan that browsers are supposed to perform to decide on the encoding and then uses that to convert to UTF-8 to pass to Gumbo.
I'm wasn't sure what the encoding property is supposed to return so I set it to the encoding of the strings in the Document itself rather than the encoding of the input document. If we want the latter, that is probably easy to change.
I'm not sure how this is supposed to interact with #to_html or other serialization methods though. Should those produce strings in the Document's encoding? I know Ruby has a concept of internal and external encoding for streams but I don't know how those should interact with this.
Basically, I don't know what the correct behavior around encodings is supposed to be but since Gumbo only supports UTF-8, I punted on it.


[Feature] Introduce TAS e2e tests with dedicated infra

What type of PR is this?
/kind feature

What this PR does / why we need it:
Add e2e test Job which will have larger structure (multiple blocks and racks) to test TAS more throughly.
Which issue(s) this PR fixes:

Relates #3480
Special notes for your reviewer:
Does this PR introduce a user-facing change?

NONE

Also please run test-tas-e2e on test-e2e for now.

So when should it become a separate target?

Also please run test-tas-e2e on test-e2e for now.

So when should it become a separate target?

After update test-infra. But I think we should merge it to main first, as other PRs will fail without the make command.

Then I should probably have the test-tas-e2e as separate target too?
The moment new test-infra target runs test-tas-e2e, we will be able to remove run-test-tas-e2e- from test-e2e.
wdyt?

/approve
Awesome, hope we can get it used more over time to test trickier scenarios. But for now this is a great starting point
cc @PBundyra


Occasional fatal error when downloading videos from Twitch

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2023.02.17 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[ ] I've read about sharing account credentials and I'm willing to share it if required

Region
No response
Provide a description that is worded well enough to be understood
I've been running into an error when downloading videos from Twitch. It doesn't happen every time, but it does occur often enough to be a nuisance.
I'm fairly certain that it only occurs if you use the "--datebefore" option and the video that you're trying to download falls outside of the designated range, but it's difficult to be absolutely certain since the error does not occur on every download.
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
[debug] Command-line config: ['-vU', '--datebefore', 'now-10days', 'https://www.twitch.tv/videos/1742266892']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8 (No ANSI), error utf-8 (No ANSI), screen utf-8 (No ANSI)
[debug] yt-dlp version 2023.02.17 [a0a7c01] (pip)
[debug] Python 3.9.2 (CPython x86_64 64bit) - Linux-5.10.0-20-amd64-x86_64-with-glibc2.31 (OpenSSL 1.1.1n  15 Mar 2022, glibc 2.31)
[debug] exe versions: ffmpeg 4.3.5-0, ffprobe 4.3.5-0
[debug] Optional libraries: Cryptodome-3.14.1, brotli-1.0.9, certifi-2020.06.20, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.2
[debug] Proxy map: {}
[debug] Loaded 1782 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2023.02.17, Current version: 2023.02.17
yt-dlp is up to date (2023.02.17)
[twitch:vod] Extracting URL: https://www.twitch.tv/videos/1742266892
[twitch:vod]<PHONE_NUMBER>: Downloading stream metadata GraphQL
ERROR: 'NoneType' object does not support item assignment
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/yt_dlp/YoutubeDL.py", line 1502, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/yt_dlp/YoutubeDL.py", line 1578, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python3.9/dist-packages/yt_dlp/extractor/common.py", line 693, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/lib/python3.9/dist-packages/yt_dlp/extractor/twitch.py", line 516, in _real_extract
    video = self._download_info(vod_id)
  File "/usr/local/lib/python3.9/dist-packages/yt_dlp/extractor/twitch.py", line 384, in _download_info
    video['moments'] = traverse_obj(data, (1, 'data', 'video', 'moments', 'edges', ..., 'node'))
TypeError: 'NoneType' object does not support item assignment

I've been having issues downloading from Twitch lately as well, using the --wait-for-video flag. This started late last week. I receive the below error before the stream even begins... sometimes it happens after a few minutes of retries, other times it can take 30min or more, but it appears to be easily reproduceable on any channel that isn't live by using the --wait-for-video flag:
[debug] Command-line config: ['--cookies-from-browser', 'firefox', '--wait-for-video', '120', 'https://www.twitch.tv/twitch', '-vU']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.02.17 [a0a7c01] (pip)
[debug] Python 3.10.6 (CPython x86_64 64bit) - Linux-5.19.0-32-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.46.0, secretstorage-3.3.1, sqlite3-2.6.0, websockets-10.3
[Cookies] Extracting cookies from firefox
[debug] Extracting cookies from: "/home/user/.mozilla/firefox/mdlwcris.default-release/cookies.sqlite"
[Cookies] Extracted 121 cookies from firefox
[debug] Proxy map: {}
[debug] Loaded 1782 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2023.02.17, Current version: 2023.02.17
yt-dlp is up to date (2023.02.17)
[twitch:stream] Extracting URL: https://www.twitch.tv/twitch
[twitch:stream] twitch: Downloading stream GraphQL
WARNING: [twitch:stream] twitch: The channel is not currently live
[wait] Waiting for 00:02:00 - Press Ctrl+C to try now
[wait] Wait period ended; Re-extracting data
<snip>
[twitch:stream] Extracting URL: https://www.twitch.tv/twitch
[twitch:stream] twitch: Downloading stream GraphQL
ERROR: twitch: An extractor error has occurred. (caused by KeyError('data')); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/home/user/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 693, in extract
    ie_result = self._real_extract(url)
  File "/home/user/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitch.py", line 974, in _real_extract
    user = gql[0]['data']['user']
KeyError: 'data'

@funky-d  Can you give some channels that are going to be live today?

@elyse0 I received the above error without the channel ever going live (or at least the channel didn't appear to go live). However, I pulled that error yesterday... I tried again after creating my post and it ran for over an hour without any error, so I'm wondering if my issue was an issue with Twitch over the weekend? Looking at my history, I first ran into the issue Friday night.
As for some channels that may be going live a few hours: willneff, fanfan, jakenbakelive or waterlynn
If you need any other info just let me know, and I'll continue to test here and will let you know if I experience my issue again.

https://www.twitch.tv/videos/1745224952
https://www.twitch.tv/videos/1740427160
https://www.twitch.tv/videos/1739499778
With all 3 I'm getting "ERROR: 'NoneType' object does not support item assignment" with Windows 2023.02.17 exe however the download starts.
Could be related, original youtube-dl 2021 exe returns the error;
"ERROR: An extractor error has occurred. (caused by KeyError('data',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output."
Verbose does not include extra information.
Error is not always thrown on original youtube-dl but if you try all 3 you eventually get it.

https://www.twitch.tv/videos/1745224952 https://www.twitch.tv/videos/1740427160 https://www.twitch.tv/videos/1739499778
With all 3 I'm getting "ERROR: 'NoneType' object does not support item assignment" with Windows 2023.02.17 exe however the download starts.
Could be related, original youtube-dl 2021 exe returns the error; "ERROR: An extractor error has occurred. (caused by KeyError('data',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output."
Verbose does not include extra information. Error is not always thrown but if you try all 3 you eventually get it.

I think download starting might be a bit misleading because i query video information first and the run the command again to download. So it probably throws the error on the first command and then doesn't when downloading. Just a coincidence that happened 3 times in a row because after i tried a couple more times it did not download a couple more times, it downloads again.

https://www.twitch.tv/videos/1745224952 https://www.twitch.tv/videos/1740427160 https://www.twitch.tv/videos/1739499778
With all 3 I'm getting "ERROR: 'NoneType' object does not support item assignment" with Windows 2023.02.17 exe however the download starts.
Could be related, original youtube-dl 2021 exe returns the error; "ERROR: An extractor error has occurred. (caused by KeyError('data',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output."
Verbose does not include extra information. Error is not always thrown but if you try all 3 you eventually get it.

I think download starting might be a bit misleading because i query video information first and the run the command again to download. So it probably throws the error on the first command and then doesn't when downloading. Just a coincidence that happened 3 times in a row because after i tried a couple more times it did not download a couple more times, it downloads again.

Yep, I could reproduce it and the linked PR will fix it

My issue has not reappeared, so it was likely an issue with Twitch last weekend and not related to OP's issue.


Don't call it ssh.sh

the script bin/ssh.sh doesn't open an ssh session, therefor calling it ssh.sh is is misnomer.  Why not avoid confusing people and call it shell.sh or something to that degree?

Agreed. @cmmarslender ?


Remove primary_is_running

This was a half-finished mechanism to allow a replica to enter hot standby mode sooner, without waiting for a running-xacts record. It had issues, and we are working on a better mechanism to replace it.
The control plane might still set the flag in the spec file, but compute_ctl will simply ignore it.

The better mechanism is being worked on at PR #7288


Pre-release docs

Hi There,
I'm working with a piece of legacy software which is using version 0.12.0 of this package.
Is there anywhere I can find the documentation for this version?
On https://laravel-json-api.readthedocs.io/en/latest it seems I am not looking at examples that relate to this much older version that I am using.
Thanks

You can view the docs folder on the 0.12.0 tag here:
https://github.com/cloudcreativity/laravel-json-api/tree/v0.12.0/docs


Add accepts-attribute to image upload.

This will give compatible browsers a hint to only allow images.

Thanks :)


Add more filter commands

This PR adds the following functionalities:

Find and filter issue list by project id
Find and filter project list by client id
Find and filter project list by client name

This PR also removes the constraint on Name objects being 1 to 3 words long with 10 characters in each to prevent overzealous input validation.

Codecov Report

Merging #138 (5846f47) into master (ed23014) will decrease coverage by 0.57%.
The diff coverage is 4.00%.

@@             Coverage Diff              @@
##             master     #138      +/-   ##
============================================
- Coverage     39.65%   39.08%   -0.58%
+ Complexity      511      505       -6
============================================
  Files           125      125
  Lines          2983     3014      +31
  Branches        371      375       +4
============================================
- Hits           1183     1178       -5
- Misses         1632     1669      +37
+ Partials        168      167       -1

Impacted Files
Coverage Δ

...address/logic/commands/issue/FindIssueCommand.java
0.00% <ø> (ø)

...ess/logic/commands/project/FindProjectCommand.java
0.00% <ø> (ø)

...seedu/address/logic/parser/IssueCommandParser.java
0.00% <0.00%> (ø)

...in/java/seedu/address/logic/parser/ParserUtil.java
27.95% <0.00%> (-0.72%)
:arrow_down:

...edu/address/logic/parser/ProjectCommandParser.java
22.85% <0.00%> (-1.15%)
:arrow_down:

...ser/predicates/IssueContainsKeywordsPredicate.java
0.00% <0.00%> (ø)

...r/predicates/ProjectContainsKeywordsPredicate.java
0.00% <0.00%> (ø)

...a/seedu/address/logic/parser/ProjectCliSyntax.java
87.50% <100.00%> (+1.78%)
:arrow_up:

src/main/java/seedu/address/model/Name.java
65.00% <100.00%> (-5.38%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Create MechJeb2-2.5.1.ckan

Added MechJeb 2.5.1 for KSP 1.0.2

Seems there's no .zip at the download location, getting 404s on my end and the buildbot have the same issue.

Link should be http://addons-origin.cursecdn.com/files/2238/724/MechJeb2-<IP_ADDRESS>.zip


incorrect syntax in docs for "Post data fuzzing"

The ampersand should not be escaped:
ffuf -w /path/to/postdata.txt -X POST -d "username=admin\&password=FUZZ" -u https://target/login.php -fc 401

does it also need -H "Content-Type: application/x-www-form-urlencoded"?

You are right @zesty,
Nice catch!.


[FLINK-35175][filesystem] Avoid reading with ByteBuffer on  hadoop version below 3.3.0

What is the purpose of the change
Hadoop whose version below 3.3.0 doesn't support reading and positionable reading  with ByteBuffer, which may cause compile error.
See https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59012&view=logs&j=87489130-75dc-54e4-1f45-80c30aa367a3&t=73da6d75-f30d-5d5a-acbe-487a9dcff678&l=3630
Brief change log

Remove support about reading and positionable reading  with ByteBuffer, and add TODO.

Verifying this change
This change could be verified by current CI.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@Zakelly @RyanSkraba Could you help to take a look ?


Tweaks to PostgreSQL docker-compose service

This PR includes two small tweaks:

remove the custom postgres user, DMRUNNER_USER
update the postgres image

On their own these changes aren't very significant, but they lay the ground work needed to fix #36, and also are needed for work to make it possible to use dmrunner on Linux.

P.S. I'm also super against commits that are just 'run codestyle', because they break history up a bit. If you're happy going back and running the formatter against each individual commit, so that they bring in the changes in the correct format, that would be fab.
I should figure out a way to enforce this.

P.S. I'm also super against commits that are just 'run codestyle', because they break history up a bit. If you're happy going back and running the formatter against each individual commit, so that they bring in the changes in the correct format, that would be fab.

That's fair, TBH I pushed this PR out in a rush because it was past my bed time...


[5.9] Split FunctionParameterSyntax and related types

Cherry-pick of https://github.com/apple/swift-syntax/pull/1455 to release/5.9

FunctionParameterSyntax was used in a number of cases (function parameter, closure parameter and enum case parameter) and because it needs to satisfy all of them all its parameters are optional. Split it into three different types that have non-optional children.
rdar://106874808

https://github.com/apple/swift/pull/64959
https://github.com/apple/swift-format/pull/505
https://github.com/apple/swift-stress-tester/pull/231
@swift-ci Please test


Added keepWithinParent to Draggable.

OK, my IDE might have gone wild with the formatting on TabbedPane (hence the huge deletions and additions count), but I'm assuming you'd have to format my changes at some point anyway, so - hopefully - no harm done. (God, I need to start using Idea.)
TabbedPane is no longer rebuilt on every tab change, so it should be significantly faster to modify its content when it already contains multiple tabs. Everything seems fine (I checked out the new TabbedPane in gdx-vis-lml-tests project), but you might want to test this before merging.

Thanks, sorry but I will have to ask you to reformat TabbedPane.  You can get IDEA (would recommend :D) or use libgdx formatter for Eclipse. Then squash that new commit with "Refactored TabbedPane". I also noticed you added final everywhere in TabbedPane methods signatures. I would prefer if you remove them.

OK, I remove the TabbedPane commit and formatted my classes with LibGDX formatter. I think you can merge this now, and I'll just update the TabbedPane in a separate PR.

Mouse wheel click on tab should close it, now it crashes with NPE:
java.lang.NullPointerException
	at com.kotcrab.vis.ui.widget.Draggable.attachMimic(Draggable.java:192)
	at com.kotcrab.vis.ui.widget.Draggable.touchDown(Draggable.java:176)
	at com.badlogic.gdx.scenes.scene2d.InputListener.handle(InputListener.java:56)
	at com.badlogic.gdx.scenes.scene2d.Actor.notify(Actor.java:181)
	at com.badlogic.gdx.scenes.scene2d.Actor.fire(Actor.java:152)
	at com.badlogic.gdx.scenes.scene2d.Stage.touchDown(Stage.java:277)
	at com.badlogic.gdx.backends.lwjgl.LwjglInput.processEvents(LwjglInput.java:303)
	at com.badlogic.gdx.backends.lwjgl.LwjglCanvas$3.run(LwjglCanvas.java:234)

It also broke close button, when you click and hold on tab (not the close button but on it's name) it should change close button color to blue. Looks like for some reason exit in TabButtonTable is called immediately after touch down.

NPE was thrown because wheel click removed the actor and it had no stage. Nothing a simple null check cannot fix.
Not quite sure I follow what's going on with the close button though.

OK, I've added input blocker to Draggable, which removes most of the style-related problems from TabbedPane. Ran into a weird SnapshotArray error along the way, but it seems to be working fine (for now).

Nice, works great now.
Is there some simple case to reproduce this issue?

gdx-lml-vis-tests, first tab closed with close button should fail to properly remove the tab. Not sure if that's some issue with my JRE/damaged libs, but I had to add the purge method to make it work properly. If you cannot reproduce this error, feel free to delete the workaround.


[1.12] Bump Mesos to nightly 1.7.x 8e8c6c0

High-level description
This is a routine bump to the latest Mesos and mesos-modules.
Related JIRA Issues
Checklist for all PRs

[ ] Included a test which will fail if code is reverted but test is not. If there is no test please explain here:
[x] Read the DC/OS contributing guidelines
[x] Followed relevant code rules Rules for Packages and Systemd

Checklist for component/package updates:
If you are changing components or packages in DC/OS (e.g. you are bumping the sha or ref of anything underneath packages), then in addition to the above please also include:

[x] Changelog:
https://github.com/apache/mesos/compare/33a1ba97041f178f8be53cdeb7cbeb7c78b89798...8e8c6c0820e1edd954ce07aa530aa64ad323f3d6

[ ] Test Results: [link to CI job test results for component]

[ ] Code Coverage (if available): [link to code coverage report]

@mesosphere-mergebot bump-ee

@mesosphere-mergebot label Ready For Review

@mesosphere-mergebot label Work In Progress

@orsenthil could we close this PR?


lacking code modularity

This codebase should have classes and inheritance in order to have "eat your own dog food"

Solved by #7


Document PARLER_LANGUAGES for parler 1.6

Fix #309
Fix #310
Addresses https://github.com/nephila/djangocms-blog/issues/307#issuecomment-235926692

Current coverage is 92.44% (diff: 100%)

Merging #313 into develop will decrease coverage by 0.07%

@@            develop       #313   diff @@
==========================================
  Files            25         25
  Lines          1363       1363
  Methods           0          0
  Messages          0          0
  Branches        166        166
==========================================
- Hits           1261       1260     -1
- Misses           54         55     +1
  Partials         48         48

Powered by Codecov. Last update ece8ee6...abb92c0

Coverage decreased (-7.3%) to 88.701% when pulling abb92c01ed3ec4fa7f6c3081bd0170c5e3f3b2fe on feature/docs_parler_version into ece8ee663fede6af85708ec918ac9d6094fbc6d3 on develop.


Instance org gets wrong after session timeout

Description of the bug
Wrong organization used for instance after session timeout
Steps To Reproduce

Log on as a user with access to multiple organizations
Select one of the organizations to represent
Start filling out an instance
Leave the browser session open and let it time out
Re-logon and select a different organization
Observe that the instance is now connected to the last organization and not the organization used when starting

Additional Information
No response

Attempting a summary of the findings, as I understand them:
Instances or data connected to the instance are not transferred to other orgs so there is no data leaked to other orgs.
This is mainly a UX bug as the frontend displays the information present in the cookie and not the instance data.

Action
Reportee in cookie
Reportee in instance data/metdata
Reportee shown in app frontend

Login and choose to represent OrgA
OrgA
N/A
N/A

Instantiate a new instance
OrgA
OrgA
OrgA

Session timeout, re-login represent OrgB
OrgB
OrgA
OrgB

Open new tab open inbox and choose to represent OrgC
OrgC
OrgA
OrgC

Frontend should display the reportee set in instance metadata if it differs from what is set in the cookie.
We need to decide if a user choose to represent OrgB and opens an app for OrgA, should we change and update the values in the users cookie or only change the value presented to the user while he/she is in the context of that instance?

Another error scenario I didn't catch in my previous comment.
If the user follows a direct link to the instance when not logged in he will be redirected to id-porten for authentication with a redirect url back to the app, but apps uses hashroutes and the hash part of the route is not part of the redirect route. Therefor a new instance is created for the organisation/person the user chooses as reportee, even if it's the same as the one he previously had started an instance for.

It's like @tjololo sums up. Although not very user friendly, it's by-design and not a security issue. But we should fix sooner rather than later. I've grouped this, and related issues under https://github.com/orgs/Altinn/projects/39/views/13.

We are concerned that this might effect the PBU project at Udir as well. It would be good to have this fixed before April.

This bug can affect the PBU project at Udir since there are schools with shared computers which means that multiple users will login on the same computer. We aren't guaranteed that they remember to actively log out after filling out the survey (or if a fire alarm were to ring and they switch computers on resuming the survey), meaning that the AltinnPartyId-cookie with the previous userid remains. When a new user logs in, some of the calls to the app fail and they are presented with a blank page. For them to get around the problem they would need to clear the cookie for AltinnPartyId, which we can't really expect of them to know.
I've added some screenshots reproducing the bug, note that the page on the left hand side is blank for the second login.

Isn't this really an issue in app-frontend-react that reports the results from the /user endpoint in the top right corner (which depends on the cookie), instead of showing the instance owner of the instance that is currently active?
As I understand the AltinnPartyId cookie, it is just a "suggestion" for what party to show the inbox for and to create new instances owned by. In all other cases you have instance.Owner.PartyId to specify the current party.

The problem for PBU at Udir is not that the wrong org is selected. The problem is that it seems to crash and give the user a white page of death.

@ivarne  I believe this should be fixed with FE updates. FE should present the instance partyID, and we should consider to update party cookie if mismatch (like in Altinn 2)

Prioritert med Udir. Foreslår et møte for å få oversikt.

I think @ivarne summary is correct.
Bug 1: The endpoint/cookie should not be used to display who is the instanceOwner in the upper right corner. There are two scenarios where I think the endpoint/cookie should be used: When instantiation a new instance, and opening a stateless application. In all other cases the instanceOwner should be fetched from the instance information. BE cannot change the cookie as the user might have instances from different parties open in multiple tabs.
Bug 2: 403 seems like the correct responsecode. User is forbidden to open another users instanse. We need to investigate the 500 response from /api/authorization/parties/current in UDIRs responses
Bug 2: I think updating/changing cookies in apps is a anti-pattern. This would lead to an unknown number of altinn 3 apps with different code and versions of altinn app nugets updating the same cookies. I suspect this will result in a debugging nightmare down the road.
I think the correct place to update the cookie is in the authentication process. Once a user logs in the value in the cookie should be checked and removed if the newly logged in user cannot represent the partyId in the cookie.

This will be resolved when

https://github.com/Altinn/altinn-authorization/issues/499
https://github.com/Altinn/app-lib-dotnet/issues/327
https://github.com/Altinn/app-frontend-react/issues/1533
are closed


Getting metadata from plugin failed with error: invalid_grant: Robot is disabled

I'm getting below error when following steps from here
https://cloud.google.com/tasks/docs/quickstart-appengine#before-you-begin
Sending task:
{ appEngineHttpRequest:
   { httpMethod: 'POST',
     relativeUri: '/log_payload',
     body: 'aGVsbG8=' } }
{ Error: Getting metadata from plugin failed with error: invalid_grant: Robot is disabled.
    at Http2CallStream.call.on (/Users/joselitonarte/Documents/Fun/web/nodejs-tasks/samples/node_modules/@grpc/grpc-js/build/src/client.js:101:45)
    at Http2CallStream.emit (events.js:203:15)
    at process.nextTick (/Users/joselitonarte/Documents/Fun/web/nodejs-tasks/samples/node_modules/@grpc/grpc-js/build/src/call-stream.js:71:22)
    at process._tickCallback (internal/process/next_tick.js:61:11)
  code: '400',
  details:
   'Getting metadata from plugin failed with error: invalid_grant: Robot is disabled.',
  metadata: Metadata { options: undefined, internalRepr: Map {} } }

@jojonarte could you double check the service account credentials that you've generated, and make sure the credentials file is pointed to using the environment variable GOOGLE_APPLICATION_CREDENTIALS (see the documentation here).
This error seems to crop up if someone is using credentials that point to a deleted account.

I got it to work now thanks


View.global is not bound to current request

Hi,
I was load testing my site with ab tool (https://httpd.apache.org/docs/2.4/programs/ab.html). I visited the site while testing and found that sometimes I seem to be logged in, sometimes not.
I started searching for the reason, and found the following:

Create a middleware
Add View.global('input', request.input('input'))
Create a view that prints it: {{ input }}
Visit your site with parameter ?input=right
Start ab -n 1000 -c 50 http://<IP_ADDRESS>:3333/?input=wrong
Hit reload sometimes and you will see it prints wrong instead of right

This way I cannot put session variables in View.global.
Also, csrf tokens will be invalid, since they do not match in the View.
It also seems like authProvider uses this for user handling.
For now I overwrote my sendView method and added these variables for myself from the request object.
Not sure if nunjucks does this right, but one possible solution would be to use a method like View.local instead of View.global that would store data bound to the Request object and would be extended by options passed to the views.
I'm in the finish line of my project... do you have any instant suggestions for me? :)

Yup that is true. A global is a global and whenever you set a new value to it the old one overrides. I know people started using globals for request specific information, which is never a reliable way since 2 requests can come at the same time.
What's the work around?
The best way is to pass the data you need to the view via sendView method, which makes sure that locals are isolated.
I cannot add any out of the box solution for it right now, since I am planning some good amount of changes for the views.

I see. May I ask some?
I always need my session data for authentication / display stuff, so I created a middleware that yields and attaches session.all() to my request, thus I don't have to yield it all the time. And now I created a middleware that overrides response.sendView method, here I can add my session to view locals since I access my request.session from my controller scope. I can also override my csrfToken and csrfField values to the correct ones. Am I right?
My other two questions are:

I guess situation is the same with flash messages but I feel a little insecure here. If I pass a local function called old to my views that returns request.old(key, defaultValue) - will it be in the right scope?
Does a global middleware always precede those defined in routes.js? (Route.group(...).middleware())

Thanks in advance.

Answers

I believe old global in the view will not point to the right request, so it is a bug.
All global middleware are executed before any middleware defined on the route.

I believe I have to jump in and fix the way you pass things onto views. There needs to be a way to have unique instance of views for each request.

Thanks!


Sticky scroll: support relative line number

Currently, the sticky scroll only supports absolute line number. Would be nice to support relative number as well.
(Probably this setting should be in respect to editor.lineNumbers setting)

Fixed via https://github.com/microsoft/vscode/pull/156992


3D axis equal

x y z axes in different scales
need to adjust

Do you mean this?
ax.set_xlim3d(0, 1000)
ax.set_ylim3d(0, 1000)
ax.set_zlim3d(0, 1000)

This scales the x, y, z axes to [0,1000).


Add support for django 3.1, remove python2 deprecations, add badges

Adds support for Django 3.0, 3.1 by removing Python 2 deprecations.

Good bye Python 2! You will be missed 😢


The built-in library 'dart:html' is not available on the stand-alone VM.

I recently migrated from Polymer 0.16+3 to 1.0.0-rc.2 in my Dart project.
I am using Dart 1.13.0-dev.5.0.
When I try to build the project I get these errors  in various polymer, polymer_interop and webcomponent files.
"The built-in library 'dart:html' is not available on the stand-alone VM.'
"The built-in library 'dart:js' is not available on the stand-alone VM."
It seems to compile when I remove these references.
Here is my pubspec.yaml file :
name: CVWebkit
description: An online jobboard for employers and contestants
dependencies:
dart_config: any
browser_detect: any
sass : any
polymer:
git:
url: https://github.com/dart-lang/polymer-dart.git
ref: 1.0.0-rc.2
reflectable: 0.2.1
web_components: ^0.12.0
dependency_overrides:
polymer_interop:
git:
ref: 0.2.0-rc1
url: https://github.com/dart-lang/polymer_interop.git
reflectable: 0.2.1
transformers:

polymer:
entry_points: web/cvwebkit.html

reflectable:
entry_points: web/cvwebkit.html

$dart2js:
minify: true

Your transformer config is invalid. Here is a working example http://stackoverflow.com/questions/32242736/import-and-make-use-of-polymer-dart-0-17-dev-preview
You need to configure the entry_point once for web_components and once for the reflectable transformer while the former expects the path to the html file and the later to the dart file.

Thank you !
It works now ! Just a question : is it because you use "reflectable" that "@observable" is regarded as an error ?
name: CVWebkit
description: An online jobboard for employers and contestants
dependencies:
dart_config: any
browser_detect: any
sass : any
polymer:
git:
url: https://github.com/dart-lang/polymer-dart.git
ref: 1.0.0-rc.2
web_components: ^0.12.0
polymer_elements:
git: https://github.com/dart-lang/polymer_elements.git
dependency_overrides:
polymer_interop:
git:
ref: 0.2.0-rc1
url: https://github.com/dart-lang/polymer_interop.git
reflectable: 0.3.0
transformers:

web_components:
entry_points: web/cvwebkit.html

reflectable:
entry_points: lib/cvwebkit.dart

$dart2js:
minify: true

observable isn't necessary or supported anymore. In Polymer 1.0 you have to explicitely notify Polymer about about changes with methods your element inherits from the PolymerElement class like set(), add(), remove(), and notifyPath() ...
You can find more details about how to work with Polymer 1.0 at https://github.com/dart-lang/polymer-dart/wiki#10-developer-guide.
There is some work in progress to allow to work with observable like in the previous Polymer version, but I don't know about the status. See #570

Thank you !
Attributes are not supported.
Here is a part of the new way : https://www.polymer-project.org/1.0/docs/devguide/properties.html

Glad to hear you got it working. Polymer 1.0 is exciting and working quite well already also the Polymer elements.
At https://github.com/bwu-dart-contributing/polymer_elements/tree/demo2/demo you can find examples about how to use the elements in Dart Polymer 1.0
I added notes in the pubspec.yaml where I run into problems.
Not sure what you mean by

Attributes are not supported.

If your issue is resolved, please close this issue. Thanks :)

If you want support for osservable in polymer 1.0 you can try
https://github.com/dam0vm3nt/polymer_autonotify
Il giorno dom 4 ott 2015 17:25 LSMetag<EMAIL_ADDRESS>ha
scritto:

Closed #613 https://github.com/dart-lang/polymer-dart/issues/613.
—
Reply to this email directly or view it on GitHub
https://github.com/dart-lang/polymer-dart/issues/613#event-426227553.


[FlatButton] Fix alignment between text and icons

Align text inside the button and also with the icon, if any. For better explanation see #3366

Alright, that sounds like a good idea https://github.com/google/material-design-lite/blob/47c215b55f2987b54c9909dae6ed31f20ee5fbb8/src/button/_button.scss#L43.
However, this has some global side effect.
For instance here https://github.com/callemall/material-ui/blob/master/src/app-bar.jsx#L58, can we use  + 1 intead of + 2 to fix this:

I will do some tests because the button isn't already aligned with the others elements like the bar tittle and with this fix get worst but the sibling element with align-self: center; and removing all that margins-top get the button perfect centered.

I will do some tests because the button isn't already aligned with the others elements like the bar tittle

I don't think that they should be centered. I would say that it's better to have the bottom of the text at the same level.

Well I spend some time changing from 2 to 1 but in my local version and nothing changes.
But I think if is already using flexbox model we should take that vantage to make that elements more flexible posible than using pixels. Imagine someone how need that button bigger or something like that or a selectfield inside the appbar (I had override the styles to be centered because was ate the top) With flexbox no mathers the size always will be centered with the other content.
I don't know if my version is different or why when I change to + 1 nothing happens (still margin-top: 8px) So feel free to close this PR and fix the alignment in a better way, also I had tested with the bottom left app bar and the button was to up, so maybe for inside the appbar this change is not te best.

But I think if is already using flexbox model we should take that vantage to make that elements more flexible possible than using pixels

That's a good point. This component needs a refactorization.

@octavioamu i'm trying to center an icon within flatbutton using flexbox but have been unsuccessful. can you give me a sample code to do this? my attempted code is here. thanks.


adding bower config

Adding a bower configuration so we can submit this to bower

Should be registered now, can you confirm?
I'm not a bower user, so I may have missed something ;)
Anyway, thanks for the PR! :)


Add new fields to Employee Class

Add Payroll, LeaveCounter, Date of Birth, and Date of Joining to Employee Class
Edit Add and Edit Command implementation for new fields
Edit JSON classes for new fields
Closes #98
Fixes #144 by using / to split tags

LGTM


Pass geoJson object instead of url in overlays

It's possible to pass a geoJson object directly to an overlay definition? Example, make this:
               overlays: {
                    traffic: {
                        name: "Traffic Jams",
                        type: "geoJson",
                        geoJson: {
                                 // geojson definition
                         }
                    }
                }

Instead of this:
               overlays: {
                    traffic: {
                        name: "Traffic Jams",
                        type: "xyz",
                        url: "http://map.be-mobile.be/customer/mobileninja/nl/los/{z}/{x}/{y}.png"
                    }
                }

It doesnt look like it. If you want it, please add it. It would go in this file.
https://github.com/tombatossals/angular-leaflet-directive/blob/master/src/services/leafletLayerHelpers.js#L45-L53

This issue was moved to angular-ui/ui-leaflet#164

I'm going to rework&redesign angular-leaflet-directive to be compatible with Leaflet v1.0. It will mantain almost all its functionality, and will be compatible with the current features of the directive, but I must start from a fresh point, so I'm going to close this issue. If you think it must be worked with the new version, please reopen it.


TravisCI Conditional Deployment

For example: when v0.14 branch had been updated, TravisCI shows an error message:

Skipping deployment with the npm provider because this branch is not permitted to deploy

However, we had already put the on.all_branches: true to the deploy section.
Need to be fixed.
See Also

Specify branches to build
Examples of Conditional Deployment
deploy only on branches matching a regex
How to start Travis CI deploy only when tag name matches a regex

After did many hard testing works,  this version of the .travis.yml work as expected:
https://github.com/Chatie/wechaty/blob/affe60e1751fd903d9ff70a835b8e9da3ed90659/.travis.yml


Health-check fails and blocks the update of an installed deviceupdate-agent package on a IoT Edge Device

On a IoT Edge device (iotedge 1.4.9) with DPS provisioning with TPM the command "sudo /usr/bin/AducIotAgent --enable-iothub-tracing --health-check --log-level 0" fails with the error message "Invalid connection info" and blocks the update deviceupdate-agent package, see https://github.com/Azure/iot-hub-device-update/blob/e6c848a911643008bc51c588cacf0507be5d80b2/packages/debian/postrm#L95
Expected Behavior
The health-check should work
Current Behavior
The health-check fails and blocks the deviceupdate-agent package update.
Steps to Reproduce

Install iotedge 1.4.9 on a PC with TPM 2.0 and configure DPS provisioning with TPM
Run "sudo /usr/bin/AducIotAgent --enable-iothub-tracing --health-check --log-level 0" on the IoT Edge device

Device Information

Host OS: Ubuntu 18.04
Architecture amd64
Provisioning Method: DPS provisioning with TPM
DU Agent Version: 1.0.0

Logs
2023-03-02T13:55:56.5203Z 5887[5887] [D] Log file created: /var/log/adu/du-agent.20230302-135556.log [zlog_init]
2023-03-02T13:55:56.5209Z 5887[5887] [I] Failed to provision a connection string from eis, Failed with error EISErr_InvalidJsonRespErr on service EISService_KeyService [GetConnectionInfoFromIdentityService]
2023-03-02T13:55:56.5209Z 5887[5887] [E] Invalid connection info. [HealthCheck]
2023-03-02T13:55:56.5209Z 5887[5887] [I] Health check failed. [HealthCheck]
2023-03-02T13:55:56.5209Z 5887[5887] [I] Agent exited with code 1 [main]
2023-03-02T13:55:56.5209Z 5887[5887] [I] Agent is shutting down with signal 0. [ShutdownAgent]
2023-03-02T13:55:56.5209Z 5887[5887] [I] De-initializing command listener thread [UninitializeCommandListenerThread]
2023-03-02T13:55:56.5210Z 5887[5887] [I] ADUC agent stopping [AzureDeviceUpdateCoreInterface_Destroy]

@shaeussler Only running as adu user is supported. Is there a reason you need to run it as root uid 0?  can also use sudo -u adu ...

@jw-msft Thanks for the response. I did not configure the edge identity service.
I installed the IoT Edge using the following instructions: https://learn.microsoft.com/en-us/azure/iot-edge/how-to-provision-devices-at-scale-linux-tpm?view=iotedge-1.4&tabs=physical-device%2Cubuntu
When I now try to update deviceupdate-agent with the command "sudo apt-get install deviceupdate-agent" I get the error message
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  deviceupdate-agent
1 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Need to get 1915 kB of archives.
After this operation, 87.0 kB of additional disk space will be used.
(Reading database ... 59359 files and directories currently installed.)
Preparing to unpack .../deviceupdate-agent_1.0.2_amd64.deb ...
******************** Running /var/lib/dpkg/tmp.ci/preinst upgrade 1.0.0 ***********************
Existing DU Agent version: 1.0.0
Unpacking deviceupdate-agent (1.0.2) over (1.0.0) ...
******************** Running /var/lib/dpkg/info/deviceupdate-agent.postrm upgrade 1.0.2 ***********************
[postrm upgrade 1.0.2] Perform new ADU agent health validation.
AducIotAgent version:
1.0.2
2023-03-13T06:47:47.3901Z 8058[8058] [D] Log file created: /var/log/adu/du-agent.20230313-064747.log [zlog_init]
Error: Time:Mon Mar 13 06:47:47 2023 File:/tmp/azure-iot-sdk-c/c-utility/adapters/socketio_berkeley.c Func:initiate_socket_connection Line:353 Failure: connect failure 111.
Error: Time:Mon Mar 13 06:47:47 2023 File:/tmp/azure-iot-sdk-c/c-utility/adapters/socketio_berkeley.c Func:socketio_open Line:829 lookup_address_and_connect_socket failed
Error: Time:Mon Mar 13 06:47:47 2023 File:/tmp/azure-iot-sdk-c/deps/uhttp/src/uhttp.c Func:uhttp_client_open Line:1134 opening xio failed
2023-03-13T06:47:47.3903Z 8058[8058] [I] Failed to provision a connection string from eis, Failed with error EISErr_ConnErr on service EISService_IdentityService [GetConnectionInfoFromIdentityService]
2023-03-13T06:47:47.3904Z 8058[8058] [E] Invalid connection info. [HealthCheck]
2023-03-13T06:47:47.3904Z 8058[8058] [I] Health check failed. [HealthCheck]
2023-03-13T06:47:47.3904Z 8058[8058] [I] Agent exited with code 1 [main]
2023-03-13T06:47:47.3904Z 8058[8058] [I] Agent is shutting down with signal 0. [ShutdownAgent]
2023-03-13T06:47:47.3904Z 8058[8058] [I] De-initializing command listener thread [UninitializeCommandListenerThread]
2023-03-13T06:47:47.3905Z 8058[8058] [I] ADUC agent stopping [AzureDeviceUpdateCoreInterface_Destroy]
[postrm upgrade 1.0.2] ERROR: ADU Agent (/usr/bin/AducIotAgent) validation failed. Aborting the upgrade.
dpkg: warning: old deviceupdate-agent package post-removal script subprocess returned error exit status 1
dpkg: trying script from the new package instead ...
******************** Running /var/lib/dpkg/tmp.ci/postrm failed-upgrade 1.0.0 ***********************
[postrm failed-upgrade 1.0.0] Perform new ADU Agent health validation.
Attempting to fix-up /usr/lib/adu/adu-shell and /etc/adu after postrm upgrade failure from 1.0.0 ...
AducIotAgent version:
1.0.2
2023-03-13T06:47:47.4122Z 8064[8064] [D] Log file created: /var/log/adu/du-agent.20230313-064747.log [zlog_init]
Error: Time:Mon Mar 13 06:47:47 2023 File:/tmp/azure-iot-sdk-c/c-utility/adapters/socketio_berkeley.c Func:initiate_socket_connection Line:353 Failure: connect failure 111.
Error: Time:Mon Mar 13 06:47:47 2023 File:/tmp/azure-iot-sdk-c/c-utility/adapters/socketio_berkeley.c Func:socketio_open Line:829 lookup_address_and_connect_socket failed
Error: Time:Mon Mar 13 06:47:47 2023 File:/tmp/azure-iot-sdk-c/deps/uhttp/src/uhttp.c Func:uhttp_client_open Line:1134 opening xio failed
2023-03-13T06:47:47.4123Z 8064[8064] [I] Failed to provision a connection string from eis, Failed with error EISErr_ConnErr on service EISService_IdentityService [GetConnectionInfoFromIdentityService]
2023-03-13T06:47:47.4124Z 8064[8064] [E] Invalid connection info. [HealthCheck]
2023-03-13T06:47:47.4124Z 8064[8064] [I] Health check failed. [HealthCheck]
2023-03-13T06:47:47.4124Z 8064[8064] [I] Agent exited with code 1 [main]
2023-03-13T06:47:47.4124Z 8064[8064] [I] Agent is shutting down with signal 0. [ShutdownAgent]
2023-03-13T06:47:47.4124Z 8064[8064] [I] De-initializing command listener thread [UninitializeCommandListenerThread]
2023-03-13T06:47:47.4124Z 8064[8064] [I] ADUC agent stopping [AzureDeviceUpdateCoreInterface_Destroy]
[postrm failed-upgrade 1.0.0] ERROR: ADU Agent (/usr/bin/AducIotAgent) validation failed. Aborting the upgrade.
dpkg: error processing archive /var/cache/apt/archives/deviceupdate-agent_1.0.2_amd64.deb (--unpack):
 new deviceupdate-agent package post-removal script subprocess returned error exit status 1
******************** Running /var/lib/dpkg/info/deviceupdate-agent.preinst abort-upgrade 1.0.2 ***********************
******************** Running /var/lib/dpkg/tmp.ci/postrm abort-upgrade 1.0.0 ***********************
******************** Running /var/lib/dpkg/info/deviceupdate-agent.postinst abort-upgrade ***********************
Errors were encountered while processing:
 /var/cache/apt/archives/deviceupdate-agent_1.0.2_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Hi @shaeussler, the DU agent needs a connection string for authentication and connection. By default, we recommend doing this using IoT Identity service (described here). But if you are using TPM directly without IoT Identity service (AIS) service, you will need to write custom code, similar to eis_util which creates the connection string with the identity service

@shaeussler: Is the above error after the identity service was configured as defined in this section: https://learn.microsoft.com/en-us/azure/iot-edge/how-to-provision-devices-at-scale-linux-tpm?view=iotedge-1.4&tabs=physical-device%2Cubuntu#provision-the-device-with-its-cloud-identity?

Yes, I configured my IoT Edge using the instructions from https://learn.microsoft.com/en-us/azure/iot-edge/how-to-provision-devices-at-scale-linux-tpm?view=iotedge-1.4&tabs=physical-device%2Cubuntu#provision-the-device-with-its-cloud-identity

I think the problem is that root is not allowed to get the identity for adu. Only the adu user can get his identity from the iot-identity-service, see https://azure.github.io/iot-identity-service/develop-an-agent.html#client-authorization

@nihemstr

Hey @shaeussler
I took a look through the above logs and wanted to confirm a few things, but before I do I wanted to address:

I think the problem is that root is not allowed to get the identity for adu. Only the adu user can get his identity from the iot-identity-service

That's not expressly true. It is true that the agent out of the box creates the principle file that authenticates ADU when it attempts to connect to the azure-identity-service ( within the azure-identity-service there are three services- the identity service (IS), key service (KS), and certificate service (CS) that we interact with ). You can read more about the principle file here at bash /etc/aziot/identityd/config.d/adu.toml. Inside of the principle file there's a field called the idType that is initially set to the user-id of the adu user, however if you want to run as root you can always change that value to a 0 indicating you're allowing any process that runs as root ( in this case the ADU process if you're running it as root ) to access the IS,KS, and CS.
We don't support that out of the box because of the security implications, but you're welcome to do so during development.
As for debugging the ADU agent it looks like your agent is getting past requesting the identity from the identity response but is instead failing to parse/getting a bad response from the key service. This might point to an issue with the way the key service has been configured or some logic path we haven't explored yet, but first we check the configuration of the IS (within the azure-identity-service) by running the following command ( assuming you've configured AIS to allow the adu user to connect, if configured for sudo access run the command without the -u adu ):
sudo -u adu curl -XGET --unix-socket /run/aziot/identityd.sock http:/foo/identities/identity?api-version=2020-09-01
The response should be the actual identity response the du-agent would receive when we ask for a response. Once we get the value we can determine what might be going wrong with the identity response.
Let me know when you get the data!
Thanks,
Nic Hemstreet

Hi @nihemstr
Thanks for your response.
Here are the requested data:
sudo curl -XGET --unix-socket /run/aziot/identityd.sock http:/foo/identities/identity?api-version=2020-09-01
{"type":"aziot","spec":{"hubName":"XXXXXX.azure-devices.net","gatewayHost":"XXXXXX.azure-devices.net","deviceId":"my-device-id","auth":{"type":"tpm"}}}

sudo cat /etc/adu/du-config.json
{
    "schemaVersion": "1.1",
    "aduShellTrustedUsers": [
      "adu",
      "do"
    ],
    "manufacturer": "XXX",
    "model": "<my-model>",
    "agents": [
      {
        "name": "my-adu-agent",
        "runas": "adu",
        "connectionSource": {
          "connectionType": "AIS",
          "connectionData": ""
        },
        "manufacturer": "XXX",
        "model": "<my-model>"
      }
    ]
  }

sudo cat /etc/aziot/identityd/config.d/adu.toml
[[principal]]
 name="IoTHubDeviceUpdate"
 idtype=["module"]
 uid= 111

id adu
uid=111(adu) gid=117(adu) groups=117(adu),106(syslog),999(aziotks),997(aziotcs),996(aziotid),118(do)

Here are some additional resources to help setup Iot Identity Service (aka AIS) for use with TPM
https://azure.github.io/iot-identity-service/pkcs11/
and for TPM2.0:
https://azure.github.io/iot-identity-service/pkcs11/tpm2-pkcs11.html
/etc/aziot/config.toml file needs:
[aziot_keys]
pkcs11_lib_path = "<path of the PKCS#11 library>"
pkcs11_base_slot = "<PKCS#11 URI of a slot where dynamically generated keys will be stored>"

As per that page, try to test with:
pkcs11-tool --module "$PKCS11_LIB_PATH" -IOT
p11tool --provider="$PKCS11_LIB_PATH" --list-token-urls

I saw that the provided id adu output shows that adu user is not in the aziottpm unix group, which seems a bit suspect.
If you do
ls -latr /dev/tpm*

are the tpm devices should be in aziottpm group?
The id adu output shows adu user is already in the aziotid, aziotcs, and aziotks groups
However, I think you may need to add adu user into aziottpm group as well, so give that a try.
Also, ensure that the azure identity .toml config file has user and group owners as aziotid and aziotid, respectively.
Closing as this is a stale issue. Feel free to open a new issue if the above does not resolve the issue.


Not working for my staging server

The extension is awesome, it works for http://graph.facebook.com/github and other public APIs the like, but when I'm view JSON response from my non-public devbox it's not working at all. Does it rely on something like HTTP headers to get things right?

Hi @gsklee it actually analyses the content to determine if it is a valid JSON or not, could you provide an example of your API output?

@tulios Something as simple as this doesn't work:
{"ts":<PHONE_NUMBER>.899794}

Try version 0.8.0, a lot of bugs were fixed.


Re-land sRGB support with iOS transparency fixes

We rolled out sRGB support in #11169 because it had issues with transparency.
We should re-land sRGB support once the issues are fixed.

@brianosman @collinjackson I assume this was done; please let me know if we're not where we want to be on sRGB, color spaces, etc.


Ability to customise the UI (e.g: change the colour of the panes, panes positions, ETC)

Which panes?

Added in next version


CSS bundle contents are tree-shaken when re-exporting a component/module

What version of Remix are you using?
1.11.1
Are all your remix dependencies & dev-dependencies using the same version?

[X] Yes

Steps to Reproduce
Re-export a component that imports a .css.ts file.
Reproducible example
Expected Behavior
Styles should be included in the css bundle.
Actual Behavior
Styles are ignored, and not included in the css bundle.

In your example I was able to work around the issue by avoiding export * in app/components/index.tsx and explicitly re-exporting the component instead:
-export * from "./Test";
+export { Test } from "./Test";

By the way, I think manually re-exporting everything like this is a better pattern anyway. I've been burned in the past with files like this that make heavy use of export * where you start to lose visibility over its API surface area, can't tell whether there are any name clashes, inadvertently make internals public API, etc. Obviously I'd prefer the esbuild bug was fixed, but I'd still recommend making this change even if the bug wasn't there.

Thanks for looking into this! Seems like a fair work around until the issue is fixed in esbuild.

Seems like the issue is fixed in esbuild version 17.7: https://github.com/evanw/esbuild/issues/1370 https://github.com/evanw/esbuild/releases/tag/v0.17.7

Version 0.17.7 introduced other tree shaking issues: https://github.com/evanw/esbuild/issues/2933

I'm closing this for now since it's an esbuild issue that can't be fixed in Remix. We've made a note of this issue in the CSS bundling setup guide to help Remix consumers avoid this.

@markdalgleish I have this specific issue with named exports but only in production mode (NODE_ENV=production) and when the component is re-exported to root.tsx. Here are a few interesting facts I noticed:

In dev mode, the CSS files are injected into the header but not in prod.
The re-exported styles are missing in the build folder after running remix vite:build.
If I import the component with React's lazy(..) function, the styles are created into the build folder and injected into the header in both dev and prod, solving the problem 🤔

Is this problem still related to the esbuild tree shacking issue? Maybe it's more related to SSR and how the root.tsx component works?
Version Info:

Remix: v2.9.2 (with Vite)
vite: v5.3.1
@vanilla-extract/css: v1.15.3
@vanilla-extract/vite-plugin: v4.0.11

UPDATE: I found out that this is not an issue! The re-exported styles are just part of the root route chunk. The problem was that we were using Ant Design, which was loaded last, causing an override over the custom Vanilla Extract styles. However, I'm still not sure why it works differently on dev 🤔
I was able to solve it by forcing Antd styles to load before <Links />:
<html lang="en">
  <head>
    <Meta />
    <link rel="stylesheet" href="/antd.css" />
    <Links />
  </head>

  <body>
    <Outlet />

    <ScrollRestoration />
    <Scripts />
  </body>
</html>


Add Game Producer / Marketing Guy

Hi thank you for curated all the insight it really helpful,
To market a game or Producing it to final product i think management skill about scaling the game size or choose it market segment is super important. And game producer or game PR guy were still in many needs by small to big studios. If we can add this to the insight i think it will be helpful to indies or jobseeker

Thanks for your suggestion. However, I don't know what to learn to become a game producer/marketer.
Anyone who can help is welcome.


Feedback and proposal for new notifications

Hi man!
Great progress on the v4-version!
I was looking at the hacks that I applied to our projects and most of these can now be facilitated with the core features of the package which is awesome! I still have one thing that I want to do and that I solved in my CustomCacheService.
Basically, we have a true/false-property on most nodes that says "Exclude from indexing" we use this recursively so if a parent node has this property checked any descendants will be excluded.
The implementation in my HackCacheService looks something like this:

Since I can't exclude all nodes of this type my solution here was to just store an empty string in the examine index and don't fetch the content using the HTTP-client.
In my case, this logic was added where the most left arrow is and if the page should be excluded I just did not perform the call to page renderer and just stored the fullHtml as an empty string.

I know that the new notifications might be a good fit for this but at the point where the notification fire I don't have access to the IPublishedContent to check the properties, I would have to re-fetch them to check the "no index"-property.
So I was thinking that there might be something that could be improved here, either:

Maybe a notification that is fired before the indexing for a certain IPublishedContent is starting to that we can return "Ignore" or something. Something like "Rendering" or similar.

Use another type in the current events that include the IPublishedContent? So that we could replace the content with an empty string. But this option is not so good since it would send an HTTP-request and then ignore the content, so would be better to be able to abort before the request is fired.

Looking forward to a new alpha with the notification stuff included =D
Cheers!

Hmm... I remember that I made the implementation Render in HttpPageRenderer virtual to be able to support this requirement.
It could just be solved like this:
public class HttpPageRendererNoIndex : HttpPageRenderer
{
    public HttpPageRendererNoIndex(
        IOptions<FullTextSearchOptions> options,
        IHttpClientFactory httpClientFactory,
        ILogger<HttpPageRenderer> logger) : base(options, httpClientFactory, logger)
    {
    }

    public override async Task<string> Render(IPublishedContent publishedContent, PublishedCultureInfo culture)
    {
        bool shouldBeExcluded = publishedContent.Value<bool>(Core.Constants.ContentProperties.NoIndexNoFollowAlias) ||
                                publishedContent.Ancestors().Any(x => x.Value<bool>(Core.Constants.ContentProperties.NoIndexNoFollowAlias));

        if (shouldBeExcluded)
            return string.Empty;

        var html = await base.Render(publishedContent, culture);

        return html;
    }
}

If you think that there is no need for a notification at the point where I was proposing, I think we can just close this issue. The solution above works well.

I think implementing your own Renderer here is the right choice. Thanks for the thoughts and suggestion. We should probably add this to the documentation :)


Identification if running on server or client lacks of assumption.

The current code to identify on which environment the application is running among other things is deferred by the existence of a module.
This is not necessarily true, especially if you use a packaging system like webpack, which uses modules on the client side.
Are there any plans to use another indicator?
File reference
Refering to the line 5 in bucky.js
isServer = (typeof module !== "undefined" && module !== null) && !(typeof window !== "undefined" && window !== null ? window.module : void 0);

We don't have specific plans for an update -- PRs are welcome, though, if you have a suggestion/proposal.

@debrutal Have you solved this problem?

Hi,
since i was not confident enough in the past to open a PR for this and i don't have access to the code anymore i can just recall my activities from 2015.
 !(typeof window !== "undefined" && window !== null ? window.module : void 0)
By removing the check for a module system, since we can have module system on the client side the part should look like this:
(typeof window !== "undefined" && window !== null );
I think that's all we did for our usecase.

I solved that problem with webpack configuration
new webpack.DefinePlugin({     'window.module': 'module', });

Unfortunately i cannot confirm this. It has been too long ago. If it's ok, i would close this since there are existing solutions.


Support for precision models other than "float"

Upcoming, I will require a fixed-precision mode of Shapely. It's something that's been discussed off and on, but now I have an actual need for it at work and am beginning to plan to make it happen.
My initial idea is that Shapely will no longer hide the GEOS context (this struct: https://github.com/libgeos/libgeos/blob/svn-trunk/capi/geos_ts_c.cpp#L137) from users, but will adapt it into a Python context manager (as in Fiona and rasterio). That context manager would be where the precision model would be specified, the usage to be something like this:
import shapely
from shapely.geometry import Point
with shapely.context(precision='fixed', scale=1.0, offset=(0.0, 0.0)):
    # make a point snapped to a fixed-precision grid
    p = Point(0, 0)

Calling shapely.context() will call initGEOS (from libgeos) and instead of creating a default geometry factory (see https://github.com/libgeos/libgeos/blob/svn-trunk/capi/geos_ts_c.cpp#L209) will create a GEOS precision model and a geometry factory based upon the precision model. All geometry operations in that context will use that geometry factory and precision model. When the with block ends, finishGEOS (https://github.com/libgeos/libgeos/blob/svn-trunk/capi/geos_ts_c.cpp#L255) will be called and the precision model and geometry factory will be deleted.
Without a context manager, the default floating point model will be used and older code will continue to work as usual. A context manager will be required to use a fixed model. The specific syntax is yet to be determined and I welcome suggestions.
News from the GEOS project is that exposing the precision model classes via the C API might be a GSoC thing for a student. The good of this is that someone will write that code and it'll be tested and be a driver for a new GEOS release. The downside is waiting for it, and what if it takes all summer? I may develop a Shapely-specific shim using the C++ classes in the meantime and then migrate over to the new GEOS code when it's ready.

+1 for a fixed precision model

This is very interesting. Has any progress been made o nthis?

Note that pygeos, which will be the basis for Shapely 2.0, already supports setting the precision as outlined above: https://pygeos.readthedocs.io/en/latest/geometry.html#pygeos.geometry.set_precision, in case you want to try it today

I think this can be closed now that we have shapely.set_precision(..) to convert a geometry to a certain fixed precision model (+ the grid_size keyword to do this on the fly in the overlay functions) ?


Negative exposure edge case

When running SigProfilerAssignment.Analyzer.decompose_fit

I've traced the cause of the issue:
single_sample.fit_signatures line 147 - The max exposure is adjusted to make a total match the total of input mutations. Because of the rounding taking place, this can send the max exposure negative in rare cases particularly where the number of mutations is small (e.g. for DBS signatures).
single_sample.remove_all_single_signatures lines 458-460 - The negative exposure is missed out from initialZerosIdx and selectableIdx  such that in line 528, newExposure is len(oldExposure)-1. This generates the size miss-match in the final line.
The combination of rounding to integer exposures in several places then correcting the max exposure for the resulting difference with input exposures is not numerically safe so this should be rewritten. I expect this has a minor effect on SBS/Indel signatures where the exposures are typically very large so correcting for rounding is a small effect but the effect on DBS can be really big.

Hi @aeverall ,
I apologize for the very late reply. Thanks for your insight into this problem and the suggestion.
I have tried to reproduce this case with a number of samples, synthetically generated scenarios, with extremely low mutation counts, high sparsity matrices, and so on, and I never was able to see a case that gives a negative exposure. Probably it is too late, we can revisit this if you have a case for us to look at.
Thanks


Autodistill: Invalid Cuda Error on Colab

Search before asking

[x] I have searched the Autodistill issues and found no similar bug report.

Bug
https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/how-to-auto-train-yolov8-model-with-autodistill.ipynb
I am trying to test this on Colab. on this section:
%cd {HOME}

from autodistill_yolov8 import YOLOv8

target_model = YOLOv8("yolov8n.pt")
target_model.train(DATA_YAML_PATH, epochs=50)

But I couldn't find the solution to this error when I started to train:

/usr/local/lib/python3.10/dist-packages/ultralytics/yolo/utils/torch_utils.py in select_device(device, batch, newline, verbose)
67             install = 'See https://pytorch.org/get-started/locally/ for up-to-date torch install instructions if no '
68                       'CUDA devices are seen by torch.\n' if torch.cuda.device_count() == 0 else ''
---> 69             raise ValueError(f"Invalid CUDA 'device={device}' requested."
70                              f" Use 'device=cpu' or pass valid CUDA device(s) if available,"
71                              f" i.e. 'device=0' or 'device=0,1,2,3' for Multi-GPU.\n"
ValueError: Invalid CUDA 'device=cuda' requested. Use 'device=cpu' or pass valid CUDA device(s) if available, i.e. 'device=0' or 'device=0,1,2,3' for Multi-GPU.
torch.cuda.is_available(): True
torch.cuda.device_count(): 1
os.environ['CUDA_VISIBLE_DEVICES']: cuda

It seems like I can work with GPU but I couldn't fix that.
Environment
No response
Minimal Reproducible Example
No response
Additional
No response
Are you willing to submit a PR?

[ ] Yes I'd like to help by submitting a PR!

Hello! There was a bug in the way in which we supported GPUs in YOLOv8 training.
To fix the issue, first upgrade to autodistill-yolov8 0.1.3:
pip install --upgrade autodistill-yolov8==0.1.3

Use this code to train:
target_model.train(DATA_YAML_PATH, epochs=50, devices=[0,1...])

Where devices is the list of CUDA device IDs you want to use in training.

I am going to close this issue but please feel free to re-open it if the solution above does not rectify the issue.


Fixed regressions from rewriting dcos job in Go.

This commit aligns the Go code more with the functionality of the Python
code. This includes some formatting issues and support for ENV and
Secrets in job definitions.

@mesosphere-ci run all integration tests


Rails 6.1 support

@alecdotninja can you please look at this PR ?

This looks great. Thank you for your work on this, @yosiat ! :tada:

:ship: This has been released in v1.0.0.


Updated command line for lessc to do --include-path= as lessc c…

Updated command line for lessc to do --include-path= as less can't parse --include-path .

Thanks a lot.
Seems like this code path is in dire need of tests.


feat!: Update crossplane-runtime to 1.14.1

Description of your changes

Update controller-runtime and other libs as well.
Remove deprecated property spec.providerRef from all CRDs

Fixes: #1934
I have:

[x] Read and followed Crossplane's contribution process.
[x] Run make reviewable test to ensure this PR is ready for review.

How has this code been tested
n.a.

How has this code been tested
n.a.

I wonder if we could have a basic e2e/integration test that installs the previous version in a kind cluster, add some resources connected to local stack, upgrade and do some basic changes

I wonder if we could have a basic e2e/integration test that installs the previous version in a kind cluster, add some resources connected to local stack, upgrade and do some basic changes

I know what you mean. We are thinking about a solution for this issue for quite a while already. Thing is that local stack only provides a subset of AWS resources and some of them only in the premium tier.
We have this integration test kinda thing in cluster/local/integration_tests.sh but it only tries to install the provider in a kind cluster and does not test the individual controllers.
There is a proposal in https://github.com/crossplane-contrib/provider-argocd/pull/89 that uses
https://github.com/maximilianbraun/xp-testing and https://github.com/kubernetes-sigs/e2e-framework/ to run E2E tests. However, it is much simpler there because you only need an ArgoCD instance instead of a whole AWS environment.

@MisterMX I was just thinking about testing a couple of services to get coverage for crossplane-runtime etc. Probably at least one service "hand written" and one generated, not full e2e coverage.


wss://emotivcortex.com:54321 error with macOS, matlab

Question:
Operating system: macOS High Sierra 10.13.5
MATLAB version: 2017b
Cortex version: Latest (Downloaded last month)
Issue:
I am trying to connect to cortex through websocket in matlab in macOS. The matlab websocket implementation I am using is:
https://github.com/jebej/MatlabWebSocket
I get the following error when I run the command
client = SimpleClient('wss://emotivcortex.com:54321');
Error: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Error: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Disconnected from server at wss://emotivcortex.com:54321
I know there have been issues posted in the past related to the certificate and some people have suggested to skip certificate verification. Is there a possible solution to this without skipping the certificate verification and in case there isn't is there an example of how to solve this?
Thanks!

hi @tanujhasija , we are working on some solutions help computer trust our certificate.

Hi @nguoithichkhampha,
Thanks for the prompt reply! If possible could you please give an approximate time by when I could expect a solution?

hi @tanujhasija by skipping the verification the connection is still secure and protected with SSL. Is there anyway you can do that for now with the MatlabWebSocket lib?

Hi @patdotme. I couldn't skip the verification using the Matlabwebsocket lib. I skipped the certificate checks from the java control panel but it did not work and continue giving me the same error.


Remote Explorer SSH Targets periodically empty

VSCode Version: 1.44.2
OS Version: OS X 10.14.6

Steps to Reproduce:

use VS Code for a while (do not use Remote Explorer)
choose the Remote Explorer pane
the SSH Targets list is sometimes empty

My system has numerous SSH targets configured in ~/.ssh/config. Sometimes VS Code will show these targets, sometimes it will not. There is no obvious reason when it fails to show them and there is no indication of errors in the UI. Quitting VS Code and restarting it seems to correct this and the list is immediately populated. This may not be trivial to reproduce.

Does this issue occur when all extensions are disabled?: Not tested

Hm, no idea. Next time this happens you could check whether there are errors in the developer tools console (Help > Toggle Developer Tools) or, see if clicking the "refresh" icon in that panel fixes it.


fix(ssh): force user for ssh commands

Changes

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#949  👈
#948
#945 : 1 other dependent PR (#946 )
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @NathanFlurry and the rest of your teammates on  Graphite


Export error using BinExport with BinNavi

Thanks to previous help, I was able to build and load zynamics_binexport_9.* in IDA Pro 6.9.  When I test the plugin in IDPro by selecting the plugin and executing "BinExport v2 Binary Export" or "Text Dump Export", or "Statistics Export" it appears to work correctly generating their respective files.
When I attempt to do the import via BinNavi however, It launches the import but returns the following error:

IDB file could not be imported
Possible causes:

Database connection problems.
Bug in the IDB exporter.

Problem impact:

The IDB file was imported partially. A raw module in an incosistent state was created. This raw module should be deleted.

Here is the command that binnavi is using to create the import process:
C:\Program Files (x86)\IDA 6.9\idaq64.exe, -A, -OExporterHost:localhost, -OExporterPort:5432, -OExporterUser:postgres, -OExporterPassword:password, -OExporterDatabase:TempDB3, -OExporterSchema:public, -S"C:\Users\XXXX\AppData\Local\Temp\binexport.idc", C:\test2\FT.i64
Inside of binexport.idc it references the correct zynamics_binexport_9.
I've tried with i64 and idb imports and the results are the same.  I've also tried ida 32 and 64 bit.  I can connect to the postgres db fine via BinNavi and PGAdmin3, so I don't think it's a connection issue.
Any ideas?
thanks

Also, I'm testing on Windows 10.

If I do the same test but modify the binexport.idc to use the old version (zynamics_binexport_5), it appears to finish without any GUI reported errors (however in the debug window it mentions missing db fields).  However, in BinNavi, module appears with a red icon by it and gui commands to perform actions on it fail or don't do anything, which would be expected if the structure of imported data has changed with different versions of BinNavi/BinExport (which would be normal).  So this test shows it can generate the imported data and connect to the postgres db correctly.
Does anybody have a known working version of zynamics_binexport_9.plw and .p64?  Thanks

Update:  It appears that it does work on some idb and p64 files.  Closing for now, will open a new issue when if necessary (and with more specific information.


update: sanctioned address list: Sep 15 2022

ref: https://github.com/0xB10C/ofac-sanctioned-digital-currency-addresses/commit/96f89d770659edad674fa2ea101a0d79a7f4c768
ofac: https://home.treasury.gov/policy-issues/financial-sanctions/recent-actions/20220915

Are these updated lists retroactively applied to the missing transactions list?

Generally, no. The tags for sanctioned transactions are only applied when processing transactions from templates or blocks.
SInce #37 it's possible to re-tag transactions by running the daemon with the configuration option retag_transactions = true once (docs missing https://github.com/0xB10C/miningpool-observer/issues/49). It will attempt to get the raw transactions from Bitcoin Core (requires txindex and non-pruned) and updates the tags in the database. This will use the current list of sanctioned addresses for transactions paying to sanctioned addresses ("to sanctioned" tag) but won't update the "from sanctioned" tag when a UTXO from a sanctioned address is spend. This would require the sanctioned UTXOs at the given time.
Currently, if a user doesn't update the list of sanctioned addresses in time, he might miss out on sanctioned transactions. I have an alert for this and while this works for me, I don't think this scales. This could be improved, e.g. by auto-updating the list of sanctioned addresses from GitHub from time-to-time in the daemon. I'm tracking this in https://github.com/0xB10C/miningpool-observer/issues/51.


feat(stream): new boxed group_top_n executor from proto

Signed-off-by: Shmiwy<EMAIL_ADDRESS>I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.
What's changed and what's your intention?
PLEASE DO NOT LEAVE THIS EMPTY !!!

impl ToStreamProst fmt::Display and PlanTreeNodeUnary for StreamGroupTopN
impl ExecutorBuilder for GroupTopNExecutorBuilder

Checklist

[x] I have written necessary rustdoc comments
[x] I have added necessary unit tests and integration tests
[x] All checks passed in ./risedev check (or alias, ./risedev c)

Documentation
If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.
Types of user-facing changes
Please keep the types that apply to your changes, and remove those that do not apply.

Installation and deployment
Connector (sources & sinks)
SQL commands, functions, and operators
RisingWave cluster configuration changes
Other (please specify in the release note below)

Release note
Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.
Refer to a related PR or issue link (optional)
#4246 #3489

@skyzh When all the tests pass, I will optimize the code and reduce code duplication


Locales (LC_MESSAGES) aren't working for certain packages

Describe the bug
When using the Finnish locale, certain packages seem to not respect the LC_MESSAGES for that language. I haven't tested many other locales yet or a lot of different packages, but I did notice a few that this seems to be the case with. I'm also not sure if this is an issue with how some packages were built or with something in the NixOS environment. These translations work on my Gentoo machines with the same package versions. I'm also very new to NixOS, so please forgive me if I've missed anything important.
Steps To Reproduce
Steps to reproduce the behavior:

Set locale to Finnish in configuration.nix: i18n.defaultLocale = "fi_FI.UTF-8";
As a non-root user type su -. Instead of saying "Salasana: ", it says "Password: ". I've tested this with "fr_FR.UTF-8" as well and verified that the source code does indeed have translations for these two languages.
A GUI package that I also noticed this with is blueman-applet from the blueman package. The tray menu should be completely translated, but is completely in English for me.

Expected behavior
Text should be translated into the locale that I am using.
Additional context
I can see that the LC_MESSAGES files for these two programs exist and contain the translations in the .mo files.
rypervenche@jodi:/nix/store$ fd blueman | rg 'fi/LC_MESSAGES'
zfmrng2pwdxqzs3cx5vi9qwcyy25i3zy-blueman-2.3.5/share/locale/fi/LC_MESSAGES/blueman.mo
lwmgh1f947ammhmslmys3qxx1qm11yd9-blueman-2.3.5/share/locale/fi/LC_MESSAGES/blueman.mo

rypervenche@jodi:/nix/store$ fd util-linux | rg 'fi/LC_MESSAGES'
v54ban0gh2iv2giy9kbl7mkckp56mldy-util-linux-2.39-lib/share/locale/fi/LC_MESSAGES/util-linux.mo
c67kb57rfi56aibv5xdb34wgbndb6sdl-util-linux-2.38.1-lib/share/locale/fi/LC_MESSAGES/util-linux.mo

Metadata
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.45, NixOS, 23.05 (Stoat), 23.05.2753.3fe694c4156b`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.5`
 - channels(root): `"nixos-23.05"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

It looks like this may be an issue with programs that use gettext with po/mo files?

I think for su it happens because shadow has different code paths for pam-style and non-pam authentication:
$ nix build --impure --expr 'with import <nixpkgs> {}; shadow.su'; LANG=fi_FI.UTF-8 LANGUAGE=fi ./result-su/bin/su
Password:
su: Authentication failure

$ nix build --impure --expr 'with import <nixpkgs> {}; shadow.su.overrideAttrs (oa: { configureFlags = oa.configureFlags ++ ["--without-libpam"]; })'; LANG=fi_FI.UTF-8 LANGUAGE=fi ./result-su/bin/su
Salasana:
su: Authentication failure

In case of PAM I think it's up to libpam modules to localize the messages. And it gets loaded from the wrong location:
$ LANG=fi_FI.UTF-8 LANGUAGE=fi strace -y -f -olog su
Password:
su: Authentication failure

$ fgrep Linux-PAM log
1180126 openat(AT_FDCWD</tmp>, "/nix/store/ibp4camsx1mlllwzh32yyqcq2r2xsy1a-glibc-2.37-8/share/locale/fi/LC_MESSAGES/Linux-PAM.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

I think normally the locale directory is passed to gettext calls via bindtextdomain(): https://github.com/shadow-maint/shadow/blob/master/src/su.c#L1008
	(void) bindtextdomain (PACKAGE, LOCALEDIR);

I see one for su, but not for linux-pam (but maybe it's well-hidden). I would guess that for Gentoo everything is stored in /usr/share/locale/fi/LC_MESSAGES/*.mo and is always visible.

As a random example of a library that does call bindtextdomain() at init is gnutls: https://github.com/gnutls/gnutls/blob/master/lib/global.c#L269
#ifdef HAVE_DCGETTEXT
	bindtextdomain(PACKAGE, LOCALEDIR);
#endif

I think linux-pam should adapt the same approach.

Asked linux-pam maintainers in https://github.com/linux-pam/linux-pam/issues/602 if they are open to add a form of lookup path.

Proposed pam change against nixpkgs as: https://github.com/NixOS/nixpkgs/pull/249487

This has been resolved with #249487. Thank you.


chore: update gunicorn config and add logging

Update gunicorn config with debug mode on, to debug worker issue in deployment.

@joshuagraber In this case, we may be able to modify the run parameters in digital ocean's app spec to do this, rather than making a change -- indeed, that's what I tried to do with ``--log-level debugand--timeout 90` (neither of which yielded anything of note).
Might save us a pull request.