Introduce pluggable configuration provider

This PR introduces a framework for injecting custom configuration resolver which allows us to resolve the configurations dynamically. The configurations be resolved are of a pattern like ${resolver_name:key_for_resolving_config_property} - We have also introduced ConfigurationResolverFactory and ConfigurationResolver which allows us to resolve configuration in runtime. We have migrated the environment loader to this framework
The PR is still in progress and the tests are pending.

cc: @electrum

@Praveen2112 can you fix failures?

@wendigo Now the CI is green

The usage of this PR is shown here - https://github.com/trinodb/trino/pull/22633

I'm not super convinced that airlift should have SPI. I think that we could add a way to plug-in an interface implementation into bootstrap like this:
Bootstrap app = new Bootstrap(...);

Injector injector = app
                .doNotInitializeLogging()
                .setConfigurationInterpolator(interpolator)
                .setRequiredConfigurationProperties(config)
                .initialize();

Where interpolator is:
interface ConfigurationInterpolator
{
        Optional<String> interpolate(String namespace, String key);
}

Ideally I'd like to provide a pre-configured interpolator to Trino connectors through the ConnectorContext that is initialized and injected through the engine.

Ideally I'd like to provide a pre-configured interpolator to Trino connectors through the ConnectorContext that is initialized and injected through the engine.

But how would we make them pluggable or have some custom implementation in the future - Won't we need classloader isolation for its implementation ? Would it be a part of Trino's plugin ? The problem is the scope of this pluggable configuration provider is not specific to Trino connectors and its ecosystem (like password-authenticator or resource-group.manager) but would be applied for config.properties for the TrinoServer as well. If we have it as a part of trino spi - then we might need to load the plugins twice - Once during the startup for configuration resolution and then once for normal execution -

Yes. And we would add an SPI in Trino

Yes and we have a version which adds a SPI in Trino as well - https://github.com/trinodb/trino/pull/22621 - But there were a few questions on how to pass the properties for the same - We had it propagated via config.properties - But we might need a separate file for configuring this ConfigurationResolver - how to configure them during the startup.

I'm fine with both the options - @electrum Can you guide us on which direction to take - We have a skeleton PR which tracks both the approaches - Having the changes in Airlift and having the changes in Trino - Both has its own set of complexity.

The reason to do it in Airlift via an SPI is to allow pluggable providers with classloader isolation that can handle interpolation for the main server configuration. For example, we'll have a plugin for AWS Secrets Manager that uses the AWS SDK. Using a plugin loader similar to Trino will prevent leaking the plugin's version of the AWS SDK into the main application's classpath.

@electrum yes, but we can add this SPI in Trino rather than in the airlift

@wendigo We have tried it a similar approach (https://github.com/trinodb/trino/pull/22621) but in order to have in Trino we have two main set of issues -

Where would we store the configuration property for this config_resolver - It could be a part of config.properties - but we need to remove the properties which are used for config_resolver - We might need modify the launcher.py to add any new additional property file.
Copying the config-plugin to a different location during the startup

In this case we might need some sort of changes in airlift and some in the Trino side. Plus this configuration resolver doesn't fully fit under Plugin lifecycle in case of Trino - These are mostly static during the startup

@electrum Thanks a lot for the review. AC

@electrum Based on the previous discussion we have migrated the configuration to TOML file and have made the EnvironmentVariableConfigurationResolver pluggable (via Config - so it would be used of configured)

@Praveen2112 how the TOML is plugged in into existing Configuration binding? Doesn't seem to be a case

The toml file path is passed via configResolver system property like how we handle for config.properties

Typo "fot" in commit title

@electrum Based on our previous conversation - Have migrated the code from Configuration resolver to Secret provider

@electrum Gentle ping for this PR

@electrum Gentle ping

@electrum Thanks a lot for the review - I have addressed most of the comments except 1 - What would be our fallback mechanism if the key for a config provider doesn't exist -

We could fail explicitly
Skip resolving them.
We could go for option 2 as a config of a similar pattern could be used by other system

@electrum Addressed all the comments - Now if the secret provider is not available we would throw an error. We have captured and throw the error via Consumer

@electrum Except this comment all others were addressed

@dain Thanks a lot for the review - Except this comment https://github.com/airlift/airlift/pull/1217#discussion_r1699364480 most of them are addressed.

@dain All the comments were addressed

@dain / @electrum All the comments were addressed.


Add support of experimental transmit_power feature to zigbee2mqtt stable.

See https://github.com/Koenkk/zigbee2mqtt/issues/2253

@danielwelch Any chance to get this pushed?
Thank you in advance!

@danielwelch Is there any particular reason not to merge this? I know this settings is marked as experimental, but it can be useful to the stable user.

I get this warning now in the system log hass.io and the non-edge is not displayed anymore in the add-ons.
20-05-23 11:42:10 WARNING (MainThread) [supervisor.store.data] Can't read /data/addons/git/7ad98f9c/zigbee2mqtt/config.json: does not match regular expression for dictionary value @ data['schema']['experimental']['transmit_power']. Got 'int(-22,19)?'

@dmitrybabeshko Unfortunately, this ended up breaking the add-on and I rolled it back. I’ll have to look into what the problem was, but hopefully can continue to work on this.

@danielwelch Sorry for this breaking change. Please check the corrected pull request https://github.com/danielwelch/hassio-zigbee2mqtt/pull/342.


fix custom dc styling

fix examples missing on the tool pages of custom dcs
fix footer/header styling/colors for stanford, feedingamerica, iitm, worldbank

https://screenshot.googleplex.com/7VeTFzMd7fDvu57
https://screenshot.googleplex.com/5ss2DzVMhmpWS2v

thanks for the reviews!


Project update

Include more Python-related ignore patterns and added a description to the README.md file about the project's goal and prerequisites

Summary by CodeRabbit

New Features

Introduced a basic HTTP server that listens for connections and responds with a "200 OK" status.
Added a comprehensive .gitignore file tailored for modern Python development, improving repository cleanliness.

Documentation

Enhanced the README.md with a new introductory section and updated prerequisites to require Python 3.10 or higher.

/describe
--pr_description.extra_instructions="
For the title, use the format [type]: [summary]
"
--pr_description.publish_labels=true
--pr_description.publish_description_as_comment=true
--pr_description.generate_ai_title=true
--pr_description.final_update_message=true

/custom_prompt
--pr_custom_prompt.prompt="ptoject tiltle"


two bucket: adding test JSON

Fixing exercism/todo#160.
Is this clear enough or does it need some clarifying comments?

I think this is clear enough. I'm sure if it's not, someone will say so when they start trying to implement a test generator. For now, let's leave it at this.
Thank you!


Editor inherits Abyss theme in find widget

@aeschli - it is caused by this generated CSS style:
.monaco-editor.vs-dark.vscode-theme-abyss-themes-Abyss-tmTheme {
    color: rgba(102, 136, 204, 1);
}

Perhaps the intent is to use .token in there?

Even better:
https://github.com/Microsoft/vscode/blob/master/src/vs/workbench/services/themes/electron-browser/themeService.ts#L365
	if (editorSettings.foreground) {
		let foreground = new Color(editorSettings.foreground);
		cssRules.push(`.monaco-editor.${themeSelector} { color: ${foreground}; }`);
		cssRules.push(`.monaco-editor.${themeSelector} .token { color: ${foreground}; }`);
	}


Support paths relative to the current file in query_file!()

This would require proc_macro::SourceFile to be stabilized (or a similar API): https://github.com/rust-lang/rust/issues/54725
Probably would require fixing https://github.com/rust-lang/rust/issues/62892
We could make it work on nightly using --cfg procmacro2_semver_exempt but that's a half-measure.

You can also make it work on nightly by using proc_macro2::Span::unwrap to get a real proc_macro::Span and then use the feature gated API. I did something similar here.
That still can't support stable, but atleast it doesn't require all transitive reverse dependencies to build with --cfg procmacro2_semver_exempt.
It doesn't seem nice to introduce behavioural differences in the same macro between stable and nightly though. If it's important enough, it might be better to add the functionality to a separate macro only available on nightly, possibly with a feature.

The current paths are at least cargo workspace relative.. for me, that's good enough to not want to do anything until we can do it properly.

It looks like we can support this soon once https://github.com/rust-lang/rust/pull/69041 lands

That doesn't give us a stable way to get the path still.

This doesn't seem like its going to happen in Rust in the near future so I'm going to mark this postponed and close. We just need to make sure our work-arounds are well documented and we're fine I think.


Move to self-hosted-mac

Signed-off-by: Vitor Bandeira<EMAIL_ADDRESS>

@maliberty FYI, after this merge (and people update their branches) I expect to not have issues with macOS. Also, faster turn-around with reduced build times.


Adding a stats page to display commit leaderboard

This PR solves issue #6
Here, I use github's graphql API to read commit history, modify and sort the same and then display it on the website.
The aforementioned page is available at /stats
Please, go through these steps before you submit a PR.

[x] My Pod Leader knows I'm working on this Pull Request
[x] I've explained what the Pull Request is adding.
[x] I've explained why this is important.

@Aryaman1706 If you can resolve the conflict and tag me once you're done with the changes and adding link on the home page.

Hey @khattakdev I made all the necessary changes. I added the section of stats in the home page itself.
Kindly add the following script to netlify build script
echo "const PAT = \"EXAMPLE_GITHUB_PERSONAL_ACCESS_TOKEN\";" > assets/js/accessToken.js
Also you will need to create a PAT and add it the aforementioned script. Here's how to do the same https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token
PS:- Do not worry about all the PATs in commit history, they are all expired.

@khattakdev I have resolved the conflicts. Kindly have a look and let me know if anything needs to be changed.


flutter doctor failed to download engine, tools and other packages

command
flutter doctor
exception
SocketException: SocketException: OS Error: (signal timeout...translated)
, errno = 121, address = storage.googleapis.com, port = 50537
#0      fetchUrl (package:flutter_tools/src/base/net.dart:18)
<asynchronous suspension>
#1      _downloadFile (package:flutter_tools/src/cache.dart:400)
<asynchronous suspension>
#2      _downloadZipArchive.<anonymous closure> (package:flutter_tools/src/cache.dart:407)
<asynchronous suspension>
#3      _withTemporaryFile (package:flutter_tools/src/cache.dart:428)
<asynchronous suspension>
#4      _downloadZipArchive (package:flutter_tools/src/cache.dart:406)
#5      FlutterEngine._downloadItem (package:flutter_tools/src/cache.dart:373)
#6      FlutterEngine.updateInner (package:flutter_tools/src/cache.dart:348)
<asynchronous suspension>
#7      CachedArtifact.update (package:flutter_tools/src/cache.dart:223)
<asynchronous suspension>
#8      Cache.updateAll (package:flutter_tools/src/cache.dart:196)
<asynchronous suspension>
#9      FlutterCommand.verifyThenRunCommand (package:flutter_tools/src/runner/flutter_command.dart:206)
<asynchronous suspension>
#10     FlutterCommand.run (package:flutter_tools/src/runner/flutter_command.dart:162)
<asynchronous suspension>
#11     CommandRunner.runCommand (package:args/command_runner.dart:194)
<asynchronous suspension>
#12     FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:277)
<asynchronous suspension>
#13     CommandRunner.run.<anonymous closure> (package:args/command_runner.dart:109)
#14     new Future.sync (dart:async/future.dart:222)
#15     CommandRunner.run (package:args/command_runner.dart:109)
#16     FlutterCommandRunner.run (package:flutter_tools/src/runner/flutter_command_runner.dart:161)
#17     run.<anonymous closure> (package:flutter_tools/executable.dart:142)
<asynchronous suspension>
#18     AppContext._run (package:flutter_tools/src/base/context.dart:76)
<asynchronous suspension>
#19     AppContext.runInZone.<anonymous closure> (package:flutter_tools/src/base/context.dart:66)
#20     _rootRun (dart:async/zone.dart:1120)
#21     _CustomZone.run (dart:async/zone.dart:1001)
#22     runZoned (dart:async/zone.dart:1467)
#23     AppContext.runInZone (package:flutter_tools/src/base/context.dart:65)
#24     run (package:flutter_tools/executable.dart:114)
<asynchronous suspension>
#25     main (package:flutter_tools/executable.dart:67)
<asynchronous suspension>
#26     main (file:///D:/dart/flutter/packages/flutter_tools/bin/flutter_tools.dart:16)
#27     _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:263)
#28     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:151)

flutter doctor
[-] Flutter (on Microsoft Windows [Version 10.0.15063], locale zh-CN, channel master)
    • Flutter at D:\dart\flutter
    • Framework revision 4da5e9b249 (3 days ago), 2017-08-04 17:44:20 -0700
    • Engine revision 73afec4081
    • Tools Dart version 1.25.0-dev.7.0
    ✗ Downloaded executables cannot execute on host (see https://github.com/flutter/flutter/issues/6207 for more information)

[✓] Android toolchain - develop for Android devices (Android SDK 26.0.1)
    • Android SDK at D:\android\SDK
    • Platform android-26, build-tools 26.0.1
    • ANDROID_HOME = D:\android\SDK
    • Java binary at: C:\work\android-studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-884-b01)

[✓] Android Studio (version 3.0)
    • Android Studio at C:\work\android-studio
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-884-b01)

[✓] IntelliJ IDEA Community Edition (version 2017.2)
    • Flutter plugin version 16.0
    • Dart plugin version 172.3544.34

[✓] Connected devices
    • None

this point: address = storage.googleapis.com, port = 50537
The downloading is not always be failed.
port?

jump from #11051
https://github.com/flutter/flutter/issues/11051#issuecomment-320694455

Related to https://github.com/flutter/flutter/issues/1800
Basically we need to be much better about handling network failures when dealing with pub and other upstream dependencies.
This also affects our build (our main source of flakiness is this kind of thing).

pull flutter, doctor
it works
There is a sample exception

Download failed: https://storage.googleapis.com/flutter_infra/flutter/f0d440bec3ed79f01890e956b07b6e8e6182db62/android-x86/artifacts.zip
HttpException: Connection closed while receiving data, uri = https://storage.googleapis.com/flutter_infra/flutter/f0d440bec3ed79f01890e956b07b6e8e6182db62/android-x86/artifacts.zip

there is a nano bug, after completed flutter ends  with a message retry...


tests: fetch state for steps without state

I just found an issue while testing state for a step that has no initial state. This PR at the moment only adds a failing test.
The problem
When there is no initial state for a step, the macro we use for fetching state throws an exception. It tries to get state for a step which has no state yet. This will also not work if a step does not have state at all, but does need access to the global state.
Possible solutions
I see two ways to fix this.
1. Make sure getStepState works as advertised
This may require adding a parameter to getCurrentStepState() in the wizard. The code would look like this:
public function getCurrentStepState(?string $step = null): array
{
    return array_merge(
        $this->allStepState[$step ?? $this->currentStepName] ?? [],
        [
            'allStepNames' => $this->stepNames()->toArray(),
            'allStepsState' => $this->allStepState,
            'stateClassName' => $this->stateClass(),
        ],
    );
}

It may be worthwhile to have this throw an exception.
Instead of the current code, the macro would return:
$this->instance()->getCurrentStepState($step);

There may be other viable options here; but this would be pretty clean and allows for fine grained testing. This might not even warrant having a macro, but having a macro is a bit cleaner in my opinion. This is my preferred solution.
2. Use what's already there
If we don't want to support getting state for specific step, we could just have developers use:
$wizard->instance()->getCurrentStepState()

And document it instead of the macro we've happed.
Not happy that this is my second PR that turns out having errors. Really sorry for that 😢 Let me know which option you prefer and I'll make sure it's sorted.

Let's go for option 1.

It seems like this PR makes the tests fail: https://github.com/spatie/laravel-medialibrary/runs/6737755363?check_suite_focus=true#step:10:28

@freekmurze can you elaborate what that PR does and how it affects this one? Not sure I follow :)

I decided to have as little logic in the macro as possible. It's up to you if this is a good thing or not.
As a bonus getCurrentStepState now supports class strings as well. Which I think from a developer point of view is easier than looking up aliases. The default behaviour is of course still the same, so there should not be any breaking changes.


Ap 257 notifcations reponse

Summary
Responses on incoming notification:

process Authentication, on failure return failed message (HTTP 401)
process HmacKey, on failure return failed message (HTTP 401)
all other notifications sent response '[accepted]' (HTTP 202)
use unified way to create JsonResponse with status code (aligned responses with docs and SW 6 Plugin)
refactor Validation to chained validation

Processing notification:

skip report notification

Tested scenarios
Test cases:

Missing Authentication exception handling
Invalid Authentication exception handling
Missing Hmac key exception handling
Invalid Hmac key exception handling
Skip Report notification

Fixed issue:  #111

@acampos1916
To be verified with pull request: https://github.com/Adyen/adyen-shopware5/pull/114

@acampos1916
To be verified with pull request: https://github.com/Adyen/adyen-shopware5/pull/114

@acampos1916
To be verified with pull request: #114
Components/IncomingNotificationManager.php

PR 115 has the save method a check to skip empty and Report Notifications
PR 114 has an altered method "save" -> "convert"

To have both PR's working the skip report notification check should be moved to the "saveTextNotification" method.

Thanks for this heads up @bortefi. We've decided to merge #115 before #114 and address the conflicts there.

@acampos1916
To be verified with pull request: #114
Components/IncomingNotificationManager.php

PR 115 has the save method a check to skip empty and Report Notifications
PR 114 has an altered method "save" -> "convert"

To have both PR's working the skip report notification check should be moved to the "saveTextNotification" method.

Thanks for this heads up @bortefi. We've decided to merge #115 before #114 and address the conflicts there.


Time Modal is not showing in android. Version is 7.6.0

Time Modal is not showing in android. Version is 7.6.0
It was working fine early. But recently i have checked and noticed that it is not show time picker modal whenever i am trying to click on time.
Any solution? Thanks!

Thanks. It is working fine in Latest version_9.1.0.


Wireless range issues

I have been having issues with only one side of my redox wireless. The right side can go well over a few feet with no missed keys, but the left side has to be within a foot for the receiver to pick it up. I know it may be a shot in the dark but any ideas of what I can try to trouble shoot? (I have already tried mucking about with the batteries and know that isn't the problem)

Maybe related to this. Still haven't found the issue.
Are you using the aluminium case?

I am using a wooden case from falbatech. I ran a few tests with it and have been finding that though the range can go pretty far but struggles with any sort of interfere  between direct line of sight.so even if I waved my hand in front of the sensor it would cut off if the sensor was over a foot away. I rearranged my setup a bit and now it rarely happens

Thanks for the insight. I'm glad you found a solution.


are_co_aligned slow and possibly unsafe

The utility function are_co_aligned, see
https://github.com/dask-contrib/dask-expr/blob/9334e062a7b41161977ca1c42176197629569cc5/dask_expr/_expr.py#L2863-L2874
is unfortunately rather slow due to the tokenization and lack of caching. in paruqet_reader benchmarks on larger datasets, I saw this slowing down the optimize step by almost a second (when using pyarrowFS such that filters are pushed down

On top of this, I believe the implementation is unsafe since it is putting Expr objects into a set. Sets and dicts are requiring both __hash__ and __eq__ to be implemented and working as the stdlib protocol defines them. While this is true for hash (it hashes the name, this is not the case for __eq__ since this just creates another Expr instance instead of returning a bool. I suspect this just tells how the set is redundant if there hasn't been ever a hash collision / duplicate object here.

The unsafe thing is relatively easy to fix (see pr). The slowdown probably needs a bit of care

Also, I think the are_co_aligned (which is a bit of a misleading function name considering that we typically talk about alignment in the context of partitions) is buggy since it doesn't include filters

Are you talking about filters in the tokenize partial call? Filters can change the number of partitions with pruning so we shouldn’t include them
I thought a little bit about the filters in read parquet, we shouldn’t use are_co_aligned there anyway, it’s too generous and allows expressions that we don’t want to allow, I have a solution for this soonish

Are you talking about filters in the tokenize partial call?

yes. I haven't thought too deeply about this but it is confusing that it isn't included while columns are. No big deal, though

Looking into this a little more deeply, it appears that the _tokenize_partial is slow because it does tokenize too many things. The parquet reader for instance is reusing one of the paramters as a cache and tokenizing this cache is what makes it costly. This info is represented in ReadParquet by overwriting the _name method but this is obviously not respected by the _tokenize_partial

I think the tokenize_partial approach is a little brittle in that sense. It compares objects without knowing anything about the objects themselves. Maybe an __approx_eq__ method on the expressions would be more sensible. This would be performing the tokenize_partial but allow for caching and overwriting.

This abstraction leak is already present with the _series argument in the ignore list.


Flocker example is deprecated as flocker does not exists anymore

The Website (https://clusterhq.com/flocker) linked in the Readme (https://github.com/kubernetes/examples/blob/master/staging/volumes/flocker/README.md) does not exist anymore.
Reason: https://techcrunch.com/2016/12/22/clusterhq-hits-the-deadpool/
Imho the example can be removed, as its obsolete.

There's still a flocker volume driver for Kubernetes: https://kubernetes.io/docs/concepts/storage/volumes/
I think this example is referenced from the relevant API field, so deleting this might not be the right move. I suggest you propose removing this driver in the main kubernetes repo first, then the example here.

OK, so maybe a deprecated warning and removing the dead link in the first place would be an idea...

I think we should come up with a depreciation plan for this one, adding a depreciation warning to the example but having it fully supported on the main repo won't be great. I'm not too worried about the broken link to clusterhq.com.


Fix list_models bool parameters

Fix https://github.com/huggingface/huggingface_hub/issues/1146.
With this PR, params fetch_config and cardData are not sent when value is False in list_models list_datasets. Also switched their type from Optional[bool] = None to bool = False for simplicity.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Thanks for the review and approval @LysandreJik, I'm merging :)


feature: Support D417 for init Parameters in numpy class docstrings

(I think this could technically be called a bug rather than a feature request, but ruff's current behavior is consistent with pydocstyle's, so let's call it a feature request :smile:)
issue:  if __init__ parameters are documented in the class docstring, they are not caught by D417
In the class docstrings section of the numpy docstring style guide, they state that __init__ parameters should actually be documented in the class docstrings:

Class docstring
Use the same sections as outlined above (all except Returns > are applicable). The constructor (__init__) should also be documented here, the Parameters section of the docstring details the constructor’s parameters.

Obviously, this one is trickier:  there's a fair degree of variation out there in source code, with many still using the __init__ docstring to document the constructor parameters. The sphinx docstring parser napoleon also supports putting the parameters in either docstring, see ExampleClass here, where they say:
    The __init__ method may be documented in either the class level
    docstring, or as a docstring on the __init__ method itself.

    Either form is acceptable, but the two should not be mixed. Choose one
    convention to document the __init__ method and be consistent with it.

Anyway, not sure how you'd like to handle this one, but thought I'd bring it to your attention.
thanks again!

edit:  here's an example of a class that I would like to throw a D417 error for a missing b argument:
class C:
    """Summary.

    Parameters
    ----------
    a : int
        Description.
    """

    # b is not documented above
    def __init__(self, a: int, b: str) -> None:
        ...

Heh, I think it’s fair to call this a bug. Should fix. A little more complex of course since it requires some bookkeeping…

what's your gut feeling?  support both __init__ and class docstrings like napoleon?  or just look in class docstrings like the spec says?

My gut is to support either, like napoleon. (Maybe biased since I tend to document the init.)

Adding to this, it would be nice to pick up missing attributes in dataclasses/attrs.
For example
@dataclass
class Foo:
    """My class.

    Attributes:
        x: The x attribute.
    """

    x: int
    y: float
    _z: bool

And perhaps if the attribute is prefixed with an _ then it is optional whether that gets included in the docstring? But I'm not sure if that's a widely accepted practice.


Could prototype router support email?

Investigate how a file could be shared securely via email.

What does Pima consider secure?
Does it adhere to SMIME standard?
Depending on hosting, we could have some kind of drop box with temporary access (files not retained long-term)
Ideal version for us is an SFTP to send to

Partner feedback indicated a download site would work for sending files securely.

Partner feedback indicated a download site would work for sending files securely.


Remove redundancies in requirements_xxx.txt files

Remove redundancies in requirements_xxx.txt files.
Currently there are 5 requirements_xxx.txt files, some with redundant packages.

requirements.txt
requirements_dev.txt
requirements_doc.txt
requirements_plus.txt
requirements_tests.txt

These can merged/pruned down to:

requirements.txt
requirements_dev.txt
requirements_plus.txt

not necessary. closing.


Add solution to Arguments Optional challenge

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] All the files I changed are in the same world language (for example: only English changes, or only Chinese changes, etc.)
[x] My changes do not use shortened URLs or affiliate links.

@RandellDawson , I pushed changes.
Thanks.

@nikkiselev We decided to replace the existing Intermediate solution with yours.
It will be  a few more days until we can merge this one, as we are awaiting another PR with changes to this same file to be merged first.

Thank you for your contribution to the page! 👍
We're happy to accept these changes, and look forward to future contributions. 🎉


Rails API with Create-React-App on Heroku

Original post: http://stackoverflow.com/questions/39774143/rails-api-with-create-react-app-on-heroku-npm-command-not-found
I created Rails API backend app and inside the app in a folder client I created create-react-app for my frontend. It works well on my local machine but does not work on heroku with an error:
npm: command not found
This is the structure of the app:
rails-app

app

controller
model
...

client

package.json
index.jsx
...

config
db
...
Procfile

Inside Procfile I have:
web: cd client && npm start
api: bundle exec rails s -p 3001

I don't know much about Heroku but presumably your server doesn't have Node.js?

Yes! A example: https://www.fullstackreact.com/articles/how-to-get-create-react-app-to-work-with-your-rails-api/
The structure is pretty much the same.

@vldsoar The Rails core team are looking to merge Webpacker when it's ready, most likely in Rails 5.1
I'd recommend deploying your frontend and API separately. The API version of Rails does a great job in removing any "view layer" dependencies.

Update: @vldsoar If this PR gets merged then this could be closed @gaearon ?
That tutorial walks you through step by step.

I have created a project using create react app and the backend rails api is on different server and i am facing cross origin unable hit any post api from

@shabanraza While this isn't a Create React App issue, you might want to checkout the rack cors gem.
Maybe this can be closed @gaearon ?


Add support for CentOS Stream 9

Description of the changes
Add support for CentOS Stream 9 using the image - quay.io/centos/centos:stream9 to GSC.

How to test this PR?
To test this PR, set follow similar procedure as followed with Ubuntu and do gsc build and signing commands.

This change is

config.yaml.template line 9 at r1 (raw file):
Previously, dimakuv (Dmitrii Kuvaiskii) wrote…
It looks like there is no official CentOS Stream 9 on DockerHub? Interesting. Do you know the reason for this? (All other base OS images we pull from DockerHub.)

Yes, this is true. There is no official image for CentOS Stream 9, so I use the community-based image for CentOS Stream 9.

@dimakuv @jkr0103
In this 32b4a96619964f3cd640302f13c9d317d98f0bcf commit, I have created a folder under CentOS called "stream". Now, all 5 Dockerfiles are located here, instead of quay.io/centos/centos. This eliminates the need to create extra folders for CentOS releases.

templates/quay.io/centos/centos/Dockerfile.build.template line 15 at r4 (raw file):
Previously, adarshan-intel (Adarsh Anand) wrote…
Can you suggest what can I do here, since removing this package shows this error?
Not sure how to fix it

If I install the "which" package, the build will be successful. As earlier it was not able to execute the command
which /bin/bash

Previously, adarshan-intel (Adarsh Anand) wrote…
Yes, fixed the issue by installing  protobuf-c-compiler, java-11-openjdk, java-11-openjdk-devel packages

@dimakuv Is this still blocking? I have installed the which,  protobuf-c-compiler package and removed the Java packages.

@dimakuv We did not face any such issue. The CI setup we have for running gsc workloads is stable. We have tested the helloworld, bash example with distro as "auto", "distro specified" in config.yaml and also tested "gsc build-gramine" feature. No issues seen

gsc.py line 268 at r10 (raw file):
Previously, mkow (Michał Kowalczyk) wrote…
Not done.

Done. Sorry my bad.

gsc.py line 248 at r14 (raw file):
Previously, adarshan-intel (Adarsh Anand) wrote…
Done.

The colon (:) will not be used because in the function template_path, we are splitting the distro based on : in fetch_and_validate_distro_support. Therefore, using colon in the template_path function as you suggested would result in an error as distro becomes exactly quay.io/centos/centos

gsc.py line 248 at r14 (raw file):
Previously, mkow (Michał Kowalczyk) wrote…
Ok, right, but what about the rest of my comment?

Done, made == comparison

@dimakuv @jkr0103 can you approve this?


Bug for decompression

You probably want to download all the files at once and run 7z x imgs.7z.001. I first want to extract a subset of the data but that causes exact same error as presented in your screenshot.


Update GeoFire.java

Replace call to missing DatabaseReference.fromStatus() method (in Firebase 3.0) with the use of DatabaseException
Replace code in method GeoFire.onDataChange(xx) with the lines below (and add an import of
import com.google.firebase.database.DatabaseException;)
                DatabaseException exception = new DatabaseException(message);
                this.callback.onCancelled(DatabaseError.fromException(exception));

I think ";" is missing at line 80.

@ErstwhileWork thanks for the contribution!  Something similar was done in #34 so I am going to close this.


[STORM-710] bin/storm command list out all the classes in the output for a command

Added -v flag for all bin/storm commands.
Long commands' outputs are removed.
Using -v flag prints traditional long output.

I like the concept, but by having storm.py pull the -v out of the command line, it now makes it impossible for any command to use -v any other way.  This is not a big deal for the typical storm commands, but for storm jar it is backwards incompatible.  This is especially a problem for something that for some reason had a -v with an argument.  I would much rather use an environment variable, or a rather log parameter like '--storm-system-verbose' which is not likely to have any collision.

@oleg03 can you send us a new patch with @revans2 feedback . Thanks.

@oleg03 I don't really like the idea of adding in a -v option that restricts all other commands, including storm jar from supporting it.  I would much rather have a long parameter that is unlikely to collide with something someone else is already using, or us a config or environment variable to trigger it instead.

Closing this. Feel free to reopen with updates.


Print ASCII tree on hypper install

This PR adds an ASCII tree of charts to be installed, when doing hypper install. Example:
$ hypper install ./local-hypper-charts/charts/demo --dry-run
The following charts are going to be installed:
demoit v0.2.0
 ├─ demo-a v0.1.0
 ├─ demo-b v0.2.0
 │  └─ demo-c v0.1.0
 └─ demo-d v0.2.0

🛳  Installing chart "demo-a" as "demo-a" in namespace "demo-a"…
🛳  Installing chart "demo-c" as "demo-c" in namespace "demo-c"…
🛳  Installing chart "demo-b" as "demo-b" in namespace "demo-b"…
🛳  Installing chart "demo-d" as "demo-d" in namespace "demo-d"…
🛳  Installing chart "demo" as "demoit" in namespace "demo"…
👏 Done!

(in console the ASCII pipes are continuous, GH markdown makes a mess).
Closes https://github.com/rancher-sandbox/hypper/issues/178.

Added skipping on printing the tree when there's only 1 chart to be installed.

Fixed it to print the chart name instead of the release name. I think it's better, as the installing chart foo as "bar" start with the chart name.


[docs] upgrades

docs

[ ] #6383
[ ] #6382
[ ] #6381
[ ] #6369
[ ] #6370
[ ] #6375
[ ] #6372
[ ] #3852
[ ] #4644
[ ] #4619
[ ] #3404
[ ] #4348
[ ] #4338
[ ] #2353
[ ] #2599
[ ] #3707
[ ] #5574
[ ] #5607

samples

[ ] #5133

Fixed a one-liner (one-worder) to kick the tires: https://github.com/dxos/dxos/issues/6370

Another simple issue: #6372

Small API generation one: https://github.com/dxos/dxos/issues/6375

https://github.com/dxos/dxos/issues/6404: collections of Ian's impressions while I still have a beginner's perspective


feat: Friendly prerender message + remove UglifyJS from SSR build

This closes #178

Heh, previously I've had See https://github.com/developit/preact-cli#prerender for further information. but I was too lazy to write it 😆

Looks great @rkostrzewski ! 😻
Quick question though: Why was uglify removed on SSR? UglifyJS was mangling the maps, ergo making tracing impossible?

@lukeed that's one of the reasons there other being that chief said so 😆, right @developit ?

@rkostrzewski Lol 😂  Just went thru that Slack. Looks like Chief said that because of the source map issues!

I notice this suggests people should check for window. I always just use this:
https://www.npmjs.com/package/window-or-global
It also makes SSR work for a bunch of features that need global, and it doesn't break even if you run things like eventListeners on global.

@thangngoc89 I've added Pre-rendering part to the README.md let me know if this is what you had in mind and I'll be merging this

@rkostrzewski LGTM


(update Readme):  Steps to add monitor and executor image

FEATURE REQUEST

What happened:

The Runnner (Executor) and Monitor (Exporter) image is hard-coded inside their builder function.

What you expected to happen:

Fetch the Runnner (Executor) and Monitor (Exporter) image for the chaosengine spec.

Anything else we need to know?:


Add a way to handle dependencies within components of a package

Is your feature request related to a problem? Please describe.
Deployments of components may fail if it depends on other components that have not been fully deployed via flux. (For example, an application relying on certain CRD's deployed as part of bigbang, helm install returns successfully so zarf moves onto the next component (app), but flux hasn't deployed everything yet)
Describe the solution you'd like
A mechanism to denote dependencies such that zarf can ensure certain resources exist before deployment of a component. Perhaps specifying namespace, deployment name, or a healthcheck/readiness endpoint that zarf can verify

This may be best handled in Flux. Flux HelmRelease custom resources allow for a dependency tree natively (see example.
If desired to have a zarf component wait until something is fully ready the scripts.after block may be used as well as an alternate solution.
components:
  - name: foo
     scripts:
       after:
         # waits until the deployment exists
         - while ! kubectl get deployment gitlab-webservice-default -n gitlab; do sleep 5; done
         # waits for the deployment to be healthy/ready
         - kubectl rollout status deployment/gitlab-webservice-default -n gitlab --watch --timeout=300s

Was not aware of the script blocks, perhaps I missed it in the docs somewhere?

Docs are still rough, so wouldn't fault you for missing it :) They're being worked on right now though.
@YrrepNoj have you by any chance looked yet into what automated tools there might be to document the zarf yaml schema, in a similar way that we automatically document the CLI?

@RothAndrew I have not yet but I imagine there are some tools that would be really useful. I'll spend a little bit of time looking into that.

We've also had some discussions around basic "ready" tests we could do, but haven't come up the right solution.  For example, having a component wait for a specific deployment to be ready might look like:
components:
  - name: foo
     waitFor:
       - deployment: example-deployment
          namespace: example

I was checking today the porter tool (https://github.com/getporter/) and seems they are having a similar need (https://github.com/getporter/porter/issues/1939). I guess @jeff-mccoy are maybe already aware that Porter is also a go tool, so would be interesting to see if there is a chance to join efforts.

@mahomedalid I learned Porter a few months ago, but haven't had any conversations with that team yet.  We probably should at some point.  CNAB seems pretty complicated, but thought it might be worth adding as a source in the future if adoption grew beyond Azure (primary use I've seen so far)

With the wait-for feature completed, I believe this can be safely closed. @Racer159 @jeff-mccoy

Closing per the above since wait-for exists.  This will also be enhanced further in https://github.com/defenseunicorns/zarf/pull/1873


Simple Bar - Big Sur

Hello, I am trying to use simple-bar on Big Sur, and I believe an error is occurring. The top left part of the bar displays the following:

Hi!
I haven't tested simple-bar on Big Sur yet since I haven't installed it and I don't know if I'ill be able to do it before the official release.
I'm planning to switch on Big Sur as soon as it is launched so I will fix this at this time.
I'll let this issue open in the mean time.

Thank you for responding. Unfortunately, I do not have the skills to help. I appreciate you sharing this software!

Also getting some weird "unexpected token const" errors regarding Pywal.js on Big Sur. Otherwise, this is great — thanks for sharing this!

@Jean-Tinland have you tested Yabai and SimpleBar on Big Sur?  Not in any rush to upgrade till i sort all of my brew packages.

There has been some issue at start but they were fixed directly in yabai (details here).
I upgraded to Big Sur and with yabai and Übersicht up to date everything is working as expected.
I just need to replicate the Pywal error and fix it if necessary before closing this issue. :)

@lukebrb I couldn't reproduce your problem.
Does your Pywal.js file looks like something like this?
export const Pywal = {
  cursor: '#d1dbe0',
  number: '#d1dbe0',
  parens: '#BFC3B9',
  result: '#94A4A8',
  comment: '#92999c',
  matched: '#94A4A8',
  function: '#376089',
  operator: '#708C9F',
  variable: '#507495',
  scrollbar: '#708C9F',
  separator: '#708C9F',
  background: '#0a0e0d',
  editorbackground: '#0a0e0d'
}

@Jean-Tinland I finally did a fresh install of Big Sur, I'm also unable to reproduce the Pywal issue with simple-bar.

Thanks @MikoMagni for testing this, I'll close this issue. Feel free to open it again if necessary.

This happened to me. The problem seems to be that the pywal-gen.sh script is adding -n to the beginning of my Pywal.js file. Changing  echo -n to just echo seems to have resolved it. I am not sure why it's literally adding -n and not just doing what the flag is supposed to do.


eval: command not found

I have this error after the last FISH update (to 3.1).
done version 1.14.3
~/.config/fish/conf.d/done.fish (line 126):
                eval $__done_notification_command
                ^
in function “__done_ended”
	called on standard input

in event handler: handler for generic event “fish_prompt”

Does $__done_notification_command evaluate to anything?

@minzastro what is the result of
echo $__done_notification_command?

@minzastro at first sight this problem seems to specific to the command that you set in
$__done_notification_command, so this issue will be closed if you don't provide more information.

The problem was with the eval command itself. It disappeared after a reboot. Have no idea what it was, actually.


Verifying DNS over TLS and DNSSEC with stubby-docker

I'm playing around with ways to verify that DNS over TLS and DNSSEC are engaged when pumping my DNS traffic through stubby-docker. The easiest way to do this verification that I know if is to use a couple of test pages provided by Cloudflare.
https://<IP_ADDRESS>/help
https://www.cloudflare.com/ssl/encrypted-sni/
I get some interesting results from those two pages when playing around with different DNS options.
DNS via unbound via stubby (e.g., unmodified stubby-docker) serving up Cloudflare DNS.

<IP_ADDRESS>/help is unable to detect a connection to <IP_ADDRESS>.
<IP_ADDRESS>/help is unable to detect DNS over TLS.
Cloudflare SNI test is unable to detect DNS over TLS.
Cloudflare SNI test is able to detect DNSSEC.

DNS via stubby (manually exposed port, skipping unbound in stubby-docker) serving up Cloudflare DNS.

<IP_ADDRESS>/help is able to detect a connection to <IP_ADDRESS>.
<IP_ADDRESS>/help is able to detect DNS over TLS.
Cloudflare SNI test is able to detect DNS over TLS.
Cloudflare SNI test is unable to detect DNSSEC.

DNS via the cloudflared proxy (DNS over HTTPS) serving up Cloudflare DNS.

<IP_ADDRESS>/help is able to detect a connection to <IP_ADDRESS>.
<IP_ADDRESS>/help is able to detect DNS over TLS.
Cloudflare SNI test is able to detect DNS over TLS.
Cloudflare SNI test is able to detect DNSSEC.

I'm not surprised that a tool that Cloudflare provides (their proxy) passes all of their tests, but I am curious as to what causes unbound+stubby or solo stubby to fail some of the tests. I'm assuming some of the additional features enabled in unbound must interfere with whatever tests Cloudflare has created to check DNS setting.
How are you testing your setup to verify that you have the protection that you think you have?

Found a test for verifying that resolvers support qname minimization and DNSSEC.
https://forum.netgate.com/topic/135477/dns-over-tls-absolute-best-configuration-for-stubby
For qname.
drill txt qnamemintest.internet.nl

For DNSSEC.
dig txt qnamemintest.internet.nl +short

All this really does is confirm that Cloudflare supports qname minimization and DNSSEC. It doesn't help tease out the differences between unbound+stubby or solo stubby as both end up at Cloudflare.
I tried to use the kdig example from the Cloudflare documentation to test the TLS connection, but kdig on my Ubuntu server doesn't seem to support any of the TLS options.
https://developers.cloudflare.com/<IP_ADDRESS>/dns-over-tls/


WIP: Heroku Backend Deployments

/cib heroku backend deployments

Branch 37-heroku-backend-deployments created!


UserWarning: The Dask Engine for Modin is experimental.

UserWarning: The Dask Engine for Modin is experimental.
UserWarning: The Dask Engine for Modin is experimental.
UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 63533 instead
UserWarning: The Dask Engine for Modin is experimental.
UserWarning: The Dask Engine for Modin is experimental.
UserWarning: The Dask Engine for Modin is experimental.
UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 63552 instead
UserWarning: The Dask Engine for Modin is experimental.
UserWarning: The Dask Engine for Modin is experimental.
UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 63565 instead
UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 63566 instead
UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 63567 instead
UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 63568 instead
Task exception was never retrieved
future: <Task finished name='Task-9' coro=<_wrap_awaitable() done, defined at C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\asyncio\tasks.py:677> exception=RuntimeError('\n        An attempt has been made to start a new process before the\n        current process has finished its bootstrapping phase.\n\n        This probably means that you are not using fork to start your\n        child processes and you have forgotten to use the proper idiom\n        in the main module:\n\n            if name == 'main':\n                freeze_support()\n                ...\n\n        The "freeze_support()" line can be omitted if the program\n        is not going to be frozen to produce an executable.')>
Traceback (most recent call last):
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\asyncio\tasks.py", line 684, in _wrap_awaitable
return (yield from awaitable.await())
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\site-packages\distributed\core.py", line 290, in _
await self.start()
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\site-packages\distributed\nanny.py", line 295, in start
response = await self.instantiate()
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\site-packages\distributed\nanny.py", line 378, in instantiate
result = await self.process.start()
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\site-packages\distributed\nanny.py", line 575, in start
await self.process.start()
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\site-packages\distributed\process.py", line 34, in _call_and_set_future
res = func(*args, **kwargs)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\site-packages\distributed\process.py", line 202, in _start
process.start()
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\multiprocessing\context.py", line 326, in _Popen
return Popen(process_obj)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\multiprocessing\popen_spawn_win32.py", line 45, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python38\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

Hi @wanghuqiang123, this is a duplicate of #843, which is a Dask issue. Further questions or discussion can be directed there. Thanks!


useGoogleMap() returns undefined

import {
  GoogleMapsProvider,
  useGoogleMap,
} from "@ubilabs/google-maps-react-hooks";
import { useCallback, useEffect, useState } from "react";

export default function Example() {
  const map = useGoogleMap();
  const [mapContainer, setMapContainer] = useState<HTMLDivElement | null>(null);
  const mapRef = useCallback((node: HTMLDivElement) => {
    node && setMapContainer(node);
  }, []);
  const mapOptions = {
    center: { lat: 59.27240992362248, lng: 15.213029186736854 },
    zoom: 16,
  };
  useEffect(() => {
    console.log(map); //undefined
  }, [map]);
  return (
    <GoogleMapsProvider
      googleMapsAPIKey={process.env.NEXT_PUBLIC_GOOGLE_MAPS_API_KEY as string}
      mapContainer={mapContainer}
      mapOptions={mapOptions}
    >

    </GoogleMapsProvider>
  );
}

Simple hooks like this aren't even usable. I wouldn't recommend this library to anyone trying it out. Don't waste your time on this and start looking for alternatives - https://www.npmjs.com/package/@react-google-maps/api

@nytrek The hooks can only be used when the provider is initialized. And then only in the child components of said provider. So your example code is lacking the right setup. Please have a look at the example in the README on the repos' homepage, right where you started.
RTFM, thanks!

I find this package working fine in one of my React project with parcel bundler, but not working in a nextjs project, both having same react version:
"next": "^14.0.4",
    "react": "^18.2.0",


Solve Day 5

Complete both parts of AoC Day 5: https://adventofcode.com/2023/day/5

Part 1
Ok, so this problem (unlike prior problems) is actually posed in such a way that it makes sense to put the dataset directly into memory, so part of the problem is figuring out the most efficient way to put the dataset into memory. Let's assume for now that we're going to make 2 classes (Almanac and AlmanacMap). AlmanacMap will have the 3 properties (Source , Destination , Range ) as described in the problem. Almanac will have two properties (Seeds <int[]>, and Maps <AlmanacMap[][]>).
This is a pretty efficient data structure in terms of memory consumption, but let's see if it supports the requirements of the problem. We pretty much are forced into forward-only traversal, do to the possibility that an incoming Source value may not have a corresponding Map value (in which case the Destination is determined to be the same value as the incoming Source). Our AlmanacMap[][] array preserves the ordering of the maps list and obviously supports forward-traversal, so we're good there.
As far as taking an incoming Source value and determining the Destination value, that can be done with a simple formula: incomingSource - map.Source < map.Range. Just use that to determine if you've found a valid map, then do outgoingDestination = map.Destination + (incomingSource - map.Source). And of course, if you don't find a valid map, just outgoingDestination = incomingSource.
For finding matching maps, it feels like we could possibly gain something by ordering the maps by their Source properties. In particular, since we know that none of the maps overlap, we could probably implement some kind of binary search algorithm to reduce the amount of time it takes to locate a matching map (or determine that one doesn't exist). That would reduce our O(n^2) traversal to O(nlogn), as we'd have to do the sort first. HOWEVER, looking at the dataset, taking the time to sort the maps would probably take longer than skipping the sort and just doing the traversal (because there are only 20 seeds we need to evaluate). If we had many thousands of seeds to evaluate, sorting and binary searching would definitely come in handy.

Overlooked the fact that they snuck in some seed values that were greater than the int32 max value. I considered switching to uint, but decided to use long just to be safe.

Part 2
I know from the JS implementation that solving this problem brute-force style (checking each and every seed) is not the way to go. We've got to implement an algorithm which loops over ranges of seeds at a time. In addition to this, we could implement the previously-mentioned sorting and better search algorithms, but I happen to know that it isn't necessary for this implementation. The ranged looping method will bring the runtime down to a few milliseconds by itself.


pkg/chart/chart: add Manifests field for untemplated manifests

This commit adds an ability to skip rendering engine for certain
manfiests, when building the chart object programmatically.
This allows to completely skip templating part of helm, while still be
able to use hooks, versioning, upgrades etc. functionality.
Assembling charts programmatically is useful, for example when user wants
to apply post-templating changes to the chart, like kustomize, to avoid
modifying upstream helm chart, but still be able to use helm for
managing the installation.
While for many charts it is fine to be processed by templating engine
multiple times, for others, it is problematic.
The example here are charts, which use similar templating syntax in ConfigMap
objects, so they have to escape {{ }} templating syntax, which is
already non-trivial with helm. The example of such chart is
prometheus-opereator, which brings alerting rules, which also use {{ }}
syntax.
Given that the implementation is very simple, I decided to create a Pull Request directly rather than creating an issue first, to emphasize, that the actual modification needed is very small.
Please let me know what you think about it.
Signed-off-by: Mateusz Gozdek<EMAIL_ADDRESS>

@bacongobbler @marckhouzam could you take a look?

hi @invidian. Could of things pop out here.
As per the CONTIRBUTING guide, we ask users to please open an issue when opening a PR, regardless of its size. That way we can have a discussion before any code is written so that you're not wasting time writing code that would eventually be thrown out or not accepted. It also allows us time to coordinate reviews across multiple PRs that may be touching the same code, or trying to fix the same areas of the codebase to address conflicting issues which can be amalgamated into a simpler fix.
We also ask for unit tests with every PR. In some cases it may be difficult to test certain behaviour, but this PR looks fairly clear in scope for testing. It gives us an opportunity to ensure regressions aren't introduced in the future.
Without any test cases, I cannot verify how I am supposed to test and verify this PR, which is why there has not been a response on it for a while.
Hope this helps.

Thanks for the reply @bacongobbler. I created #8497 to discuss it then. It seems that the PR might not be needed, as in the meanwhile post-rendering functionality has been added to Helm and I may utilize that. If I can get my things to work without this patch, I'll close the PR.

okay. Let's close this for now and continue the discussion in #8497 before we come to a solution. Thanks.

@bacongobbler sorry for keeping it unanswered for so long. I created a draft PR with mentioned changes (https://github.com/kinvolk/lokomotive/pull/727), but I couldn't get it to work for some reason. I'll try to figure it out and then get back to #8497. Thanks!


Address some feedback from Ved: "Chapter 5 Some of the requirements a…

…re missing the RFC2119 keywords to indicate whether they are mandatory or optional."

(will add Ved here when he accepts collaborator role)


style: fix style warnings

fixed all style warnings and allows build to fail on future style warning.

lgtm


bug(tabs): customizable tokens background-color and foreground-color have no effect

Is this a regression?

[ ] Yes, this behavior used to work in the previous version

The previous version in which this bug was not present was
No response
Description
When trying to override tabs background and foreground colors
  @include mat.tabs-overrides(
    (
      background-color: red,
      foreground-color: white,
    )
  );

nothing happens.
Overridden colors take effect only when apply backgroundColor directive on tabs component.
Moreover after applying backgroundColor directive customizable tokens active-label-text-color and active-indicator-color aren't being applied anymore.
  @include mat.tabs-overrides(
    (
      background-color: red,
      foreground-color: white,
      active-label-text-color: green,
      active-indicator-color: green,
    )
  );

In this case using directive color on tabs component solve the problem.
Reproduction
StackBlitz link: https://stackblitz.com/edit/2xvcvpsg
Steps to reproduce:
It's self-explanatory
Expected Behavior
Customizable tokens should apply regardless directives backgroundColor and color
Actual Behavior
Customizable tokens are applied only when directives backgroundColor and color are used on tab component.
Environment
Angular CLI: 19.0.4
Node: 22.11.0
Package Manager: pnpm 9.13.2
OS: linux x64
Angular: 19.0.3
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... platform-server, router
Package                           Version
@angular-devkit/architect         0.1900.4 (cli-only)
@angular-devkit/build-angular     19.0.4
@angular-devkit/core              19.0.4
@angular-devkit/schematics        19.0.4
@angular/cdk                      19.0.2
@angular/cli                      19.0.4
@angular/material                 19.0.2
@angular/material-luxon-adapter   19.0.2
@angular/ssr                      19.0.4
@schematics/angular               19.0.4
rxjs                              7.8.1
typescript                        5.6.3
webpack                           5.97.1
zone.js                           0.15.0

Verified this is issue is occuring for the tab style override mixin

Overrides work if you add backgroundColor="primary" | "accent" | "warn" to the element.

.mat-mdc-tab .mdc-tab__ripple::before  has opacity:0 which is why values of token active-ripple-color is not reflecting.


thirdweb stamp on marketplace-v3 contracts

would be nice to do ascii or at least do thirdweb.com imo


Installation and Testing Improvements

Issues Found

Model Registration Mismatches

Architecture name mismatch in checkpoint: expected 'IPEC-decoder_dctform-rgb-share-size4'
Required adding correct registration decorator in JDEC.py:@register('IPEC-decoder_dctform-rgb-share-size4')  # Changed from 'jdec'
class JDEC(nn.Module):

Secondary mismatch with 'swinv2_rgb_nomore_deepir_rgbver' encoder model
Quick fix implemented by using existing 'swinv2_group_embedded' architecture which shares the same underlying structure

Environment Setup Issues

Missing directory structure for temporary files
Added missing bin directory creation for JPEG processing through gitkeep

Testing Performed
Tested on Ubuntu 20.04 environment
Successfully processed test images

@TheForgotten69 Thank you so much for working on this and discovering these errors in the repo!
I downloaded the fixed checkpoint file (thanks @WooKyoungHan !) and couldn't run it. But with your patches, it finally works!
I can confirm my list of available models is the same as yours:
Available models: dict_keys(['mlp', 'mlp_1dconv', 'swinv2_group_embedded', 'IPEC-decoder_dctform-rgb-share-size4'])
@WooKyoungHan Do you have any ideas why swinv2_rgb_nomore_deepir_rgbver does not exist for us but swinv2_group_embedded exists? I think we might be using the wrong model. I think nomore patches is a big part of JDEC's architecture, isn't it?
But at least, these patches allowed me to do some first testing with JDEC and I am really impressed. It totally removes the artifacts. I will be watching this pull request with big interest so we can figure out the final problems here! :)
If you have some local code that is not merged and fixes these problems, please be sure to upload it to GitHub in case this Pull Request is not necessary with your own local code. :)

I just did another quick check and see that swinv2_group_embedded comes from these code locations:
JDEC/models/swinirv2.py
901:@register('swinv2_group_embedded')

JDEC/configs/train_JDEC.yaml
37:      name: swinv2_group_embedded

So it seems like it may be the correct model name after all? So we don't need to find swinv2_rgb_nomore_deepir_rgbver? Please investigate @WooKyoungHan  :)

This is a problem that occurred when I uploaded it without updating it. This is a case in which an old version of the register name was applied. It can be solved simply as above, but it seems cleaner from previous experience to solve it at the register declaration stage. I will organize it well and upload it right away. I want to express my sincere gratitude to all of you who are interested in my thesis.

@WooKyoungHan Thank you, I am happy to hear this. And I agree, it's better to solve this naming problem at the @register because the swinv2_rgb_nomore_deepir_rgbver name is more accurate to be using.
I have tried a few quick tests of the model at q30 and q85 by the way. It's superb at removing JPEG artifacts, but I am a bit worried that it also removes a lot of high frequency details (like skin texture) at q85 so I am wondering if that's normal or something can be done to keep more high freq details?

Perhaps my method of solving the part is more of a temporary method than a fundamental problem. In this regard, the fundamental way to solve the problem is re-training with 'jdec,' which is the most appropriate method (I apologize for this and will consider re-learning within a few weeks).

Thanks for bringing up this issue.
These days, I am thinking a lot about the high-frequency details problem. This is because increasing the PSNR (reducing errors in per-pixel units) is not equivalent to increasing the high-frequency details (although it is highly correlated). Further, jdec is an INR-based network, which is known to be vulnerable to this problem (spectral bias). Although JDEC mitigated this using the method CCF, the unresolved spectral biases will cause unsatisfactory results.
So, how do we solve this problem? The fastest and most effective way would be to add the GAN method. Many studies have reported that the same method as the GAN works well in this respect. Also, GAN discriminators are known to 'like' high-frequencies (but in fact, discriminators are also difficult to judge high-frequency as fake). Although this may not be good in terms of restoring real images, it seems to be a better way to observe more 'fine' images in our eyes. This is going to be a question of which metric is better for us.
I'm thinking about how to restore high frequency through GAN-free methods. In most cases, the fact is that if we make the network huge, we'll have fewer errors in the per-pixel units and better details, so it's good to express high-frequency. But it's hard to say that this is methodologically efficient.
The kernel method is the one I thought about the most. Although it is necessary to consider whether the kernel method is effective in jpeg, which is an ill-posed problem, I think it would be fun to think about generating high-frequency random components and cut them well with the kernel to make details.

Perhaps my method of solving the part is more of a temporary method than a fundamental problem. In this regard, the fundamental way to solve the problem is re-training with 'jdec,' which is the most appropriate method (I apologize for this and will consider re-learning within a few weeks).

Ahh. Is there any quality-problem with the current checkpoint file, or is it only "wrong" "layer name"?
If you only want to change the layer name to something that's "nicer sounding" ("jdec"), maybe you can easily rename the layer inside the checkpoint file without retraining it?

Well, I see that the paper says:

Implementation Detail
We use 112 × 112 patches as inputs to our network. This size is chosen because it is the
least common multiple of the minimum unit size of color
JPEG (16 × 16) and the window size of our Swin architecture [27] (7 × 7). The network is trained for 1000 epochs
with batch size 16. We optimize our network by Adam [20].
The learning rate is initialized as 1e-4 and decayed by factor
0.5 at [200, 400, 600, 800].

So it seems like it can use different input sizes. But that the JPEG files dimensions must be multiples of 112?
I hope you can clarify whether this network can:

Use arbitrary input dimensions which are not multiple of 112, such as 707 x 1231? In other words, real-world JPEGs.
Can use large input dimensions like 6k x 6k?

@WooKyoungHan What do you think of this PR? It's still worth to be merge imo as the current code won't work as is.


Separate tests and package

Fix the fact that using dotfile prevents the package to be installed in a texmf directory. require is used instead (source).
Also,

Move every test files into a tests directory, together with the Makefile ;
Rewrite the Makefile ;
Separate the package from the files in the Travis test, to simulate a real installation.

It seems OK.

Yup, let's go :)


Fix: Read XDG_DATA_HOME first

Previously the program read XDG_DATA_HOME after reading XDG_DATA_DIRS,
ignoring user's custom desktop entry files.

Yes, that's right, XDG_DATA_HOME should have the priority. Thanks for the fix, merged.


EchoBot throws exceptions during the initialization and terminates the call on joining.

We try to run the EchoBot example (GitHub: microsoftgraph/microsoft-graph-comms-samples/Samples/PublicSamples/EchoBot/).
We have a Microsoft 365 Developer sandbox as a part of Visual Studio subscription, and a tenant in the sandbox.
We use Postman to hit the bot.
Versioning:
- We use Windows 11, Visual Studio 2022 Professional, C#, and .NET version 6.0.
- README in the EchoBot example recommends .NET 4.7.1, but it looks like it's just wrong.
(The project file for the EchoBot example contains net6.0).
- We downloaded .NET 4.7.1 SDK from the NET web site, installed it, switched to .NET Framework 4.7.1,
did the NuGet clean and restore. Does not compile.
Ngrok:
- We have Ngrok (free version) running and providing two endpoints locally, one for HTTPS, and one for TCP.
Certificates:
- We created a self-signed certificate for the TCP domain and installed it in the local certificate store for
both User / Personal and Local Machine / Personal.
- The certificate is being picked up by the bot (without the certificate installed, the bot does not start at all).
Graph Application and Azure Bot registrations:
- The application is registered in MS Entra for our sandbox tenant, all the permissions are granted and consented.
- The Azure Bot is also registered in the Azure portal.
- We also added the Microsoft Teams Channel to the bot channels in the bot registration, enabled calling in the Calling tab,
and added the correct domain name to Webhook (for calling) field: https://.ngrok-free.app/api/calling
Local environment:
- The application / client ID and the application / client secret are added to the .env file in the bot project.
- The Ngrok endpoints and ports are added to the .env file.
- The certificate thumbprint is added to the .env file.
When we run the bot from Visual Studio in the debug mode, during the bot initialization, the following 12 different types of exceptions are logged:
1. onecore\net\netprofiles\service\src\nsp\dll\namespaceserviceprovider.cpp(613)\nlansp_c.dll!00007FFF45F1F6BD:
(caller: 00007FFF619FACF6) LogHr(1) tid(59e8) 8007277C No such service is known. The service cannot be found in the specified name space. (3)
2. Microsoft C++ exception: std::system_error at memory location 0x00000050203BA850.
3. Exception thrown at 0x00007FFF5FE1FABC (KernelBase.dll) in EchoBot.exe: WinRT originate error - 0x80040155 :
'Failed to find proxy registration for IID: {79EAC9E4-BAF9-11CE-8C82-00AA004BA90B}.'.
4. Exception thrown at 0x00007FFF5FE1FABC (KernelBase.dll) in EchoBot.exe: 0x80040155: Interface not registered. (2)
5. MediaPerf is not registered: no key found at SYSTEM\CurrentControlSet\Services\MediaPerf\Performance
6. Exception thrown at 0x00007FFF5FE1FABC (KernelBase.dll) in EchoBot.exe: 0x000006D9: There are no more endpoints available from the endpoint mapper.
7. Exception thrown at 0x00007FFF5FE1FABC (KernelBase.dll) in EchoBot.exe: 0x000006BA: The RPC server is unavailable. (3)
8. 'System.InvalidOperationException' in System.Diagnostics.PerformanceCounter.dll: Category does not exist.
9. 'System.IO.IOException' in System.Net.Sockets.dll, System.Net.Security.dll, System.Private.CoreLib.dll
10. 'System.InvalidOperationException' in Unity.Container.dll.
No public constructor is available for type Microsoft.Extensions.Options.IPostConfigureOptions`1[Microsoft.Extensions.Logging.LoggerFilterOptions]
(and for other interfaces, about 60 of them).
Inner Exception: InvalidRegistrationException: Exception of type 'Unity.Exceptions.InvalidRegistrationException' was thrown.
11. System.IO.IOException in System.Net.Security.dll, System.Private.CoreLib.dll. Received an unexpected EOF or 0 bytes from the transport stream. (5)
12. System.IO.IOException: 'Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..'
But despite of all the exceptions, the bot starts:
Microsoft.Hosting.Lifetime: Information: Now listening on: http://[::]:9442
Microsoft.Extensions.Hosting.Internal.Host: Debug: Hosting started.
And the GET health check request from Postman hits Ngrok, hits the bot, and returns 200 OK to the Postman.
During the bot initialization, the BotMediaStream is created, the audio socket is initialized,
the bot sends status active for media, and the media player is created.
Testing;
- We start the bot in Visual Studio (as administrator) locally.
- We start the Teams client locally, in our sandbox tenant.
- We start a private scheduled meeting in Teams (Calendar -> Schedule -> Join Meeting).
- We use the meeting JoinURL for the bot to join the meeting call from Postman with a JSON POST request
{ "joinURL": <JOIN_URL> }.
- It hits Ngrok, hits the bot, the bot returns 200 OK and reports that it joined the call:
CallController: Joined the call, call ID: 1e005980-8e2c-4831..., URL: https://teams.microsoft.com/l/meetup-join/19%3Ameeting_NjZh...
- The bot also receives two notifications via api/calling/notification endpoint after the join call request and replies with 202 Accepted:
PlatformCallController: Notification result: StatusCode: 202, ReasonPhrase: 'Accepted', Version: 1.1, Content: System.Net.Http.EmptyContent.
- We monitor the wire in WireShark for the media TCP ports.
The problem:
It tries to establish the call:
17:01:28:954 CallHandler: Call status updated to Establishing.
And then immediately terminates it:
17:01:29:954 CallHandler: Call status updated to Terminated - Server Internal Error. DiagCode<EMAIL_ADDRESS>And then throws the following exceptions:
Exception thrown: 'System.Threading.Tasks.TaskCanceledException' in System.Private.CoreLib.dll
A task was canceled.
Exception thrown: 'System.IO.IOException' in System.Net.Sockets.dll
Unable to read data from the transport connection:
The I/O operation has been aborted because of either a thread exit or an application request.
THE BOT DOES NOT APPEAR IN THE MEETING, and NO MEDIA IS SENT TO THE BOT.
Questions:
1. All the exceptions are triggered deeply in the native libraries. Can it mean that there are some versioning issues?
2. What can cause the exceptions in the bot during the initialization? Are those exceptions significant?
3. Can those exceptions cause that the bot is not joining the call and also the media stream issues?
4. Is it OK to use a self-signed certificate?
5. What would you recommend to further debug the issue?
Thank you!

I'll start with question 4 it is not ok to use a self signed certificate. Regarding the other questions probably they resolve themself if you use a certificate with a certificate to the root chain. Maybe as tip with ngrok try to get a connection to 0.tcp.-endpoint of ngrok by restarting on the other endpoints there sometimes seem to be problems with the firewall settings from ngrok.

Thank you for the advice! We'll try a root chain certificate.
Our Ngrok is configured to generate the following two endpoints:
TCP:  tcp://.tcp.ngrok.io: -> localhost:8445, N = 0,2,4.6,8, PORT is a random int usually between 10000 and 30000.
HTTP: https://.ngrok-free.app -> http://localhost:9442, DOMAIN is a random string like d7b9-63-209-137-19
We generate the certificates for domain .tcp.ngrok.io, the TCP connection domain, no wildcard *. Is that correct?
Thank you!

I think your tenant situation is not a problem as long as you gave admin permission for the teams client.
Regarding the certificate for the tcp ngrok address, as it might be rather complicated to get a valid certificate for the ngrok domain you can use a cname on a custom domain that then points to e.g. 0.tcp.ngrok.io and get the certificate for your custom domain(this implies using the custom domain instead of the ngrok domain in the bot)

Thank you for your help! We'll try it.
A deployment question. Does it have to be deployed in Azure VMSS? It it possible to deploy it on-premises on a Windows Server in Kubernetes?

In our case, we'll have to redirect the media stream to our on-premises platform for processing and analysis anyway, so the distance is not very important. Anyway, thank you for the advice, we'll consider it.

BTW, is it possible to use HTTPS for the media stream instead of TCP? Web Sockets, for example?

No, it seems like the media platform is somewhere based on WCF nowadays core WCF so you can not replace the TCP endpoint with websockets, hopefully Microsoft will refactor the media platform one day and pay some technical debt and remove the dependency to WCF.

Finally, we have our root chain certificate issued and the DNS cname records added.
It works now (in the non-debug mode).
Thank you very much for your help!
In the debug mode it triggers the same exceptions.
Is it supposed to work in the debug mode?
It would provide a lot of useful information...
Does the audio stream provide the participant diarization?
Thank you!

Our real-time media bot throws the following exception. Could anybody help us to explain what we do wrong?
The MS servers sometimes forcibly closes connection. (The bot is able to join a Teams meeting and it receives the audio.)
Thank you!
[24-12-04 17:02:56.5681] fail: PTBotService.Bot.MediaLogger[0]
[AvMP][AppId:63b1acf6-88cb-4b73-a92a-e9eb26872450] TL_ERROR(TF_COMPONENT)
30684.122::12/04/2024-17:02:56.567.00000009 (MPAZAPPHOST,LogError:AzAppMPHostLogger.cs(39))
[EventManager] Sending of notification message failed to Uri :
https://002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701/in_7/media/v2/mpproxy/1/mpcontext/9c869ec7-5161-4b0c-9ed5-67e79cf96eeb/mpEventPostback
for HttpMethod: POST
with exception : System.AggregateException: One or more errors occurred. (The SSL connection could not be established, see inner exception.)
(The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.)  --->
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.  --->
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..  --->
System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Security.SslStream.g__InternalFillHandshakeBufferAsync|189_0[TIOAdapter](TIOAdapter adap, ValueTask1 task, Int32 minSize)     	at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)     	at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)     	at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)     --- End of inner exception stack trace ---     	at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)     	at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)     	at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)     	at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)     	at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAndLogAsync(HttpRequestMessage request, CancellationToken cancellationToken, String msgId, String localMsgId)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.MessagesIdentificationMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.TaskExtensions.RunTaskFunc[T](Func1 taskFunc, CancellationTokenSource cts, TaskCompletionSource1 tcs, Clock clock, ILogSource log, String taskDescription)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.AsyncRetryMessageHandler.<>c__DisplayClass7_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.Rtc.MPService.Internal.ExternalLibs.RetryUtil.RunWithRetries[T](IAsyncRetryPolicy1 policy, Func1 retryableMethod, Clock clock, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Microsoft.Rtc.MPService.Internal.ExternalLibs.RetryUtil.RunWithRetries[T](IAsyncRetryPolicy1 policy, Func1 retryableMethod, Clock clock, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.ContextForwardingMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationTokenoriginalCancellationToken)
at Microsoft.Rtc.Internal.MediaProcessor.WebAPI.MtlsHttpRequestSender.InternalSendAsync[T](T message, Uri requestUri, HttpMethod httpMethod, Uri implicitCallbackLink, String contentType, String authToken)
at Microsoft.Rtc.Internal.MediaProcessor.WebAPI.MtlsHttpRequestSender.SendAsync[T](T message, Uri requestUri, String authToken, HttpMethod httpMethod)
at Microsoft.Rtc.Internal.MediaProcessor.WebAPI.EventManager.SendNotificationAsync[T](T notificationMessage, Uri requestUri, HttpMethod httpMethod, Boolean sendNotificationUsingCompositeTransport, Uri implicitCallbackLink, String authToken)  --->
(Inner Exception #1) System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.  --->
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..  --->
System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Security.SslStream.g__InternalFillHandshakeBufferAsync|189_0[TIOAdapter](TIOAdapter adap, ValueTask1 task, Int32 minSize)     	at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)    at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)     	at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)     --- End of inner exception stack trace ---     	at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)     	at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)     	at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)     	at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)    at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAndLogAsync(HttpRequestMessage request, CancellationToken cancellationToken, String msgId, String localMsgId)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.MessagesIdentificationMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.TaskExtensions.RunTaskFunc[T](Func1 taskFunc, CancellationTokenSource cts, TaskCompletionSource1 tcs, Clock clock, ILogSource log, String taskDescription)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.AsyncRetryMessageHandler.<>c__DisplayClass7_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.Rtc.MPService.Internal.ExternalLibs.RetryUtil.RunWithRetries[T](IAsyncRetryPolicy1 policy, Func1 retryableMethod, Clock clock, CancellationToken cancellationToken)<---   --->
(Inner Exception #2) System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.  --->
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..  --->
System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Security.SslStream.g__InternalFillHandshakeBufferAsync|189_0[TIOAdapter](TIOAdapter adap, ValueTask1 task, Int32 minSize)     	at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)     	at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)     	at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)     --- End of inner exception stack trace ---     	at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)     	at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)     	at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)     	at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)     	at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAndLogAsync(HttpRequestMessage request, CancellationToken cancellationToken, String msgId, String localMsgId)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.LoggingMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.MessagesIdentificationMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Rtc.MPService.Internal.TaskExtensions.RunTaskFunc[T](Func1 taskFunc, CancellationTokenSource cts, TaskCompletionSource1 tcs, Clock clock, ILogSource log, String taskDescription)
at Microsoft.Rtc.MPService.Internal.ExternalLibs.AsyncRetryMessageHandler.<>c__DisplayClass7_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.Rtc.MPService.Internal.ExternalLibs.RetryUtil.RunWithRetries[T](IAsyncRetryPolicy1 policy, Func1 retryableMethod, Clock clock, CancellationToken cancellationToken)<---

@genemgh a few questions:

What nuget version of Microsoft.Skype.Bots.Media are you running?
What target framework (is it net framework or net6.0?)
What version of TLS / SSL are you configured with on your machine?

Hi @adityaramgopal !
Answering you questions:

Microsoft.Skype.Bots.Media version: <IP_ADDRESS>.alpha
It’s Net 6.0.
TLS 1.2 and TLS 1.3 are enabled on my machine (Windows 11)
Thank you!

@genemgh do you have any firewall rules on outbound traffic?

@adityaramgopal
In fact, I do. Our IT team manages them.
Which port should be open to communicate with the MS media server?
The media we need from a Teams meeting comes through to our local bot, so it looks like it does not happen every time...
Firewall would block all the media. Is that correct?

@genemgh can you temporarily disable all outbound firewall rules and see if the exception is thrown?

I'm not supposed to do it. And it's a security risk.

without eliminating that as the reason it's going to be hard to figure out the root cause.

It's an HTTPS request which goes through port 443, which is always open. So, I don't think it's firewall.
And usually there are better ways to debug than disabling your firewall :-) WireShark?
Thank you anyway.

Are you saying that you allow all outbound traffic to port 443 and don't constrain them to specific IP addresses?

The media we need from a Teams meeting comes through to our local bot, so it looks like it does not happen every time...

Also, can you clarify what you mean by this comment. This issue reproduces sometimes but not always? Or does it reproduce in a specific machine (say local) but not in another machine?

What I mean is that I'm not sure how often that http request fails (on one single machine). And I'm not sure how this request is related to the media stream. But we are getting the media stream, so it means that the request is either not critical or does not fail every time. Do you know what that request does and how it's related to the media stream?

To my understanding, if an outbound firewall rule blocks an outgoing request, the request won't be sent.
In our case, it looks like it's reaching the remote http server, and the remote host closes the connection.
So, it does not look like a firewall issue.

@genemgh  can you share a sample call id of a successful call vs a failing call? Might help us debug further. Maybe a wireshark trace of a failing call might also help

In order to reproduce it, we tried to hit the same URL directly from Curl in the verbose mode (both from Windows and WLS):
Windows> curl -vv -d '{}' -H "Content-Type: application/json" https://002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701/in_7/media/v2/mpproxy/1/mpcontext/9c869ec7-5161-4b0c-9ed5-67e79cf96eeb/mpEventPostback

Host 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701 was resolved.
IPv6: (none)
IPv4: <IP_ADDRESS>
Trying <IP_ADDRESS>:10701...
Connected to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com (<IP_ADDRESS>) port 10701
schannel: disabled automatic use of client certificate
ALPN: curl offers http/1.1
Recv failure: Connection was reset
schannel: failed to receive handshake, SSL/TLS connection failed
closing connection #0
curl: (35) Recv failure: Connection was reset

WSL Ubuntu:~$ curl -vv -d '{}' -H "Content-Type: application/json" https://002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701/in_7/media/v2/mpproxy/1/mpcontext/9c869ec7-5161-4b0c-9ed5-67e79cf96eeb/mpEventPostback

Trying <IP_ADDRESS>:10701...
Connected to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com (<IP_ADDRESS>) port 10701 (#0)
ALPN, offering h2
ALPN, offering http/1.1
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
TLSv1.0 (OUT), TLS header, Certificate Status (22):
TLSv1.3 (OUT), TLS handshake, Client hello (1):
OpenSSL SSL_connect: Connection reset by peer in connection to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701
Closing connection 0
TLSv1.0 (OUT), TLS header, Unknown (21):
TLSv1.3 (OUT), TLS alert, decode error (562):
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701

WSL Ubuntu:~$ curl -k -vv -d '{}' -H "Content-Type: application/json" https://002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701/in_7/media/v2/mpproxy/1/mpcontext/9c869ec7-5161-4b0c-9ed5-67e79cf96eeb/mpEventPostback

Trying <IP_ADDRESS>:10701...
Connected to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com (<IP_ADDRESS>) port 10701 (#0)
ALPN, offering h2
ALPN, offering http/1.1
TLSv1.0 (OUT), TLS header, Certificate Status (22):
TLSv1.3 (OUT), TLS handshake, Client hello (1):
OpenSSL SSL_connect: Connection reset by peer in connection to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701
Closing connection 0
TLSv1.0 (OUT), TLS header, Unknown (21):
TLSv1.3 (OUT), TLS alert, decode error (562):
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 002-usea.noam.prd.api.cos.mediapaas.infra.teams.microsoft.com:10701

WSL Ubuntu:~$ telnet <IP_ADDRESS> 10701

Trying <IP_ADDRESS>...
Connected to <IP_ADDRESS>.
Escape character is '^]'.
Connection closed by foreign host.


Clean-up dockerfiles

Cleaning up docker files which expose mcr references.
Implement the suggested work around which involves pulling the image desired and re-tagging locally.

@joe-braley @karianna now the Dockerfile can't be built by our users. And our users can't tell what are the base images of our images.

The resulting image in MCR would have the information no?  Or did I misread what this PR does...

This PR removes default values from the Dockerfiles.
And as for the docker image history command, I am quite sure it won't show the name of the base image.

As for hiding references to MCR images, why is this a problem?


SCP-1823 untyped evaluation in plc-agda

So far I have reinstated untyped evaluation in plc-agda. It takes a typed term, erases it, reduces it and then prints it out using my 'ugly printer'.
This is ok for testing against other plc-agda modes but to test against the plutus-core evaluator will require reusing plutus-core untyped machinery.

Pre-submit checklist:

Branch

[ ] Commit sequence broadly makes sense
[ ] Key commits have useful messages
[ ] Relevant tickets are mentioned in commit messages

PR

[ ] Self-reviewed the diff
[ ] Useful pull request description
[ ] Reviewer requested

If you updated any cabal files or added Haskell packages:

[ ] nix-shell shell.nix --run updateMaterialized to update the materialized Nix files
[ ] Update hie-*.yaml files if needed

If you changed any Haskell files:

[ ] nix-shell shell.nix --run fix-stylish-haskell to fix any formatting issues

If you changed any Purescript files:

[ ] nix-shell shell.nix --run fix-purty to fix any formatting issues

Pre-merge checklist:

[ ] Someone approved it
[ ] Commits have useful messages
[ ] Review clarifications made it into the code
[ ] History is moderately tidy; or going to squash-merge

This works with the plc examples. The way an error is returned is slightly different though.

hydra timed out on the cost-model. git commit --amend; git push -f.


Media Uploads for scheduled statuses

Models for handling media uploads that will be attached to scheduled status updates.

Needs to be able to handle images and video.
Needs to generate thumbnails
Needs to be able to validate in advance if an upload is accepted by the target service.
Needs to handle the fact that media will be uploaded to multiple destinations, sometimes before the status itself is posted.
Needs cleanup scripts to discard original media on a schedule.
Needs to cleanly support remote storage through something like Django-storages, etc.

I originally wanted to have a single upload model, but the challenges involved in doing that seem like that's me trying to be too clever.
Two options:

Have separate models based on the type of media upload so that we don't lose existing conveniences for things like ImageField.
Create a new Django model field type that can determine whether the content is image or video and extract properties without having to fully retrieve it from storage again.
Create a single model and then use proxy models with distinct functions/properties for the media type.

Currently leaning towards option one, but that feels ugly to me.

Also have to decide how we want to track where the files are already published. For example, to schedule a post in Mastodon, you need to upload the media and submit the post with the scheduled time in advance.
For Twitter, due to API limitations, you need to wait to upload the media and post the status until publish time.
For Instagram, if we do that, you need to store it locally until publish time, and (if not video content) needs to be a curl-able square jpeg exposed publicly.