Detecting AU_MEDICARE through Patterns, Context, and Checksum

An Australian Medicare number is a type of Personal Identification Information (PII) used in the Australian Medicare system. With this number, you can identify if someone has opted into the system, gain access to other insurance plans, and ensure Medicare pays first on the bill. Without this number, one would lose access to their insurance and have to pay medical bills entirely with their own money. If someone steals the number, they can rack up charges with insurance providers under someone else's name. If someone's medicare number is breached, it puts their insurance and savings at risk, making it very important to keep protected.

Is there any authentication barriers for this PII?

Yes, there will be multifactor authentication.


Last is overriding sort

find.sort(:created_at => 1).last will override sort if already set

Nothing to do about it, mongodb doesn't have last


[10.0.x] ISPN-10852 Upgrade to protostream-4.3.0.Final

https://issues.jboss.org/browse/ISPN-10852

@gustavonalle I've mistakenly made this PR on top of master instead of 10.x. I'll cherrypick the commit myself. Thanks!


Pin Docker :latest image tags to digests

The :latest tag changes, so future pulls of this image may retrieve a different image with different (and possibly erroneous, unexpected, or dangerous) behavior. Pin the image to an image digest for deterministic behavior. See also: Hadolint error DL3007.
Created by Sourcegraph batch change christine/pin-docker-images-0723.

Please close out asap


the size of feature maps before deform conv

I want to use deform conv in a single classification network, but i don't know if the size of feature map(7x7)
before deform conv is suitable or not. Could anyone help?

@hjy1312  I trained a classification network using mdcn with offset and mask in stage5 of a normal Resnet50.  But I view the offset and mask the model have learned，its rediculous.. I cannot get meaningful offset and mask through training.
What about your result?

@zhanglonghao1992 Actually it got worse after i used the deform conv in the last 4 conv of my resnet34 network. I suppose the reason could be that my feature map is too small.

@hjy1312  I tried 1414 and 77 but couldn't get a well-trained network.


Add deecting heap allocations / boxing in code

Issue Description
Is it possible to currently configure omnisharp or add functionality to have similar results like this resharper extension https://github.com/controlflow/resharper-heapview ? I know a bit of C# and JS so I could help with adding this, but I'd need some tips on where to start adding it in roslyn/omnisharp codebase.

You need to ensure you have Roslyn Analyzers enabled in OmniSharp, by adding the following to your omnisharp.json file
{
    "RoslynExtensionsOptions": {
        "enableAnalyzersSupport": true,
    }
}


Removes duplicate line

Just removes a duplicated line in the channels section of the amqp concepts page.
Also thanks very much for a very good tutorial! It was a good read, and helped me learn.
Cheers
// Sebastian

@cruzzan Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

Thank you!

@cruzzan Thank you for signing the Contributor License Agreement!


Why is total_iters unrelated to batch size and number of data?

Hi, I have been using this code to train a custom dataset recently. I noticed that in train.py,  total_iters is definded as total_iters = cfg.optim["epochs"]* OFFCIAL_EPOCH_LENGTH. This is confusing for me. As to my knowledge, total_iters should be equal to epoch * number of training data / batch size.
Also, when training with multiple GPUs, I observed that each GPU will run total_iters instead of total_iters / number of GPUs. It also confuses me.
Is anyone know what is the purpose of this design?
In addition, if I want to train my custom dataset with the iters which recognized by the public, what should I do? Just set the total_iters as  epoch * number of training data / batch size? Is there any other setting I need to modify? Thanks!

Hello, I'm also confused on a related problem. In the ssl_default_config.yaml, the params are batch_size_per_gpu: 64 and OFFICIAL_EPOCH_LENGTH: 1250. And in the README, it says Run DINOv2 training on 4 A100-80GB nodes (32 GPUs) in a SLURM cluster environment with submitit.
So, the global batch size is 32x64=2048, and times the OFFICIAL_EPOCH_LENGTH, it is 2560000, which seems about twice on the number of ImageNet. Could you please help check this? Thanks.

I have the same question，the training time is not related with GPU number in the release code


improve documentation about dynamic expressions for import()

This is working as intended. There is no technical problem with import(expr) and this used to work in I think webpack 1 and 2 or so, but we disabled this behavior as this is in 99% not the behavior you want.  So we only handle partially dynamic imports by default.
But hey if you want to enable webpack handling unknown expressions too set: module.unknownContextRegExp: /^\.\/.*$/
Originally posted by @sokra in https://github.com/webpack/webpack/issues/4292#issuecomment-671580420

I think it will be great to docs

I'll see if I can document it.

Unfortunately I can't make the option work as I expected, if anyone are interested in documenting it, please go ahead.
And here's the related section https://webpack.js.org/api/module-methods/#dynamic-expressions-in-import


error KAPT version 2.5.2

java.lang.Exception: Error while processing annotations: java.lang.IllegalArgumentException: NavGraph [main] has no start route specified
use for initialize navgraphSafeRouteNavHost

sorry im just missing config,
navGraph = RouteNavGraph(name = "nameYourNavGraph", start = true)


New/Old/Revert "feature: add old password field to profile page" Pull Request

Needed to revert, because I merged with the main repo and all changes of my code were gone.
This reverts commit f1084655395bc5a0a0bf10d9927255147eb0c41f.
And is also the requested pull request you wanted from me @vishalverma9572

Thanks! @TommiNICE


get_dependency

Would it be possible to add a get_dependency method to return the raven client? For times when you want to send a non exception message.
client.captureMessage('Somebody did something sorta strange.')

I understand the desire here. I wonder if a better way to do it would be to use a SentryHander and the logging framework.
I've just raised a PR against nameko https://github.com/onefinestay/nameko/pull/313 that loads LOGGING config on service startup.
The following config.yaml configures nameko_sentry and adds a logging SentryHandler at the same time.
SENTRY:
    DSN: &sentry_dsn
<EMAIL_ADDRESS>    CLIENT_CONFIG: &sentry_client_config
        site: 'Test Site'

AMQP_URI: amqp://guest:guest@localhost/
LOGGING:
    version: 1
    disable_existing_loggers: false
    formatters:
        simple:
            format: "%(levelname)s:%(name)s - %(message)s"
    handlers:
        console:
            class: logging.StreamHandler
            formatter: simple
        sentry:
            <<: *sentry_client_config
            dsn: *sentry_dsn
            class: raven.handlers.logging.SentryHandler
        sentry_restricted:
            <<: *sentry_client_config
            dsn: *sentry_dsn
            level: WARNING  # restricted to WARNING and above
            class: raven.handlers.logging.SentryHandler
    loggers:
        sentry:
            level: INFO
            handlers: [sentry]
            propagate: False
    root:
        level: INFO
        handlers: [console,sentry_restricted]

With this particular config, warnings and above in any logger go to sentry, and there's a specific sentry logger will sends info level and above, e.g.
import logging

logging.getLogger("app").info("logs to console only")
logging.getLogger("sentry").info("logs to sentry only")
logging.getLogger("nameko").warn("logs to console and sentry")

The only snag with the above is that the built-in raven.handlers.logging.SentryHandler is incompatible with eventlet. It works but you get a traceback on teardown. This can be fixed by subclassing the built-in one to force the handler to use the eventlet http transport:
from raven.handlers.logging import SentryHandler as RavenSentryHandler
from raven.transport import EventletHTTPTransport

class SentryHandler(RavenSentryHandler):
    def __init__(self, *args, **kwargs):
        kwargs['transport'] = EventletHTTPTransport
        super(SentryHandler, self).__init__(*args, **kwargs)

If this is a useful solve for your problem we could add the above subclass into nameko_sentry.

Using the logging module seems like a more natural approach.
There is one other case for having direct access to the raven client. Typically during an exception or message I'll merge is the user context like so...
self.client.context.merge({'user': player_id})
Which then populates the user detail in sentry. This could probably be handle by exposing a method in the dependency to merge context.

Using context.merge before sending an exception doesn't seem quite right. The client object is shared so if you have exceptions in multiple concurrent workers there's no guarantee that the merge will happen at the correct time.
It would be better to pass the user information when you call captureException. As it stands that requires a small modification to nameko-sentry, or a subclass. Using the logging handler though, it looks like you can just provide it using the extra kwarg.
As an aside, are you keeping your user_id in the data attribute of the worker_ctx? There's a general purpose solution whereby all information in the "context data" gets passed into the sentry client.

This is addressed by #7.
As of nameko 2.5.1, all worker lifecycle methods happen within the same greenthread, meaning that raven's thread locals keep worker contexts isolated. As a result we can safely expose the client to the worker.

I'll write up some docs and cut a release soon.


cabal test takes ages to complete and uses only 2 cores

When running cabal test on my 8x Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz with 32 GB RAM I had to cancel it after some 18 minutes:
$ cabal test
Build profile: -w ghc-9.2.2 -O1
In order, the following will be built (use -v for more details):
 - horde-ad-<IP_ADDRESS> (lib) (configuration changed)
 - horde-ad-<IP_ADDRESS> (lib:testLibrary) (configuration changed)
 - horde-ad-<IP_ADDRESS> (test:shortTestForCI) (configuration changed)
 - horde-ad-<IP_ADDRESS> (test:extremelyLongTest) (dependency rebuilt)
Configuring library for horde-ad-<IP_ADDRESS>..
Preprocessing library for horde-ad-<IP_ADDRESS>..
Building library for horde-ad-<IP_ADDRESS>..
Configuring library 'testLibrary' for horde-ad-<IP_ADDRESS>..
Preprocessing library 'testLibrary' for horde-ad-<IP_ADDRESS>..
Building library 'testLibrary' for horde-ad-<IP_ADDRESS>..
Configuring test suite 'shortTestForCI' for horde-ad-<IP_ADDRESS>..
Preprocessing test suite 'extremelyLongTest' for horde-ad-<IP_ADDRESS>..
Building test suite 'extremelyLongTest' for horde-ad-<IP_ADDRESS>..
[...]
Preprocessing test suite 'shortTestForCI' for horde-ad-<IP_ADDRESS>..
Building test suite 'shortTestForCI' for horde-ad-<IP_ADDRESS>..
Running 1 test suites...
Test suite shortTestForCI: RUNNING...
Running 1 test suites...
Test suite extremelyLongTest: RUNNING...
^C

More importantly, this test suite appears to utilize just 2 CPU cores. It turns my laptop into a small cooker, though: +96 °C aka +204.8 °F (CPU temp.).

Right. I think the 2 cores come from running two test suites. I think there are ways to parallelize each of the suites. Anyway, if your computer is not up to the task, please disregard the extremelyLongTest.

BTW, perhaps cleaning the fan might do the trick?
The tests really are computation-intensive: the computer learns to recognize and then recognizes man thousands of digits from pictures, using many different methods. With a short test you wouldn't be able to tell if the neural network learned anything.


Fixes #2120 Add cursor toggle to TreeViewFileSystem Scenario

Fixes #2120- Add cursor toggle to TreeViewFileSystem Scenario
@tig regarding issue #2120, there is already a feature for this (DesiredCursorVisibility).  The default state of this is Default (i.e. it has a cursor).  This PR adds a menu item to the scenario that lets user turn cursor on and off.
If you would like the default to be Invisible let me know and I can update this PR.
Pull Request checklist:

[x] I've named my PR in the form of "Fixes #issue. Terse description."
[x] My code follows the style guidelines of Terminal.Gui - if you use Visual Studio, hit CTRL-K-D to automatically reformat your files before committing.
[x] My code follows the Terminal.Gui library design guidelines
[x] I ran dotnet test before commit
[x] I have made corresponding changes to the API documentation (using /// style comments)
[x] My changes generate no new warnings
[x] I have checked my code and corrected any poor grammar or misspellings
[x] I conducted basic QA to assure all features are working

What is the customer scenario where a cursor is useful on Treeview?

What is the customer scenario where a cursor is useful on Treeview?

Primary use case is during multi selection to determine where the active row is.  Even when multiple rows are selected there is still a 'primary' row which is the one tracked by SelectedObject and the one which the selection will be adjusted from if user uses Shift up/down.
You can see a similar effect in Visual Studio Code, but in this case the active row has a bright border.

I'm happy to go either way on this though.  Setting the default to Invisible could be a good change for new users.  The setting was originally added in response to #1559

What is the customer scenario where a cursor is useful on Treeview?

Primary use case is during multi selection to determine where the active row is. Even when multiple rows are selected there is still a 'primary' row which is the one tracked by SelectedObject and the one which the selection will be adjusted from if user uses Shift up/down.
You can see a similar effect in Visual Studio Code, but in this case the active row has a bright border.

I'm happy to go either way on this though. Setting the default to Invisible could be a good change for new users. The setting was originally added in response to #1559

Is there another scenario beyond multi-select where it's useful? Perhaps the right thing to do is only enable it when multi-select is active?

Is there another scenario beyond multi-select where it's useful?

I can't really think another use case other than stylistic choice.  Also because there is a public DesiredCursorVisibility property then removing it would be a breaking change.  Changing its default value might also be considered breaking but is much more minor.

Perhaps the right thing to do is only enable it when multi-select is active?

As I was describing the use case I did think of this but I dislike 'magic' properties (where changing one public property MultiSelect auto changes another DesiredCursorVisibility).  Especially you don't want :
tv.DesiredCursorVisibility = CursorVisibility.Invisible;
tv.MultiSelect = true;

To behave differently from:
tv.MultiSelect = true;
tv.DesiredCursorVisibility = CursorVisibility.Invisible;

Also relevant to this discussion is that MultiSelect defaults to true.
Let me know what you want to do, options seem to be:

Leave unchanged
Change DesiredCursorVisibility default to Invisible
Remove DesiredCursorVisibility and render cursor based solely on MultiSelect
Something else

@tznind see the TextView or the TextField where are conditions to show or hide the cursor. There is also a backup field where the cursor is restored (visible) accordingly with the user settings. What @tig suggest is only show the cursor if multi select is enabled and hide otherwise.

There is also a backup field where the cursor is restored (visible) accordingly with the user settings. What @tig suggest is only show the cursor if multi select is enabled and hide otherwise.

Ok I think I understand... do you mean that DesiredCursorVisibility becomes a setting that only applies when multi select is on? And if single select is on then the DesiredCursorVisibility value is ignored and the cursor is suppressed?

Ok I think I understand... do you mean that DesiredCursorVisibility becomes a setting that only applies when multi select is on? And if single select is on then the DesiredCursorVisibility value is ignored and the cursor is suppressed?

Right, but take an account the the user may be configured the DesiredCursorVisibility to other than the default. Then you must use a private backup key to save the user setting before the change and use that backup key to restore it. For e.g. the user may be set a block visible cursor. Before you set to invisible you have to save the user setting in that field. Before restore to visible you have to set to the block cursor and not the default.

Wouldn't it work just to change the get property:
/// <summary>
/// Get / Set the wished cursor when the tree is focused.
/// Only applies when <see cref="MultiSelect"/> is true.
/// </summary>
public CursorVisibility DesiredCursorVisibility {
	get {
		return MultiSelect ? desiredCursorVisibility : CursorVisibility.Invisible;
	}
[...]

Probably yes.

@tznind don't forget that call Application.Driver.SetCursorVisibility (returnedValue) or Driver.SetCursorVisibility (returnedValue) after you get the value to effectively apply the changes.

Ok I made the following changes of behavior:
TreeViewFileSystem now has menu items for toggling:

Cursor (on/off)
MultiSelect (on/off)

The changes to TreeView are:

Cursor now defaults to off (Invisible)
Cursor stores but ignores any changes to DesiredCursorVisibility while MultiSelect is off and always uses Invisible in this state.

Cursor visibility is set in the Enter event of the control and the set on DesiredCursorVisibility.


add translate method to quantum circuits

Summary
resolves issue #6787
Details and comments

@kdk Good point. I've added tests for those cases.

I agree with the comment https://github.com/Qiskit/qiskit/pull/6788#discussion_r694347268 and responses that calling transpile is probably more stable and also easy enough. Given that, and that we would like to keep tabs on the amount of methods on QuantumCircuit (and that this is pretty out-of-date) I'll go ahead and close this issue.
A useful alternative could be to make BasisTranslator easier to call by giving it a default equivalence library.


Biopython issue with module three_to_one

Hello,
Which version of Biopython should be installed? I am running BindCraft locally and I end up with an attribute error: AttributeError: module 'Bio.PDB.Polypeptide' has no attribute 'three_to_one'
Thanks for the help,
Best
Irene

You need version 1.78. I will update the repo next week to work with newer versions of biopython


Unable to install or Use NuGit package on Windows

Description
When trying to install NewRelic.MAUI.Plugin into our MAUI project via Nugit; the installation fails with the stack trace below. When a Mac installs the Nugit package it installs, but windows still is unable to build the project, with the same issue.
Our project uses .net 8
Enable Long Path has been turned on for windows.
Restoring packages for C:\WIP\ProjectName\ProjectNameMAUI\ProjectNameMAUI.csproj...
GET https://api.nuget.org/v3-flatcontainer/newrelic.maui.ios.binding/index.json
GET https://nuget.telerik.com/v3/package/newrelic.maui.ios.binding/index.json
NotFound https://nuget.telerik.com/v3/package/newrelic.maui.ios.binding/index.json 56ms
OK https://api.nuget.org/v3-flatcontainer/newrelic.maui.ios.binding/index.json 82ms
GET https://api.nuget.org/v3-flatcontainer/newrelic.maui.ios.binding/7.5.0/newrelic.maui.ios.binding.7.5.0.nupkg
OK https://api.nuget.org/v3-flatcontainer/newrelic.maui.ios.binding/7.5.0/newrelic.maui.ios.binding.7.5.0.nupkg 31ms
System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Users\UserName.nuget\packages\newrelic.maui.ios.binding\7.5.0\lib\net7.0-ios16.1\NewRelic.MAUI.iOS.Binding.resources\NewRelic.xcframework\ios-arm64_x86_64-simulator\NewRelic.framework\Modules\NewRelic.swiftmodule\arm64-apple-ios-simulator.private.swiftinterface'.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)
at System.IO.File.Create(String path)
at NuGet.Packaging.StreamExtensions.Testable.MmapCopy(Stream inputStream, String fileFullPath, Int64 size)
at NuGet.Packaging.StreamExtensions.Testable.CopyToFile(Stream inputStream, String fileFullPath)
at NuGet.Packaging.PackageFileExtractor.ExtractPackageFile(String source, String target, Stream stream)
at NuGet.Packaging.PackageArchiveReader.CopyFiles(String destination, IEnumerable1 packageFiles, ExtractPackageFileDelegate extractFile, ILogger logger, CancellationToken token)    at NuGet.Packaging.PackageReaderBase.CopyFilesAsync(String destination, IEnumerable1 packageFiles, ExtractPackageFileDelegate extractFile, ILogger logger, CancellationToken cancellationToken)
at NuGet.Packaging.PackageExtractor.<>c__DisplayClass5_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Common.ConcurrencyUtilities.d__51.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    at NuGet.Common.ConcurrencyUtilities.<ExecuteWithFileLockedAsync>d__51.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Packaging.PackageExtractor.d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Commands.ProjectRestoreCommand.d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at NuGet.Commands.ProjectRestoreCommand.d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Commands.ProjectRestoreCommand.d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Commands.RestoreCommand.d__86.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Commands.RestoreCommand.d__68.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Commands.RestoreRunner.d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.Commands.RestoreRunner.d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at NuGet.Commands.RestoreRunner.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.PackageManagement.DependencyGraphRestoreUtility.d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.PackageManagement.NuGetPackageManager.d__89.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.PackageManagement.NuGetPackageManager.d__75.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.PackageManagement.VisualStudio.NuGetProjectManagerService.<>c__DisplayClass22_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at NuGet.PackageManagement.VisualStudio.NuGetProjectManagerService.d__33`1.MoveNext()
Time Elapsed: 00:00:02.0205157
========== Finished ==========

@kruszkab it is Maui Framework issue, please check with Microsoft for this. https://github.com/dotnet/maui/issues/17828

this is deplicate issue . #24


What is the LightGBM splitting criterion?

In the source code(i.e. feature_histogram.hpp), I find the splitting criterion function named GetLeafSplitGainGivenOutput. Specically,  the splitting criterion is -(2.0 * sg_l1 * output + (sum_hessians + l2) * output * output)。
If I understand it correctly, that is to say, the above form is different from the formula for calculating the split gain of xgboost.
My question is why does LGBM calculate the split gain using the above form ? Is there a mathematical derivation?
Thank you for your great help.

close a duplicated one.


Including pstl

AIUI std::execution (added in C++17) requires pstl. Given there may be more usage of std::execution in the future ( for example https://github.com/conda-forge/cython-feedstock/pull/75 ), maybe it would be worthwhile to build clang with pstl support.

Transferred this to the right feedstock, because it's really a libcxx issue.

As of libcxx 17 there are the first bits of PSTL support. Even if the rest gets done until llvm 18, it would probably experimental for a release or two. We could still start to build the experimental bits if we want to.


Issue with functions

I'm working on recreating the whole ConseilJS package in the dart called Tezster_dart.
I have already had the 1st version deployed. I got stuck at this point.

I could not understand the difference between those lines.

 r.greater(0) and r > 0
r.and(127) and r & 127
r.shiftRight(7) and r >> 7

I could not find the alternative, can you guide me through this problem.
https://github.com/Cryptonomic/ConseilJS/blob/f7192a8a32cbf90f6b583a850d85b2cf6b12102a/src/utils/CryptoUtils.ts#L96

Hi @Rakshak1344, this function handles large numbers in javascript with bignumber.js and smaller numbers natively in javascript.


Fix backwards incompatible pandas usage.

Experiment with latest CI break.

We have a pandas break on this run: https://github.com/quantumlib/Cirq/runs/4914290551?check_suite_focus=true

https://github.com/quantumlib/Cirq/runs/4914785919?check_suite_focus=true as well:
pandas.errors.InvalidIndexError: (slice(None, None, None), None)

This PR will fix those.


tailscale GUI on macOS doesn't display exit nodes from headscale

Bug description
I use headscale as a control server and an official tailscale client from the App Store as a client, and I can't choose exit node from the client's GUI, list of available exit nodes is empty.

At the same time, I can see the list of exit nodes available and can choose an exit node using CLI:
/Applications/Tailscale.app/Contents/MacOS/Tailscale up --login-server=https://my-headscale --exit-node=apx

As a result, it works great! But in GUI I can see only the chosen exit node and it shown as offline, despite it works correctly:

I reported this to tailscale/tailscale#5628 but they suppose that there is a problem in control server:

I bet Headscale isn't setting the online/offline bits or something.

To Reproduce

Install and configure headscale on a remote server (this step may be omitted, please contact me if you'd like to reproduce this using my headscale instance).
Install a Tailscale client on macOS machine.
Configure the Tailscale client using .mobileprofile or a CLI command:

defaults write io.tailscale.ipn.macos ControlURL https://my-headscale

Login to control server and ensure you can see enabled exit nodes using a CLI command:

/Applications/Tailscale.app/Contents/MacOS/Tailscale status

Check the list of exit nodes in GUI.

Actual result
List in GUI is empty.
Expected result
List in GUI contains the same exit nodes as in console.
Context info
Control server: headscale v0.16.4 on CentOS Linux 8
Client: tailscale 1.30.1, App Store version on macOS 12.5.1
Kernel version: 4.18.0-305.12.1.el8_4.x86_64
headscale and caddy configs

@nsauk hey, can you try to run 0.17.0-alpha2?

@juanfont issue still reproduced on 0.17.0-alpha2.

@nsauk, did you enable routes in headscale CLI? You need to do that in order for an exit node to become available.
headscale routes enable -i <machine_id> -a

@kazauwa yes I did. I have no problems with using these exit nodes, I can choose them with tailscale CLI and routing works, but I can't see them in the UI of macOS client. As I pointed out in the description, I originally reported this to tailscale/tailscale#5628 but they suppose that there is a problem in control server.

Hmm, I got it: I use IPv4 routes only and tailscale on macOS doesn't consider such exit nodes as available.
@kazauwa thanks for your command example with the -a parameter.
@juanfont may we know, if this thing is different between tailscale and headscale implementations?

@nsauk
I'm almost sure that the implementations are the same, because Tailscale docs mention both IPv4 and IPv6 default routes.

We might not be sending all the info for the macOS client. Let's keep it open.

@juanfont, do you send the Online bool?

@bradfitz apologies for the delay. Yes, we do.
@nsauk are the clients shown online when checking status in the CLI?

@juanfont let me chime in, since I'm also experiencing this issue and was re-creating tailnet for a few times, before I stumbled on this issue.
Client are shown as active but offline from the cli. Same in MacOS app GUI: I can see a dot near client name in Network devices list, indicating it's active, but can't connect to respective exit node(greyed out, says it's offline).
Android client sees exit nodes just ok.

Just checking in on this.  I've now tried with the last 3 release versions of headscale up to the most recent 0.19.0-beta1, and on OSX the exit node shows in the UI after logging in from the terminal only the first time.  After selecting not to use the exit node though, it disappears from the menu so I can't go back to using it.  When I try to re-run the tailscale up command specifying the exit node I get an error saying the node is not advertising an exit node, even though nothing changed on the server or the exit node client.  The only way I've figured out to make it work again is to restart the headscale server and login again, requiring the full setup procedure, but then I only get one session with the exit node.  The route is configured properly because I have used the exit node successfully many times, and all our linux clients work fine.

OK, so I'm wondering if this might be related to IPv4 vs IPv6 issues.  When I copy the exit node's address via the UI menu in tailscale it copies an IPv6 addr.  On the headscale server the only route I had enable was the IPv4 route.  When I checked the tailscale status --json from my OSX client it showed that the exit node machine was not advertising as an exit node.  (Even though we have been successfully using it as an exit node for weeks with these settings... consistently in Linux, but spotty on OSX.). Noticing the route I enabled the IPv6 route on the headscale server.  Now when I check the status from the OSX client it shows the machine as advertising as an ExitNode, it also shows in the tailscale UI, and I'm successfully using it again.
Hopefully this helps point to where the bug may lie.  Thanks for creating headscale!

I'm not following this bug closely (nor do I use Headscale) but keep in mind that Tailscale only considers a node to be an exit node if it advertises both the IPv4 <IP_ADDRESS>/0 and IPv6 ::/0 route:
https://github.com/tailscale/tailscale/blob/988801d5d97287a276fc74cddb09a0e0ddf8afb7/net/tsaddr/tsaddr.go#L247-L259
(This has nothing to do with whether the exit node actually has IPv6 access itself, but it needs to advertise the route)


Ap/show unresolved

Customer scenario
Changing package version from known to unknown version causes unresolved dependency to disappear from solution explorer
Bugs this fixes:
https://github.com/dotnet/sdk/issues/451
Workarounds, if any
Reload project
Risk
Minimal
Performance impact
none

@dotnet/project-system @srivatsn @natidea

@MattGertz for RTM consideration - If someone edits the project file to change the version of a package, the node disappears from the solution tree instead of showing up as unresolved with a warning sign.

As bar check.

This was rejected in shiproom. I'm retargeting this PR to the post-dev15 branch.

You'll need to rebase.

@abpiskunov Could you please fix the conflicts and update the PR?

this PR is obsolete. There is a PR https://github.com/dotnet/roslyn-project-system/pull/1880/files that addresses this issue in the new  design

Closing then.


Following the license change, modify the boilerplates in crypto/lhash/

Merged.
8573be06a99cd1260b4988e9fa58e75df32c3be4 Following the license change, modify the boilerplates in crypto/lhash/


MinifyCss is now a property on  WebOptimazerScssOptions class.

Embedded generated map files were being stripped out in the minifying process.  This allows for the minification process to be enabled or disabled when using the pipeline extensions.  The default value is set to true (to minify) to honor the previous default behavior of always minifying.

Thanks!


Update feedback.md

Fixed markup for section 4.2:https://docs.mendix.com/developerportal/collaborate/feedback#processing

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Support Lux.jl

When tracing through a (function) that calls a Lux model, a variety of warnings are thrown about using fallback matmul functions because of eltypes that don't reduce to (I guess) common BLAS routines (i.e. Float64 and the tracer types).
Overloading either Lux evaluation functions in an extension, or even matmul-esque functions would be greatly appreciated!

Thanks for the report Andrew! Could you provide us with a stack trace and ideally also a minimal example that reproduces the issue?
E.g. a Lux model stripped down to the relevant layer(s) evaluated on random inputs?
matmul generally works, so there must be something else going on here as well.

yes, I will try. I have a large backlog of MWEs I need to make but should be getting some time to address them soon!


Wrong rules.time config variable results in serious client's bugs

Description
Client get a bugs with UI and chat channels when there are a wrong variable was set for rules.time cvar. This cvar requires simple type to be set, for example 100.0, but not 100. If you set 100, you will get only error in client console and no rules will be shown. When game starts and player rejoins server, his chat channels disappears and UI no longer work. Why engine doesn't check cvars for correct variable types or not converting them? We didn't know that this bug appeared because of that shit for 1 month
Reproduction
Setup server and add this cvar in server config
[rules]
time = 45

Screenshots
The error in console

No UI windows and no chat channels

Any comment?


Enquiry on Trilinear Kernel

Dear author,
I am interested in your paper and I do not understand what is trilinear kernel is.
I found the implantation from models.py where there is one step to init the kernel.
def trilinear_kernel(self, ts, num_channels):
        gt_values = torch.zeros_like(ts)

        gt_values[ts > 0] = (1 - (num_channels-1) * ts)[ts > 0]
        gt_values[ts < 0] = ((num_channels-1) * ts + 1)[ts < 0]

        gt_values[ts < -1.0 / (num_channels-1)] = 0
        gt_values[ts > 1.0 / (num_channels-1)] = 0

        return gt_values

However, I do not figure out what the meaning of the above code. It should be the gt values of one trilinear kernel.
Then, I write one demo to test it. As follows:
import torch

def trilinear_kernel(ts, num_channels):
    gt_values = torch.zeros_like(ts)

    gt_values[ts > 0] = (1 - (num_channels - 1) * ts)[ts > 0]
    gt_values[ts < 0] = ((num_channels - 1) * ts + 1)[ts < 0]

    gt_values[ts < -1.0 / (num_channels - 1)] = 0
    gt_values[ts > 1.0 / (num_channels - 1)] = 0

    return gt_values

ts = torch.rand((1, 2000))
torch.manual_seed(1)
ts.uniform_(-1, 1)
print(trilinear_kernel(ts,9))

The problem is the result is 0 no matter how much times I run.

Hi, I found that the provided code did not use the trilinear kernel or trilinear_init.pth(just load it), but used the linear layer three times. Is using three linear layers as the trilinear kernel?

Hi Jeeking,
The trilinear kernel in this case is just atriangular kernel applied to t
(so not tri but 1). However, it can be generalized to apply also to x and y
but was not considered in this work.
On Sat, 1 Aug 2020, 15:55 JeeKing<EMAIL_ADDRESS>wrote:

Hi, I found that the provided code did not use the trilinear kernel or
trilinear_init.pth(just load it), but used the linear layer three
times. Is using three linear layers as the trilinear kernel?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/uzh-rpg/rpg_event_representation_learning/issues/8#issuecomment-667535969,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFBHKKBPQ3SBG7U63FUDMVLR6QNFZANCNFSM4PIU735A
.

Thank you for your prompt reply.

if we want to apply to  x and y,
values = t * self.value_layer.forward(t-i_bin/(C-1))
we just need to use x and y instead of t in this code. right？

Yes exactly
On Sat, 1 Aug 2020, 16:11 JeeKing<EMAIL_ADDRESS>wrote:

if we want to apply to x and y,
values = t * self.value_layer.forward(t-i_bin/(C-1))
we just need to use x and y instead of t in this code. right？
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/uzh-rpg/rpg_event_representation_learning/issues/8#issuecomment-667537964,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFBHKKGXU7UG7JCAXDT4AALR6QPADANCNFSM4PIU735A
.

Thus, what is the meaning of gt_values. Is there one formula to define it?

the gt_values are not all 0.
print(torch.max(trilinear_kernel(ts,9)))
output: tensor(0.9907)

the gt_values are not all 0.
print(torch.max(trilinear_kernel(ts,9)))
output: tensor(0.9907)

Yes, I found that. But I cannot understand how to get the gt_values since they seem to be calculated from the fixed equations. And the channel number is one parameter for this.
gt_values[ts > 0] = (1 - (num_channels - 1) * ts)[ts > 0]
gt_values[ts < 0] = ((num_channels - 1) * ts + 1)[ts < 0]

gt_values[ts < -1.0 / (num_channels - 1)] = 0
gt_values[ts > 1.0 / (num_channels - 1)] = 0

@yanxiang-wang  方便留微信或者QQ吗，有些问题想请教 :blush:


Invalid HTTP request form (expected: authority or absolute, got: relative) from Burp

Steps to reproduce the problem:

I used mitmproxy as forwarding proxy from Burp but I keep getting these error.

And this is error I get from console

mitmdump -p 8779 -s proxy.py

Proxy server listening at http://*:8779
<IP_ADDRESS>:56560: clientconnect
 << Invalid HTTP request form (expected: authority or absolute, got: relative)
<IP_ADDRESS>:56560: HTTP protocol error in client request: Invalid HTTP request form (expected: authority or absolute, got: relative)

I only get this error when use mitmproxy as upstream proxy on burp in Repeater tab from Burp

System information
Mitmproxy: 3.0.2
Python:    3.7.0
OpenSSL:   OpenSSL 1.0.2p  14 Aug 2018
Platform:  Darwin-18.0.0-x86_64-i386-64bit

I'm not sure how Burp's forwarding proxy is supposed to work, but this looks like a configuration issue. Please take a look at https://docs.mitmproxy.org/stable/concepts-modes/.

@jkryanchou
Could you share your solution or how it worked out for you ?

@patrickjahns it turns out that you should use absolute paths rather than relative paths. like
Do it
GET https://xxx.google.com/foo/bar
hostname: xxx.google.com
... ...

instead of
GET /foo/bar
hostname: xxx.google.com
... ...

System information
Mitmproxy: 5.0.0
Python:    3.6.7
OpenSSL:   OpenSSL 1.1.0j  20 Nov 2018
Platform:  Windows-10-10.0.19041-SP0
I have met this issue too.but your solution is work@ @jkryanchou
thanks

@jkryanchou
i face the same error
Mitmproxy: 4.0.4
Python:    3.8.10
OpenSSL:   OpenSSL 1.1.1f  31 Mar 2020
Platform:  Linux-5.4.0-139-generic-x86_64-with-glibc2.29
how to change
GET /foo/bar hostname: xxx.google.com ... ...
to
GET https://xxx.google.com/foo/bar hostname: xxx.google.com ... ...

how to let client use GET https://xxx.google.com/foo/bar hostname: xxx.google.com ... ...


Training with smaller dataset

Hi,
I was trying to train deeplab2 model only for semantic segmentation with mapillary dataset.
I am getting decent results with training with a large dataset (8200 images).
Since I need a model to be trained for class 'Tunnel' alone, I  selected the images and labels with tunnels and tried to train with those images (total size of dataset in around 210) for a single class. I have used all images for training and no testing and evaluation split done.
But am getting a result like the semantic map covering the whole image. I have tried with iterations as low as 2000 and as high as 60k but same result. Am I doing anything wrong here? I have used learning rate of 0.001.
when I trained with 60k iterations the training loss was gradually reducing from 0.65 to 0.03. But result was as stated above

Hi @nithinme3,
Apologize for the late reply.
Regarding your problem, it is not clear to us how to efficiently solve it (since it is not related to the codebase itself).
Nevertheless, you could probably try something like tuning the loss weight for the tunnel class, which appears less frequently than the other class.
Additionally, you could check some methods related to training with imbalanced classes (e.g., long-tailed classes).
Cheers,

Closing the issue, as there is no active discussion for a while.
Please feel free to open a new issue if needed.


Have 'Download updates automatically' also install the update

Description of the new feature / enhancement
If the 'Automatically download updates' is set to 'Yes', have PT install the update and then let me know it has (and if it needs the computer restarted, like Windows updates).
Scenario when this would be used?
Any time an update is downloaded.
Supporting information
I do get the emails when an update is published, but I don't always get to the install. I'd rather it auto install so i can keep current.

We aren't quite there yet due to UAC dialog and elevation requirements.  We are getting there.  /dup #15517


Use Markdown Readme on PyPI

[X] Added entry into CHANGELOG.md

Changes proposed in this pull request:

Use the Markdown README directly on PyPI

@pr-omethe-us/chemked

Excellent!


Archive/Note old gitlab project

Leave a note on the original gitlab project directing people here, and making it clear that this is the canonical version.
https://gitlab.com/jerometwell/pynonymizer

This has been done.


Allow a  for .input-group-addon by removing margin

When using a label as input-group-addon there will be a default margin-bottom of .5rem via the reboot, https://github.com/twbs/bootstrap/blob/v4-dev/scss/_reboot.scss#L280.
This will lead to undesirable white space below the label and the input will scall bigger then needed. By removing the margin, it will play nice with label elements

Happy to accept this, but could you please change that line to:
margin-bottom: 0; // Allow use of <label> elements by overriding our default margin-bottom

And then rebase & squash your commits? Since we're setting only margin-bottom, we only need to override that particular direction and not all margin. That also adds a comment then so we can quickly understand what's going on in the future.

Tweaked per @mdo and merged as a62f70b42065b2f7512fd22b07270da039add6ba.

:+1:


Way larger buffer to deal with spikes and collector downtime.

:+1:, though I think the stuff in spec/test_server.rb can be removed as it's not being used anywhere ( collecting increment commands into an increments accessor )

You're right, that didn't end up being used.


[#242] Avoid reload of full-res image

Closes #242

Coverage decreased (-0.007%) to 89.007% when pulling 7c327be6fe75632919f292a75c05b8db3e263132 on feature/242-avoid-reload-of-full-res-image into bc42465512c87efdf67db512b9f233d983eaab17 on master.


Run kitchen ci verification test. Initial commit.

Run kitchen ci verification test. Initial commit.
Reduced inductor jar size by half by removing unwanted dependencies.

👍


Create default task specs (IPFS-flavored Wasm, HTTP, DNS)

Summary

Include Task Types

Problem / Need / Opportunity

ETA: AAAA-MM-DD

Closing in favor of https://github.com/ipvm-wg/spec/issues/18.


plugin dev: TypeScript errors around run({ log }) removed (Confused)

https://github.com/snowpackjs/snowpack/blob/36c2a6772bc87ed1ea0e1ef626fea5cb8a399935/snowpack/src/types.ts#L117-L119
https://github.com/snowpackjs/snowpack/pull/671/files#diff-3a6ddf726e2b1de00744da2e6c1e06e378461be1ab1e0f125e63180819048123R43-R46
It appears that the runOptions.log doesn't have a type, yet many plugins use this function. Is there some other plan around the corner for logging from my plugins?
If there is any interest in receiving some help to add some of these missing types to the definitions, I'd be happy to propose changes that can then be reviewed, but ultimately, it seems like it will be hard to prevent regressions since the "official" plugins in this repo aren't checked against the types.

Yes! This was never really meant to go be used by others, and uses some internal Snowpack interface. A big v3.1 feature that I'm gunning for will be a new dashboard & reporting interface to fix this.

Yes! This was never really meant to go be used by others, and uses some internal Snowpack interface. A big v3.1 feature that I'm gunning for will be a new dashboard & reporting interface to fix this.


Distributing loop iterations across processors without assuming number of processors available.

Consider the following loop nest that represent the outer parallel loops obtained by  tiling a linalg.matmul
%c0 = constant 0
%c2 = constant 1
%m = dim %A, %c0
%n = dim %B, %c1
scf.parallel (%iv0, %iv1) =  (%c0, %c0) to (%m, %n) step (%ts1, ts2)  {
  %svA = subview %A[%iv0, 0][...][1, 1]
  %svB = subview %B[0, %iv1][...][1, 1]
  %svV = subview %C[%iv0, %iv1][...][1, 1]
  linalg.matmul ins(%svA, %svB : ...) outs(%svC)
}

This loop is distributed across different processes on CPU side as follows
%y_id = iree.workgroup_id {dimension : "y"}
%iv0 = muli %y_id, %ts1
%x_id = iree.workgroup_id {dimension : "x"}
%iv1 = muli %x_id, %ts2
%svA = subview %A[%iv0, 0][...][1, 1]
%svB = subview %B[0, %iv1][...][1, 1]
%svV = subview %C[%iv0, %iv1][...][1, 1]
linalg.matmul ins(%svA, %svB : ...) outs(%svC)

This makes the assumption that the number of tiles to distribute is equal to the number of threads available in the runtime, so there needs to be at least as many threads as ceil(%m / %ts1) x ceil(%n / %ts2).
An alternative approach is to distribute the tiled code as follows
%y_id = iree.workgroup_id {dimension : "y"}
%x_id = iree.workgroup_id {dimension : "x"}
%nprocs_y = iree.workgroup_count {dimension : "y"}
%nprocs_x = iree.workgroup_count {dimension : "y"}
%offset_y = muli %y_id, %ts1
%offset_x = muli %x_id, %ts2
%step_y = muli %nprocs_y, %ts1
%step_x = muli %nprocs_x, %ts2
scf.for %iv0 = %offset_y to %m step %step_y {
  scf.for %iv = %offset_x to %n step %step_x {
    %svA = subview %A[%iv0, 0][...][1, 1]
    %svB = subview %B[0, %iv1][...][1, 1]
    %svV = subview %C[%iv0, %iv1][...][1, 1]
    linalg.matmul ins(%svA, %svB : ...) outs(%svC)
  }
}

Here the generated code works with any value of the number of threads available at runtime. It also means that the value of the number of threads dynamically available needs to be provided as an argument to the kernel.
History Note The GPU code-generation used exactly the codegeneration approach described in the third snippet above, but moved to generate the code similar to the second snippet. This required some mechanism to relay to the runtime the number of workgroups to use (see Issue #3737 for more details). This allowed reducing control overhead in the generated code while also making some transformations like hoisting etc. unnecessary.

Addressed in https://github.com/google/iree/commit/334aa4a5ac033399255fbd836d4fbe807c741230


[WIP] Add initial implementation of Query expression

Purpose
This PR contains the initial implementation of query expression.
Approach
N/A
Samples
N/A
Remarks
Refer, related issue https://github.com/ballerina-platform/ballerina-lang/issues/20722 ...
Check List

[x] Read the Contributing Guide
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@108d6e1). Click here to learn what that means.
The diff coverage is 0%.

@@            Coverage Diff            @@
##             master   #20845   +/-   ##
=========================================
  Coverage          ?   14.59%
=========================================
  Files             ?       51
  Lines             ?     1398
  Branches          ?      214
=========================================
  Hits              ?      204
  Misses            ?     1178
  Partials          ?       16

Impacted Files
Coverage Δ

tool-plugins/vscode/src/extension.ts
0% <0%> (ø)

...plugins/vscode/src/syntax-highlighter/scopeTree.ts
0% <0%> (ø)

...ugins/vscode/src/syntax-highlighter/highlighter.ts
0% <0%> (ø)

...plugins/vscode/src/syntax-highlighter/activator.ts
0% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 108d6e1...9f940b0. Read the comment docs.


when trying to download comments from a video te following error occured on my terminal

Part 2 - using the theorem to show some cool stuff: https://youtu.be/o3QBgkQi_HA
And a bit extra on inversion: https://youtu.be/qmfFH1SyXP0
Traceback (most recent call last):
File "/usr/local/bin/yttool", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/yttool.py", line 940, in main
cmt.recursecomments()
File "/usr/local/lib/python3.8/dist-packages/yttool.py", line 409, in recursecomments
self.recursecomments(subcc, level+1)
File "/usr/local/lib/python3.8/dist-packages/yttool.py", line 409, in recursecomments
self.recursecomments(subcc, level+1)
File "/usr/local/lib/python3.8/dist-packages/yttool.py", line 409, in recursecomments
self.recursecomments(subcc, level+1)
[Previous line repeated 972 more times]
File "/usr/local/lib/python3.8/dist-packages/yttool.py", line 392, in recursecomments
cmtjson = self.yt.getcomments(cc, self.xsrf)
File "/usr/local/lib/python3.8/dist-packages/yttool.py", line 179, in getcomments
return self.httpreq(url + "?" + urllib.parse.urlencode(query), postdata.encode('ascii') )
File "/usr/local/lib/python3.8/dist-packages/yttool.py", line 155, in httpreq
response = self.opener.open(req, **kwargs)
File "/usr/lib/python3.8/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/usr/lib/python3.8/urllib/request.py", line 1393, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
r = h.getresponse()
File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
response.begin()
File "/usr/lib/python3.8/http/client.py", line 331, in begin
self.headers = self.msg = parse_headers(self.fp)
File "/usr/lib/python3.8/http/client.py", line 225, in parse_headers
return email.parser.Parser(_class=_class).parsestr(hstring)
File "/usr/lib/python3.8/email/parser.py", line 67, in parsestr
return self.parse(StringIO(text), headersonly=headersonly)
File "/usr/lib/python3.8/email/parser.py", line 56, in parse
feedparser.feed(data)
File "/usr/lib/python3.8/email/feedparser.py", line 176, in feed
self._call_parse()
File "/usr/lib/python3.8/email/feedparser.py", line 180, in _call_parse
self._parse()
File "/usr/lib/python3.8/email/feedparser.py", line 295, in _parsegen
if self._cur.get_content_maintype() == 'message':
File "/usr/lib/python3.8/email/message.py", line 594, in get_content_maintype
ctype = self.get_content_type()
File "/usr/lib/python3.8/email/message.py", line 578, in get_content_type
value = self.get('content-type', missing)
File "/usr/lib/python3.8/email/message.py", line 471, in get
return self.policy.header_fetch_parse(k, v)
File "/usr/lib/python3.8/email/_policybase.py", line 316, in header_fetch_parse
return self._sanitize_header(name, value)
File "/usr/lib/python3.8/email/_policybase.py", line 287, in _sanitize_header
if _has_surrogates(value):
File "/usr/lib/python3.8/email/utils.py", line 57, in _has_surrogates
s.encode()
RecursionError: maximum recursion depth exceeded while calling a Python object

i was trying to download the comments from the numberphile video that use as an example

thanks for letting me know.  It seems something has changed. I will look into it.

It seems youtube now requires the use of the action_get_comment_replies parameter to comment_service_ajax

I pushed an update, let me know if this works for you now.

worked thanks but I have to manually install it to start working

I now updated the pypi package too.


XUnit CI reporter uses xmldom

Fixes https://github.com/airportyh/testem/issues/529
Use XML DOM rather than string concatenation. Also make xml schema correct (missing required attributes). Augmented readme with example output

very nice!


Integrity fails to clone private repository

Problem
Integrity fails to fetch a private repository:
Integrity::CommandRunner::Error: Failed to run 'git clone git@github.com:lodi-g/repo builds/10': Cloning into 'builds/10'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

What was done

Generated a key with the user running bundle exec rackup
Added the key to github
Can run git clone git@github.com:lodi-g/repo from the command line (literally pasting the line Integrity tries to execute is working).

Something I did wrong, or is there an inner problem?
Thank you

I generated another key and it solved itself. Sorry for the inconvenience.


Grant flow Bad Request

Hi Simon,
I want to try the "grant flow" support. But Im stuck and not sure what Im doing wrong.
I did all of the required steps on AAD and the Powershell cmdlets described on:
https://learn.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth#use-client-credentials-grant-flow-to-authenticate-imap-and-pop-connections
My account setup in the emailproxy.config:
<EMAIL_ADDRESS>permission_url =
token_url = https://login.microsoftonline.com/common/oauth2/v2.0/token
oauth2_scope = https://outlook.office365.com/.default https://outlook.office365.com/IMAP.AccessAsUser.All https://outlook.office365.com/POP.AccessAsUser.All https://outlook.office365.com/SMTP.Send offline_access
redirect_uri = http://localhost
client_id = xxxxxxxxxxxxxxxxxxx
client_secret = yyyyyyyyyyyyyyyy
Not sure if I have to add the ".default" URI into the oauth2_scope.
When I start the emailproxy.py --no-gui ,
telnet localhost 1995 and provide my testuser credentials following happens.
Caught exception while requesting OAuth 2.0 credentials for<EMAIL_ADDRESS><HTTPError 400: 'Bad Request'>
Any idea which step Im probably missing?

Hi, thanks!
My config looks good now.
But still get an error when I try to connect over telnet.
2022-10-06 11:43:11: Caught exception while requesting OAuth 2.0 credentials for<EMAIL_ADDRESS><HTTPError 401: 'Unauthorized'>
Im pretty sure I did the steps on AAD correctly.

It's hard to diagnose issues with the client credentials grant flow because the setup is so opaque. But I'd assume that here there is some AAD misconfiguration causing the authorisation failure. Note that this error is raised by the proxy when requesting credentials, rather than actually using them, so check you've properly set up and given admin authorisation for your AAD client.
The encrypt_client_secret_on_first_use option is only needed if you are starting with an unencrypted secret. The proxy will automatically use the encrypted secret value if it is present.

Hi Simon,
Its working now, Im not sure where the issue was but I wrote a Powershell script to configure all of this in AAD so I dont forget any steps on AAD in the future.
Maybe the script is useful for someone else too.
It's not beautiful and some if statements to prevent obvious errors like "Duplicate Appname etc." are missing yet but its working at least.
`# Requires AzureADPreview Module;
Install-Module AzureADPreview
Install-Module -Name AzureRM
User must be able to create apps and consent (Global Admin, ...)
Param (
$appName = "YourAppName",
$redirecturi = "http://localhost",
$fullaccessuser = ('test1@tld','test2@tld')
)
#Login to Azure Active Directory
$credential = Get-Credential
Connect-AzureAd -Credential $credential;
#MS Graph stuff:
#Microsoft Graph delegated permissions
$servicePrincipalNameGraph = "Microsoft Graph";
$servicePrincipalNameOauth2PermissionsGraph = @("IMAP.AccessAsUser.All", "POP.AccessAsUser.All", "SMTP.Send");
Get MS Graph
$servicePrincipalGraph = Get-AzureADServicePrincipal -All $true | Where-Object { $_.DisplayName -eq $servicePrincipalNameGraph };
#MS Graph permissions
$reqGraph = New-Object -TypeName "Microsoft.Open.AzureAD.Model.RequiredResourceAccess";
$reqGraph.ResourceAppId = $servicePrincipalGraph.AppId;
$servicePrincipalGraph.Oauth2Permissions | Where-Object { $.Value -in $servicePrincipalNameOauth2PermissionsGraph} | ForEach-Object {
$permission = $
$delPermission = New-Object -TypeName "Microsoft.Open.AzureAD.Model.ResourceAccess" -ArgumentList $permission.Id,"Scope" #delegate permission (oauth) are always "Scope"
$reqGraph.ResourceAccess += $delPermission
}
#EXO stuff:
#Office 365 Exchange Online app permissions
$servicePrincipalNameExchangeOnline = "Office 365 Exchange Online";
$servicePrincipalNameOauth2PermissionsExchangeOnline = @("IMAP.AccessAsApp", "POP.AccessAsApp");
#Get Office 365 Exchange Online
$servicePrincipalExchangeOnline = Get-AzureADServicePrincipal -All $true | Where-Object { $_.DisplayName -eq $servicePrincipalNameExchangeOnline };
#Exchange Online permissions
$reqExchangeOnline = New-Object -TypeName "Microsoft.Open.AzureAD.Model.RequiredResourceAccess";
$reqExchangeOnline.ResourceAppId = $servicePrincipalExchangeOnline.AppId;
$servicePrincipalExchangeOnline.AppRoles | Where-Object { $.Value -in $servicePrincipalNameOauth2PermissionsExchangeOnline} | ForEach-Object {
$permission = $
$appPermission = New-Object -TypeName "Microsoft.Open.AzureAD.Model.ResourceAccess" -ArgumentList $permission.Id,"Role"
$reqExchangeOnline.ResourceAccess += $appPermission
}
#Define the Clientsecret here
$secretstartdate = Get-Date
$secretenddate = $secretstartdate.AddMonths(24)
$description = "OAuth_secret"
New-AzureADApplication -DisplayName $appName -AvailableToOtherTenants:$true -PublicClient:$false -ReplyUrls $redirecturi -RequiredResourceAccess $reqGraph, $reqExchangeOnline;
$newapp = Get-AzureADApplication -SearchString $appName;
$clientsecret = New-AzureADApplicationPasswordCredential -ObjectId $newapp.ObjectId -StartDate $secretstartdate -EndDate $secretenddate -CustomKeyIdentifier $description;
"`n"
"ClientId: " + $newapp.AppId;
"ClientSecret: " + $clientsecret.Value
"TenantId: " + (Get-AzureADTenantDetail).ObjectId;
"`n"
"You can check your AAD app here: https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationMenuBlade/CallAnAPI/appId/" + $newapp.AppId + "/objectId/" + $newapp.ObjectId + "/isMSAApp/";
#Give it some time to appear in azuread so the next step will run successfull
Start-Sleep -Seconds 60
#Add you as owner
$AppOwner = get-azureaduser -all $true -Filter "UserPrincipalName eq '$($credential.UserName)'"
Add-AzureADApplicationOwner -ObjectId $($newapp.ObjectId) -RefObjectId $($AppOwner.ObjectId)
#Grant Admin consent
Login-AzureRmAccount -Credential $credential
$context = Get-AzureRmContext
$tenantId = $context.Tenant.Id
$token = [Microsoft.Azure.Commands.Common.Authentication.AzureSession]::Instance.AuthenticationFactory.Authenticate($context.Account, $context.Environment, $TenantId, $null, "Never", $null, "74658136-14ec-4630-ad9b-26e160ff0fc6")
$headers = @{
'Authorization' = 'Bearer ' + $token.AccessToken
'X-Requested-With'= 'XMLHttpRequest'
'x-ms-client-request-id'= [guid]::NewGuid()
'x-ms-correlation-id' = [guid]::NewGuid()}
$url = "https://main.iam.ad.ext.azure.com/api/RegisteredApplications/$($newapp.AppId)/Consent?onBehalfOfAll=true"
Invoke-RestMethod -Uri $url -Headers $headers -Method POST -ErrorAction Stop
#Register Service principal
Connect-ExchangeOnline -Credential $credential
$serviceprincipalexo = New-ServicePrincipal -AppId $newapp.AppId -ServiceId $newapp.ObjectId -Displayname $newapp.DisplayName -Organization $tenantId
#Grant your application's service principal full access to the specified mailboxes.
ForEach ($fullaccessuser in $fullaccessusers) {
Add-MailboxPermission -Identity "$fullaccessusers" -User $serviceprincipalexo.Id -AccessRights FullAccess
}`

Thanks for the contribution!
(I edited it slightly to make a code block rather than individual lines – you can use three backticks to do this if needed)

@D4rki91
`#EXO stuff:
#Office 365 Exchange Online app permissions
$servicePrincipalNameExchangeOnline = "Office 365 Exchange Online";
$servicePrincipalNameOauth2PermissionsExchangeOnline = @("IMAP.AccessAsApp", "POP.AccessAsApp");
#Get Office 365 Exchange Online
$servicePrincipalExchangeOnline = Get-AzureADServicePrincipal -All $true | Where-Object { $_.DisplayName -eq $servicePrincipalNameExchangeOnline };
#Exchange Online permissions
$reqExchangeOnline = New-Object -TypeName "Microsoft.Open.AzureAD.Model.RequiredResourceAccess";
$reqExchangeOnline.ResourceAppId = $servicePrincipalExchangeOnline.AppId;
$servicePrincipalExchangeOnline.AppRoles | Where-Object { $.Value -in $servicePrincipalNameOauth2PermissionsExchangeOnline} | ForEach-Object {
$permission = $
$appPermission = New-Object -TypeName "Microsoft.Open.AzureAD.Model.ResourceAccess" -ArgumentList $permission.Id,"Role"
$reqExchangeOnline.ResourceAccess += $appPermission
}`
Are you purposefully configuring your registered application to support both 'Delegation' and 'Flow'?
if you are using only one you dont need both do you?
Thanks in advance for the clarification

Is there a mistake in this script
`Param (
$appName = "YourAppName",
$redirecturi = "http://localhost",
$fullaccessuser = ('test1@tld','test2@tld')
)
#Grant your application's service principal full access to the specified mailboxes.
ForEach ($fullaccessuser in $fullaccessusers) {
Add-MailboxPermission -Identity "$fullaccessusers" -User $serviceprincipalexo.Id -AccessRights FullAccess
`
should be
`Param (
$appName = "YourAppName",
$redirecturi = "http://localhost",
$fullaccessusers = ('test1@tld','test2@tld')
)
#Grant your application's service principal full access to the specified mailboxes.
ForEach ($fullaccessuser in $fullaccessusers) {
Add-MailboxPermission -Identity "$fullaccessuser" -User $serviceprincipalexo.Id -AccessRights FullAccess`

@D4rki91
Instead of creating a specific list of users... and granting full permissions to all of them...  would it work  instead to add "full_access_as_ap" into your Exchange Online Permissions"?  If so this would make it more maintainable too...

Hey,
Thanks for your input.
I know you don't have to configure both. :)
It was just a quick and dirty script from me, so my colleagues can register Enterprise Applications faster and without asking me or reading my "HowTo".
I made a simple if statement to configure either this Grant Flow stuff or only the MS Graph permissions.
The script is still not pretty I know, but all of these weird AzureAD and AzureRM cmdlets are confusing. :)
As long as it works Im happy.
# Requires AzureADPreview Module;
# Install-Module AzureADPreview
# Install-Module -Name AzureRM
# User must be able to create apps and consent (Global Admin, ...)

Param (
$appName = "EXOAuth",
$redirecturi = "https://oauth.exorack.de",
$fullaccessuser = ('test1@tld','test2@tld'),
$grantflowrequirements = $false
)

#Login to Azure Active Directory
$credential = Get-Credential
Connect-AzureAd -Credential $credential;

#MS Graph stuff:

#Microsoft Graph delegated permissions
$servicePrincipalNameGraph = "Microsoft Graph";
$servicePrincipalNameOauth2PermissionsGraph = @("IMAP.AccessAsUser.All", "POP.AccessAsUser.All", "SMTP.Send");

# Get MS Graph
$servicePrincipalGraph = Get-AzureADServicePrincipal -All $true | Where-Object { $_.DisplayName -eq $servicePrincipalNameGraph };

#MS Graph permissions
$reqGraph = New-Object -TypeName "Microsoft.Open.AzureAD.Model.RequiredResourceAccess";
$reqGraph.ResourceAppId = $servicePrincipalGraph.AppId;

$servicePrincipalGraph.Oauth2Permissions | Where-Object { $_.Value -in $servicePrincipalNameOauth2PermissionsGraph} | ForEach-Object {
    $permission = $_
    $delPermission = New-Object -TypeName "Microsoft.Open.AzureAD.Model.ResourceAccess" -ArgumentList $permission.Id,"Scope" #delegate permission (oauth) are always "Scope"
    $reqGraph.ResourceAccess += $delPermission
}

If($grantflowrequirements -eq $true){
#EXO stuff:

#Office 365 Exchange Online app permissions
$servicePrincipalNameExchangeOnline = "Office 365 Exchange Online";
$servicePrincipalNameOauth2PermissionsExchangeOnline = @("IMAP.AccessAsApp", "POP.AccessAsApp");

#Get Office 365 Exchange Online
$servicePrincipalExchangeOnline = Get-AzureADServicePrincipal -All $true | Where-Object { $_.DisplayName -eq $servicePrincipalNameExchangeOnline };

#Exchange Online permissions
$reqExchangeOnline = New-Object -TypeName "Microsoft.Open.AzureAD.Model.RequiredResourceAccess";
$reqExchangeOnline.ResourceAppId = $servicePrincipalExchangeOnline.AppId;

$servicePrincipalExchangeOnline.AppRoles | Where-Object { $_.Value -in $servicePrincipalNameOauth2PermissionsExchangeOnline} | ForEach-Object {
    $permission = $_
    $appPermission = New-Object -TypeName "Microsoft.Open.AzureAD.Model.ResourceAccess" -ArgumentList $permission.Id,"Role"
    $reqExchangeOnline.ResourceAccess += $appPermission
}
    }
    else{Write-Host "Grantflow requirements disabled"}
#Define the Clientsecret here
$secretstartdate = Get-Date
$secretenddate = $secretstartdate.AddMonths(24)
$description = "OAuth_secret"

New-AzureADApplication -DisplayName $appName -AvailableToOtherTenants:$true -PublicClient:$false -ReplyUrls $redirecturi -RequiredResourceAccess $reqGraph, $reqExchangeOnline;
$newapp = Get-AzureADApplication -SearchString $appName;
$clientsecret = New-AzureADApplicationPasswordCredential -ObjectId $newapp.ObjectId -StartDate $secretstartdate -EndDate $secretenddate -CustomKeyIdentifier $description;

"`n"

"ClientId: " + $newapp.AppId;
"ClientSecret: " + $clientsecret.Value
"TenantId: " + (Get-AzureADTenantDetail).ObjectId;

"`n"

"You can check your AAD app here: https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationMenuBlade/CallAnAPI/appId/" + $newapp.AppId + "/objectId/" + $newapp.ObjectId + "/isMSAApp/";

#Give it some time to appear in azuread so the next step will run successfull
Start-Sleep -Seconds 60

#Add you as owner
$AppOwner = get-azureaduser -all $true -Filter "UserPrincipalName eq '$($credential.UserName)'"
Add-AzureADApplicationOwner -ObjectId $($newapp.ObjectId) -RefObjectId $($AppOwner.ObjectId)

#Grant Admin consent
Login-AzureRmAccount -Credential $credential
$context = Get-AzureRmContext
$tenantId = $context.Tenant.Id
$token = [Microsoft.Azure.Commands.Common.Authentication.AzureSession]::Instance.AuthenticationFactory.Authenticate($context.Account, $context.Environment, $TenantId, $null, "Never", $null, "74658136-14ec-4630-ad9b-26e160ff0fc6")
$headers = @{
  'Authorization' = 'Bearer ' + $token.AccessToken
  'X-Requested-With'= 'XMLHttpRequest'
  'x-ms-client-request-id'= [guid]::NewGuid()
  'x-ms-correlation-id' = [guid]::NewGuid()}

$url = "https://main.iam.ad.ext.azure.com/api/RegisteredApplications/$($newapp.AppId)/Consent?onBehalfOfAll=true"
Invoke-RestMethod -Uri $url -Headers $headers -Method POST -ErrorAction Stop

If($grantflowrequirements -eq $true){
#Register Service principal
Connect-ExchangeOnline -Credential $credential
$serviceprincipalexo = New-ServicePrincipal -AppId $newapp.AppId -ServiceId $newapp.ObjectId -Displayname $newapp.DisplayName -Organization $tenantId

#Grant your application's service principal full access to the specified mailboxes.
ForEach ($fullaccessuser in $fullaccessusers) {
Add-MailboxPermission -Identity "$fullaccessusers" -User $serviceprincipalexo.Id -AccessRights FullAccess
}
    }
    else{Write-Host "Grantflow requirements disabled"}

Hey, I'm pretty sure I've tried that and it did not work. :(

I have yet to set this up... looking for guidance on how to do it properly.   Would appreciate any guidance/help.
I would like to avoid having to maintain the application every time a user is created or removed from the organization.  As it is now that is required because it needs the principal to have access to the mailboxes... correct?    How are you maintaining it today yourself?
Was hoping "full_access_as_ap" would help avoid such a maintenance issue.  Is it possible for you to double check to see if  the "full_access_as_ap" helps?

@D4rki91
Simply "if" works fine... just wanted to clarify what is required for which scenario...  script is nice to automate and not use the UI.
From here: https://learn.microsoft.com/en-us/graph/auth-limit-mailbox-access
There are scenarios where administrators may want to limit an app to only specific mailboxes and not all Exchange Online mailboxes in the organization. **Administrators can identify the set of mailboxes to permit access by putting them in a mail-enabled security group. Administrators can then limit third-party app access to only that set of mailboxes by creating an application access policy for access to that group**
We should be able to make use of this:

create a Mail Enabled Group of only the users we want the principal to have access to
assign the principal access to the Mail Enabled Group (not to individual users)
add/remove users from the group --> no need to change/maintain the principals ongoing permissions.   AD itself can be used to manage the Mail Enabled Group membership

Can you add this to your script? (creating a mail enabled security group and assigning the principal permissions to it)

@D4rki91
In your script:
Param (
$appName = "YourAppName",
$redirecturi = **"http://localhost",**
$fullaccessuser = ('test1@tld','test2@tld'),
$grantflowrequirements = $false
)

This will only work where the proxy is deployed on a location where port 80 traffic is not already in use...  (quite commonly used)
Can you update the script to take a DomainName and Port in the params ?  e.g.   http://domainName:Port)

@D4rki91
My understanding is that when creating/registering an application the redirectURI is optional... and only used to make sure traffic is restricted to that URI only....  but in general the proxy choose the redirectURL for the application... would it be better not to define it in the application at all?  Does it work not to put one in the application and let the proxy dictate the redirectURL?

@D4rki91
In your script:
Param (
$appName = "YourAppName",
$redirecturi = **"http://localhost",**
$fullaccessuser = ('test1@tld','test2@tld'),
$grantflowrequirements = $false
)

The redirectURL will only work where the proxy is deployed on a location where port 80 traffic is not already in use... (quite commonly used)
Can you update the script so $redirectURI takes a DomainName and Port in the params ? e.g. http://domainName:Port)

Sure I can take a look at it, so it will accept DomainName and Port.
We are using the external auth method so in our construct it's not required.
The STunnel on the Client System is setting up a secure connection to our public FQDN "OAuth.domain.tld" and there is a SNAT in our Watchguard Firewall that will redirect it to the Proxy Server. Then I just have to send the User the "login.microsoft" URI and after he successful logged in I my ".php.ini" has a simple script that will write the "https://oauth.domain.tld/?code=xyz" part into a CSV file on the Server so I just have to enter it into the proxy application.
It's a really comfortable way to authenticate Usermailboxes in our scenario and of course we restricted the acces with proxy actions like pattern matches on our Firewall to make it as secure as we can. :)
To make sure STunnel can set up a secure connection we use "winacme" on the proxy server to autorenewal the certificate .pem files the Proxy needs.

Thanks for the detailed response... still coming up to speed on all of the OAuth flow nuances... would you mind creating a diagram for me so I can better understand?
Here is a diagram I have created for the Client Flow configuration

I've tested it, it works with DomainName and Port like that.
There was a missing if statement so the script will register the Application with "grant flow = $false". Before it ran into a variable = "null" error.
Also I corrected the "fullaccesuser" mistake inside the foreach loop.
Requires AzureADPreview Module;
Install-Module AzureADPreview
Install-Module -Name AzureRM
User must be able to create apps and consent (Global Admin, ...)
Param (
$appName = "TestApp",
$redirecturi = "http://localhost:8081",
$fullaccessuser = ('test1@tld','test2@tld'),
$grantflowrequirements = $false
)
#Login to Azure Active Directory
$credential = Get-Credential
Connect-AzureAd -Credential $credential;
#MS Graph stuff:
#Microsoft Graph delegated permissions
$servicePrincipalNameGraph = "Microsoft Graph";
$servicePrincipalNameOauth2PermissionsGraph = @("IMAP.AccessAsUser.All", "POP.AccessAsUser.All", "SMTP.Send");
Get MS Graph
$servicePrincipalGraph = Get-AzureADServicePrincipal -All $true | Where-Object { $_.DisplayName -eq $servicePrincipalNameGraph };
#MS Graph permissions
$reqGraph = New-Object -TypeName "Microsoft.Open.AzureAD.Model.RequiredResourceAccess";
$reqGraph.ResourceAppId = $servicePrincipalGraph.AppId;
$servicePrincipalGraph.Oauth2Permissions | Where-Object { $.Value -in $servicePrincipalNameOauth2PermissionsGraph} | ForEach-Object {
$permission = $
$delPermission = New-Object -TypeName "Microsoft.Open.AzureAD.Model.ResourceAccess" -ArgumentList $permission.Id,"Scope" #delegate permission (oauth) are always "Scope"
$reqGraph.ResourceAccess += $delPermission
}
If($grantflowrequirements -eq $true){
#EXO stuff:
#Office 365 Exchange Online app permissions
$servicePrincipalNameExchangeOnline = "Office 365 Exchange Online";
$servicePrincipalNameOauth2PermissionsExchangeOnline = @("IMAP.AccessAsApp", "POP.AccessAsApp");
#Get Office 365 Exchange Online
$servicePrincipalExchangeOnline = Get-AzureADServicePrincipal -All $true | Where-Object { $_.DisplayName -eq $servicePrincipalNameExchangeOnline };
#Exchange Online permissions
$reqExchangeOnline = New-Object -TypeName "Microsoft.Open.AzureAD.Model.RequiredResourceAccess";
$reqExchangeOnline.ResourceAppId = $servicePrincipalExchangeOnline.AppId;
$servicePrincipalExchangeOnline.AppRoles | Where-Object { $.Value -in $servicePrincipalNameOauth2PermissionsExchangeOnline} | ForEach-Object {
$permission = $
$appPermission = New-Object -TypeName "Microsoft.Open.AzureAD.Model.ResourceAccess" -ArgumentList $permission.Id,"Role"
$reqExchangeOnline.ResourceAccess += $appPermission
}
}
else{Write-Host "Grantflow requirements disabled"}
#Define the Clientsecret here
$secretstartdate = Get-Date
$secretenddate = $secretstartdate.AddMonths(24)
$description = "OAuth_secret"
If($grantflowrequirements -eq $true){
New-AzureADApplication -DisplayName $appName -AvailableToOtherTenants:$true -PublicClient:$false -ReplyUrls $redirecturi -RequiredResourceAccess $reqGraph, $reqExchangeOnline;
}
else
{New-AzureADApplication -DisplayName $appName -AvailableToOtherTenants:$true -PublicClient:$false -ReplyUrls $redirecturi -RequiredResourceAccess $reqGraph;}
$newapp = Get-AzureADApplication -SearchString $appName;
$clientsecret = New-AzureADApplicationPasswordCredential -ObjectId $newapp.ObjectId -StartDate $secretstartdate -EndDate $secretenddate -CustomKeyIdentifier $description;
"`n"
"ClientId: " + $newapp.AppId;
"ClientSecret: " + $clientsecret.Value
"TenantId: " + (Get-AzureADTenantDetail).ObjectId;
"`n"
"You can check your AAD app here: https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationMenuBlade/CallAnAPI/appId/" + $newapp.AppId + "/objectId/" + $newapp.ObjectId + "/isMSAApp/";
#Give it some time to appear in azuread so the next step will run successfull
Start-Sleep -Seconds 60
#Add you as owner
$AppOwner = get-azureaduser -all $true -Filter "UserPrincipalName eq '$($credential.UserName)'"
Add-AzureADApplicationOwner -ObjectId $($newapp.ObjectId) -RefObjectId $($AppOwner.ObjectId)
#Grant Admin consent
Login-AzureRmAccount -Credential $credential
$context = Get-AzureRmContext
$tenantId = $context.Tenant.Id
$token = [Microsoft.Azure.Commands.Common.Authentication.AzureSession]::Instance.AuthenticationFactory.Authenticate($context.Account, $context.Environment, $TenantId, $null, "Never", $null, "74658136-14ec-4630-ad9b-26e160ff0fc6")
$headers = @{
'Authorization' = 'Bearer ' + $token.AccessToken
'X-Requested-With'= 'XMLHttpRequest'
'x-ms-client-request-id'= [guid]::NewGuid()
'x-ms-correlation-id' = [guid]::NewGuid()}
$url = "https://main.iam.ad.ext.azure.com/api/RegisteredApplications/$($newapp.AppId)/Consent?onBehalfOfAll=true"
Invoke-RestMethod -Uri $url -Headers $headers -Method POST -ErrorAction Stop
If($grantflowrequirements -eq $true){
#Register Service principal
Connect-ExchangeOnline -Credential $credential
$serviceprincipalexo = New-ServicePrincipal -AppId $newapp.AppId -ServiceId $newapp.ObjectId -Displayname $newapp.DisplayName -Organization $tenantId
#Grant your application's service principal full access to the specified mailboxes.
ForEach ($fullaccessuser in $fullaccessusers) {
Add-MailboxPermission -Identity "$fullaccessuser" -User $serviceprincipalexo.Id -AccessRights FullAccess
}
}
else{Write-Host "Grantflow requirements disabled"}

Hey, I tried to create a diagram. I'm very bad at this and also haven't a propper tool for that right now. :D
But I think it should clarify our solution.


notFoundTemplate never loads

My data is definitely undefined but this doesn't work right:
Using version 1.0.12

Sorry, I didn't realize how much the API had changed, it works fine


Global text changes: report provider, report consumer

Within Release 5.1, naming conventions to be amended. In place of the current mix of content provider, publisher, platform, we will use 'report provider' and 'publisher platform' as appropriate. In place of library we will use 'report consumer' to reflect that people outside of libraries also use COUNTER Reports.

As the terms occur in conjunction, this issue includes using 'COUNTER Registry' in place of COUNTER register of compliant content providers.


Where I can see the code behind azure yaml  pipelines?

[Enter feedback here]
I am getting this error on my azure build pipeline:
Value cannot be null, parameter name: key.
I am clueless on how to get more information on this error. I have used jenkins before this and a lot of time taking a look inside code gives the answer, does code behind azure build pipeline is opensource like jenkins, if yes where I can find it.
Thanks
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7458d5b2-1bb7-7d87-7ce6-2b0c430b0155
Version Independent ID: edf5bbd3-a3f0-5426-e985-31290f8afd05
Content: YAML schema - Azure Pipelines
Content Source: docs/pipelines/yaml-schema.md
Product: devops
Technology: devops-cicd
GitHub Login: @steved0x
Microsoft Alias: sdanie

Thanks for your question. It looks like you're working through an issue with your scenario or implementation, rather than an issue with the documentation. Here are a couple of options where you might consider asking your question:

Azure DevOps Support Bot
Azure DevOps on Stack Overflow


Specify all variables as non-nullable

🗣 Description
This pull request specifies all variables as non-nullable.
💭 Motivation and context
This means that we need not worry about whether variables are explicitly specified as null.  Since this is not something we were expecting, it makes sense to avoid the situation.
See also the Terraform documentation.
🧪 Testing
All automated tests pass.
✅ Pre-approval checklist

[x] This PR has an informative and human-readable title.
[x] Changes are limited to a single goal - eschew scope creep!
[x] All relevant type-of-change labels have been added.
[x] I have read the CONTRIBUTING document.
[x] These code changes follow cisagov code standards.
[x] All new and existing tests pass.

The nullable argument is only available in Terraform v1.1.0 and later, but our skeleton is defined as required_version = "~> 1.0".  If we want this feature, we need to update our required_version.  Anyone have any concerns with doing that?

The nullable argument is only available in Terraform v1.1.0 and later, but our skeleton is defined as required_version = "~> 1.0". If we want this feature, we need to update our required_version. Anyone have any concerns with doing that?

See commit c551420.

Hi Shane @jsf9k, the Readme file still shows the reference to terraform ~> 1.0

Hi Shane @jsf9k, the Readme file still shows the reference to terraform ~> 1.0

See commit f9bf064, which I just pushed up.

Hi Shane @jsf9k, under directory examples/basic_usage: same for Readme file and version.tf there reference to  terraform ~> 1.0

Hi Shane @jsf9k, under directory examples/basic_usage: same for Readme file and version.tf there reference to terraform ~> 1.0

Please see commit 139eefd.


Support expression as a variable?

Can we support expression as a variable? Or is there any reason it was not meant to be supported?
Expression subExpression = new Expression("1 + c").with("c", "2");
Expression expression = new Expression("a * b");
expression.with("a", "1")
                 .and("b", subExpression)
	         .eval();

If expression as a variable is supported, we can get all expressions from the root expression.

This is most likely a duplicate of #9, and there is #143 which I was completely not aware of.
Though, the closure of #9 it is now 5 years ago and a few things have changed since then, I would be in favor of reevaluating this idea (evaluating, mind you).
However, implementing this might be a little bit more complicated than one might assume on first sight.

I'm aware of #143's feature, It helped a little bit, with it we can introduce explanatory variables to make the expression easier to understand.
But sometimes some calculate process is too complicated, so split it into multiple expressions helps a lot ( and with the ability to pull them out as a whole)
I didn't find #9 before, I agree that it's not hard to get this done on a higher level. if this feature was not intended to add, I'll just wrap Expression to make it work.
Thanks.

I think the quickest way for you to get that would be to have a custom LazyNumber implementation, something along these lines:
Expression expression = new Expression("a+b")
        .with("a", new LazyExpression("1+1"))
        .with("b", "5");

public class LazyExpression implements LazyNumber {
    protected String expression = null;

    public LazyExpression(String expression) {
        this.expression = expression;
    }
    public BigDecimal eval() {
        return new Expression(expression).eval();
    }
    ...
}

However, there is no way to get variables from an Expression once they have been added. I believe my "Scope" PR contained such possibilities, but that one never got finished.

Thanks, I'll gonna save those variables in the wrapper, sorta like this. But it was not tested yet.
public class Formula {

    private Expression expression;
    private TreeMap<String, Parameter> vars = new TreeMap<>();

    private Formula(String expr) {
        expression = new Expression(expr);
    }
    public Formula with(String name, Formula formula) {
        vars.put(name, new Parameter(name, formula.expression.eval(), formula.expression.getExpression()));
        expression.with(name, new LazyNumber() {
            @Override
            public BigDecimal eval() {
                return formula.expression.eval();
            }

            @Override
            public String getString() {
                return formula.expression.getExpression();
            }
        });
        return this;
    }
}

Looks good, minor detail is that you most likely don't want a TreeMap, a simple HashMap should suffice.

I'll close this. If it is still wanted, please re-open and submit a PR

Hi, It seems LazyNumber was removed in 3.x, this approch is no longer viable.
Is there any way to do this in 3.x?

Thanks, I'll gonna save those variables in the wrapper, sorta like this. But it was not tested yet. However, variables was private in 2.1, it was protected in the master branch.
public class Formula {

    private Expression expression;
    private TreeMap<String, Parameter> vars = new TreeMap<>();

    private Formula(String expr) {
        expression = new Expression(expr);
    }
    public Formula with(String name, Formula formula) {
        vars.put(name, new Parameter(name, formula.expression.eval(), formula.expression.getExpression()));
        expression.with(name, new LazyNumber() {
            @Override
            public BigDecimal eval() {
                return formula.expression.eval();
            }

            @Override
            public String getString() {
                return formula.expression.getExpression();
            }
        });
        return this;
    }
}

With 3.x, there is the possibility of passing sub-expressions as parameters directly, see the documentation for EXPRESSION_NODE data type
Does this what you are aiming for?
Example:
Expression expression = new Expression("a*b");

ASTNode subExpression = expression.createExpressionNode("4+3");

EvaluationValue result = expression
    .with("a", 2)
    .and("b", subExpression)
    .evaluate();

System.out.println(result); // prints 14

Hi, I've read about the ASTNode example in the document, but I think it has some differences.
With the previous implementation user can get the expression and variables of sub-expressions, it seems not possible with ASTNode.
What am I tring to achieve is to display expression, result and variables to user, some times the expression is too complicated, it'll be useful to break the expression to serval expressions.
Can we support 'Expression' as a parameter for 'with' instead of(or along with) 'ASTNode'?

Getting all the variables for a tree starting with an ASTNode is very easy, have a look at the code for getUsedVariables().
And the expression itself is what you pass to the constructor or createExpressionNode("a+b"), so I think you have all at hand.
Maybe adding a public static getUsedVariablesForASTNode(ASTNode root)  convenient function or something like this could help you avoiding copying the code.

Expression expression = new Expression( expressionString: "(a+b)*c");
ASTNode subExpression = expression.createExpressionNode("d+e");
EvaluationValue result = expression.with(variable: "a","1")
 and( "b", 2)
 and ("c", subExpression)
 and("d",  3)
 and ("e", 4)
.evaluate() ;

For example,I can get all varialbes ”a, b, c" via expression.getUsedVariables(). and can get the value of "a, b" via expression.getDataAccessor().getData().
But I also want to get the expression and the evaluated value of "c" ( "d+e" and 7), it seems not possible atm?
And I seems to me ASTNode is more like a internal type,  Maybe add a new type such as SubExpression  for end user is more suitable IMO.

There is also a way to work with subexpressions using two expressions. Here you have two expressions and can use all the methods, including getUsedvariables():
        Expression expression = new Expression("a+b+c");
        Expression subExpression = new Expression("a-x");

        Map<String, Object> variables = new HashMap<>();

        variables.put("a", 3);
        variables.put("b", subExpression.getAbstractSyntaxTree());
        variables.put("c", 4);
        variables.put("x", 1);

        assertThat(expression.withValues(variables).evaluate().getStringValue()).isEqualTo("9");

        assertThat(expression.getUsedVariables()).containsExactlyInAnyOrder("a", "b", "c");
        assertThat(subExpression.getUsedVariables()).containsExactlyInAnyOrder("a", "x");

But there's no easy way to get the value of subExpression?

There is also a way to work with subexpressions using two expressions. Here you have two expressions and can use all the methods, including getUsedvariables():
        Expression expression = new Expression("a+b+c");
        Expression subExpression = new Expression("a-x");

        Map<String, Object> variables = new HashMap<>();

        variables.put("a", 3);
        variables.put("b", subExpression.getAbstractSyntaxTree());
        variables.put("c", 4);
        variables.put("x", 1);

        assertThat(expression.withValues(variables).evaluate().getStringValue()).isEqualTo("9");

        assertThat(expression.getUsedVariables()).containsExactlyInAnyOrder("a", "b", "c");
        assertThat(subExpression.getUsedVariables()).containsExactlyInAnyOrder("a", "x");

Hi, it seems we can't get the value of subExpression easily(without put all those variables to subExpression again).
The result of expression.getDataAccesor().getData("b") is 0 instead of 2.

There is also a way to work with subexpressions using two expressions. Here you have two expressions and can use all the methods, including getUsedvariables():
        Expression expression = new Expression("a+b+c");
        Expression subExpression = new Expression("a-x");

        Map<String, Object> variables = new HashMap<>();

        variables.put("a", 3);
        variables.put("b", subExpression.getAbstractSyntaxTree());
        variables.put("c", 4);
        variables.put("x", 1);

        assertThat(expression.withValues(variables).evaluate().getStringValue()).isEqualTo("9");

        assertThat(expression.getUsedVariables()).containsExactlyInAnyOrder("a", "b", "c");
        assertThat(subExpression.getUsedVariables()).containsExactlyInAnyOrder("a", "x");

The data storage is not part of the expression, but part of the configuration. By default each expression gets a new instance of a default configuration, which also has a new instance of the MapBasedDataAccessior, which holds the data in a Map.
If you want to share the configuration (and data) between different expression, you can create a common configuration and pass this to the expression constructor.
expression.getDataAccessor().getData("b")  will return the sub-expression, i.e. the root ASTNode of the expression, it will not be calculated automatically. (Remember: the getData() method returns an EvaluationValue, which can be a number, string, array, structure or a sub-expression (ASTNode).
Here's an extended example:
    @Test
    void testSubExpression() throws ParseException, EvaluationException {

        // shared configuration, has a an instance of the data accessor
        ExpressionConfiguration configuration = ExpressionConfiguration.defaultConfiguration();

        Expression expression = new Expression("a+b+c", configuration);
        Expression subExpression = new Expression("a-x", configuration);

        Map<String, Object> variables = new HashMap<>();

        variables.put("a", 3);
        variables.put("b", subExpression.getAbstractSyntaxTree());
        variables.put("c", 4);
        variables.put("x", 1);

        assertThat(expression.withValues(variables).evaluate().getStringValue()).isEqualTo("9");

        assertThat(expression.getUsedVariables()).containsExactlyInAnyOrder("a", "b", "c");
        assertThat(subExpression.getUsedVariables()).containsExactlyInAnyOrder("a", "x");

        // all the data is accessible through expression
        assertThat(expression.getDataAccessor().getData("a").getStringValue()).isEqualTo("3");
        assertThat(expression.getDataAccessor().getData("b").getExpressionNode()).isEqualTo(subExpression.getAbstractSyntaxTree());
        assertThat(expression.getDataAccessor().getData("c").getStringValue()).isEqualTo("4");
        assertThat(expression.getDataAccessor().getData("x").getStringValue()).isEqualTo("1");

        // all the data is also accessible through subExpression
        assertThat(subExpression.getDataAccessor().getData("a").getStringValue()).isEqualTo("3");
        assertThat(subExpression.getDataAccessor().getData("b").getExpressionNode()).isEqualTo(subExpression.getAbstractSyntaxTree());
        assertThat(subExpression.getDataAccessor().getData("c").getStringValue()).isEqualTo("4");
        assertThat(subExpression.getDataAccessor().getData("x").getStringValue()).isEqualTo("1");

        // subExpression has to be evaluated manually
        assertThat(expression.evaluateSubtree(expression.getDataAccessor().getData("b").getExpressionNode()).getStringValue()).isEqualTo("2");

        // or in this case, simpler:
        assertThat(subExpression.evaluate().getStringValue()).isEqualTo("2");
    }

Thanks, sharing configuration works.
And there's another question, why checked exception was introduced in 3.x while exceptions in 2.x were unchecked exception?

Error behavior was a bit inconsistent in 2.x, There are now two exceptions that can be used to recover from possible failures.
One possible failure can occur when parsing an expression, in that case the ParseException is thrown. This exception is also thrown when the validate() method is called.
The other possible failure can occur when validating an expression, depending on the input values. In that case, an EvaluationException is thrown.
Both exceptions hold valuable information that can be given to a user so that he/she knows what went wrong at what exact position of the entered expression.
Both are checked exceptions, because both are predictable and a proper exception handling should be done by the calling system.


Doc is wrong that you cannot install yt-dlp on Termux

Checklist

[X] I'm reporting a bug unrelated to a specific site
[X] I've verified that I'm running yt-dlp version 2021.10.22. (update instructions)
[X] I've checked that all provided URLs are alive and playable in a browser
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Description
After installing Termux from F-Droid and running pkg update, you can install yt-dlp with no problem.
It also works with Termux from Google Play, but you have to update sources manually for `pkg
Verbose log
Doc bug

I only meant that IF you have trouble installing, u can install with --no-deps. I'll reword the docs to clarify

actually, I'll just remove the mention of termux altogether